Archive for the ‘linux’ Category
Contributing to Musicbrainz
Posted on: January 31, 2013
- In: linux | ubuntu
- Leave a Comment
Hacking on code is not the only way to contribute to the Free Software / Open Source Software community. Many applications rely on external datasets to provide some or most of their functionality and a contribution there helps all the projects downstream.
Ubuntu’s default music application Rhythmbox as well as Banshee, KDE’s Amarok and a host of smaller programs like Sound Juicer all offer the ability to rip CDs. For this feature to be seen to “work” in the eyes of the user the software needs to correctly identify the CD and add the appropriate album details and track names. To do this, these programs all query the Musicbrainz database and the quality of that response essentially decides the users experience of the ripping process; Will it be a one click “import”, or 10 minutes of filling in track names while squinting at the CD jacket? God help you if you have tracks with accented characters and an English keyboard.
What all this means is that every contribution to Musicbrainz results in a better experience for users of ALL of these programs. When someone somewhere decides to rip their favourite obscure CD, and the software magically fills in the track names and album details, its a pretty happy moment. So if you have wanted to contribute to the Free/Open Source Software community but don’t have the programming chops to do it, contributing to Musicbrainz is one way to help out.
The Musicbrainz dataset is under the Creative Commons CCO license which places all the data in the public domain. This means that your contributions will stay open to the public and we won’t have another CDDB/Gracenote situation where people contributed to a database that ended up charging for access. All you need to get started is to create a Musicbrainz account.
A typical contribution looks something like this. I’ll decide to rip one of my CDs and pop it in the drive. I launch Rhythmbox which will tell me if its not recognized:
When you click the “Submit Album” button the application will send you to the Musicbrainz website with the Table Of Contents (TOC) information from the CD in the url. Once on the site you will need to search for the Artist or Release to add the TOC to:
Most of the time there will be an entry in the database for the Artist and all that needs to happen is to add the TOC that you are still carrying from page to page in the url to one of the Artists CDs. In cases where the search returns multiple matches I usually explore the different options by ctrl+clicking on the results to open them in separate tabs.
Click through the artists albums until you find the one you are looking for, or add one if you need to. In this case there was one already there (and all the details including the catalog number matched) so my next step was to click the “Attach CD TOC”. This takes the TOC you can see in the address bar and attaches it to that release.
You will be asked to add a note describing where this information you are providing is coming from. In this case its coming from the CD. Add the note and you are done. What make contributing to Musicbrainz particularly gratifying is that next time you put in the CD, it is recognized right away. My favourite kind of gratification: instant. You can also immediately see the results in Rhythmbox, as well as Banshee and any other application that uses Musicbrainz.
Its pretty great thinking that the few minutes invested in this process not only solves your immediate problem of having an unrecognised CD, but also makes software all over the Linux ecosystem just a tiny bit better. That’s a win/win situation if I’ve ever seen one.
Getting to know SQLite3
Posted on: December 11, 2012
- In: linux
- Leave a Comment
I’m finding SQLite3 super useful lately. Its great for any kind of experimentation and quick and painless way to persist data. There are just a few things I needed to wrap my head around to start to feel commfortable with it.
As with most things on Debian based systems, installing is really easy:
sudo apt-get install sqlite3 libsqlite3-dev
My first real question was about datatypes. What does SQLite support? It was a bit mysterious to read that SQLite has 5 datatypes (null, integer, real(float), text, blob) but then see a MySQL style create table statement like this work:
create table people( id integer primary key autoincrement, name varchar(30), age integer, awesomeness decimal(5,2) );
How are varchar and decimal able to work? Worse still, why does something like this work:
create table people( id integer primary key autoincrement, name foo(30), age bar(100000000), awesomeness baz );
As it happens SQLite maps certain terms to its internal datatypes:
If the declared type contains the string “INT” then it is assigned INTEGER affinity.
If the declared type of the column contains any of the strings “CHAR”, “CLOB”, or “TEXT” then that column has TEXT affinity. Notice that the type VARCHAR contains the string “CHAR” and is thus assigned TEXT affinity.
If the declared type for a column contains the string “BLOB” or if no type is specified then the column has affinity NONE.
If the declared type for a column contains any of the strings “REAL”, “FLOA”, or “DOUB” then the column has REAL affinity.
Otherwise, the affinity is NUMERIC.
So the foo, bar and baz columns above, being unrecognized, would have received an affinity of numeric, and would try to convert whatever was inserted into them into a numeric format. You can read more about the in’s and outs of type affinities in the docs, but the main thing to grasp up front is that syntax-wise you can usually write whatever you are comfortable with and it will probably work, just keep in mind that affinities are being set and you will know where to look when you see something strange happening. For the most part this system of affinities does a good job of not violating your expectations regardless of what database you are used to using.
The other thing to get is that SQLite determines the datatype from the values themselves. Anything in quotes is assumed to be a string, unquoted digits are integers, or if they have a decimal, a “real” while a blob is a string of hex digits prefixed with an x: x’00ff’.
So the safest/easiest thing might just be to leave the column definitions out altogether so they will all have an affinity of none and let the values speak for themselves.
The rest of my learning about SQLite is really a grab bag of little goodies:
Getting meta info about tables, indexes or the database itself is done with a pragma statement.
For example, if I want information about the table data:
sqlite> pragma table_info(people); 0|id|integer|0||1 1|name|foo(30)|0||0 2|age|bar(100000000)|0||0 3|awesomeness|baz|0||0
You can get that same list of info within Ruby like so (after running “gem install sqlite3″):
require 'sqlite3'
@db = SQLite3::Database.new("cats.db")
table_name = "cats"
@db.table_info(table_name)
A complete list of pragma statements can be found in the docs.
To open or create a database simply run sqlite3 with the name of the file:
mike@sleepycat:~☺ sqlite3 cats.db
And finally if you have a file with sql statements you would like to run on a database:
mike@sleepycat:~☺ sqlite3 cats.db < insert_all_the_cats.sql
Its been good to get to know SQLite3 a little better. Before this I had only really come in contact with it through my Rails development work and knew it only as the in-memory test database or the one I would use when I couldn’t be bothered to set up a “real” database. The more I look at it the more its seems like a really powerful and useful tool.
Opening the results of a grep in Vim
Posted on: May 28, 2012
This week while removing some deprecated syntax from our old tests I found myself wondering how to take the results of my grep command and open the files returned in vim. Moments like this are great for getting to know your tools better so I decided to figure it out. Opening the files was not that difficult to figure out:
vim -o `grep -ril Factory.build test/`
This did exactly the right thing, a case insensitive (-i), recursive (-r) search of the test directory for the problematic syntax (Factory.build) returning only filenames (-l) and then opening the results in vim. I was able to quickly make my changes with a search and replace. With that done you can then write all buffers and close vim with:
:xa
I really love finding stuff like that. While that solved the immediate need, as written, it would not be able to deal with file names that have spaces in them. Since that sort of thing bugs me, I set up some test files and gave it a try:
mike@sleepycat:~/projects/vimtest☺ tail ./*
==> ./fileone <==
blah
blah
cats
blah==> ./filethree <==
blah
rabbits
blah==> ./file two <==
blah
racoons
dogs and cats
blah
After a little tinkering I figured it out with a little help from the xargs man page. Its not something I would want to type often but this will make a very useful alias or script some time:
grep -Zril cats . | xargs -0 sh -c 'vim -O "$@" < /dev/tty' vim
Bash Boilerplate
Posted on: May 18, 2012
- In: linux
- Leave a Comment
I’ve ended up doing a lot of bash scripting lately and am really shocked at how difficult it is to sort out all most mundane things like handling long and short options and some other cases like ensuring the script is run as root. So here is my bash boilerplate; 45 lines of time and sanity saving bash script. It has been lovingly assembled in Franenstinian fashion from about a dozen blogs and stackoverflow answers and owes a great debt to missiondata’s excellent post on getopt. Thank you internet.
I have several scripts using variations on this now that I use reasonably regularly and its a real time saver to use this as my starting point. If you notice a mistake or something that is flat out a bad idea please let me know. I’d like to make this as solid as possible.
#!/usr/bin/env bash
# set the script to exit immediately on error
set -eusage(){
cat <<'EOT'
Call this script with...
EOT
exit 0;
}# exit if there are no arguments
[ $# -eq 0 ] && usageset -- `getopt -n$0 -u -a --longoptions "help dogs: cats:" "hd:c:" "$@"`
# $# is the number of arguments
while [ $# -gt 0 ]
do
case "$1" in
-d|--dogs) dogs="$2"; shift;;
-c|--cats) cats="$2"; shift;;
-h| --help) usage;;
--) shift;break;;
*) break;;
esac
shift
donecleanup_before_exit () {
# this code is run before exit
echo -e "Running cleanup code and exiting"
}
# trap catches the exit signal and runs the specified function
trap cleanup_before_exit EXIT# if this script needs to be run as root
# we can check like this:
# if [ ! `id -u` -eq 0 ]; then
# echo "You need to call this script using sudo."
# exit 1
# fi# OK now do stuff
echo "$cats and $dogs living together, mass hysteria!"
- In: linux
- Leave a Comment
Recently I listened Bryan Lunduke’s talk “Why Linux Sucks“. One of the arguments he made was that the open source/free software development model has produced a lot of great software but struggled to produce highly specialised and sophisticated software like Photoshop or some of the top audio/video editing apps.
The argument is certainly food for thought and he points out that the projects like the Linux kernel that are making progress very quickly are largely commercially backed. From there he talks about the need for the developers to get paid and lauds the Ubuntu Software Centre for offering the ability to sell software as a potential solution. I have been digesting this for a little while now and I think the concept of selling software can be deeply problematic.
The difficulty I have is that the economic incentives are for the developer to maximise the amount of money they collect by treating each version as a separate product. When you sell someone a particular version, the will keep that version until they feel that it is worth it to shell out for the next one. My Dad for instance ran Office 97 for almost 10 years, because he never felt the new features were worth the money.
The problem with this situation lies in the fact that old software lingering on your system is a vector for attack. Microsoft’s failure to entice my Dad into upgrading means that my Dad’s computer is now at risk of infection and compromise by all sorts of malware. Once he is infected and his computer starts attacking other people, this becomes everyone’s problem.
I think one of the strengths of the Linux platform is that all this “free as in speech” software has largely translated into “free as in beer” software which in turn means the barrier to staying up-to-date is nearly non-existent. Your package manager keeps installing the latest version for free and you don’t need to worry about virus’s and malware and I don’t need to worry that your computer is attacking mine. Throwing paid for software into this ecosystem seems like a way to compromise on of the best things about the platform. While Bryan Lunduke’s talk was good for stimulating some discussion I have come to feel that he has misdiagnosed the problem.
At a recent Android meetup I met a guy who was being paid by IBM to work on PhoneGap (now known as Apache Cordova). When I asked him why IBM would do this his answer was that IBM has hundreds of people that are trained and familiar with web technologies and almost none familiar with Objective C and the rest of the Apple technologies. With a need to make a bunch of apps on a variety of mobile devices and a a bunch of trained web developers on staff they looked around for a technology that could bridge that gap and found PhoneGap. After a little poking around they determined that it didn’t do everything they wanted but was close enough that paying someone to work on improving PhoneGap was cheaper than retraining all their web developers to learn Java and Objective C.
Coming back to the lack of good image/video/audio editors on Linux, it seems to me, then, that we have a chicken and egg type problem. The editors on Linux don’t attract financial backing because they are not good enough to be considered “good enough” for basic professional use. While that sounds a little circular, I think its sound. Just like IBM looking at PhoneGap and deciding that it was close enough to what they wanted that with minor investment they could make it fit their needs, GIMP and other programs could find that both monetary and code contributions could start rolling in once they cross that “close enough” threshold for their industry.
A good example is the GIMP. Much maligned for its interface, its real problem is that its lack of CMYK support has meant that anyone that deals with printing presses has been unable to use it. With the GIMP teams recent announcement that 90% of GIMP core has been ported to GEGL which allows them to operate in different colour-spaces like CMYK. This means that price sensitive people in the printing industry and can finally consider using GIMP in their daily workflow and may just find that its “good enough”. When they do, they too may find that throwing a few dollars towards the project to smooth out some rough edges will make the same kind of sense to them as it did to IBM.
For my part I took Bryan’s advice and took a look at the Ardour audio editor he used as one of his examples. He mentioned that it was getting some traction in the industry and was in need of donations to keep the developer working on it full time. I decided to donate monthly to the project, hoping that it too will cross that threshold into “good enough” and start attracting some investment from professionals looking for improvements. Here’s hoping!
Getting Sikuli running on Ubuntu
Posted on: March 16, 2012
I have been wanting to play with the automation tool Sikuli for a while now. The few minutes I have dedicated to it have ended in frustration and promising myself that I would sit down and make a serious attempt at it some evening soon.
That evening finally happened. My attempts at the simplest script had ended each time with the error:
[error] Stopped [error] Error message: Traceback (innermost last): (no code object) at line 0 SyntaxError: ('future feature with_statement is not defined', (' ', 1, 6, ''))
A little duckduckgoing revealed that the problem is that the Ubuntu/Debian repositories have a old version of Jython which is causing Sikuli problems. It took a little experimentation (and upgrading both Sikuli and Jython) but this is what got my Sikuli script working:
wget -O jython_installer-2.5.2.jar http://downloads.sourceforge.net/project/jython/jython/2.5.2/jython_installer-2.5.2.jar?r=http%3A%2F%2Fwiki.python.org%2Fjython%2FDownloadInstructions&ts=1331868765&use_mirror=voxeljava -jar jython_installer-2.5.2.jar #run through the installer, I installed it into my home directory...
sudo ln -s /home/mike/jython2.5.2/jython /usr/local/bin/jython #then I created a link to it in my path
sudo apt-get install libcv2.1 libcvaux2.1 libhighgui2.1 # install some of the computer vision libraries Sikuli uses
Once that setup was done, I also downloaded and unzipped the latest version of Sikuli and then ran the script to start the ide:
Sikuli-X-1.0rc3 (r905)-linux-x86_64/Sikuli-IDE/sikuli-ide.sh
And it worked! Now I can finally experiment!
- In: linux | rails | ruby | ubuntu
- Leave a Comment
I just stumbled across this error in mid-yak-shave for one of my projects. While trying to get my app up and running to test something I get the usual native extensions whining:
Installing sqlite-ruby (2.2.3) with native extensions
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension./home/mike/.rvm/rubies/ruby-1.9.2-p290/bin/ruby extconf.rb
checking for main() in -lsqlite… no
checking for sqlite.h… no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.
After a you see this sort of thing a few times you pretty instinctively bang out the command:
sudo apt-get install libsqlite3-dev
I was pretty takenaback to get the same error after installing the dev libraries. It turns out that I was installing the gem “sqlite-ruby” when I needed “sqlite3” is what you need if you are using Ruby 1.9.2.
Firefox 4 Beta on Ubuntu
Posted on: June 30, 2010
- In: firefox | linux | ubuntu
- Leave a Comment
I am playing with the 64 bit version on Ubuntu Karmic and so far is seems pretty great. You can grab a copy here and, once unzipped into a directory you can run it by doing
chmod +x firefox
./firefox
Its generally feels snappy; opens quickly and loads the pages quickly. No crashes yet either. One serious downer is the lack of a 64 bit flash plugin which Adobe has recently withdrawn. Vendors have been selling 64 bit systems for years now, but for some reason Adobe hasn’t been able to release an official 64 bit flash plugin yet. So Mozilla finally launches a 64 bit browser and great swathes of the internet are unusable to anyone who uses it. Nice. No wonder Steve Jobs doesn’t like these guys. The speed the 64 bit gives is needed when dealing with Javascript intensive sites. The javascript driven Visual editor on WordPress, while sluggish at the best of times, is painfully slow with the Beta of 4. Even switching back to the tab running the WordPress editor is sluggish. On the other hand Google maps is noticeably faster and smoother at nearly everything.
I am also excited by the new Add-ons manager. Mozilla is taking things that typically open a new window and making them run in a new tab instead. This is excellent since the Add-ons definitely require more space than the dialog they were appearing in allowed for. Its interesting that for all the seeming obviousness of the tabbed window design and how long its been around, browser makers are still working on fully integrating the concept into the browser.
My only disappointment so far is likely not an issue with Firefox at all. When I first saw the mockups I was excited to see that Mozilla was following Chrome’s lead in moving the tabs into the otherwise underused title bar. While I love the screen real estate that Chrome gives, it hasn’t been enough to tempt me away from all the amazing plugins that Firefox has.
Version 4 was supposed to mean that I got to have my cake and eat it too, adding the look of Chrome to the browser I already use.However while Windows users get the Chrome like interface, on Linux they have merely simulated the effect by hiding the bookmarks toolbar. Irritatingly, the Windows 32bit version gets the full treatment, tabs on top, File, Edit and Bookmarks all hidden away until you hit the ALT key. Its pretty sexy.
Disappointing but my suspicion is that it likely has something to do with the way windows are handled by Gnome, rather than anything to do with Firefox. Hopefully that will be sorted out with Gnome 3. For the moment, fast is good enough.
Dear Canonical: Need Data.
Posted on: June 11, 2010
- In: linux | ubuntu
- Leave a Comment
Dell’s partnered with Canonical in 2007 was pretty exciting news. In my professional life I have spent close to half a million on computer hardware with the majority of it going to Dell. All my personal computers have always come from them as well. When I decided to get myself a new laptop, I went to straight to Dell.ca/ubuntu to pick out my dream machine.
After a little poking around I realised that they only sell Ubuntu on the low end machines. At the time (almost a year and a half ago now) they sold Ubuntu preinstalled on at least one desktop machine and on maybe three different laptops. Their best one was the lowest end of the XPS line and had very few customisation options.
While I really wanted to buy a system with Ubuntu preinstalled so Dell would know that a market for this stuff actually exists, I also knew I would be doing myself a disservice by buying something less powerful than I needed. After agonising over it I bit the bullet and bought the top of the range XPS with most of the bells and whistles… and Windows Vista.
Looking again, Dell seems to have reduced even that meagre offering and is now only offering Ubuntu preinstalled on the Dell mini netbook and a low-end Inspiron laptop. Aside from the fact that Ubuntu is almost impossible find for anyone not typing in the URL directly, the worst part is the Inspiron 15 with Ubuntu is $579 while the Inspiron 15 with Windows 7 is $569. Perhaps I’m old fashioned, but a computer with a $110 copy of Windows 7 on it should be more expensive than one without. It would be nice if one of the most obvious benefits of Linux were a little more obvious in the pricing. For a partner, Dell seems to have some funny ideas about what will help Ubuntu sell well.
While I believe Dell was correct in sensing that there is a market out there, and gutsy enough to try it out, its Ubuntu offerings seem to be languishing. Partly its pricing silliness and poor marketing but mostly it seems like they are misreading the market. Its been my experience that Linux users don’t buy low end hardware. Low end hardware in the Linux community seems to be something that is either gifted or salvaged, not something you purchase. When purchasing, its mid range to high end systems they are after. With that in mind, I can’t help but feel that Dell has missed the mark with their offering.
I suspect they missed it for the same reason most other companies still think there is no money to be made in the Linux world; there’s very little data. While Canonical is starting to gather a little data for their servers, I think the desktop probably needs it more. Perhaps its time for an Ubuntu version of the Steam Hardware Survey. Maybe they should consider some demographic surveys as well. I think its time we find out big the GNU dollar is.
I think this needs to happen before companies fall prey to a self fulfilling prophecy; Offer a product blindly, receive an underwhelming response, and conclude that there really is no money to be made from the 12 million+ Ubuntu machines and the unknown number of users of other distros. This kind of data would be invaluable not only to developers considering a Linux version of an existing program, but also to partners like Dell who just seem to need a bit of a nudge in the right direction.
What say you Canonical?
- In: Geeky | linux
- Leave a Comment
Today Phoronix “officially” announced that Steam is coming to Linux. While their definition of “official” doesn’t seem to require any input from Valve (the company that makes Steam), it does seem likely that they are correct given the screenshots and whatnot circulating on their site.
Its been pointed out before (I can’t remember where) that gamers are a perfect target market for Linux. They are more technical than the average user, looking for performance and geeky. I also think Linux and gaming are a match made in heaven, and now that it seems like it may actually happen, it does make you wonder about the ripple effect this will have across the industry.
For Microsoft there are a few interesting implications. First, one of the top games companies has gone cross-platform. To do this you would have to ditch Microsoft’s DirectX and code using OpenGL. Losing a company like Valve is bad, but its made worse since Steam is now cross-platform (currently on Windows and Mac, Linux rumours aside) and as such incentivizes other developers using Steam as a sales platform to follow suit.
Judging from the recent success of Wolfire’s Humble indie bundle, Mac and Linux users are worth the trouble of reaching out to. Had Valve been a little faster at making Steam cross-platform, some of the nearly $1.2 million they Wolfire raised could have gone to Valve. All the games in the bundle are available via Steam already, but only for Windows. Hopefully other developers were paying attention.
Also, pretty much every Linux user I know has a Windows machine they keep around for gaming. There is also a pretty sizeable number of Mac users out there that do the same. If you can get all (or even most) of your favourite games for your chosen OS, why keep Windows around?
Linux users may have some additional reasons to keep a Windows box around (Photoshop jumps to mind) but Mac users will likely just jettison Windows entirely the first chance they get. This effect will probably be noticeable only in some indirect ways: more pressure on OEM’s like Dell or HP to deliver machines without Windows, sluggish sales of the next version of Windows. While it won’t be huge, I think this effect would probably be big enough to be noticed by Microsoft. That said, I don’t think you could ever make a direct causal link.
The effect on the Linux community is also interesting to think about. Will we see stripped down gaming distros, tweaked to get the highest possible Frames Per Second, running Xfce (or even TWM), Steam and not much more? Imagine having both game and OS compiled from source specifically to squeeze every bit of performance out of your processor. Gentoo… I think we have found your calling. Of course, if gamers come to expect the ability to significantly optimize their operating systems, they may start demanding that of their video drivers as well…
Even though Steam is still vaporware at the moment, it’s awfully fun to speculate about. While it’s still possible that it won’t actually happen, one thing if for sure; Steam on Linux would be a very big deal.
