Caliban - Opinion and Righteous Anger

In the 2.5 years since I left Google, stopped working and moved away from Silicon Valley, my motivation to write code has waned and reached an all-time low ebb.

There are many reasons for this. For example, I'm father to a busy toddler these days, I'm no longer surrounded by brilliant geeks who fire my imagination and make me research new technology, and I rarely happen upon a need for a piece of code that hasn't yet been written.

Indeed, with very few computer systems to set up at home and no professional programming assignments to send me off on weird and wonderful coding tangents at home, the only piece of code that has seen regular development over the last year has been my TV guide data grabber for the Dutch cable TV network, UPC. Unless you were a Dutch MythTV user whose cable company was UPC, you wouldn't have noticed this. It's not exactly been my top download.

Even maintenance of my old projects has pretty much ground to a halt. You see, most open source software is born of a personal need. Necessity is the mother of invention, as they say. If code to solve a certain problem didn't exist, I had to invent it, assuming it was within my capability; which it fortunately sometimes was.

Ruby/LDAP is a good case in point. Although I didn't start Ruby/LDAP from scratch, it amply illustrates what I'm talking about.

After submitting a series of patches to Ruby/LDAP in 2003 and becoming frustrated at their slow rate of inclusion, the then maintainer tired of my e-mails and offered me the role of maintainer of the software. He had lost interest in the project, because he had long since stopped working with LDAP. See what I mean?

Anyway, I jumped at the chance. In the next year, development was rapid. I added a lot of features and fixed a legion of bugs. The development roadmap was easy. The use of LDAP at Google was exploding at the time, much of which was happening under my guidance and supervision. Every time a new need was found that logic dictated should be solved at the library level, it went on the Ruby/LDAP to-do list and was implemented by me not long after. I didn't need to spend much time thinking about which new features should be added, because they simply presented themselves to me in the course of doing my work.

These days, things are a little different. Not only do I no longer have a need to manipulate vast quantities of data in LDAP directories, I no longer even have ready access to working LDAP servers full of real, live data. Granted, I could set up a server at home, populate it with junk, and use that to test my library against, but without real-world problems occurring to me, it's not obvious which course development should take. In this scenario, the only real work that happens is the fixing of bugs.

It's the same with most of the software I have produced in recent years. The conclusion is simple. Once I cease to use a particular technology, my interest in it eventually drops below the level required to continue maintaining software. Sometimes the technology is no longer used because I am no longer in a professional environment (Ruby/LDAP, Ruby/CorporateTime), but it can also be because I have ceased to use it at home (bash completion.

Another reason software development can grind to a halt is because it reaches the point that it satisfies all of my personal needs (Ruby/Password, Ruby/DICT, Ruby/Finance, acoc, the very ancient signature.

The upshot of all of this is that most of the software that is under my wing for maintenance has badly stagnated.

In spite of promises to the contributors and enthusiasts of my various projects, I have languished and failed to make available public source repositories, mailing lists and the like. Such resources would have allowed the development of the software to continue, in spite of my current apathy.

Part of the problem has been in not wanting to let my babies go. bash completion, in particular, was started by me from scratch and has been very successful. Whilst I haven't made a cent from it, it features in many Linux distributions, has been in the Freshmeat most highly rated projects top 10 for many years and has featured in several magazine articles. Not bad for a piece of code that I started just to keep myself busy in the evenings, whilst killing time for a month in Canada, waiting to re-enter the US (it's a long story).

Rightly or wrongly, I'm quite proud of that software, but that pride has got squarely in the way of its continued development. In not wanting to relinquish the reins of control and any vague geek recognition that might have afforded me, I have instead allowed it to stagnate. Its last release was over 18 months ago.

Well, I have seen the error of my ways and am now prepared to act. As of this moment, I am declaring all of my software eligible for adoption by new parents, with the exceptions of my MythTV data grabber and Ruby/Amazon. This latter project has, in spite of everything I've said above, seen some recent work and I still intend to release a new version within the foreseeable future (but please don't hold your breath).

In particular, bash completion and Ruby/LDAP urgently need loving new homes. If you are reading this and would like to become the new maintainer of either project, please get in touch. Be forewarned, however, that you will need a fairly convincing argument to become the new owner of bash completion, as it needs a skilled and dedicated developer (yes, yes, even though I haven't been that myself recently). Ideally, you will have been a reasonably prolific past contributor to the project, thereby having already proven your candidature.

If I don't find new homes for these waifs and strays soon, I will scour my old e-mail for likely candidates and approach those myself. In fact, I may do this anyway.

So, it's open season on my old software. Let me know if you're interested, explain why and where you plan to take future development, and you could find yourself the owner of a mouldering CVS repository.

I found out at the weekend that we're soon to be offered ten days of free viewing of the Film1 film and sport channels as a promotional stunt. Now, I have no interest in the sport channels, but there could be some good films on the film channels.

Unfortunately, during most of the ten day free period, we'll be in the UAE and Oman. So, how to ensure that we don't miss any good stuff?

Enter the latest version of tv_grab_nl_upc, 0.6.1, which is the grabber I wrote to feed my MythTV system with programme schedule data for UPC's digital television network. This new version is able to look up the IMDB viewer rating for each of the films that it finds. This rating is actually a very good indicator of the true quality of the film in question, as it reflects the opinion of real viewers and, usually, with sufficient quantity that a reliable average results.

The only thing that remains to do is produce a custom recording rule in MythTV; and here it is:

program.stars >= 0.75 AND MONTH(program.starttime) = 2
  AND DAYOFMONTH(program.starttime) >= 16 AND DAYOFMONTH(program.starttime) <= 25
  AND channel.callsign LIKE 'F1%

This says to record any programme that has a star rating equal to or higher than 0.75 (equivalent to a 7.5/10 rating on IMDB), when the start of the broadcast is in February, between the 16th and the 25th of the month, and the broadcast channel is any of the Film1 channels.

Using tv_grab_nl_upc and the above rule, we should have a few decent films to watch when we return from holiday.

Again, I find the above a great example of the power and flexibility of MythTV. No other PVR gives you this level of control.

I noticed today that our MythTV box was going to fail to record the Christmas special of The Real Hustle. Why? Because the programme's name was The Real Hustle Christmas and we had instructed the machine to record only The Real Hustle, which as far as MythTV is concerned, is an entirely different programme.

Now, one doesn't want to have to create a second companion rule for each recording rule, purely to catch the Christmas specials. Nor does one want to have to create a power rule instead of a normal rule, so that one can match the basic programme name followed somewhere by the word Christmas.

What does one really want to do? Ideally, you want to record any programme that has Christmas in the title, if, when the string Christmas or Christmas Special is removed, the remainder of the title matches the name of any programme for which one has already created a recording rule.

To do this, create a custom rule, perhaps called Christmas Specials. In the search phrase, place the following:

program.title LIKE '%Christmas%'
  AND REPLACE(REPLACE(program.title, ' Special', ''), ' Christmas', '') =
    REPLACE(record.title, ' (Power Search)', '')

Dissecting this, we first check to see whether a given programme in the TV guide contains the word Christmas in the title. If it does, we remove the suffix Special, if present, from the title. Next, we remove the suffix Christmas from the title. That leaves us with the stub of the programme's title, which, with a bit of luck, will be the same as the title of a normal episode of the programme.

On the right-hand side of the equality check, we join with the record table to see whether there's a recording rule with the same name as the title stub of the Christmas programme. The REPLACE function first removes the text that MythTV itself adds to the name of any rule that is a power search, as this will otherwise potentially cause valid matches to be missed.

If we find a match, bingo, the Christmas special is caught in the net and MythTV will record it.

As I've said before, absolutely anything is possible with MythTV's recording scheduler, as long as you can figure out how to express yourself in SQL. This is the first time I've found a use for a database table other than program.

Above, you can see Rockbox running on my iAudio X5 player. The code was taken directly from today's CVS and cross-compiled on my laptop for the Coldfire CPU. The album art patch by Nicolas Pennequin was also applied. This allows one to configure the WPS (While Playing Screen) to display a bitmap image related to the track currently being played. The main purpose of this, obviously, is to allow one to display an image of the album from which the track is taken.

I needed a quick way to get album covers onto my X5, so I wrote some Ruby code that uses my very own Ruby/Amazon to pull album images from Amazon. Basically, the approach is to take one track from each album directory and inspect the Ogg Vorbis tag of that track. From this, artist and album title data is derived. Then, an Amazon search is performed to find the album in question.

Some albums are available exclusively on either Amazon UK or Amazon US, so I first try to pull the album cover from Amazon UK and resort to the American site only if I can't find it there.

The album cover images arrive from Amazon in JPEG format, but Rockbox needs BMP files, so I use ImageMagick to convert from one to the other and then save the image as cover.bmp in the same directory as the album.

The Ogg Vorbis tag data isn't accurate or specific enough in some cases to locate the album on Amazon. Occassionally, the album isn't even available from Amazon, which obviously also results in a failure to locate its cover art. In the end, approximately 75% of my albums end up having a cover image retrieved. It remains to be seen how many were mismatched with the wrong album!

I've had no time to hack together my own WPS file for the X5, so I'm currently using Markus Haselboeck's boeselhack_v1 theme. It's pretty nice, I think you'll agree. Having a small image of the album cover on the screen really is a nice touch and gives the software a very professional edge.

Here's a full photo of the X5 running Rockbox, just to place the above in context.

The new DVD player for the MythTV box turned up on Friday... and turned out to exhibit the same problem: blinking lights, but no action. After disconnecting the SATA cable and the power cable a hundred times, swapping those cables with those of the hard drive, and disconnecting and reconnecting random leads from the motherboard, I did get the DVD drive tray to slide open once. I put in the FC4 installer, rebooted and got a single line of the boot display to show up, after which the boot failed, citing the DVD drive once again.

A couple more attempts at booting the machine ensued, then the video card started to scream at me that its power cord wasn't connected. This was accompanied by a subtle, yet distinct air of electrical burning. Somehow -- I really have no idea why -- the video card has now decided to ally itself with the other rebellious hardware and turn to shit in my hands. It thinks it has no power, even though the cord is properly connected. A new cord and a different socket on the power supply fail to remedy the situation.

At this point, I have to suspect the motherboard after all. Things are breaking in such strange and unpredictable ways, and I'm really not that clumsy that I can realistically blame myself for each new problem that arises. But what next? Order a replacement motherboard and hope that everything magically just works? I've poured so much time, energy and money into this project already. It looks like I now need a new video card, too.

No, that's it for me, I think. The whole project has worn out my patience, so I'm shelving it for now. The working pieces of hardware will become spares for some future computer (since they're all either incompatible with or superfluous for my other machines), but essentially it's a lot of money down the drain at the moment.

The whole exercise has left a nasty taste in my mouth. This is the very last time I attempt to build a new PC from the ground up. Unless one has multiple spares of every part on hand with which to test and rule out causes, it's just not worth the potential grief.

I'm sticking to software from now on; it's what I'm good at.

The new drive for the MythTV box arrived today and works! Well, let's not be hasty; the BIOS recognises it. I won't know whether it truly works until I actually try to put an operating system on it.

To that end, I thought I'd try to start the FC4 install from a bootable DVD. FC5 comes out next Wednesday, but this would at least tell me that the system was finally ready to go from a hardware perspective. I pressed on the eject button of the DVD player to open the tray and... nothing. The LED on the front of the drive blinked a few times, but that was it.

After searching the Internet, I came across a document describing Plextor's blink codes. It describes a different drive to the one I have, but Plextor's blink codes seem to be the same across all models.

First of all, I was getting three blinks, which means ejection or loading error.

After manually ejecting the tray, inserting a DVD and then closing the tray, I then get eight blinks, which means spindle failed. None of the suggestions on Plextor's site provided any relief.

Why me? Why me? I ask.

I've already had to send back the TV tuner card and order a replacement. That arrived today, together with the new hard drive. Tomorrow, I'll have to send back the first hard drive and now also the DVD drive. More bloody delays and frustration ahead. All of these hardware problems would be bad enough if I had bought the parts locally, but having to send stuff back through the post is very tedious and time-consuming.

I'm beginning to wonder whether a working MythTV box will ever see the light of day by way of my hands.

I've ordered a new hard drive to replace the useless article that I couldn't cajole into action yesterday. Let's hope the new one shows more signs of life. I hope it's not just a case of some obscure incompatibility between the drive and the motherboard, because I've ordered the exact same model of drive again.

I've also ordered a new TV tuner card to replace the broken one I had to send back.

The package has already been despatched, so I may be able to report more as early as tomorrow. Until then, I'm keeping my fingers crossed that the new hard drive will actually spin up.

After putting the bits of our future MythTV box back together, the system now powers up, but the SATA hard drive is not detected by the motherboard. Yawn. The SATA DVD drive, on the other hand, is recognised, so I tried the hard drive with that device's cables, but to no avail.

With no other SATA computer in the house, I have no idea whether it's a duff drive or some other, more obscure problem. There are plenty of stories out there about the Seagate ST3500641AS not working with Macs, due to the use of Spread Spectrum Clocking (SSC), but this isn't a Mac, of course.

It looks like I'm going to suffer another hardware-induced delay while I get this figured out.

Our prospective MythTV box is more or less ready for the software. The shop that was investigating the source of my problem had called to say that they had got to the bottom of it, so off I went in our new car (which still has fewer than 100 km on the clock) to collect it.

It turns out that some kind of short-circuit was the issue. They took everything out, put it back in and then the system worked. I had pulled out virtually everything, but had left the motherboard in place. I can't imagine what could have been causing a short-circuit, but I have to hand it to them: they found the problem and I didn't. The bloke who had actually investigated the problem wasn't present today, so I couldn't ask whether he had managed to trace the exact location of the problem.

Anyway, I said I'd talk about the hardware a little bit once I'd got the system built, so here we go. Let's start with a list of all the parts.

• The outer case was one of the most expensive parts for this DIY PVR adventure; in fact, the most expensive component (with the hard drive a close second). It's a black OrigenAE X11 box. It comes with a VFD on the front, with its own user guide. The VFD will be used to display time, channel and programme information. It also has an IR unit, so that the box can accept commands from a remote control. All of this is supposedly compatible with Linux. We'll see. The box also features front-panel USB and audio inputs, which could prove handy later on.
• The motherboard I chose is an Asus P5P800 SE, a Socket 775 board with on-board Gigabit LAN and sound. It can take 2 Serial ATA devices, plus the usual IDE ones. It appealed to me because it was quite cheap, had lots of PCI slots, no PCI-E slots (which I don't need), plus AGP for the video card. I read that Linux doesn't yet play well with some PCI-E video cards, so I decided to go with tried and trusted technology. This box doesn't need cutting-edge hardware, anyway. This motherboard is also lacking Firewire support, which would have been nice, since the X11 case has a front-panel Firewire port.
• For the CPU, I had originally chosen an Intel Pentium 4 2.6 Ghz unit. Unfortunately, this went out of stock at the on-line shop from which I was purchasing all of the parts, so I then chose the more expensive 2.8 Ghz model. Finally, I noticed that the 3 Ghz model was only €20 more and had an extra megabyte of Level 2 cache, so I changed the order again. This is the one I finally bought.
• Because this system will live in the living-room, it needs to be as quiet as possible. For this reason, I purchased an Arctic Cooling Freezer 7 Pro heatsink/fan unit to replace the standard Intel one. The fan's mounting system is particularly nice and should help to reduce a lot of the noise.
• The power supply is a Tagan TG480-U15. There's no point skimping on this part and it doesn't use any more electricty if the parts it's powering require less than its full capacity, so I went for a 480 watt unit. It's a nice-looking unit, as PSUs go.
• A gigabyte of Kingston RAM will provide volatile memory.
• The hard drive had to be beefy, because this PVR will be seeing a lot of work. Speed was important, as was capacity. I opted for a 500 Gb Seagate Barracuda. It's a 7200 RPM drive with an 8.2 ms seek time. I'm not sure how silently it will run, so we'll have to see.
• For playing DVDs and archiving to DVD, I chose a Plextor PX-755SA rewriter. It can handle DVD±RW, including the dual layer variants and is a SATA device, rather than IDE.
• The video card is an ASUS N6800XT/TD, a 128 Mb 8x AGP card with an NVIDIA chipset.
• The TV tuner card is a MythTV stalwart, the Hauppauge PVR-350, complete with IR unit and remote-control. I chose this, rather than the double tuner PVR-500, because UPC's digital TV offering allows only one single tuner set-top box per household. True, I could connect a second tuner to the analogue cable TV, but I'm going to wait and see how this works out for now. Even if I did that, the PVR-500 has only a single signal input, which it then splits internally. That would make it impossible to connect plain old analogue cable and digital cable to a single card, anyway. I'll need another DVR-350 if I want to take that route.
• Naturally, this box needs to be networked in order to pick up its programme guide, amongst other things. After careful consideration, I finally picked the Netgear WG311, primarily because it works with the MadWifi driver, with which I have had good results.
• Last but not least, the MythTV box will need to be able to change channels on the digital TV set-top box when it needs to record a programme starting on another channel. Many set-top boxes have an old-fashioned RS232 interface for this, but UPC's Thomson-made box of tricks doesn't. What to do? After some scouting around, my best bet seemed to be Mike over at irblaster.info. He makes IR transmitters that yoiu can connect at one end to the serial port of your PC, whilst at the other of the cable is an IR emitter. Basically, you drape the cable from behind over the front of the set-top box and then send IR signals down it. Essentially, you're glueing a remote-control to the receiving eye of the set-top box. It's a dirty hack and will be the least elegant part of the whole set-up, but at least it will work. This is the only piece of hardware purchased from outside The Netherlands. I paid by Paypal and the device arrived within just a few days: very prompt service, indeed.

I must confess that picking out the hardware wasn't much fun. A lot of research was required to avoid buying parts that would not (properly) function under Linux, would not work with each other, would not generate an excessive amount of noise, etc. Then there was the physical hassle of building the system, followed by the grief and delay when the whole combo refused to play ball. I'm just not a hardware guy at the end of the day. Give me a box with working hardware, but no operating system, and let me work my software magic on it.

Anyway, we're more or less at that stage now. I say more or less, because I had to send the broken TV tuner card back to be exchanged. I don't yet have the replacement and it's a pretty essential part, so I'm stuck for a few more days.

Furthermore, Fedora Core 5 will be released on 15th March and it would be nice if that could go straight onto the system without needing to upgrade from FC4 or FC5 test3. What's a few more days? That'll give me time to replace the TV card, anyway.

The last six months, I've been enjoying (well, mostly) an extended break away from pretty much anything to do with computers. I've read my e-mail and done plenty of on-line reading, but all geek activities were halted and any resemblance I may once have borne to a coder or a sysadmin was no longer in evidence.

Happily, my workstation is now reinstalled and awaiting further instructions. Next week, I hope to embark on the arduous process of integrating all of the patches I've received over the last six months for bash completion, Ruby/LDAP, Ruby/Google and some of my other projects. New releases should follow within the foreseeable future.

Ruby/Amazon is another project of mine that has been generating a lot of e-mail lately. A fair number of people would like to see me update the code to use version 4 of the Amazon API (it currently supports version 3), but it's a major overhaul that I just haven't had the time or inclination to take on in the last six months. After all, I did become a father, re-emigrate across the world and buy a house.

Anyway, I've still not been convinced to sink the necessary time into Ruby/Amazon, but I'm closer to being convinced now than I was three months ago. One person even offered to pay me on a contract basis to do the work, which was very flattering, but it's not an issue of the money. It's simply that it's quite an undertaking to do it properly and that's time I wouldn't be able to spend on other things.

We'll just have to see how that one pans out. No promises, but if life settles down a little and I manage to produce new releases of my less involved projects in the next few weeks, I might reconsider my position on Ruby/Amazon.

I should also start work on that MythTV box I keep talking about. So many things to do and only 86400 seconds in a day.

I've received numerous patches since we moved back to Amsterdam. bash completion, Ruby/Finance, Ruby/Google and Ruby/LDAP have all received patches that I need to integrate.

Unfortunately, all of my computers are still in storage, so I can't work on these projects at the moment. Have no fear, however; as soon as they are back on-line, I will pick up the reins and start integrating the fixes and enhancements that people have sent me. As for the time frame for this, I don't expect to be in a position to work on these projects again until at least late December. It's more likely that I won't recommence work until the new year.

If you're someone who has sent me a patch, please be patient. I will respond eventually.

Whilst browsing around, looking for a new version of the firmware for my iRiver H140, I chanced upon Rockbox, which, in its own words, is Open Source replacement firmware for the Archos Jukebox 5000, 6000, Studio, Recorder, FM Recorder, Recorder V2 and Ondio MP3 players.

What that description fails to mention is that Rockbox is also in the process of being ported to the iRiver H1x0 series of audio jukeboxes. The code is still in alpha state, but this comparison of features in the original firmware and the Rockbox firmware makes for impressive reading.

I was intrigued enough to give it a whirl. One very slick feature of Rockbox is that it turns your iRiver into a dual-boot system. If you depress the Record button whilst turning on the system, it boots the original iRiver firmware instead of the Rockbox firmware. That's handy to get access to the FM tuner, for example, which still isn't implemented in the Rockbox firmware.

Another nice feature is the very fast boot time, which is around 5 seconds, comparing very favourably with the original iRiver firmware boot time of about 20 seconds. There's also the oft-requested feature of truly gapless playback, which is nice for ripped DJ mix CDs, amongst other things.

Other cool features are some trivial games and demons (Mandelbrot, Snake, Patience, etc.), the ability to customise the main display, by using a so-called WPS (While Playing Screen) template. This basically allows you to design how you want the main screen to look, including title and artist information, battery life, etc. It's incredibly flexible, even allowing you to scroll items and alternate between multiple items per line.

Here's a sample .wps file with the resulting display on the iRiver:

· %s%?it<%in. %it|%fn> · %s%?ia<%ia|%d2> · %s%?id<%id|%d1> %pc/%pt %?pp<[%pp/%pe]|> %pb %?It<Coming up in %pr:|%acRetrieving data> %s%?It<· %In. %It|%?Fn<· %Fn|%acfor next song...> %s%?Ia<· %Ia|%?D2<· %D2|> %s%?Id<· %Id|%?D1<· %D1|>> %?It<%?pn<· [%pn]|>|%?pn<Playlist: %pn|>> %pm %acBattery: %bt (%bl) %fc %?fv<%ffHz| %ffHz> %?fv<%fbK VBR| %fbK>

Rockbox offers a large choice of fonts to use on the display. In the above photo, you can see the Atadore font.

Rockbox also has support for interesting features, such as crossfade and replay gain, things completely lacking from the official firmware.

There are a few glitches in the Rockbox firmware and I've managed to crash my player a couple of times, but it offers enough niceties that I think I'm going to stick with it from this point forward. And, when I need the FM radio or something else not yet supported by Rockbox, it's trivial to boot back to the official iRiver firmware.

If you have an H1x0 player (or one of the other supported players), have a look at Rockbox. It'll breathe new life into an already great device and set it even further apart from the iPod.