Caliban - Opinion and Righteous Anger

As you may know, I use a Netgear ReadyNAS RND4410 for my internal network's mass storage. I'm very happy with the device, but it is a bit inflexible when it comes to back-ups.

The problem is that most forms of back-up that it provides for don't support the deletion of files that are no longer on the source. For example, if file A and B were backed up last night, but file A was deleted on the source by somebody this morning, tonight's backup should back up only file B (and even then, only if it has since changed) and delete file A from the back-up destination.

This level of control calls for the common free software program, rsync, to be used for back-ups. The ReadyNAS does support rsync, but Netgear's interface to it via the Web-based FrontView software is less than ideal. That's because it doesn't allow one to specify the use of rsync's many optional flags and parameters. In particular, it doesn't allow the essential --exclude flag to be used to omit certain directories from the back-up. I'm not being picky; I actually have a back-up job that will fail without the use of this flag.

Happily, though, the ReadyNAS is Linux-based and Netgear nowadays provide an EnableRootSSH patch, which will, oddly enough, allow you to ssh into your ReadyNAS as root.

I'd been resisting the temptation to do this for some time, because I had no good reason to do so. The ReadyNAS is sold as an appliance and one isn't really supposed to go prodding at its internals. The potential for rendering one's device non-functional is definitely there. Of course, I know what I'm doing (famous last words, I know), but I still have a healthy respect for devices supposed to operate as black-box appliances.

Nevertheless, I needed more flexible rsync functionality for my back-ups. An alternative to poking around on the Netgear would be to schedule the back-up jobs on the clients themselves, pushing the data to be backed up to the ReadyNAS instead of having the ReadyNAS run the back-up job and pull the data from the clients.

I wanted my back-ups centralised, however, so I installed the EnableRootSSH patch and went gently wandering across the file-system.

I found what I needed and was able to add the functionality I needed with 15 lines of Perl. Now, it's possible to define a set of extra options to be passed to an rsync back-up job when it's invoked.

I've posted details of how to do this to Netgear's ReadyNAS forum, so I won't repeat them here. I mention the hack here only to gain a bit of publicity for it, as I'm sure I'm not the only person who needs this extra functionality.

Of course, a much better solution would be for Netgear to integrate this into their FrontView Web-based interface. I'd much rather be able to use the supplied tools than have to resort to hacks like this.

Still, at one level, it is nice that Netgear have allowed this kind of thing to be done. It encourages experimentation, development and user community growth.

Another thing that kept me busy over the last few days was enhancing, tv_grab_nl_upc, my program for fetching television guide data for UPC's digital cable network in The Netherlands and outputting the results in XMLTV format.

Of all the stuff I've written, this is possibly the most obscure and least useful to anyone else (of course, I've said that before and been wrong). Nevertheless, necessity is the mother of invention, and I need this program, so I've been working on improving it over the Christmas period.

Specifically, this new version uses some heuristics to try to derive a subtitle for the programme in question. Ideally, this should be the episode name for a series, the destination city/country for a travel programme, etc. Sometimes, UPC includes this kind of information as part of the programme's description, but programmatically determining whether the first sentence of the description is suitable for use as a subtitle is a tricky business; hence the many man hours that have gone into the 0.5.0 release, as I had to perform a fair amount of textual analysis and parser refinement before reaching the point that plucking text from the description to use as the subtitle was logical enough to be more use than hindrance.

Anyway, 0.5.0 of tv_grab_nl_upc is now out, even if I'm the only one who cares.

For a number of reasons, I've been dissatisfied with using our Hauppauge PVR-350's on-board MPEG-2 decoder and TV Out port for MythTV. For one thing, the MythTV mailing-list makes it clear that support for the decoder of this device is on the way out. None of the developers use one, so the code is unmaintainable.

With this in mind, I purchased an XFX GeForce FX5200 AGP video card for €39 from Alternate, which has a well-supported nVidia chipset. The card arrived in the post yesterday, so I installed it in the system and got to work configuring it at the end of the evening.

With the loss of the PVR-350's on-board MPEG-2 decoder, playback now has to be handled by the CPU, but we have a 3 Ghz Pentium 4 in the box, so it can take it in its stride. This is pretty much the only disadvantage to switching to this video card.

Here are some of the advantages:

• The FX5200 has a DVI Out socket, so we're now connecting to the TV using DVI. The result is that X and MythTV now look superb on our 94 cm Philips LCD TV. The image is crisp and clear and we can drive the TV at its native resolution, which appears to be 1280x720. Related to this, menus and widgets no longer run off the screen and fonts are properly scaled. Bliss!

• Thanks to DDC, the monitor can now tell the video card which video modes it can handle, so the Xorg configuration is now very simple and doesn't even need HorizSync or VertRefresh parameters for the TV. Here's the relevant section from Xorg.0.log;

  (--) NVIDIA(0): Connected display device(s) on GeForce FX 5200 at PCI:1:0:0:
  (--) NVIDIA(0):     Philips FTV (DFP-0)
  (--) NVIDIA(0): Philips FTV (DFP-0): 135.0 MHz maximum pixel clock
  (--) NVIDIA(0): Philips FTV (DFP-0): Internal Single Link TMDS
  (WW) NVIDIA(0): Mode "1280x800" is too large for Philips FTV (DFP-0);
  (WW) NVIDIA(0):     discarding.
  (WW) NVIDIA(0): Mode "1280x768" is too large for Philips FTV (DFP-0);
  (WW) NVIDIA(0):     discarding.
  (II) NVIDIA(0): Assigned Display Device: DFP-0
  (II) NVIDIA(0): Validated modes:
  (II) NVIDIA(0):     "1280x720"
  (II) NVIDIA(0): Virtual screen size determined to be 1280 x 720
  (--) NVIDIA(0): DPI set to (76, 76); computed from "UseEdidDpi" X config option
  

  I'm not sure why those first two modes are tried, since they wouldn't offer the correct 16:9 aspect ratio. If anyone knows, please tell me. Nor do I understand why 1920x1080i isn't tried or selected, since our TV is capable of that resolution. I'm not too bothered, though, since we don't have any input sources that can provide that resolution.

• MythTV is slowly moving towards OpenGL for its themes and menus, so having an nVidia card with its hardware acceleration allows us to enjoy all of the OpenGL goodies. TV playback can also use certain OpenGL functions to reduce jitter and make other improvements to the image. And, very pleasingly, we can now enjoy all of the OpenGL visualisations offered by the MythMusic module. The TV has become a big flat disco ball for listening to my music collection!

• Although MPEG-2 decoding is now handled by the CPU, we can still offload some of the work to the video card's GPU by using MythTV's XvMC support. A lot of people seem to have trouble with XvMC, but I found it very easy to get working. The only issue was that the OSD (on-screen display) became greyscale rather than colour, but that's a known issue and there's an easy fix for it.

• During playback, fast-forwarding at greater than 3x speed now properly displays the current location in the recording. With The PVR-350, fast-forwarding beyond 3x speed would freeze the playback image, leaving you with only the playback timer to hazard a guess at where in the stream to resume playback. This was very annoying and confined us to 3x fast-forwarding much of the time.

• During playback, rewinding at any speed now properly displays the current location in the recording. Similar to the previous point, but even more annoying, was that the PVR-350 would freeze the playback image when rewinding at any speed. If you wanted to replay the last couple of scenes in a programme, you would thus have to guess how long they were and resume playback after rewinding that many minutes.

• Not using the PVR-350 MPEG-2 decoder for playback means that the audio and video are now separable. This, in turn, means that we can now send audio to our sound system, rather than being restricted to using the TV's built-in speakers.

• As a result of the previous change, the internal MythTV volume controls now also work. Previously, we had to adjust the volume via the TV, because the PVR-350's hardware decoder volume was beyond the control of MythTV.

• The ivtv driver decoder errors that have plagued us since I first set up this system almost a month ago are now happily also a thing of the past. We would regularly get errors like these in the log:

  Sep 18 14:36:52 tourbillon kernel: ivtv0 warning: DEC: Sched Buffer end reached 0x0ad51267
  Sep 18 14:36:52 tourbillon kernel: ivtv0 warning: DEC: Mailbox 10: 0x00000000 0x0ad41267 0x0ad41267 0x00000000
  Sep 18 14:48:32 tourbillon kernel: ivtv0 warning: DEC: Decoder Invalid type 0xd8031707?
  Sep 18 14:48:32 tourbillon kernel: ivtv0 warning: DEC: Decoder Invalid type 0x0e86df64?
  

  In practical terms, these manifested as occasional picture freezes during playback. Sometimes we'd get several within a few minutes. At other times, we'd watch a whole programme without one. They were very unpredictable.

  A quick hit of the Rewind button was necessary to unjam the system. It was irritating, but we were able to live with it. Now we're no longer using the hardware MPEG-2 decoder of the PVR-350 card, however, we're avoiding whatever it was in the driver or the firmware that was causing these.

• Time-stretching now works properly. Time-stretching is the ability to play back video at faster than normal speed, but with the appropriate audio compensation, so that the pitch remains constant.

  The idea is that, if you're short on time, you can get through a 30 minute programme in, say, 20 minutes if you pay good attention and can stand listening to people who talk very quicklyt. At 1.5x speed, for example, everyone sounds like the people who read the disclaimer at the end of American radio adverts. You know the ones, where it sounds as if all the gaps between the words in their sentences have been sucked out.

• Because we're using a plain old video card now, the entire Linux boot process can be followed on the TV screen. If there's ever a problem that causes the system to fail to boot, I'll now be able to debug it without a blindfold.

And that's about it. I reprogrammed our Harmony 885 universal remote-control to use the external sound system instead of the TV's audio for MythTV playback and to select the DVI input on the TV instead of the relevant SCART connector, and now we're all set. I just need to go and buy a new DVI cable, as I borrowed the one from my computer upstairs for testing purposes.

What do you do when you're missing your wife and darling daughter? In my case, you keep busy.

I had intended to go into the crawlspace under our house at the weekend to lay down some CAT5e cable. The intention was to run Ethernet to the spot directly under where our TV is located, then up through a tiny hole in the floor that the previous owners must have (had) drilled. That would give us the much needed wired network access that we're missing in the living-room.

Well, when I went to do the work, I noticed that I had purchased the wrong kind of cable; I had bought solid cable instead of braided. So, I went back to Media Markt yesterday and switched my solid cable for a 50m reel of braided.

Today, I had no good reason to put off the work any longer, so with the family out of the way, I got down to some undisturbed work in the cellar.

The crawlspace under our house is really quite vile. It's not just a flat, dusty space down there. Oh no, it's full of rubble, old plastic bags, bits of old wire and God knows what else; probably a load of rat turds and asbestos, I shouldn't wonder.

I'd bought a nice, powerful Maglite torch yesterday, an essential piece of kit when crawling around on your belly and elbows in the pitch black. It has a rotary head, so that you can turn from a wide angle beam into a sharp, bright point of light. I'd been bitching for ages that we didn't own a decent torch, but thanks to a local ironmonger's, that's now been put to rights.

Anyway, I was under the house for only about fifteen minutes and there were no unexpected hitches. It was easy to draw the cable across the breadth of the house and feed it up into the living-room.

Upon exiting the crawlspace, I immediately headed upstairs for a shower. My clothes, skin and hair were covered in... I don't even want to know what.

Back downstairs, I carefully stripped the cable in the living-room and crimped an RJ-45 connector onto the end, using an excellent guide, turned up by Google. If you think that I know the colour-coding order of an Ethernet cable off by heart, you'd be wrong.

With one end done, I went back downstairs into the cellar and crimped a connector onto the other end. I then hooked up that end to the router.

Back in the living-room, I plugged the other end of the cable into the on-board Ethernet jack of the MythTV box, tried to bring up the link and... bingo!

Sep 20 16:43:32 tourbillon kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex

I've already decommissioned the 802.11b WLAN card inside the MythTV box, as it's no longer needed. It wasn't working well any more anyway, because it was suffering interference from the wireless speakers of our recently purchased Logitech Z-5450 5.1 speaker set.

With a 100 Mbit link from the MythTV box, I'll now be able to stream TV programmes to my workstation on the top floor, which could be handy if Sarah is commandeering the real TV downstairs.

In the future, I suspect we'll eventually move to gigabit Ethernet and a distributed MythTV set-up, so that we can stream programmes and DVDs to televisions anywhere in the house. We could even make the front-ends discless, so that there would be almost no noise emanating from the cases.

Anyway, that stuff is all some way off yet. I'm just happy to finally have Ethernet where the TV is.

The site seems to be running nicely on the old server in the cellar. I'm sure browsing photos isn't a terribly fast experience any more, but until I can find a reasonably priced hosting provider I can trust, this is the way it has to be. As detailed here, my last dedicated hosting provider turned out to be less than dedicated and not much of a host to his paying guest.

As I've mentioned before, the new hardware that Web Host Plus put me on after managed.com sold my hard drive to them was less than reliable. I suspected a few causes, one of which was bad RAM. This theory now seems to have been borne out by an experiment I did. Before I rsynced our photo gallery across the Atlantic, I rebooted the problem box in New Jersey and reduced its operational RAM from 2 Gb to 512 Mb. In that new configuration, I was able to spend many a joyous hour copying my precious data over the transatlantic pipe without the originating box going AWOL. QED, I'd say.

I'll be cancelling service with managed.com, Web Host Plus or whoever's running the show now as soon as my next invoicing cycle starts. Good riddance to bad rubbish, as they say.

In the meantime, any DNS oddness you may have been seeing should now be a thing of the past. All slaves are in sync and handing out correct data. Incorrect data in the caches of other DNS servers should now also have expired. Normal service has now been resumed.

Our photo gallery is back on-line.

I was able to bring it back much more quickly than I had first anticipated, because I used an old Gallery 1.x back-up to seed the albums in the Gallery 2.x installation. That copied hundreds of extra, unnecessary files, but they were easily removed afterwards by rsync.

I did this by getting a list of all the album directories on the remote server:

cd /var/www/g2data/albums
find -type d > /tmp/file_list

I then copied this over to the new server with the Gallery 1.x back-up:

for i in `cat /tmp/file_list`; do
  album=${i##*/}
  src=`find /var/www/html/albums -type d -name $album`
  [ -n "$src" ] && rsync -av $src/ /var/www/g2data/albums/$i
done

In the end, I needed to copy from New Jersey only the photos we had taken since mid-February, which is when I had done a full back-up in preparation for migrating from Gallery 1.x to 2.x.

Somehow, one of the tables in the MySQL database had got corrupted in the move:

060518 18:02:38 [ERROR] Got error 134 when reading table './gallery2/g2_ImageBlockCacheMap'

This was easily corrected:

mysql> repair table g2_ImageBlockCacheMap;
+--------------------------------+--------+----------+--------------------------------------------+
| Table                          | Op     | Msg_type | Msg_text                                   |
+--------------------------------+--------+----------+--------------------------------------------+
| gallery2.g2_ImageBlockCacheMap | repair | warning  | Number of rows changed from 45465 to 45460 |
| gallery2.g2_ImageBlockCacheMap | repair | status   | OK                                         |
+--------------------------------+--------+----------+--------------------------------------------+

And, with that, the rescue and salvage operation to yank caliban.org from the incompetent clutches of the unholy alliance of Managed.com and Web Host Plus is 95% or more complete.

Once the residual DNS propagation issues evaporate, I'll be able to fully exhale once again.

Much of the last 24 hours has been spent seizing those rare moments during which my server -- migrated through no desire of mine to Web Host Plus -- is up and on the network, and using them to perform a migration of my own, namely to the server in my cellar.

I'm knackered, but a lot has been accomplished today. DNS and e-mail have now been fully migrated, including Web mail and mailing lists. The Web site, too -- which you're now reading -- is also up and running on the new (well, actually quite old) server.

The main thing that's not yet back up is our gallery of photos. That's because it's 19 Gb of data, which would be slow to copy from a reliable server on a fast network. Well, I have to copy it from a machine that keeps crashing and is not on a fast network. It could be a couple of weeks before I manage to get all of my data off it... if I'm lucky. I don't want to even contemplate the notion of not being able to recover all of my data.

I thought I'd left this kind of sysadmin drudgery behind me when I stopped working. Indeed, I moved my domain to dedicated hosting to reduce the downtime and maintenance that I had repeatedly endured when I hosted it myself on a domestic DSL line. Little did I know that I would get to enjoy such advantages for barely a year before falling victim to the worst kind of professional incompetence: that with no sense of responsibility for one's customers.

And so caliban.org is back on a domestic DSL line, albeit one that has proved itself more reliable than those I had in the US. The upstream bandwidth is also somewhat better.

Anyway, don't bother looking for new photos -- or old ones -- in the near future. I'll announce when -- if? -- they're once again available.

Yes: wankers. Wankers! WANKERS! WANKERS! WANKERS!

Who am I talking about? Managed.com, of course, the company to which I give good money each month to host this site.

What happened?

Well, managed.com decided to move its network from California to New Jersey. At least, that's as much as they told us, the paying customers.

In preparation for this, they sent all of their customers an e-mail asking to be supplied with the customer's root password via plain text e-mail. For those of you who aren't in the field of computer system and network administration, let me state that this is a violation of one of the most basic and universally lauded principles of the profession: never, under any circumstances, send passwords in the clear.

And yet, my hosting provider was asking me to do just this. In hindsight, that should have been enough to spur me into action. I should have found another hosting provider, right there and then, and moved my data prior to the migration. But I decided to wait until after the migration to seek a better provider. As always, laziness, compounded by a failure to recognise the urgency of the situation, won out.

Anyway, managed.com were supposed to back-up their customers' data, firstly with a full back-up and then, shortly before the migration, with a further incremental back-up. The migration was supposed to be barely noticeable, with a guaranteed maximum of two hours of downtime.

I was sceptical, but kept my fingers crossed.

Can you believe that managed.com didn't tell its customers in their notification e-mail when this migration would actually take place? We were left to guess. E-mails to them on the subject went unanswered, as did requests for a secure channel through which to supply one's root password.

When I noticed one day that my machine had been rebooted without my permission, I incorrectly assumed the migration had already taken place. If I'd known at that time that things would be moving to New Jersey, not just around the corner in California, I could have run a traceroute and seen that my machine had not actually gone anywhere. At that time, however, I thought they were just moving locally. What else could I think? Managed.com had told me virtually nothing in their e-mail.

caliban.org mysteriously went off the network on 9th May. It remained inaccessible for almost three days. So much for the two hours of guaranteed downtime.

All of my e-mails to managed.com went unanswered in this period. Only when I threatened them with legal action (a trick I picked up in America), did they finally respond by rebooting the machine and getting it back on-line.

Naïvely, I thought that would be an end to my problems. Yes, that was very naïve of me.

You see, managed.com restored my service from a week old back-up. I've no idea what happened to the promised incremental back-up. It was probably never made and, even if it was, it would have had to be of the last week's worth of data, not just the day before the migration. I suspect it was never even made, however.

The net effect? I found I was missing a week's worth of e-mail, multiple DNS changes had been lost, the last week's worth of blog entries had effectively never been written, and sundry other less serious issues now needed to be fixed, such as recent software updates becoming undone.

More e-mails to managed.com went unanswered. Due to an oversight on my part, my own off-site back-ups had not taken place in recent times, so I had no private back-up from which I could recover my data. Typical.

I began work on the system to repair the damage my hosting provider had done to it, but before I could achieve very much, the system went down again. The system was off-line again for more than a day. Once again, e-mail threats were required to get it back on-line.

So what's going on?

Exploration of my system's log messages shows that the new hardware on which my data resides is not the same as the old. For one thing, the system has a different Ethernet card. Now, either that card is flaky or the Linux driver for it is, because the system regularly gives up the ghost and all but crashes: TCP connections to open ports hang without response; processes can no longer be forked; even syslogging stops.

Yet, even if the new hardware had presented no problems, it's inconceivable that a company would move a working Linux (or any other) system to new hardware and just expect it to work. What if I had not had the driver for the new network card compiled for my kernel? My machine would have had absolutely no way of ever getting back onto the network after the migration. It's sheer luck that I can sometimes still log into my machine and that it's not completely dead to the world.

So, the networking on the new hardware is extremely unreliable. rsyncs regularly fail with checksum errors. The more network traffic one pumps over the interface, the more such errors occur. Eventually, the system becomes unstable and eventually unreachable.

It's also possible that the machine has bad RAM or ineffective cooling, either at the CPU or the data centre level. Witness these messages, culled from my log in a rare moment of accessibility.

May 15 06:39:58 ulysses CPU0: Temperature above threshold
May 15 06:39:58 ulysses CPU0: Running in modulated clock mode

The system is now on heavy-duty medication: cold reboots, at first twice daily, but that proved inadequate, so cron now reboots the machine every hour. That's the only way to avoid the machine locking up completely, which then puts me at the mercy of managed.com to reboot it. That's something that now seems to take more than 24 hours to accomplish.

Clearly, this appalling state of affairs can't be allowed to continue, so I'm already on the look-out for alternative hosting providers.

A year ago, when I selected this company to host my services, people seemed happy with it. I, too, was happy with the service until earlier this year. In the last couple of months, however, things have been going downhill, which is never a good portent for the future. Nevertheless, I was not prepared for what has now befallen me. These people are lacking even the most basic system administration skills.

So, what happened? Well, a little research shows that managed.com is not really performing a migration. The hard drives and the data have moved to the other side of the country, yes, but not because managed.com is doing it. No, managed.com has been sold, you see? My data now turns out to be at the mercy of Web Host Plus, so the current disaster is actually largely due to their mismanagement and incompetence.

In fact, it turns out that a great many people are in a similar or even worse state, thanks to this bunch of clowns. Sixty-three pages of utter misery and appalling professional disregard of one's customers come to light.

Anyway, to say that I am in the market for a new hosting provider is an understatement. If you have any recommendations, I'd be glad to hear them. Ideally, they should not be located in the US, due to that country's Draconian legal stance with regard to privacy.

Thanks to Google, I was able to rescue the missing blog entries from the Google cache. I had to add back the article comments by hand, which caused the loss of the original time of entry, but at least the text of the article itself has been recovered.

The week of missing e-mail, on the other hand, is simply gone. Calls to Web Host Plus to make available the missing incremental back-up simply fall on deaf ears.

I'm utterly appalled to experience first-hand how this company has lost my data and now ignores my complaints. I'm left bewildered as to the precise ratio of incompetence to deliberate professional disregard, but I am 100% sure that I have to get my data away from this bunch of wankers as soon as I possibly can.

Until that time, expect the server to be up and down like a yo-yo.

I went back to troubleshooting my expensive box of dysfunctional hardware today, but I actually couldn't test any further. The next thing to check was the power supply, but I didn't have a spare ATX unit to pop in.

Sadly, I had to bite the bullet and take the thing into a trusted PC shop. There, the bloke tried a new power supply, but it made no difference. That leaves pretty much just the CPU and motherboard to test. I bet it ends up being the sodding motherboard. That'll give me another thing to send back to the place I purchased it. What a hassle. Buying hardware on-line is fine when everything works, but when it doesn't...

Anyway, they probably won't even start to look at it until early next week, so I'll forget about it for the next few days.

The first new release of bash completion in more than six months is a reality.

This release features many new completion functions. Many others have been improved or optimised. In addition, a large number of bugs have been fixed, including all known compatibility issues with bash 3.1, the latest version of the shell.

An almost complete change log for this release follows:

• Completion for minicom(1), mtr(8), sysctl(8), smartctl(8), vncviewer(1), invoke-rc.d, update-rc.d and dpkg-source has been added.
• gdb completion of second parameter was broken when first parameter contained white space.
• gdb completion wasn't completing second parameter correctly when it was a file, rather than a PID.
• Ruby ri completion has been broken for some time. This is now fixed.
• Various fixes to work around change in how POSIX quoting is handled in bash 3.1.
• subversion completion has been reimplemented from scratch and integrated into the main file.
• iconv(1) completion has been improved.
• yum(8) completion has been updated for current version of yum.
• ant completion will now make use of complete-ant-cmd.pl, if available.
• cvs(1) completion has been improved with 'update' and 'stat' completion.
• 'aptitude show' now works in the same way as 'apt-cache show'.
• make(1) now also completes on file names.
• MPlayer will now also complete on .flac, .mpc and .3gp files.
• wine will now also complete on .exe.so files.
• unzip will now also complete on oowriter's .ott files.
• xine et al will now complete on .mng files.
• The list of programs completing on .dvi files has been expanded.
• The range of files on which timidity and evince complete has been expanded.
• mkisofs completion now defaults to treating results as file names.
• $DEBUG has been renamed $BASHCOMPLETIONDEBUG to avoid namespace clashes with other software.
• man(1) completion now works correctly on OpenBSD.
• svk and Mercurial completion have been added to contribs.
• Many other small optimisations and fixes.

Most of Sunday was spent assembling our new MythTV box. When I switched it on that evening, an LED on the motherboard lit up and the VFD display at the front of the case displayed a couple of messages, but that was as far as it went. No fans spun up, no sounds emanated from the motherboard; zip, nada, diddly.

I've replaced the RAM, but that didn't help. I tried short-circuiting the start-up pins on the motherboard, but that didn't cause a boot, either. I've double-checked all the connections and everything is fine. I tried reseating the CPU. All to no avail.

The TV tuner card was also broken, so I've sent that back to be exchanged.

My fingertips are still sore from all of the messing around, but at some point I need to try to rule out the power supply as the dodgy factor. If that passes muster, I'll have to suspect the motherboard.

How tedious.

I had hoped to be well into the software installation at this point, but I don't even have the hardware working. I find it really tedious to build computers and this is one of the main reasons why. When things don't work, it's boredom at best and misery at worst, trying to figure out what the problem is.

Dealing with hardware is definitely one of the least enjoyable parts of system administration; as far as I'm concerned, at least; some people love it, of course.

After nearly seven months without a printer, I'm happy to say that we are once again able to print documents from our computers.

I took receipt of an Epson Stylus Photo R800 on Saturday, but had to wait until today to purchase a USB A-B cable. Printers aren't supplied with cables any more, it seems. I ordered the printer on-line and it arrived just in time on Saturday for the shops to be closing. It can also be connected over Firewire, but I decided to hook it up over USB instead.

I looked at a bunch of printers before deciding on this one. I considered the Epson R245 and the R320, which were attractive because of their support for memory cards. Using those, you can stick your Compact Flash card straight into the printer, view your photos on a small LCD screen and print from there. It's a gimmick, albeit a nice one.

Higher-end printers don't have such features, because the thought is that serious photographers would never print straight from the card, anyway. Most serious photographers (of which I am not one) take their photos in RAW format (which is not actually a standard and means something different on every make of camera), post-process them in something like PhotoShop, and only then send them to the printer.

Of course, I don't need to spend €1000 on a printer, but I did want something a little bit better than can be had for €100. Photo print quality was important, but equally important was support for Linux. For this reason, I had to rule out the Canon IP5000 that Fenella suggested. Canons are very poorly supported in Linux.

I also looked at an HP Photosmart 8250, but read too much conflicting information about the quality of its prints. I also didn't want to have to fart around with yet another system of printer drivers. Berkeley LPR, LPRng, System V printing, CUPS, Omni, Foomatic... I've had enough of making UNIX systems print properly over the years. Printing isn't exciting; it isn't sexy; it isn't even interesting. It just needs to work.

And so it ended up being the R800, a decently priced printer with high quality photo prints. Eight separate UltraChrome ink cartridges take care of that, although one of those is actually a gloss optimiser cartridge that avoids bronzing on glossy paper.

Its top resolution is 5760x1440 dpi with a 1.5 picolitre droplet size, but I shudder to think how long it takes to churn out a photo at that resolution. Maybe I'll be pleasantly surprised, though. So far, all I've printed out was a test page.

I researched the Linux support for this printer extensively before deciding to purchase it. Although the printer's been around since 2004, Linux support for it is quite new. Epson apparently has some sort of driver for it, but I wanted to use CUPS, which is as close to a decent printing system as UNIX has ever had. Actually, it is decent, if a little difficult to recall the details.

gimp-print 4.2.7, which is on my Fedora Core 4 system, doesn't support this printer, so I had to download and compile gutenprint 5.0.0-rc2. gutenprint is actually gimp-print, with its name changed to remove the understandable confusion that some people had in thinking that one could only use this package to print from The Gimp.

Anyway, once this bleeding-edge copy of gutenprint had been installed, with all of its PPD files, I was able to configure a printer queue for the R800, using CUPS' rather nice Web interface. A test page rolled out shortly afterwards.

And, just to show that Linux does -- after a little bit of work -- support this printer well, here's an example of how the escputil utility (part of gutenprint) can be used to read the ink levels of the cartridges:

# escputil -i -r /dev/usb/lp0
Escputil version 5.0.0-rc2, Copyright (C) 2000-2003 Robert Krawitz
Escputil comes with ABSOLUTELY NO WARRANTY; for details type 'escputil -l'
This is free software, and you are welcome to redistribute it
under certain conditions; type 'escputil -l' for details.

         Ink color       Percent remaining
            Yellow                     100
           Magenta                     100
              Cyan                     100
       Matte Black                     100
       Photo Black                     100
               Red                     100
              Blue                     100
   Gloss Optimizer                     100

Handy, eh?

Anyway, no longer will I have to bike over to Jo's house when I want to print an important document. Thanks for that, Jo. You're free of me now!

After much dithering, I have finally ordered nearly all of the hardware needed to build our MythTV box, an open source equivalent of the TiVo, with which we had so much fun in the US.

I'd been enamoured with the idea of building my own intelligent DVR ever since Peter, my office mate back in Mountain View, unpacked his MythTV hardware in front of me and explained that he was going to build one. That was last July or so, right after having a baby and right before leaving the country. Needless to say, I had no time back then to be building new computers.

I did wonder if I should purchase the hardware for such a box before leaving the US, given the cheaper prices over there, but I decided not to. After all, I might never get around to actually building the box at all, which would be rather a waste. And, even if I did get around to it, better hardware would likely have hit the market in the interim.

So, within a few days, I should have the hardware in house and can start assembling the system. I hate building computers, so I'm looking forward to having that part done and then getting onto the software configuration part, which is much more my thing. I'll detail the hardware once I have it all assembled, as I'll then be able to report on any oddities that I experienced while putting the system together.

DVR DIY is much more expensive than simply purchasing a TiVo, but then again, that isn't an option in The Netherlands. TiVo doesn't sell their system over here. One or two people offer a Windows Media Center Edition computer in the form of a household appliance, and that can do more or less the same thing, but those are expensive for what they are and, whilst it would be easy to set up, I can't quite bring myself to run Windows on a box that will be central to our TV viewing. Within a very short span of time, I would inevitably run into annoyances or shortcomings in the software and I'd be stuck. With MythTV, since it's open source, I can just fix the problems.

Besides which, I can make the MythTV box do so much more. Since it's just a Linux box at its core, it can do anything Linux can. For example, how about a webcam on top of the box, so that we can call up the family in the US and watch them on TV while we chat to them? How about streaming music to the TV from our file server in the cellar? Anything is possible, really.

One problem that will be a little tricky to solve is getting the MythTV box to change the channel of the UPC Mediabox, the set-top box that decodes the encrypted digital TV signal. I'm going to need some form of IR blaster for that part. I have one in mind, but it will mean having it shipped from the US.

Anyway, this should be a lot of fun to set up (as well as a black hole for my time), but by the end, we should have a box that will do pretty much anything we want, when it comes to finding programmes to watch and record. It'll be very cool to have essentially assembled the system from scratch, having picked out the hardware by hand and then installed the software over the top.

Watch this space.

The upgrade from Gallery 1.x to 2.x was quite involved. As has happened before (and to my great chagrin), albums and photos whose titles, summaries and descriptions contained foreign letters like 'ð', 'þ' and 'œ', as well as accented letters like 'é' and 'ö' suffered GBH in the transition.

Fixing this meant writing a pile of Ruby DBI code and testing it very well before letting it lose on the MySQL database in which everything is now stored by Gallery 2.x. The end result, happily, is that things should now be more or less back to normal.

There may still be a few sporadic instances of spurious characters appearing, which seems to have been caused by some titles, summaries and descriptions accidentally undergoing double-encoding to UTF-8, so that the previously encoded multibyte UTF-8 form of a few foreign and accented characters was re-encoded, producing strings of gibberish. If you find any of these, please let me know which photos are affected and I'll correct them by hand.

You'll find all of our photos in our gallery. If some albums seem slow, that's because thumbnail generation is delayed until the first viewer enters that album. Once an album has been viewed by someone, the thumbnails are stored for future use, so subsequent page renderings will be much quicker.

I have just released Ruby/LDAP 0.9.3.

This is mostly a bug-fix release. The changes from 0.9.2 are listed below:

• LDAP::Schema#names and LDAP::Schema#attr will now allow names with hyphens and/or underscores.

• A warning about @sasl_quiet when run in debug mode has been silenced.

• Uninitialised data structures in LDAP::SSLConn#bind and LDAP::SSLConn#simple_bind have been fixed.

• Ruby/LDAP now builds properly with OpenLDAP 2.3.

• Build-time options --with-ldap-incdir and --with-ldap-libdir have been replaced by --with-ldap-include and --with-ldap-lib. This is a consequence of making extconf.rb more standard.

• The Windows build has been improved, so that it should now at least build without error. Whether it will work is another matter.

It's good to get a new release of this out the door, even if the changes are very minor. I did all the major work I wanted to do on this library in previous versions, mostly for the 0.9.0 release. In particular, adding the ability to manipulate LDIF data was very important to me at the time. Now that I'm not working and actively using LDAP, I'll have to find my motivation to work on the code from some other source.

In the future, it would be nice to improve the quality of the Windows build. Various people report varying degrees of success in getting the software to work under Windows, but since I have no Windows build system, I can't really work on the code. Progress is pretty much restricted to integrating any patches I get sent.

As a longer term project, it would be nice to be rewrite Ruby/LDAP in pure Ruby, rather than wrapping the C libraries, as is currently the case. Ruby/LDAP will always have better performance than a pure Ruby solution, however, so that's probably best started as a separate project.

Sarah went out this morning to her first La Leche League meeting (you should blog about that, Sarah), so I seized the opportunity to upgrade Gallery from version 1.5.2-pl2 to 2.1rc1. The new version has lots of nice new features, including vastly improved security when it comes to keeping one's photo repository outside of the Web server root.

On the other hand, it required the configuration of a MySQL database and, for some odd reason, this version has lost the ability to auto-rotate portrait photos on import, based on a bit in the EXIF tag. Why wasn't this feature carried over from version 1?

Anyway, all of this was in preparation for uploading the long-overdue photos detailing months eight and nine in the life of the world's most beautiful baby.

In addition to these two new albums, the Tall Timbers album contains a lot of new photos from our recent trip to see Fenella, Tim, Cameron and Willow, out in rural Maryland, USA. Some of those are really lovely, so if you're the kind of person who coos over cute children, you won't want to miss that one.

Phew! That should keep my mother-in-law quiet for another month.

The time has finally come to resurrect my old office from Mountain View. I put my desk together Monday evening and then set about unpacking my computers and turning them back on after nigh on six months of disuse. My file server happily whirred into action, using the new power cord I purchased from Media Markt on Saturday.

My workstation, however, was a different story. The dreaded click of death familiar to anyone who has ever lost a hard drive in active duty defiled my ears with its filthy rattle. My heart sank. I knew immediately what this meant.

That's what can happen when a hard drive kept in constant use for months on end is suddenly spun down, packed into a box, shipped across the Atlantic in God knows what kind of temperatures, kept in storage for months in God knows what kind of temperatures, then plucked from a box months later, connected to the mains and optimistically called back from its slumber as if nothing had ever happened. Some drives can't take that kind of abuse and give up the ghost. Mine was one of them. Most of the stuff on there had been backed up, anyway. My home directory was networked from jiskefet, which was, itself, backed up before leaving the US.

In short, this isn't a disaster, but it is a major irritation, because now I have to install Linux again from scratch on my workstation. That means compiling a bunch of local programs, too.

So, I had to go out Tuesday morning to buy a new hard drive. I picked up a 400 Gb Maxtor from MyCom on the Ceintuurbaan. I also bought a USB 2.0 and Firewire card to put in jiskefet, the file-server. That machine is also used for making back-ups to the external hard drive that I purchased a few days before our departure from the US. I had noticed back then that making these back-ups was painfully slow and it turned out that its USB ports were only USB 1.1 devices. I should have bought a USB 2.0 PCI card and fitted it there and then, but there were so many things to do in those last few days that it never happened. Anyway, jiskefet now has such a card fitted. I'll connect the external hard drive to it tomorrow.

Anyway, to cut a long story short, I've fitted my workstation with the new 400 Gb drive and started to reinstall the operating system, but it's a slow process, choosing all those packages and the like. I'm putting Fedora Core 4 back on it, even though 5 will be out in a few days. Friends are currently raving about Ubuntu, but I don't have time to look into a new distro at this point in time.

Another thing happening on Tuesday was that the electricians came back to complete the Ethernet work they had begun for me. I can now happily report that the house is currently wired with CAT5 in most rooms. Wireless 802.11 is all well and good, but sometimes one wants the stability and security of a wired connection. Besides, more and more household appliances can be connected to a network these days, so it's good to have ports in most rooms.

I also had to buy a new mouse, because the old cordless one I was using had a power adapter that couldn't handle 220V. Anticipating this kind of thing, I had bought a voltage converter at Fry's before leaving California, but the shape of the adapter plug meant I couldn't plug it into the voltage converter. Doh!

To fix the problem, I bought myself a Logitech G7 mouse. This doesn't plug into the mains at all. Instead, its base station charges battery packs that then slide into the mouse. The base station is powered by one of the PC's USB ports. It's a great mouse, but there was nothing wrong with the old one. It would have been sufficient to purchase a Logitech 220V adapter, but, of course, no-one sells those separately.

I hope to find the time to finish installing my workstation tomorrow, at which point I can begin configuring the system again. With a little luck, I'll be able to stop the day's other chores from getting in the way.

I'd forgotten to change the time on my digital camera from PDT to CEST, so all of the photos I've taken since getting here have timestamps and EXIF headers that are nine hours off. Rather than live with this, I quickly cooked up the following one-liner to rectify matters:

 for i in *.jpg; do cp $i /tmp; jhead -ta+9:00 /tmp/$i; \
 time=$(ruby -e "puts Time.at("$(($(stat -c '%Y' $i)+9*60*60))")"); touch -d "$time" /tmp/$i; done

There's got to be a better way, though, right? I mean, it would be nice to feed the output from stat right back into touch's -d option without having to use Ruby to convert it to a human-readable string. Note that the file must be touched after one has modified its EXIF header, not before, because modifying the header also resets the modification timestamp.

Anyway, at 03:00, this is the best I can manage and it gets the job done.

Picking up my e-mail on my new Nokia Communicator 9500 was becoming annoying, because the self-signed certificate on my mail server is untrusted by my device. In these circumstances, the 9500 will ask you evey time it picks up your e-mail whether this untrusted certificate should be used. Unfortunately, it offers no option to register the certificate as trusted until its expiry.

If you're in a similar situation and, assuming you're running Linux on your mail server, here's what to do.

Firstly, convert the mail certifcate from PEM to DER format:

 openssl x509 -in /usr/share/ssl/certs/mail.pem -inform PEM -out /tmp/mail.der -outform DER

Next, copy the DER certificate from your mail server to your phone. I scp'ed it from my mail server to my laptop and then sent it via Bluetooth to my phone, where I saved it to my MMC card.

Finally, go to Control Panel|Security|Certificate manager on the 9500 and select Add. Select the file containing your certificate and add it. You should now be able to see it in the list of certificates. Now, select your certificate from the list and choose View details followed by Trust settings. Change the setting for Secure networking from No to Yes.

At this point, you should be able to pick up your e-mail without confirming each time that you want to trust the untrusted certificate. If it still doesn't work, make sure that you have filled in as the server name the exact same name used in the certificate, not an alias that points to the same IP address. The 9500 will use the certificate only if the server name it contains matches that in your e-mail settings.

I don't like to leave things to chance, and I really don't like to leave my data to chance, so I decided that I need an extra layer of security with our upcoming move. With all of my computers soon to be in the hold of a ship, what happens to my data if the ship sinks? Exactly.

I'm planning to carry my laptop onto the plane, but it has no room for my data (loaded with OGG files, you see), so I needed an alternative. A quick trip to Fry's later and I was the proud owner of a 400Gb USB Seagate drive.

That baby has now been formatted with a single ext3 file-system and is now happily soaking up all of my CVS source code, OGG files, photos, Web site pages, home directories and what have you. There'll be bags of room to spare when it's finished, too. The only bummer is that my fileserver appears to have only USB 1.1 ports. It's a good thing I'm not in a hurry, but I'll have to fix that at some point in the future.

When we leave for Amsterdam, that drive is going with me onto the plane. I don't want a dirty great thing like that in my hand luggage, so I'll probably entrust it to the hold of the plane. What are the chances that our suitcases get trashed and the boat carrying our belongings sinks or suffers flooding? Never say never, I suppose, but it seems unlikely. And if the plane goes down, well then, I won't be needing my data after all.

Today is the 6th annual System Administrator Appreciation Day, so go on, appreciate me!

Briton Gary McKinnon was recently arrested and is awaiting extradition to the US. He is charged with penetrating US military networks and causing huge amounts of damage, although this BBC radio interview with him paints a more subtle picture.

Whilst he did, indeed, penetrate US military computers, he did so with the default administrator passwords of Windows systems that had been connected to the public Internet. Gary is facing 70 years in a US prison. If he goes down, they ought to send the system administrator of those boxes to share a cell with him, because allowing systems like that to hang on a public network is nothing short of criminal negligence.

Anyway, the interview is fascinating, because Gary claims to have been in search of and found proof of US involvement in the use of extraterrestrial technology. Gary talks candidly and lucidly about his experiences and paints a picture of himself as a tragically obsessed individual whose life became synonymous with the pursuit of more information. If he is to be believed, he caused no damage at all, beyond creating a clinical need to reinstall the systems through which he passed, as these had now been compromised and could no longer be trusted. Of course, any system whose administrator password has not been changed before it is connected to a public network is, by definition, untrustworthy and must be regarded as compromised.

Fascinating listening. Check it out.

I was lucky enough to have a seat today in Kirk McKusick's course, FreeBSD Kernel Internals, based on his book, The Design and Implementation of the FreeBSD Operating System. Today was the first day of the course, which has been organised by Google and is being held on-site on Thursdays in February.

How nice to be able to spend one day a week listening to a recognised expert on the UNIX operating system, as he runs through the design and implementation of the system that's been keeping me in work all these years. And to think I get paid to sit there and listen to him, too. Poor me.

The Finance department was having an ice-cream party when we stopped for a break in the mid-afternoon, so we even gatecrashed their party and tanked up on dessert.

It's a hard life sometimes.

Computers can do so much these days, or can they? Some things are just too hard to set up for the basic need they address.

I spent a significant chunk of this evening configuring my laptop to use my GPRS mobile phone as a modem. It's a Sony Ericsson P910i. The Google employee ski-trip is coming up a couple of days from now and it'll be nice to be able to get on-line from the hotel room; or the bus on the way up, for that matter. Yes, it's sad, but I like to be able to get on-line at the drop of a hat and Opera on my phone just doesn't cut it. Using the command line over ssh is even worse, as the phone's keyboard is too awkward to use and doesn't even support the sending of control characters.

Anyway, after recompiling my kernel to support Bluetooth (how'd that happen? I was using Bluetooth three months ago and I had support configured in then.), I configured Bluetooth support. This involved scanning for my phone, picking up its MAC address, querying it to find out on which channel it supports DUN (dial-up networking), and then binding the Bluetooth RFCOMM system to that channel, which gave me a /dev/rfcomm0 device to play with.

Then it was a simple (simple?) matter of running pppd over that device. pppd is one of those pain-in-the-arse programs that is just never simple to configure or debug. chat scripts full of Hayes AT commands; lots of low level options to mess with; copious debugging output to wade through; configuration scripts that are called after a connection is established. And then the damn thing fails to write a /etc/ppp/resolv.conf, so I have to do it manually.

Why can't things be easier? I first configured pppd back in 1995, if I remember correctly. Now, some 10 years on, you'd think I'd be an expert, but some things just remain hard, because you never really use them enough to build up expertise with them.

I'm reminded of 1997, when I wrestled endlessly to get ipppd (the synchronous version of pppd) working with a passive Teles ISDN card plugged into one of my ISA slots. What an unbelievable amount of hassle that was to get working. Ethernet is very simple by comparison.

But I digress. Computers are still in their infancy, really. Yes, they can do a lot, but we humans have to do an awful lot to get our computers to do anything. And I should smile, really, because it's the fact that not everyone has knowledge about such things that keeps people like me in demand with employers. Ideally, though, things would be different and everything would just work.

Anyway, like most computer users, I eventually got the thing I was messing around with to work. I'll now have Internet access from the hotel room in Tahoe, assuming my phone has reception in that area.

I didn't get enough sleep last night. (Note to self: try to avoid sounding like the person at work who blames all of his mistakes on lack of sleep and excessive devotion to the cause.)

Jules, Geoff and I went out for some Indonesian food at Orient, one of my favourite restaurants (on the Van Baerlestraat), then biked into De Pijp for some drinks at my old stomping ground, Café Krull. By the time we got back to our hotel, it was very late, but I still had a few things to do on the computer, like chat to my lovely wife, back in California.

SANE 2004 is progressing nicely. Gerald Carter's Tuesday Samba tutorial was very good, but largely irrelevant to me. We don't use Samba at Google and, besides, the tutorial was really for people who are currently running 2.2 servers and want to know what they can expect from 3.0.

Gerald Carter also hosted my Wednesday tutorial, which was about implementing [Open]LDAP. This was much more relevant to me, but my own knowledge of LDAP is fairly extensive, so I didn't pick up much new from the session. Nevertheless, it was still useful to listen to Gerald speak, as he recapped a lot of the work I've been involved in over the years and basically covered all of the material from his book in just eight hours. It was quite an involved talk, as you might imagine.

The tutorials ended yesterday with the Free Software Bazaar, which basically consisted of a talk by Richard Stallman. I'd seen the same talk before at Google a couple of months ago, but it was still amusing and gratifying to hear him compare one's moral obligation to copy software for one's neighbour to one's moral obligation to save a drowning man, unless that man be Bush, Rumsfeld, Ashcroft or Kerry. I do respect people who dare to speak their mind

The conference proper began today. The keynote by Paul Kilmartin was about eBay's infrastructure through the years and went into a surprising amount of detail. Not a bad talk for a bigwig.

Wietse Venema discussed lessons learnt from open source security programming later in the morning, which was another very well-attended talk. He discussed the release and publicity surrounding each of TCP Wrappers, SATAN and Postfix and contrasted them.

Now it's the late afternoon and I'm starting to feel woozy. I've got to hang in there, though, because the social event is coming up this evening, which translates into a boat trip along the IJ river. That'll be a lot of fun if I can stay conscious.

On a sadder note, the occupants of a car returning to Amsterdam from Paris after dropping off Richard Stallman were involved in an accident that cost one of them his life; a compelling reminder of how suddenly one's fortunes can permanently change.

Tomorrow is the final day of SANE. It's hard to believe that Geoff and I began our journey to The Netherlands almost a week ago. Time passes very quickly in Amsterdam, unlike in Silicon Valley, where the weeks seem to drag on forever. On the other hand, it's gratifying to know that I'll see Sarah again on Monday, as I'm missing her and looking forward to being reunited.

After a weekend spent showing Geoff around Amsterdam, we're at SANE 2004 for the next 5 days. We're currently in the Black Hats tutorial, which rather amusingly started with a number of examples of how you can use Google to find password files and such like. I knew that people did this, but it was amusing to see all of the good hacks in a single session. Some very creative use of Google's advanced features was demonstrated.

Incidentally, there's a public transport strike in Amsterdam today, but we hired bikes and rode here, so experienced no hassle.

It's good to be home and it makes me look forward to returning here on a permanent basis, the changeable and gloomy weather notwithstanding.