Caliban - Opinion and Righteous Anger

In the absence of female distraction, I took the opportunity in recent weeks to upgrade all of the machines in the house to Fedora 7, including the MythTV box.

A couple of years ago, with a few weeks still to go before the birth of Eloïse , I took advantage of the calm before the storm to move my domain to dedicated hosting at managed.com. Unfortunately, and as I've documented in the past, that company turned out to be less than dedicated, so after a year, caliban.org ended up back in our cellar, hosted over a domestic DSL line.

With the girls out of the country for a while, the time was ripe to move the domain back out to dedicated hosting. The DSL line has been incredibly reliable, but there's always the chance that it will go on the blink while we're travelling. Moving house would also automatically mean downtime, which is out of the question when one is responsible for one's own domain e-mail. Downtime means lots of bounced e-mail, not to mention an unreachable Web site.

I'd done my homework before the girls left for their trip, so with the new contract signed, the slow process of copying all of our files over the slow upstream DSL link to the new server began. The process wouldn't be completed until approximately ten days later.

As you read this, all services have been migrated to the new machine (and have been for over a week).

For you, the user, there shouldn't be much noticeable difference, except that browsing our photo gallery should be considerably faster than before. For me, however, it's nice to know that the continuity of our e-mail and Web site is no longer tied to our home DSL being up.

The site seems to be running nicely on the old server in the cellar. I'm sure browsing photos isn't a terribly fast experience any more, but until I can find a reasonably priced hosting provider I can trust, this is the way it has to be. As detailed here, my last dedicated hosting provider turned out to be less than dedicated and not much of a host to his paying guest.

As I've mentioned before, the new hardware that Web Host Plus put me on after managed.com sold my hard drive to them was less than reliable. I suspected a few causes, one of which was bad RAM. This theory now seems to have been borne out by an experiment I did. Before I rsynced our photo gallery across the Atlantic, I rebooted the problem box in New Jersey and reduced its operational RAM from 2 Gb to 512 Mb. In that new configuration, I was able to spend many a joyous hour copying my precious data over the transatlantic pipe without the originating box going AWOL. QED, I'd say.

I'll be cancelling service with managed.com, Web Host Plus or whoever's running the show now as soon as my next invoicing cycle starts. Good riddance to bad rubbish, as they say.

In the meantime, any DNS oddness you may have been seeing should now be a thing of the past. All slaves are in sync and handing out correct data. Incorrect data in the caches of other DNS servers should now also have expired. Normal service has now been resumed.

Our photo gallery is back on-line.

I was able to bring it back much more quickly than I had first anticipated, because I used an old Gallery 1.x back-up to seed the albums in the Gallery 2.x installation. That copied hundreds of extra, unnecessary files, but they were easily removed afterwards by rsync.

I did this by getting a list of all the album directories on the remote server:

cd /var/www/g2data/albums
find -type d > /tmp/file_list

I then copied this over to the new server with the Gallery 1.x back-up:

for i in `cat /tmp/file_list`; do
  album=${i##*/}
  src=`find /var/www/html/albums -type d -name $album`
  [ -n "$src" ] && rsync -av $src/ /var/www/g2data/albums/$i
done

In the end, I needed to copy from New Jersey only the photos we had taken since mid-February, which is when I had done a full back-up in preparation for migrating from Gallery 1.x to 2.x.

Somehow, one of the tables in the MySQL database had got corrupted in the move:

060518 18:02:38 [ERROR] Got error 134 when reading table './gallery2/g2_ImageBlockCacheMap'

This was easily corrected:

mysql> repair table g2_ImageBlockCacheMap;
+--------------------------------+--------+----------+--------------------------------------------+
| Table                          | Op     | Msg_type | Msg_text                                   |
+--------------------------------+--------+----------+--------------------------------------------+
| gallery2.g2_ImageBlockCacheMap | repair | warning  | Number of rows changed from 45465 to 45460 |
| gallery2.g2_ImageBlockCacheMap | repair | status   | OK                                         |
+--------------------------------+--------+----------+--------------------------------------------+

And, with that, the rescue and salvage operation to yank caliban.org from the incompetent clutches of the unholy alliance of Managed.com and Web Host Plus is 95% or more complete.

Once the residual DNS propagation issues evaporate, I'll be able to fully exhale once again.

Bas pointed out to me this morning that those of you who use SPF as an anti-spam measure may be flagging mail you receive from me as spam.

This is due to inconsistencies in the answer that DNS is currently returning as the canonical address of caliban.org. These problems should evaporate by the end of the weekend, at which time all DNS servers that serve name information for caliban.org should be synchronised.

Much of the last 24 hours has been spent seizing those rare moments during which my server -- migrated through no desire of mine to Web Host Plus -- is up and on the network, and using them to perform a migration of my own, namely to the server in my cellar.

I'm knackered, but a lot has been accomplished today. DNS and e-mail have now been fully migrated, including Web mail and mailing lists. The Web site, too -- which you're now reading -- is also up and running on the new (well, actually quite old) server.

The main thing that's not yet back up is our gallery of photos. That's because it's 19 Gb of data, which would be slow to copy from a reliable server on a fast network. Well, I have to copy it from a machine that keeps crashing and is not on a fast network. It could be a couple of weeks before I manage to get all of my data off it... if I'm lucky. I don't want to even contemplate the notion of not being able to recover all of my data.

I thought I'd left this kind of sysadmin drudgery behind me when I stopped working. Indeed, I moved my domain to dedicated hosting to reduce the downtime and maintenance that I had repeatedly endured when I hosted it myself on a domestic DSL line. Little did I know that I would get to enjoy such advantages for barely a year before falling victim to the worst kind of professional incompetence: that with no sense of responsibility for one's customers.

And so caliban.org is back on a domestic DSL line, albeit one that has proved itself more reliable than those I had in the US. The upstream bandwidth is also somewhat better.

Anyway, don't bother looking for new photos -- or old ones -- in the near future. I'll announce when -- if? -- they're once again available.

Yes: wankers. Wankers! WANKERS! WANKERS! WANKERS!

Who am I talking about? Managed.com, of course, the company to which I give good money each month to host this site.

What happened?

Well, managed.com decided to move its network from California to New Jersey. At least, that's as much as they told us, the paying customers.

In preparation for this, they sent all of their customers an e-mail asking to be supplied with the customer's root password via plain text e-mail. For those of you who aren't in the field of computer system and network administration, let me state that this is a violation of one of the most basic and universally lauded principles of the profession: never, under any circumstances, send passwords in the clear.

And yet, my hosting provider was asking me to do just this. In hindsight, that should have been enough to spur me into action. I should have found another hosting provider, right there and then, and moved my data prior to the migration. But I decided to wait until after the migration to seek a better provider. As always, laziness, compounded by a failure to recognise the urgency of the situation, won out.

Anyway, managed.com were supposed to back-up their customers' data, firstly with a full back-up and then, shortly before the migration, with a further incremental back-up. The migration was supposed to be barely noticeable, with a guaranteed maximum of two hours of downtime.

I was sceptical, but kept my fingers crossed.

Can you believe that managed.com didn't tell its customers in their notification e-mail when this migration would actually take place? We were left to guess. E-mails to them on the subject went unanswered, as did requests for a secure channel through which to supply one's root password.

When I noticed one day that my machine had been rebooted without my permission, I incorrectly assumed the migration had already taken place. If I'd known at that time that things would be moving to New Jersey, not just around the corner in California, I could have run a traceroute and seen that my machine had not actually gone anywhere. At that time, however, I thought they were just moving locally. What else could I think? Managed.com had told me virtually nothing in their e-mail.

caliban.org mysteriously went off the network on 9th May. It remained inaccessible for almost three days. So much for the two hours of guaranteed downtime.

All of my e-mails to managed.com went unanswered in this period. Only when I threatened them with legal action (a trick I picked up in America), did they finally respond by rebooting the machine and getting it back on-line.

Naïvely, I thought that would be an end to my problems. Yes, that was very naïve of me.

You see, managed.com restored my service from a week old back-up. I've no idea what happened to the promised incremental back-up. It was probably never made and, even if it was, it would have had to be of the last week's worth of data, not just the day before the migration. I suspect it was never even made, however.

The net effect? I found I was missing a week's worth of e-mail, multiple DNS changes had been lost, the last week's worth of blog entries had effectively never been written, and sundry other less serious issues now needed to be fixed, such as recent software updates becoming undone.

More e-mails to managed.com went unanswered. Due to an oversight on my part, my own off-site back-ups had not taken place in recent times, so I had no private back-up from which I could recover my data. Typical.

I began work on the system to repair the damage my hosting provider had done to it, but before I could achieve very much, the system went down again. The system was off-line again for more than a day. Once again, e-mail threats were required to get it back on-line.

So what's going on?

Exploration of my system's log messages shows that the new hardware on which my data resides is not the same as the old. For one thing, the system has a different Ethernet card. Now, either that card is flaky or the Linux driver for it is, because the system regularly gives up the ghost and all but crashes: TCP connections to open ports hang without response; processes can no longer be forked; even syslogging stops.

Yet, even if the new hardware had presented no problems, it's inconceivable that a company would move a working Linux (or any other) system to new hardware and just expect it to work. What if I had not had the driver for the new network card compiled for my kernel? My machine would have had absolutely no way of ever getting back onto the network after the migration. It's sheer luck that I can sometimes still log into my machine and that it's not completely dead to the world.

So, the networking on the new hardware is extremely unreliable. rsyncs regularly fail with checksum errors. The more network traffic one pumps over the interface, the more such errors occur. Eventually, the system becomes unstable and eventually unreachable.

It's also possible that the machine has bad RAM or ineffective cooling, either at the CPU or the data centre level. Witness these messages, culled from my log in a rare moment of accessibility.

May 15 06:39:58 ulysses CPU0: Temperature above threshold
May 15 06:39:58 ulysses CPU0: Running in modulated clock mode

The system is now on heavy-duty medication: cold reboots, at first twice daily, but that proved inadequate, so cron now reboots the machine every hour. That's the only way to avoid the machine locking up completely, which then puts me at the mercy of managed.com to reboot it. That's something that now seems to take more than 24 hours to accomplish.

Clearly, this appalling state of affairs can't be allowed to continue, so I'm already on the look-out for alternative hosting providers.

A year ago, when I selected this company to host my services, people seemed happy with it. I, too, was happy with the service until earlier this year. In the last couple of months, however, things have been going downhill, which is never a good portent for the future. Nevertheless, I was not prepared for what has now befallen me. These people are lacking even the most basic system administration skills.

So, what happened? Well, a little research shows that managed.com is not really performing a migration. The hard drives and the data have moved to the other side of the country, yes, but not because managed.com is doing it. No, managed.com has been sold, you see? My data now turns out to be at the mercy of Web Host Plus, so the current disaster is actually largely due to their mismanagement and incompetence.

In fact, it turns out that a great many people are in a similar or even worse state, thanks to this bunch of clowns. Sixty-three pages of utter misery and appalling professional disregard of one's customers come to light.

Anyway, to say that I am in the market for a new hosting provider is an understatement. If you have any recommendations, I'd be glad to hear them. Ideally, they should not be located in the US, due to that country's Draconian legal stance with regard to privacy.

Thanks to Google, I was able to rescue the missing blog entries from the Google cache. I had to add back the article comments by hand, which caused the loss of the original time of entry, but at least the text of the article itself has been recovered.

The week of missing e-mail, on the other hand, is simply gone. Calls to Web Host Plus to make available the missing incremental back-up simply fall on deaf ears.

I'm utterly appalled to experience first-hand how this company has lost my data and now ignores my complaints. I'm left bewildered as to the precise ratio of incompetence to deliberate professional disregard, but I am 100% sure that I have to get my data away from this bunch of wankers as soon as I possibly can.

Until that time, expect the server to be up and down like a yo-yo.

I've spent the last couple of days tweaking a few small aspects of the site. The CSS that lays out the main page's three column look is now as it should be, I believe. Previously, it was a rather hit-or-miss affair, with the potential for the AdSense to overlap the main, centre column at low resolutions. Everything should display properly now at any resolution.

The Amazon links in the sidebar now provide an image of the product to which they refer in the small pop-up that appears when you run your mouse over the link.

In case you're interested, I thought I'd document the various bits and bobs on which this site runs. The following components are used to bring you caliban.org.

The server is Apache 2.0.52, running with PHP 4.3.11 and mod_ruby 1.2.4.

The blog is constructed around Movable Type 3.17, with MT-Blacklist 2.04b and the latest beta snapshot of SpamLookup providing anti-spam capabilities. New blog entries are entered using Markdown, rather than raw HTML. I find Markdown to be much more convenient, as it simplifies the commonly needed stuff, whilst still allowing the full spectrum of HTML to be used as necessary.

The Amazon products are periodically pulled from Amazon's AWS API, using Ruby/Amazon 0.9.0 and stored in YAML format. These are then reloaded and processed dynamically when the front page is requested, thanks to some server-parsed Ruby scripting within the page, interpreted using eRuby. The final display of the data in a nice little pop-up is handled by version 4.17 of Erik Bosrup's excellent overLIB library for JavaScript.

The stock quotes come from version 0.2.1 of Ruby/Finance. The quotes are regularly refreshed and saved as YAML. The server-parsed Ruby code loads the file and displays the quotes whenever a page request is made.

Last night, I moved e-mail service to the new hosted server. That was more or less painless. The only grief I caused myself was in upgrading Postfix as part of the process, which rendered my version of one of the configuration files incompatibly old. Once I figured that out, all was well.

Since I run the Cyrus IMAP server on the back-end, I'd been forwarding my own personal e-mail off to a separate server that does traditional spool delivery, since that's how I like to read and manage my own mail. The goal now, however, was complete independence from my DSL line, so I had to move away from this model and consolidate spool delivery with the database-driven Cyrus model. I completed this work this afternoon.

With that, caliban.org is now completely independent of my Speakeasy DSL line. It has good bandwidth and is hopefully more or less immune to power blackouts. Reverse DNS has been set up and I have new secondary DNS servers for the domain, as the previous ones had been somewhat unreliable.

Only a few stragglers are still hitting the Web site at the old address. They're given an HTTP 301 to send them to the new server. Within a few days, those should completely dry up.

I'm glad to have all of this out of the way before the baby arrives, as there will be precious little time available once he's here.

Hopefully, those of you who've struggled with viewing our photos in the past will enjoy the increase in speed. Multi-megapixel photos should now load a lot more quickly.

If you've had any connectivity problems with this site over the last couple of days, that's because I've been busy moving the site to a new home, over at managed.com.

Every time we move house, I have the headache of trying to keep caliban.org on the net. That's because it's been hosted on domestic DSL since I moved to the US, so every time we move house, I have to get DSL up and running in the new house before it's shut off in the old one. This is a real pain.

Not only that, but domestic DSL doesn't offer you much upstream bandwidth to serve files, which can be a real bottleneck with RPM packages and our photo gallery. Whenever the site was hit hard by crawlers, browsing the Internet from home or working over ssh from the office was agony. With our move to managed.com, that should be a thing of the past.

The Web site has already been moved, along with DNS service. The last thing to go will be e-mail, which should be completed by the end of the weekend, if not before.

Getting the site ready to roll at the new location has involved me syncing large parts of the site to my laptop, which I then bring to work, where I rsync the data over to the new server. Like I said, the upstream bandwidth of domestic DSL is pathetic, so this was the only way to transfer the data in a reasonable amount of time. To copy it from home would probably have taken a couple of weeks.

Anyway, this will free us up to move house at will. Whatever we do, the site will stay up and the e-mail will keep flowing. We're paying $95 per month for the privilege, but I think it's money well spent for the peace of mind it gives us.

Since I wrote about my new anti-spam measures, the spam has been furiously banging up against my virtual front door.

Talking to a colleague on IRC tonight, I was inspired to write a quick Ruby script to report the progress since last Sunday:

 #!/usr/bin/ruby -w

 reject = Hash.new( 0 )

 while line = ARGF.gets
   case line
   when /un(verified|deliverable) address/
     next
   when /554 Service unavailable.* (blocked using .+?);/
     reject[$1] +=1
   when /NOQUEUE: reject:(?:.+?:.+?: )(.+?)[;:] from/
     reject[$1] +=1
   when /reject: header .+helo=.+?: (.+)$/
     reject[$1] +=1
   end
 end

 total = 0
 reject = reject.to_a.sort { |a,b| a[1] <=> b[1] }
 reject.each do |x|
   printf( "%-74s%5d\n", x[0], x[1] )
   total += x[1]
 end

 printf( "\n%-74s%5d\n", "Total blocked:", total )

Here are the results:

Bad attachment with file name extension: bat                                  1
Bad attachment with file name extension: cpl                                  1
Sender address rejected: need fully-qualified address                         2
Sender address rejected: Improper use of SMTP command pipelining              3
Bad attachment with file name extension: exe                                  5
Bad attachment with file name extension: com                                  8
Relay access denied                                                           9
Bad attachment with file name extension: scr                                 12
Bad attachment with file name extension: pif                                 25
Helo command rejected: Improper use of SMTP command pipelining               27
blocked using sbl-xbl.spamhaus.org                                           30
Helo command rejected: Host not found                                        58
Helo command rejected: need fully-qualified hostname                        124
blocked using dnsbl.sorbs.net                                               155
blocked using bl.spamcop.net                                                290
Sender address rejected: Domain not found                                  1619
Recipient address rejected: User unknown in local recipient table          6659

Total blocked:                                                             9028

All in all, I'm very pleased. Very little spam is making it through now. For the spam that does make it into the system, I also upgraded to a recent CVS snapshot of SpamAssassin this afternoon, so most of it still gets zapped before making it to the in-box of any of my users.

I spent a chunk of the weekend working on impoving caliban.org's anti-spam measures. I'd been running with the same configuration for quite a while, which was basically a basic Postfix configuration on the front-end and SpamAssassin on the back-end.

This worked well, but I decided it was time to consider using some real-time blackhole lists (or RBLs, as they're called). Since MAPS turned into a subscription-only service, I haven't been using blackhole lists for the outright rejection of mail, as I didn't think any of them were as trustworthy as MAPS.

I haven't had to do much professional SMTP server administration since 2000, so it had had been quite a while since my last assessment of other RBLs. In light of this, I decided to have a look at how professionally the currently active RBLs are administered.

I was pleased by what I found. A few of the lists appear to be very professionally administered, so I decided to use them for front-end mail rejection as opposed to just back-end spam tagging. Previously, I was using several blacklists in combination with SpamAssassin to tag messages as possible spam, but the messages were then subject to many other tests to determine whether they exhibited other spam-like characteristics.

Using up-front rejection, however, the mere presence of the sending host on one of these lists results in an immediate rejection of the message being offered. There's no second chance to inspect the actual header and body of the e-mail, so one needs a very high degree of confidence in the quality of the blackhole lists being used as the basis for this decision.

SpamAssassin's use of blacklists is a little different, anyway, as it will check the host in the chronologically first Received line to see whether the originating host is on a blacklist. Envelope-time rejection, however, as performed by an MTA, checks only the IP address of the connecting host and possibly the domains that it claims in the HELO and MAIL FROM lines.

I upgraded to Postfix 2.1 a few weeks ago. After this weekend's fine-tuning, the relevant part of main.cf now looks like this:

strict_rfc821_envelopes = yes
strict_mime_encoding_domain = yes

# Use this when reject_unknown_hostname is true
unknown_hostname_reject_code = 550

# Use this when reject_unknown_sender_domain is true
unknown_address_reject_code = 550

# Use this when reject_unverified_sender is true
unverified_sender_reject_code = 550

smtpd_client_restrictions = permit_mynetworks
# Lots of 'good' sites have broken reverse DNS
#                           reject_unknown_client

smtpd_helo_required = yes

smtpd_helo_restrictions = permit_mynetworks
                          reject_unauth_pipelining
                          reject_invalid_hostname
                          reject_non_fqdn_hostname
# This traps too many poorly configured good guys
#                         reject_unknown_hostname

smtpd_sender_restrictions = permit_mynetworks

smtpd_recipient_restrictions =  permit_mynetworks
                                reject_unauth_destination
                                check_recipient_maps
                                reject_multi_recipient_bounce
                                reject_non_fqdn_sender
                                reject_non_fqdn_recipient
                                reject_unknown_sender_domain
                                check_client_access hash:/etc/postfix/client_access
                                reject_rbl_client bl.spamcop.net
                                reject_rbl_client dnsbl.sorbs.net
                                reject_rbl_client rhsbl.sorbs.net
                                reject_rbl_client sbl-xbl.spamhaus.org
                                reject_unverified_sender
address_verify_map = btree:/etc/postfix/verify

In the above, I have optimised the order of the controls for network traffic and, by extension, response time. For example, there's no point running most of the tests if we already know that the intended recipient does not even exist. For this reason, most tests are deferred until the RCPT TO comes in.

First of all, we require an SMTP HELO and demand that the other side strictly comply with RFC821. For good measure, we also disallow unauthorised pipelining of SMTP commands. This, alone, will catch some very poorly written spamware in the act.

Next, when the HELO arrives, we check for a validly formed, fully-qualified hostname. Again, a few clueless spammers may be caught out here, but there are no significant gains. Ideally, we'd also reject anyone who passes us a hostname with no matching A record in DNS, but plenty of good sites don't have their act together here and it quickly became apparent that I was going to reject a lot of good mail with this in place.

Next, we let the MAIL FROM stage pass without action, as we are waiting for the RCPT TO before performing our main suite of tests. Once we get the data from the RCPT TO, we'll perform our MAIL FROM checks only if we decide we still need to.

At the RCPT TO stage, we order the tests for minimum network load and processing. We check that the remote side is not trying to relay and then the intended recipient does, in fact, exist. Next, we check for a fully qualified sender domain in the MAIL FROM as well as a fully qualified recipient in the RCPT TO. We also refuse any e-mail that is a multi-recipient bounce, which is another technique used by spammers for squeezing their messages into your server on a technicality. None of these checks requires any additional network traffic.

Next, we check DNS for an A record for the sender domain. If we find one, we then check a whitelist of domains that we always want to accept mail from. Basically, we want to spare them from the upcoming tests, which we can't guarantee they will pass and we always want to receive e-mail from these systems.

Now, the tests get a little heavier on the network. In turn, we check SpamCop, SORBS and Spamhaus for the IP of the remote host. If found, we reject the message on the spot. From my research, these lists demonstrate that they are fairly administered and that the chance of false positives is small. That wasn't always the case with SpamCop, but they seem to have improved a lot in recent times.

Any mail that has made it this far is doing pretty well, but there's one final test we subject it to at this point. We take the address in the MAIL FROM and use it to make an SMTP connection back to the sender, going as far as issuing a RCPT TO with that address, but not following up with the usual DATA section.

The purpose of this exercise is to ascertain whether the ostensible sender has a real account that can be delivered back to. A sender who cannot receive replies is considered spam and we reject the original incoming message. A cache of legitimate sender addresses is built up in order to minimise the amount of work that is needed for the verifcation process.

The directives that set reply codes to 550 are there to ensure that all rejections are permanent. Postfix defaults to a 450 response on these types of rejections, which signifies a transient error. That means the remote host is basically told to correct the problem and try again. Of course, that will never happen, so we cold-heartedly reject the message as soon as we find a problem with it. Otherwise, the remote side will periodically try to resend it and we will keep on needing to check it and reject it.

Since instituting these changes, I've been rejecting an awful lot of e-mail. Some of it is e-mail that SpamAssassin would previously have trapped anyway, but now I'm preventing it from ever entering the system. It's better to reject spam at SMTP-time, rather than accepting it into the system, as the latter course of action is rightfully interpreted by spammers as successful delivery. They don't care that the mail was later filtered and not read; they care only that it was accepted by the receiving server.

Postfix really is a superb piece of software and I highly recommend it. Few MTAs offer so much fine-tuning in the fight against spam, whilst still maintaining legible configuration files.

Well, not quite new management, but I thought I'd try something new. I'm using Dave Thomas's RubLog to give the site a bit of a new look. All of the old pages are still active. The main links that used to form the footer of the front page can now be found in the This Site sidebar, which is off to the left of the page.

I've also changed the way I add content to the site. I'm still using vim as my editor, but I no longer enter HTML directly. Instead, I use Markdown, which is a simplified mark-up language, run through a Ruby library called BlueCloth. Thanks to RubLog's converters, this is all extremely easy and automatic.

Sarah's currently away in England and Ireland, so I'm all on my Todd. I get to live the bachelor lifestyle for about 12 days, at which time I fly home to Amsterdam, where we'll meet up again. I can't wait to get back home for a few days and escape the madness of the US.