Caliban - Opinion and Righteous Anger

It's been pouring with rain here all day, so I seized the opportunity to knock out a quick release of Ruby/AWS.

Version 0.4.0 allows the user to retrieve existing shopping-carts from AWS using the new Amazon::AWS::ShoppingCart::Cart#cart_get method. I hadn't originally implemented this method, because I considered it unnecessary in view of the fact that shopping-cart state was already maintained in Cart objects. Should these objects cease to exist, however, there was no way to resurrect the object from AWS's servers. That has now been remedied.

This version also requests the newest version of the Amazon AWS API, namely 2008-06-26 and fixes a bug in the Amazon::AWS::ShoppingCart::Cart#cart_modify method.

As of this release of the software, I consider Ruby/AWS to be in beta. In other words, you may now consider it fit for non-critical use. It had been that way since version 0.3.0, really, but I was being cautious.

If you're a user of the remote shopping-cart functionality, you should upgrade.

It's only been a week since the release of Ruby/AWS 0.3.2, but 0.3.3 is here already.

Actually, there are almost no visible changes for the user. The only development of note is that the library is now available as a so-called gem for the RubyGems packaging system.

People have been on at me for years -- basically, since RubyGems first became usable -- to package my Ruby software as gems, but I've lacked the motivation to get to grips with a new packaging system.

It turns out that I overestimated the work involved. You can learn to create RubyGems in under an hour. The system is much less complex than building RPMs, for example. I really should have done this sooner.

Ruby/AWS 0.3.2 has been released.

This release addresses (de)serialisation issues with Marshal and YAML, caused by the inability of objects spawned from dynamically defined classes to be reinstantiated at load-time.

The latest release of Ruby/AWS is out. It fixes a few bugs and improves remote shopping-carts by adding support for the Save For Later area of the cart.

Cart#modify now takes an extra parameter in order to support this new functionality. Cart#include? now checks both the active and the saved area of the cart before returning true or false, and there are two new methods, Cart#active? and Cart#savedforlater? for checking the two areas.

Numerous bugs were fixed, too, so all users should upgrade.

Ruby/AWS now has its own mailing-list, where the development and use of this library can be discussed.

I've finally managed to put together version 0.3.0 of Ruby/AWS. The crux of this release is the addition of support for remote shopping-carts. Check out the new Amazon::AWS::ShoppingCart module and the Amazon::AWS::ShoppingCart::Cart class.

The amount of free time I have in any 24 hour period is drastically reduced these days. If this were still 2004, this release would have appeared a lot more quickly. Nowadays, however, the amount of time I can spend on coding and related activities (writing documentation, testing, etc.) is quite limited.

We've also had Sarah's folks over here for the last couple of weeks and been treated to no fewer than thirteen consecutive days of uninterrupted sunshine. Neither fact has been conducive to productive coding.

Anyway, in spite of all of this, the new release is finally a fact. It does feel great to be knocking out useful code again.

With the implementation of remote shopping-carts, the AWS v4 API is now more or less fully supported, save for a few tiny gaps in the functionality of a couple of operations. If I'm not mistaken, Ruby/AWS now supports all of the functionality of its predecessor, the now obsolete Ruby/Amazon, plus a lot more that simply wasn't available via the old AWS v3 API. This is a significant milestone.

This release of Ruby/AWS interfaces with the latest revision of AWS v4, namely the 2008-04-07 revision. I've finally written a few unit tests, too, to prevent regressions from one release to the next.

Another useful addition in this release is the new AWSObject#each iterator method, which yields each |property, value| of the AWSObject. This makes it trivial to iterate over an item's properties.

In addition to the new functionality, a few bugs have been fixed and minor improvements made. In particular, error-checking when performing MultipleOperations and batched operations has been improved.

Version 0.2.0 of Ruby/AWS has been released.

If was quite pleased with the previous version, I'm very pleased with this one. The code has really been cleaned up and a lot of functionality has been added in this release.

Here's a list of the major changes:

• Many more types of operation are now supported. In fact, everything except shopping cart operations is now supported.

• Symbols can now be used instead of Strings as parameters when instantiating operation and response group objects.

• Image objects can now retrieve their images and optionally overlay them with percentage discount icons.

• Compatibility fixes for Ruby 1.9.

• Dozens of other fixes and minor improvements.

There's still no support for shopping carts, but that will change in version 0.3.0. Thanks for your patience.

The arrival of new offspring understandably inspires a certain amount of fundamental self-reflection. How have I lived my life? Am I a good father? How do I want to live the remainder of my life? Can I better myself?

Can a leopard change its spots? Imagine my surprise when one day, unexpectedly, my Ruby interpreter provided the answer.

[ianmacd@frankincense]$ irb
irb(main):001:0> self = self.better
SyntaxError: compile error
(irb):1: Can't change the value of self
self = self.better
      ^
        from (irb):1

Beta books are a great idea. Why don't more technical publishers (or even publishers of any work of non-fiction) do this?

Take the Pragmatic Programmers, Dave Thomas and Andy Hunt, and their publishing company, the Pragmatic Bookshelf. Dave's currently working on the third edition of Programming Ruby, updated for Ruby 1.9. The finished product will be in the shops a few months from now.

Nothing unusual about that, you might say, but rather unconventionally, the book is already available for sale. How is that possible?

Firstly, the Pragmatic Programmers have taken the entirely logical step of selling PDF copies of their books. If you buy the paper + PDF bundle, you get them for less than the sum of the two. A PDF of a technical book is a grand thing, because it's a lot easier to use a computer to search a file than it is to use one's fingers and eyes to search a stack of paper.

PDFs are also cheap to produce and not just user-friendly, but environmentally friendly, too. Extending the idea, why not produce PDFs of books that aren't quite ready yet. Offer them to your readers and, as with a piece of beta software, you'll get errata reports back. Reader feedback is important to an author, so why not get that feedback while you write the book, instead of after it's published, by which point it's only useful for the next edition, which is almost certainly a few years away. And that edition will have its own problems, too.

So, I already have my copy of the third edition of Programming Ruby and am happily using it. Whenever the manuscript is updated, I get an e-mail, which allows me to go to the Web site of the Pragmatic Programmers and regenerate the PDF for myself.

I think PDFs of technical books make perfect sense. Beta PDFs of not yet finished books make even more sense, if you can improve on perfect.

I'm coding again and it feels good.

The only serious coding I'd done since leaving Google was tv_grab_nl_upc, an XMLTV Dutch TV programme guide data grabber for use with MythTV. It's a decent piece of code, but it hasn't exactly found a wide audience.

One of my most popular projects of all time has undoubtedly been Ruby/Amazon, a Ruby interface to Amazon's Associate Web Service. Ruby/Amazon was originally written and released in 2004 as an autodidactic exercise for me to gain experience with programming for Web services.

It was written for Amazon's AWS v3 API. Amazon announced soon after the release of Ruby/Amazon that it would soon be launching AWS v4, but would continue to maintain the AWS v3 interface until further notice. That notice came in the first half of 2007, when Amazon announced that it would finally shut down access to the AWS v3 API on 31st March 2008.

Close to a year's notice would have given a normal person ample time to work on upgrading his code to use the new API, but I'm no mere normal person, so the code predictably continued to gather moss under my custodianship.

I did finally do some significant work on rewriting the library for AWS v4 while Sarah and Eloïse were in Chicago for a few weeks last summer, but once they had returned, the code stagnated again. By the time I'd found my latent coding stride -- something I can only work up to in relative solitude -- the house became a hive of activity once more.

The unavailability of that state of solitude, a state I find essential in order to foster concentration deep enough to produce quality code, has, as far as I'm concerned, been the most debilitating aspect of parenthood. If I'd had an office five minutes' walk from here, that would have been enough to solve the problem, but being in the same house has proved, for me, to be too big a distraction.

Nevertheless, for whatever reason, in the last ten days of March this year, I was suddenly gripped by the urge to get working on the library again and knock out a working version for AWS v4 before the AWS v3 shut-down deadline at the end of March.

Version 0.0.1 of Ruby/AWS, the sequel to Ruby/Amazon, finally saw the light of day on 24th March. The code was ugly -- embarrassingly so -- and was publicly released for the sole reason of providing Ruby/Amazon users with a migration path to the new API. Yes, I should have given people more than a week to migrate -- I should have done this work two years ago -- but at least I didn't leave them completely in the lurch.

Why the name change to Ruby/AWS? At the time, AWS was the only Amazon Web API, so it made sense to call my Ruby interface to it Ruby/Amazon. In the intervening years, however, AWS has become just one of many Amazon Web APIs. Therefore, it's more accurate to call the new library Ruby/AWS. An even better name might be Ruby/Amazon/AWS. Take your pick.

Ruby/Amazon has, in some ways, been my most successful piece of code to date. Not only was it downloaded and used by a lot of people, but it also scored me an opportunity to write about my own software for an issue of Dr. Dobbs' Journal a few years ago. Writing an article for a serious publication is always great, but writing about your own code is even more enjoyable.

Over the last three years, I've had many questions about updating the library for AWS v4 and even an offer of paid contract work to do it, but the motivation has always sadly been lacking. Then, suddenly, with the arrival of Lucas mere days away, I was suddenly possessed by the urge to do the right thing and not just let the code fade into oblivion. I suppose I needed the boost to my self-esteem, because a decent amount of my self-respect is derived from my intellectual muscle, if you will; muscles that have atrophied and become a bit flabby in recent times.

I was also spurred on by the increasing number of mails I was receiving as the end of March deadline approached, most of which basically said, 'Help, my code is about top stop working.' I felt a certain obligation to my users not to leave them out in the cold. For whatever reason, people weren't entirely content with the other projects that had sprung up to fill the vacuum in recent years.

Anyway, yesterday saw the release of version 0.1.0 of Ruby/AWS, a version that no longer causes me embarrassment. It doesn't (yet) support the full v4 API, but it's pretty good at what it does.

What it does is ItemSearch, ItemLookup, SellerListingSearch, BrowseNodeLookup and ListSearch. It also supports batch requests and (as of 0.1.0) multiple operations.

Conspicuous by its absence is support for remote shopping carts. This will come later.

Version 0.1.0 has a new recursive XML parser that dynamically creates classes and instantiates objects from them on demand. This is an improvement from versions 0.0.1 and 0.0.2, as well as from Ruby/Amazon, which were all only semi-dynamic in their class definition. This one change allowed me to ditch most of the work I did last summer, which involved manually defining a large number of classes to match the data sets that could be returned by the API.

Anyway, it does feel good to have produced a new piece of code. Almost no code from Ruby/Amazon was recycled in the process, because the v3 and v4 APIs are totally different.

This Web site is already running the new code to display links to Amazon products in the left sidebar and Ruby/AWS has already been placed in the Fedora distribution (releases 7 and later), replacing the now obsolete Ruby/Amazon.

If you need programmatic access to Amazon, Ruby/AWS may be your thing.

Another thing that kept me busy over the last few days was enhancing, tv_grab_nl_upc, my program for fetching television guide data for UPC's digital cable network in The Netherlands and outputting the results in XMLTV format.

Of all the stuff I've written, this is possibly the most obscure and least useful to anyone else (of course, I've said that before and been wrong). Nevertheless, necessity is the mother of invention, and I need this program, so I've been working on improving it over the Christmas period.

Specifically, this new version uses some heuristics to try to derive a subtitle for the programme in question. Ideally, this should be the episode name for a series, the destination city/country for a travel programme, etc. Sometimes, UPC includes this kind of information as part of the programme's description, but programmatically determining whether the first sentence of the description is suitable for use as a subtitle is a tricky business; hence the many man hours that have gone into the 0.5.0 release, as I had to perform a fair amount of textual analysis and parser refinement before reaching the point that plucking text from the description to use as the subtitle was logical enough to be more use than hindrance.

Anyway, 0.5.0 of tv_grab_nl_upc is now out, even if I'm the only one who cares.

I noticed only today that the article I wrote about Ruby/Amazon for Dr. Dobbs Journal is on-line, and probably has been for a year or more. That's nice to see.

That's one thing I do lament about my career as a system administrator/programmer thus far: I haven't done much writing for professional publications and I've never held a presentation at a conference. I have no RFC to my name and, if you scan the shelves of your local computer bookshop, you won't find any volumes bearing my name.

The lack of conference presentations is largely a result of my own choice, as I have never even submitted a proposal, allergic as I am to the idea of speaking in public. That's something I have yet to overcome. It's a shame, because when I worked for Google, you only had to put your e-mail address to a proposal to gain an invitation to speak, because a Google speaker at a conference works like a magnet. Everyone is eager to hear how things work behind the scenes at that company. Unfortunately, I was too shy to take advantage of this.

It would have been nice, however, to have done more professional writing. I still could, of course, but I'm no longer active in the field, in the thick of it, drilling deep into a particular area of the job. As such, no particular subject seems obvious now to write about. Being in semi-retirement, it's going to take a large effort on my part to change that.

I mentioned a few days ago that I'd been working on a grabber to pull TV programme timetable data for the channels in UPC's digital TV package.

Well, with the family out of the country, I seized the opportunity to clean up the code, squash a couple of bugs and add some hopefully useful command-line options. With that work done, all that remained was to put up a page, telling people how to use the software. And with that work now done, I can announce it, which is what I'm doing.

I don't expect a lot of interest in this software. After all, how many users of MythTV can there be in The Netherlands? And how many of those are subscribers to UPC's digital TV package? And how many of those care about the channels for which data can't be obtained from the usual source, tvgids.nl? Well, I'd be surprised if I needed more than one hand to count them all.

For those people, but mostly for myself and Sarah, I give you tv_grab_nl_upc.

Ruby/LDAP 0.9.5 has been released.

There are no actual changes or enhancements this time around. Rather, all of the work in this release went into getting the code to build and work on Windows platforms. Accordingly, this version should build and work straight out of the box on Windows systems. It's been tested on Windows XP SP2, using SVC C++ 6.0 to build it.

If the code also works with other breeds of Windows and compiler, this will be a nice milestone to have reached, as the library has not worked on Windows since I took over maintenance of the code, somewhere back at the end of 2004, I think.

The overdue task of maintenance on my code base continues.

Some time last October, Yahoo made changes to its finance site, which broke the currency conversion module of my Ruby/Finance library. I've finally got around to fixing this and the result is the newly released version 0.2.2.

Ruby/Finance is a partial port of Perl's Finance::Quote. I initially wrote it so that I could track Google stock movements as the company became publicly traded, which is why it hasn't seen a lot of expansion since first being released.

I must get around to fixing the AEX module, too, at some point, as that's been broken for ages. That will require a large rewrite of the module, however, as the underlying service used by the module has disappeared.

I wasn't planning another release of this package quite so quickly, but a user asked me how to restrict the number of results returned by the server, prompting me to ask myself, "Yeah, how do you do that?"

I quickly remembered how to set the number of results, but then discovered that this doesn't actually help with Ruby/LDAP. The reason is because the server will return the status of LDAP_SIZELIMIT_EXCEEDED along with the search results when its maximum number of results has been exceeded. Ruby/LDAP had the unfortunate property of throwing an exception to the status LDAP_SIZELIMIT_EXCEEDED, as it does with any status that isn't LDAP_SUCCESS. Any results obtained up until that point were discarded.

The 0.9.4 release corrects this problem in all search methods, to whit LDAP::Conn#search, LDAP::Conn#search2, LDAP::Conn#search_ext and LDAP::Conn#search_ext2. It's now up to the user to check the error status after performing a search. This can be done by invoking LDAP::Conn#err. If that returns LDAP::LDAP_SIZELIMIT_EXCEEDED instead of LDAP::LDAP_SUCCESS, then the results set was truncated by the server. This allows users to request a tiny subset of results, so as to spare the server from unnecessary work. Of course, if you know your modern LDAP, you'll know that the paged results control could be used to achieve the same goal, but fixing this issue makes for an easier alternative (and not all servers support the paged results control, anyway).

There are a couple of other minor changes in this release, but nothing user-visible.

You can obtain Ruby/LDAP via the project home page.

I promised I'd return to coding and here's the first fruit of my labours, a new version of Ruby/Google.

Version 0.6.0 adds a new method, Search#utf8(src). If src is not nil, query strings used by the Search#search and Search#spell methods will be converted from encoding src (e.g. ISO-8859-15) to UTF-8 prior to being sent to Google.

Additionally, something broke Ruby/Google in recent months, possibly a low-level change Google made to their Web API. I had a patch available fairly quickly to fix the issue, but I'm happy to say that it's now been integrated into 0.6.0.

My thanks go to Klaus Stein for alerting me to the UTF-8 issue that led to this release.

It does feel good to be coding again, I must confess. I'm looking forward to updating some of my other projects in the coming days and weeks.

Patches are supposed to fix things, not leave them broken in new and interesting ways, so I thought I should redo yesterday's patch in such a way that it would actually have value. If you are scratching your head after applying yesterday's patch, please reapply it.

If you're a Ruby/Google user and recently discovered that the library had stopped working, you'll probably be interested in this patch to version 0.5.1 that fixes things. It appears that Google made some silent changes at their end, probably to the WSDL file that they use. This caused Ruby/Google to fail.

I would release a new version of the software containing the patch, but since my computers (including my CVS repository) are currently in the hold of a ship somewhere on the mid-Atlantic, I'm unable to issue a new release of the software for the time being.

My article on Ruby/Amazon has finally made it into print in the February issue of Dr Dobb's Journal.

This is my first published work for a computer magazine in just under ten years. It's hard to imagine it was that long ago that I wrote a monthly column about the Internet for the now defunct PCW Plus, a magazine devoted to users of the dearly beloved Amstrad PCW.

I'd still like to write a book at some point, if I can ever manage to summon the energy. I used to think I'd one day write a book about LDAP or Bourne shell scripting, but there are good books available on those subjects these days.

No, it's more likely I'll turn my hand to newspaper journalism at some point in the future, once I have more free time available.

I've put out a series of patches to ruby-ldap 0.8.3 to allow the easy use of controls. Controls take advantage of the extensible nature of LDAPv3 to provide functionality not part of the original protocol specification.

Specifically, I had a need at work to use the Paged Results control, described in RFC2696. ruby-ldap 0.8.3 allows the client to set controls at the search level via LDAP::Conn#searchext and LDAP::Conn#searchext2, but it has no way to return controls sent by the server to the client as a side-effect of the search. Furthermore, I wanted to be able to set controls at the session level and have those be effective during LDAP::Conn#search and LDAP::Conn#search2, which are methods I use much more frequently.

Anyway, if you need to use LDAP controls from Ruby (and let's face it, who doesn't?), this if for you.

Ruby/Finance is finally available for public consumption. I released version 0.1.0 yesterday evening to a deafening roar of silence.

Currently, it handles basic Yahoo Finance stuff, such as currency conversions and stock data retrieval from the American, Australian, Asian and European markets. As such, it replicates some of the functionality of Perl's Finance::Quote and is somewhat based on its design. In fact, whilst running unit tests, I was amused to find that I had even managed to faithfully port one of Finance::Quote's bugs.

For the foreseeable future, I'll be adding more of Finance::Quote's functionality to provide Ruby users with the same range of features that the Perl crowd enjoy. After that, who knows?

The next module I'll be implementing is one to retrieve stock data from Amsterdam's AEX exchange.

I've been farting around, looking for another programming project to sink some time into. I haven't felt like doing any programming for a couple of months now, and I needed something to get my teeth into enough that it would draw me back in and make me excited to play with my computer once more.

More and more, I find myself losing interest in computers, not just professionally, but to a lesser degree, personally, too. I hope that's just a symptom of feeling jaded and burnt out. I'd hate to permanently lose interest in a hobby I've had for most of my life, but we all get older and some of us even get a little wiser in the process, so maybe that's the direction I'm heading in. I don't think so, though; I think this is just a temporary malaise.

Anyway, I released version 0.8.3 of Ruby/Amazon a few days ago, but that was hardly a major event. It's already the most complete high-level language interface to Amazon's Web Services, so this release was just to add support for HTTP proxy servers. Some poor unfortunates still have to use those, I suppose.

Now back to my story. With Sarah's company (well, her employer's company, to be precise) having gone public last week, I fished around on RAA for a library to help me write a stock price grabber. To my surprise, there wasn't yet anything available.

Well, it didn't take me long to hack up a few lines of code to grab the current price and plonk it in the sidebar on our front page, as well as e-mail a copy to Sarah at work when the integer dollar price changes.

That set me thinking, though. E-mail is all well and good, but I wanted some kind of scrolling applet on my desktop. Unfortunately, ruby-gnome2 doesn't seem to support the GNOME panel yet, so that was out as a possibility. Oh well, I thought, there must be a GKrellM plug-in out there and, sure enough, I quickly found what I was looking for: GkrellStock.

Upon untarring the archive, it quickly became apparent that this software would need Perl's Finance::Quote., which I vaguely remembered once having read a little about. Anyway, once I'd acquired that, I was up and running with GkrellStock.

Finally, I'm getting to the point of the story. With Sarah's company already floated on the stock market and mine destined to do the same at some point in the coming months, the need for a Ruby library to handle one's financial networking is greater than ever. And thus was begun the effort to port Finance::Quote to Ruby.

After a few hours of hacking, I have implemented about 15% of what Finance::Quote can do, but even this is enough to have GkrellStock now work via Ruby/Finance or whatever it ends up being called.

I don't normally port things, as it's too much like reinventing the wheel and thus usually strikes me as a poor investment of my time. However, since the Ruby world is so lacking in this area, it seemed appropriate to put some time into the project. Depending on how much time I can spend on it this week, I should have something I feel comfortable having the hoi polloi gawk at pretty soon.

It really is quite nice to be hacking again.

Since I wrote about my new anti-spam measures, the spam has been furiously banging up against my virtual front door.

Talking to a colleague on IRC tonight, I was inspired to write a quick Ruby script to report the progress since last Sunday:

 #!/usr/bin/ruby -w

 reject = Hash.new( 0 )

 while line = ARGF.gets
   case line
   when /un(verified|deliverable) address/
     next
   when /554 Service unavailable.* (blocked using .+?);/
     reject[$1] +=1
   when /NOQUEUE: reject:(?:.+?:.+?: )(.+?)[;:] from/
     reject[$1] +=1
   when /reject: header .+helo=.+?: (.+)$/
     reject[$1] +=1
   end
 end

 total = 0
 reject = reject.to_a.sort { |a,b| a[1] <=> b[1] }
 reject.each do |x|
   printf( "%-74s%5d\n", x[0], x[1] )
   total += x[1]
 end

 printf( "\n%-74s%5d\n", "Total blocked:", total )

Here are the results:

Bad attachment with file name extension: bat                                  1
Bad attachment with file name extension: cpl                                  1
Sender address rejected: need fully-qualified address                         2
Sender address rejected: Improper use of SMTP command pipelining              3
Bad attachment with file name extension: exe                                  5
Bad attachment with file name extension: com                                  8
Relay access denied                                                           9
Bad attachment with file name extension: scr                                 12
Bad attachment with file name extension: pif                                 25
Helo command rejected: Improper use of SMTP command pipelining               27
blocked using sbl-xbl.spamhaus.org                                           30
Helo command rejected: Host not found                                        58
Helo command rejected: need fully-qualified hostname                        124
blocked using dnsbl.sorbs.net                                               155
blocked using bl.spamcop.net                                                290
Sender address rejected: Domain not found                                  1619
Recipient address rejected: User unknown in local recipient table          6659

Total blocked:                                                             9028

All in all, I'm very pleased. Very little spam is making it through now. For the spam that does make it into the system, I also upgraded to a recent CVS snapshot of SpamAssassin this afternoon, so most of it still gets zapped before making it to the in-box of any of my users.