Using the VMS PATCH utility to fix a VAX LISP issue

In my previous post about VAX LISP, I complained about a misfeature of the editor that prevented binding editor functions to the Ctrl-S and Ctrl-Q keys.  In the manual, it is stated that these keys cannot be bound due to "system restrictions". It appears, though, that it is a measure to prevent users from footing themselves in the foot because Ctrl-S and Ctrl-Q could be sent by terminals to implement software flow control.  Software flow control is optional, though; it can be switched on and off for each terminal line and it is not needed for virtual terminal connections (Telnet, DECNET) as flow control is implemented by the network layer and networked terminal sessions usually have enough buffer space.  Thus, reserving Ctrl-S and Ctrl-Q really only makes sense for asynchronous terminal lines that require software flow control, and there is no reason to reserve these characters for all terminal sessions, regardless of technology.
I am used to having Ctrl-S and Ctrl-Q be bound to certain functions from GNU Emacs, and I want to use the same bindings with VAX LISP as they've become part of my muscle memory.  Thus, I wanted to remove the explicit checks for these keys in the EDITOR::BIND-COMMAND and BIND-KEYBOARD-FUNCTION functions.  Lacking the source code to VAX LISP, I could not just remove the checks and recompile, so I needed a different way.
This is where the VMS PATCH utility comes into play.  It is a standard tool that is used to modify program executables, usually to remove bugs without requiring a full reinstallation of the software package that needs to be fixed.

Finding what to patch

In order to remove the checks for Ctrl-S/Q in EDITOR::BIND-COMMAND and BIND-KEYBOARD-FUNCTION, I disassembled the two function using the standard VAX LISP function DISASSEMBLE.  VAX assembler code is easy to read, so it was rather straightforward to find out how the checks work.  Here is the relevant extract of the disassembly output from the VAX LISP BIND-KEYBOARD-FUNCTION function:

000FC  MOVQ   BIND, -(STACK)
000FF  MOVL   -10(FRAME), -(STACK)
00103  MOVZBL #89, -(STACK)
00107  MOVZBL #99, -(STACK)
0010B  MOVAB  18(STACK), FRAME
0010F  MOVL   'CHAR/=, LEX
00113  MOVL   4(LEX), FUNC
00117  JMP    @4(FUNC)
0011A  CMPL   RES, SV
0011D  BEQL   121
0011F  BRB    145
00121  MOVL   FRAME, -(STACK)
00124  MOVL   #A28, -(STACK)
0012B  MOVQ   BIND, -(STACK)
0012E  MOVL   '"The character argument must be a control character (other than~%~
            CTRL/S or CTRL/Q): ~S", -(STACK)

Apparently, #89 and #99 refer to Ctrl-S and Ctrl-Q, and the CHAR/= function is called to check whether the argument is one of those characters.
The easiest way to get rid of the check seemed to me to replace the BEQL 121 instruction, which jumps to printing the error message and returning from the function, to BRB 145, which is the jump to the rest of the function, which performs the actual binding.

Locating the code

To make the actual modification using the PATCH utility, I now needed to know where in the executable file these instructions can be found - the addresses in the DISASSEMBLE output are only relative to the start of the function, and the LISP.EXE executable file does not include any symbols.  I thus picked out a few instructions that seemed characteristic and assembled them to machine code using the VMS macroassembler.  Then, I searched for this instruction sequence in LISP.EXE using a trivial perl program, which gave me the offset of the function that I needed to update.
Knowing the offsets of the locations in the binary file that I needed to change, I could now go about and use the PATCH utility in interactive mode to devise the actual patch.  The PATCH manual suggests that the VMS standard debugger would be used for patch development, but as I did not have the symbol table for LISP.EXE, using PATCH itself for development was more appropriate as it can work in /ABSOLUTE mode which uses file offsets for loctions.
First, I needed to find and disassemble the code using the locations I had devised.  As I only had the offset of the MOVZBL #89, -(STACK) instruction and VAX instructions have variable lengths, I needed to probe a bunch of addresses before I could come up with this:

PATCH>e/i 00403E68:00403E9A
00403E68:  MOVQ    R10,-(R7)
00403E6B:  MOVL    B^0F0(R8),-(R7)
00403E6F:  MOVZBL  #089,-(R7)
00403E73:  MOVZBL  #099,-(R7)
00403E77:  MOVAB   B^18(R7),R8
00403E7B:  MOVL    B^30(R11),R9
00403E7F:  MOVL    B^04(R9),R11
00403E83:  JMP     @B^04(R11)
00403E86:  CMPL    R6,AP
00403E89:  BEQL    00403E8D
00403E8B:  BRB     00403EB1
00403E8D:  MOVL    R8,-(R7)
00403E90:  MOVL    #00000A28,-(R7)
00403E97:  MOVQ    R10,-(R7)
00403E9A:  MOVL    B^38(R11),-(R7)

This is exactly the same instruction sequence that I disassembled using VAX LISP above. As you can see, it is significantly harder to read the disassembly produced by PATCH as it does not show symbolic register names, and it also does not translate data pointers like in the last instruction - VAX LISP showed the error message where PATCH prints B^38(R11).  The addresses in the disassembly, however, are absolute offsets into the LISP.EXE file, and that was what I needed.

Creating the actual patch

Equipped with this information, I could create the actual patch to change VAX LISP so that it no longer disallows binding of Ctrl-S and Ctrl-Q, which I reproduce below:

$ PATCH/ABSOLUTE LISP$SYSTEM:LISP.EXE/OUTPUT=LISP$SYSTEM:
REPLACE /INSTRUCTION
^X00403E5A
'BLEQ ^X00403E5E'
EXIT
'BLEQ ^X00403EB1'
EXIT
REPLACE /INSTRUCTION
^X002F3246
'BEQL ^X002F324A'
EXIT
'BRB ^X002F3265'
EXIT
UPDATE
EXIT

I used the REPLACE/INSTRUCTION command which requires the specification of an address, the current instruction(s) at the location and the new instruction(s) to put at the location.  PATCH would only perform the replacement if the instructions currently in the file match what is specified, otherwise it would abort with an error.  This is useful to prevent the patch from patching the wrong file or the wrong version of a file.

Better patches

As I did not have the source code or the symbol table for LISP.EXE when I developed this patch, I needed to resort to patching in absolute mode which is kind of brittle.  PATCH can also work in symbolic mode if a symbol table is available for the executable file, which allows patches to work even if the executable has been linked differently than the executable used to develop the patch (i.e. you could write a patch for a library function that'd work for executables that were linked statically against the library).  Also, PATCH allows patches to insert more instructions than were originally present in the executable.  It does this by adding additional disk space to the executable to hold the new, longer code, and then patch the required jump instructions to the patch area into the original location.

Closing thoughts

It is quite nice to be able to fix a bug in or remove a feature from a commercial program without having the source code, just by working with a bit of disassembly and on-board utilities. Back in the day, it was also useful to be able to distribute fixes to customers in the form of tiny patches. These could even be sent as a printed letter and typed in by the customer on a terminal, with reasonable effort and good chances of success.

Evaluating VAX LISP 3.1

The back story

Back in the day, when there still existed a number of different computer architectures and operating systems, I was a huge fan of the VAX/VMS operating system. VMS was designed together with the VAX architecture, and it used specific mechanisms offered by the processor to implement its I/O, security and other services. As it was still common at the end of the 1970ies, portability was not a design objective, and computer companies often had both hardware and software teams working together to implement computing solutions, from hardware to operating systems, development environments and applications.
I became a VMS fan mostly for three reasons: VMS machines were popular in the early packet switched networks that I could access, their default installation often left accounts with default passwords open, and VMS has an excellent online help system which helped me to learn and program VMS without having access to printed manuals.

I also liked the VAX for its orthogonal architecture, which made it somewhat similar to my favorite 8 bit microprocessor at the time, the Zilog Z80.  Back then, being able to program a machine in assembly language was still somewhat desirable, and an orthogonal instruction set with many general purpose registers and fancy addressing modes allowing complex programs to be written in assembly language appealed to me, too.
Frankly, though, I never wrote a lot of assembly language code. Instead, C quickly became my language of choice even on VMS, and when g++ 1.42 became available on VMS, I mostly became a full  time C++ programmer.  This was when object oriented programming just became the fad, with magazines like the C/C++ Users Journal and the Journal of Object Oriented Programming being the prime sources of inspiration. Around that time, I read Grady Booch's excellent book Object-Oriented Analysis and Design with Applications, which explained OO programming using examples in a variety of then-popular OO languages like Smalltalk, C++ and Object Pascal. It also had a chapter on using CLOS as the example language. To me, CLOS was completely out of reach and locked in the ivory tower, requiring utterly expensive machines and software licenses that I'd never hope to have access to, so I read the chapter more as science fiction than as something of practical relevance.
Thus, I stuck with C++ and later Perl on VMS, which was amusing enough, also for being a rather exotic blend.

Around 2001, I was introduced to Lisp by the way of VMS. At the time, my programming gigs were all on Unix or Windows and my affection to VMS became part of my hobby as a collector of vintage computers.

At some point, I brought a MicroVAX with me to a vintage computer convention, and a friend had his MacIvory Symbolics Lisp Machine on display. As the two systems were from the same era, we thought that we should try to bring up some network between them.  My VAX ran a fairly recent version of VMS, though, and when we tried to establish a DECNET connection, the Lisp machine would signal an error and put us into a debugger. What got me hooked on Lisp was that my friend could display the source code of the failing DECNET device driver, figure out that the problem was the too-new DECNET version number that the VAX had reported in the connection setup packet, fix the Lisp code in the DECNET driver of the Lisp Machine and then continue the file transfer operation that had signaled the problem.  I certainly understood that this would theoretically be possible back then, but I had never seen a system that would have that level of integration of operating system and programming language. I was intrigued.
Fortunately, machines had become much cheaper and faster in the decade since I had first read about CLOS, and the open source movement had made Unix and Common Lisp affordable. I went for CMUCL and eventually became a full time Common Lisp developer for several years.

The holy combination

I had been looking for VAX LISP for some time, as LISP and VMS sounded like a blend of very likeable technologies, so I was very happy when someone found the installation tapes and made them available through the comp.os.vms news group in 2017.  I played around with it a little, but unfortunately the manuals that were publicly available at the time were only for VAX LISP V2.1, and V3.1 was supposedly different enough to make me want the updated manual set.  It took a bit of research and emailing until I got in touch with the former architect of VAX LISP at digital equipment, and he was kind enough to borrow me the complete manual set for version 3.0, which I scanned and OCRed so that it is now available on bitsavers.  Thus, i felt equipped to give VAX LISP a test drive as a vacation project.

About VAX LISP

VAX LISP is an implementation of Common Lisp, but it implements it as described in the first version of “Common Lisp the Language” (CLtL1). This means that it feels like Common Lisp as it was standardized nine years later, but with several important features missing. Among these missing features are:

• LOOP, the general purpose, well-hated macro for iteration.  You get simple LOOP, DOLIST, DOTIMES and of course the dreadful DO and DO* couple.
• The Common Lisp Object System (CLOS), although there is a version of PCL coming with VAX LISP in the examples directory.  There also is a version of the Flavors object system included.
• The condition system. VAX LISP allows you to bind the *UNIVERSAL-ERROR-HANDLER* special variable to a function that’d be called when ERROR, CERROR or WARN would be called and that could take a non-local jump using THROW.
• DESTRUCTURING-BIND and lambda lists with &KEY, &REST etc.  Lambda lists like we know them today are only available in DEFUN

Furthermore, the package system is not yet where it is today, there are no generalized sequences and there are probably a several other bits and pieces that are missing.

Notable features of VAX LISP are:

• Integrated, user-extensible editor with both EDT (VMS’s standard editor) and Emacs emulation modes, both with a terminal and a DECwindows backend. There is extensive documentation on programming the editor in LISP and I’ve added several features myself, which was a joy.
• Interface to the standard DECwindows widget set. This required the implementation of a multithreading system so that DECwindows code, written in C, could be called from LISP and DECwindows handlers could be written in LISP and called from C.
• A foreign function interface implementing the VMS procedure calling standard, making it possible to call VMS system services and library routines as well as code written in other VMS languages.

Additionally, it comes with a good bunch of example code that is quite helpful when trying to figure out how to use the language, although the quality of the samples vary.

A Project to Try It Out

I wanted to get a feel of how programming in VAX LISP works, so I decided to write a simple http server in it.  This would give me the opportunity to try out the system interface as I would have to call into the TCP/IP subsystem.  As I wanted to evaluate VAX LISP during a vacation trip, I used SIMH on my MacBook instead of a real VAX, running VMS V5.5-2.

The Development Experience

Initially, I had hoped that I could use the graphical version of VAX LISP for development, but I did not manage to set up a fully functional DECwindows workstation environment and fell back to the terminal version.  The downside of this is that I had to alternate between working in the editor and in the REPL, as the editor does not have a convenient way to evaluate expressions and display the results built-in.  It would probably not be very hard to add that, but I wanted to spend most of my time working on the web server.

The REPL

The terminal REPL of VAX LISP does not have a lot of convenience features.  LISP expressions are entered using the standard VMS line editor, which means that no completion is available and all you get is a way to recall the previously entered line.  Annoyingly, if a line is longer than whatever your current terminal width is, there is no way to edit text before the line break.  Obviously, no work has been put into making the terminal REPL nice.

The only notable convenience is that one can bind keys (ASCII codes, really) to call LISP functions, and one use case suggested in the manual is to bind, say, Ctrl-E to call the 'ED function.  That way, switching to the editor from the REPL is very quick and I use that feature all the time.  Beyond that, the REPL experience is rather bare-bones and only slightly better than, say, a bare SBCL or Clozure CL REPL because of VMS's standard line-editing facility, which gives you a one-line recall buffer.

The Editor

VAX LISP comes with an editor that is written in LISP itself and that is extensively documented so that users can extend and adapt it as desired.  By default, it tries to emulate EDT, which is the original VMS editor, but EDT is rather quirky and pretty much unusable without a DEC keyboard as it relies on certain function keys that are not present on modern keyboard layouts.  The VAX LISP editor also supports an Emacs compatibility mode, because even back in the late 1980ies, most LISP programmers would be used to Emacs and not be willing to use another editor.  In Emacs emulation mode, I found it rather easy to write and edit code and I even did some substantial refactorings of my http server code base with it without too much grief.  I spent a bit of time of adding a few features that I am used to from GNU Emacs.  Writing editor extensions in the same language and environment as the actual project was rather joyful.

The documentation to the editor is extensive, but it leaves a number of questions unanswered.  I also find its organization a bit odd, with a whole chapter on editor concepts that comes after all the nitty gritty details of editor customization explanations as separate "Part II" of the manual.  I'd expect the description of editor concepts to be coming first, before applying the concepts would be explained.  Also, the explanation how editor major and minor modes work is rather fuzzy.

One annoying misfeature of the editor is that it explicitly disallows binding Ctrl-S and Ctrl-Q, and the manual claims that this is due to "system restrictions".  While it is true that Ctrl-S and Ctrl-Q were used to implement software flow control on terminal connections that lacked hardware flow control, software flow control is an optional feature of the VMS terminal driver and can easily be switched off if out-of-band flow control is available for a given terminal connection.  This would be easy to fix if one hard the VAX LISP source code, but that seems to be lost, unfortunately.

Maybe not so unexpectedly for an editor written in the 1980ies, the VAX LISP editor does not have an "undo" facility.  This dampened my enthusiasm when I started writing code, but in practice, it has not hurt me too much.  When I made larger changes, I saved my buffers beforehand, and as the VMS file system automatically keeps old versions of files until they're purged, I could go back to the previous state if I wanted to roll back.

VMS and the Internet Protocols

My romanticized memories of VMS date back from a time when there was no Internet, and the Internet Protocols (IP) were just one set of protocols that were used in global networking.  DEC had its own protocol suite, DECNET, and was committed to making DECNET OSI compatible, as they believed that OSI would become the dominant wide-area networking protocol suite of the future.  DECNET was integrated into VMS quite nicely, and it was easy to implement peer to peer connections from any language, even from command line scripts (which enabled the creation of early networked malware like WANK).  Implementations of other networking protocols like RSCS (for BITNET/EARN) and IP (for Arpanet) were available for VMS, but DEC just treated them as secondary networking systems that did not deserve a lot of attention.

To implement my http server in VAX LISP, I needed an implementation of IP, so I installed "DEC TCP/IP Services for VMS" (traditionally referred to as UCX, "VMS/Ultrix Connection") as an add-on product to my base VMS installation.  UCX implements a standard set of IP clients and servers like FTP, Telnet, NFS etc. and supports two programming interfaces, one similar to the BSD socket calls meant to be used from programs written in C, and another one based on VMS system services, meant for programs written in other languages.  Boldly, I decided that I wanted to use the System Services interface, also in the hope that it would be as nice and convenient as the DECNET programming interface.  I was quite wrong.

Apparently, the System Services interface to UCX is a layer on top of the socket interface, so you get the "best" of both worlds:  BSD sockets with its quirky sequence of required calls to socket(), listen() and accept(), asking you to specify the length of the "listen queue" and to set the "reuse address" option in your server code because, well, because that is how it wants its chicken to be waved to work well.  And then the beautiful VMS $QIO interface with its opaque "function code" and the parameters named P1 to P6 which would be used differently based on the function code supplied.

DEC, in its lack of enthusiasm for supporting IP on VMS, really made sure that the combination of $QIO and sockets would be as ugly as possible.

So, the required sequence to listen to incoming TCP connections and then accept them requires the program to:

• Assign a channel to the UCX network device
• Set the channel to be a TCP/IP socket
• Enable the REUSEADDR option, which is already inconvenient in C with the setsockopt interface, but with $QIO you have to create an additional descriptor to point to the socket option
• Set the local address of the socket, again using an additional descriptor to point to the local address
• Set the socket to listen, using another $QIO invocation, now passing the maximum queue length

These steps directly correspond to what you'd have to do when using the BSD socket interface, only that with the System Services interface, you need to call $QIO all the time.  What is actually done underneath is then determined by which of the P1 to P6 parameters you set and how, so if you call $QIO with the IO$_SETMODE function code and specify a non-zero P5 parameter, you are calling setsockopt() and if you specify a non-zero P3 parameter, you're calling listen() and so on.

Of course, there are few things that can't be syntactically improved with a bunch of LISP macros and in my http server, the whole sequence does not look so bad, but seriously:  This is one of the worst APIs that I have seen in my life and the experience put quite a damper on my romanticized view on VMS and DEC in general.

Serving Files

A http server commonly serves content from files, so I wanted mine to do so as well so that it could deliver a HTML page and some additional resources to the client.  On VMS, files are not the simple byte streams that we think about when talking about files nowadays.  Rather, the Record Managment System (RMS) maintains the file structure for applications and supports various file organizations like line-sequential and indexed with fixed and varying record lengths etc.  It is still possible to read the raw data stored in a file, but that will include the bookkeeping information of RMS.  This means that the http server also needs to know something about the structure of the file being served and possibly perform some conversion instead of just forwarding the file data to the client directly.

Binary Files

Serving binary files like images requires the http server to just read the files in a block-wise fashion, but as VAX LISP does not support block-wise reading, this needs to be done with direct calls to RMS.  This gave me an opportunity to learn working with RMS, as its API is quite different from, say, the POSIX file API with its simple open()/read()/write()/close() interface.

As RMS supports different file organizations, there are many parameters that an application program needs to specify when it wants to operate on a file.  Instead of putting all these different parameters into many functions, RMS uses in-memory structures (blocks) that describe the operations and parameters.  These blocks could be defined using the facilities of the application programming language (i.e. struct definitions and numeric constant macros in C), or they could be described using the File Definition Language (FDL).  FDL is a domain specific language that describes file and record structures.  Applications could invoke the FDL interpreter to turn textual file structure descriptions into binary RMS blocks that it could then use for accessing files.

As a result, the RMS API entry points only require specifying pointers to the blocks that describe the operation, which makes the call sites look rather clean.  As, the entry points themselves clearly describe the operation, this makes the LISP code to serve a binary file using RMS calls rather clean-looking.

Text Files

Serving text files like HTML or CSS files is rather more challenging in VAX LISP, considering that the on-disk data of RMS text files cannot simply be copied to the HTTP client.  This is because, depending on the file's RMS record format, text files are stored with the length of each line stored in the file in binary form before the actual characters in the line.  For that reason, I chose to use the text file I/O mechanism of VAX LISP to read and provide for some explicit buffering to avoid writing each line to the client in a separate $QIO call.  I am not sure whether that strategy really provides performance benefits or whether UCX's own buffering would be as good.  The resulting code does not really look terribly complex, though.

Roundup

In the end, I've now implemented a basic http server in VAX LISP, called rasselbock.  The development experience was quite good, and I found it to be fun to write some code in a Lisp that felt like the Common Lisp I've been using professionally for several years, but that lacks several of the features that I took for granted.  I think that in the end, VAX LISP provides enough tooling for serious programming work.  It was also an interesting experience to write some code in a language for which the Internet offers no support at all.  All I had was the manuals, some sample code and DESCRIBE, and that was sufficient to create something which (kind of) works.

My vacation comes to an end so I'm going to stop at this point for now.  In the future, I'm going to put rasselbock onto a real VAX, and I also want to spend some time working with the DECwindows version of VAX LISP as that will probably be a much nicer development experience.  Furthermore, I would like to try converting rasselbock to run asynchronously, as that one of the cool things in VMS is that all its system services can be used asynchronously.

I need to thank Walter van Roggen who was kind enough to borrow me the VAX LISP V3.0 manual set for scanning, and Michael Kukat who parted with his remaining VAX hardware so that I have a real machine to run VAX LISP and rasselbock on.  I'd also like to thank Richard Wells who pointed me at the vectorized version of the digital logo which led me to become a producer of t-shirts.

If you made it this far, you might also be interested in my follow-on post, in which I describe how I patched the editor so that I could bind Ctrl-S and Ctrl-Q.

No ECLM in 2014

As many of you may know, there was the plan to have the European Common Lisp Meeting happen in Berlin in October 2014. I was part of the organizing team, and we've tried our best to find high-quality speakers to make the event be at least as good as the previous ECLMs.
Unfortunately, and you've determined that already from the title and the previous wording, we've failed. We simply did not find enough interesting commercial projects that have not been presented on one of the previous ECLMs. As we do not want to compromise on the content, we thus have decided that there will be no ECLM this year. We hope to learn about new and exciting developments in the course of this year so that we can have an ECLM in 2015.

Dealing with Excel files from Common Lisp - Using ABCL and Apache POI

In my day job, I mostly program in Common Lisp. Most of what I do is file and database work which Common Lisp is pretty well-suited for. Sometimes, though, I have to deal with data that comes in Excel files, and in the past that meant loading the files into Excel, exporting it into some plain text format and then working with those plain text files from Common Lisp.

While this works, it is a manual and error prone process. Also, Excels plain text export mechanism often mangles the data in undesired ways, which requires yet more additional manual steps (or VB scripting), which I'd rather like to avoid for processes that need to be automatic. Thus, I was on the looks for a way to access the Excel files directly with a Common Lisp program.

Writing a new Excel file parser was out of the question - I have real customer needs to fulfill, and implementing a capable Excel file reader is a large infrastructure project. So I looked into using an existing Excel file reading library instead. There are numerous options, commercial and non-commercial, and I've looked into one of the libraries written in C, but the requirement to more or less manually create FFI stubs and foreign structure layouts for a library that in itself was not documented very well and also did not look like being very accessible made me look for other options.

Apache POI

One of the more prominent open source libraries for accessing Microsoft Office files is Apache POI. It has been around for over 10 years and supports most MS Office formats, including the old OLE2 Excel format as well as the newer OpenXML format (that, despite using XML as the base format, is a horribly complex mess that I hope to never have to deal with directly). Apache POI is a Java library, so it can't directly be used from SBCL, which is the Common Lisp implementation that I normally use.

Armed Bear Common Lisp (ABCL)

In the recent months, I have noticed that there was quite some activity around Armed Bear Common Lisp (ABCL). I had tried an earlier release of it, and while it somewhat worked, it seemed to have a fair number of restrictions that made it unsuitable for me at the time. In particular, ABCL lacked support for the Metaobject Protocol which is something that I often use, either directly or as a library dependency. Also, the older version that I tried could not load the Postmodern library that we use to access our Postgres database, which was the final show stopper. But all that was before the recent 1.1.1 release of ABCL.

ABCL is hosted on the Java Virtual Machine (JVM), and maybe the biggest advantage of that is that access to other JVM-hosted code is straightforward and easy from Common Lisp programs running in ABCL. Thus, using Apache POI should be a snap. Also, as ABCL is becoming a reasonably complete implementation of the Common Lisp standard now, I had hopes to be able to use some of my existing infrastructure code in the program that dealt with Excel files.

To make it short: ABCL works great now. It took me very little time to translate the calls that I found in some Apache POI example program to Common Lisp, and I could also use all of the Common Lisp libraries that I needed for the task. There are some important libraries that don't work on ABCL yet (i.e. CXML-STP, CL+SSL), but I don't need these right now. And ABCL, during development, behaves like any other reasonable Common Lisp implementation in that it supports SLIME.

The Excel reading was a snap and the read process is reasonably fast, but ABCL's startup times are a bit annoying. There currently is no way to do the equivalent of "saving the world" on ABCL, so one has to load all required software at startup time. We're using ASDF for that, and it seems that some of the slow startup times need to be attributed to it. The Excel file reader will run as a batch job, so the startup times don't matter for our production uses, but testing the scripts was a tad tardy.

To illustrate how easy accessing Excel files from Common Lisp is, here is some example code that dumps the first worksheet of an Excel file to the standard output in a Tab separated values format:

;; -*- Lisp -*-

(defpackage :export-tsv
  (:use :cl))

(in-package :export-tsv)

(defun init-classpath (&optional (poi-directory "~/poi-3.9/"))
  (let ((*default-pathname-defaults* poi-directory))
    (dolist (jar-pathname (or (directory "**/*.jar")
                              (error "no jars found in ~S - expected Apache POI binary ~
                                        installation there"
                                     (merge-pathnames poi-directory))))
      (java:add-to-classpath (namestring jar-pathname)))))

(defun process-file (pathname)
  (let* ((file-input-stream (java:jnew "java.io.FileInputStream"
                                       (namestring pathname)))
         (workbook (java:jstatic "create"
                                 "org.apache.poi.ss.usermodel.WorkbookFactory"
                                 file-input-stream))
         (sheet (java:jcall "getSheetAt" workbook 0))
         (formatter (java:jnew "org.apache.poi.ss.usermodel.DataFormatter" java:+true+))
         (total-row-count (java:jcall "getLastRowNum" sheet)))
    (dotimes (row-number total-row-count)
      (let* ((row (java:jcall "getRow" sheet row-number))
             (column-count (java:jcall "getLastCellNum" row)))
        (dotimes (column-number column-count)
          (unless (zerop column-number)
            (write-char #\Tab))
          (write-string (java:jcall "formatCellValue"
                                    formatter
                                    (java:jcall "getCell" row column-number))))
        (terpri)))
    (java:jcall "close" file-input-stream)))

Before the process-file function can be used, init-classpath must be called to add the Apache POI jars to the Java class path.

As you can see, the program is rather short, and even if all comments are stripped, the Java version contains a lot more ceremony. No surprise here, maybe you wanted to have your prejudice confirmed :).

ABCL will now have a firm place in my toolkit. Big shouts go to the maintainers who did a great job lifting ABCL up to a level where it will be very useful to me.

DRAKMA documentation updates

The DRAKMA documentation has received a big overhaul to bring the docstrings and the HTML documentation in sync again. I also restructured the large example section in the beginning so that the individual examples are indexed. I also added changed two examples so that they deal with JSON data as JSON is so popular nowadays.

DRAKMA and JSON

A user suggested that DRAKMA should handle JSON data like text data, i.e. that the content type "application/json" should, by default, be on the list of text content types. The issue with that suggestion is that JSON data is not following the encoding rules for text content type. By default, JSON data is encoded in UTF-8. It is possible to use UTF-16 and UTF-32 by the way of a byte order mark transmitted in the beginning of the file. For textual content types, the character set is determined by the charset parameter in the Content-Type header, which is not used with "application/json".

I have thought about adding a DRAKMA:HTTP-JSON-REQUEST function that would deal with JSON encoding properly, but I think such a function would rather belong into a JSON library than into DRAKMA, so it is left out for now.

How useful are docstrings, really?

Getting the docstrings and the documentation into sync again was pretty laborious, and I fear that I have been missing quite a few things that were in one version of the documentation, but not in the other. So far, Edi and I have used a forward-annotation strategy where the initial version of the documentation was created off the docstrings in the source code, and then manually edited to contain links and other niceties that real documentation deserves. This, of course, is a pretty bad strategy, as edits must be repeated or the docstrings need to contain some form of markup so that no edits are needed.

Going forward, I will probably start to generate docstrings from the XML documentation source. That way, the documentation source can contain links and other, more advanced markup that is not useful in docstrings, and the two representations will not go out of sync. The docstrings will no longer be inline with the source code, but go into a separate file that is generated as part of the release process.

Any thoughts? Use comments or send email.

lparallel and how I was thinking too complicated

Yesterday, I posted a short article that showed how I used lparallel to speed up a lengthy computation by reading input data files in parallel. A few readers responded, and the response from James M. Lawrence, who is the author of lparallel, is reproduced here because it contains source code that can't be properly formatted in blogspot comments (any suggestions regarding something better?).

[citing James M. Lawrence]
Here is a map-reduce-style version:

(defun slurp-alist (hash-table alist)
  (loop
    :for (key . data) :in alist
    :do (setf (gethash key hash-table) data))
  hash-table)

(defun load-files-in-parallel (pathnames)
  (flet ((parse (pathname)
           (let ((result nil))
             (read-and-parse-file
              pathname
              (lambda (key data)
                (push (cons key data) result)))
             result)))
    (reduce #'slurp-alist
            (lparallel:pmap 'vector #'parse pathnames)
            :initial-value (make-hash-table))))

The benefit here is that worker threads do not have to contend for a queue lock. A possible drawback is that the key-data pairs survive until the final reduce, whereas the pairs have a shorter lifetime when passed through a queue. Of course performance will depend upon the scale and shape of the data.

Benchmarks with early versions of lparallel showed better performance with a vector-based queue, however since then I found a simple cons-based queue to be generally faster, which is currently the default. There is still a compile-time option to get the vector-based queue, and unless that is enabled the number passed to make-queue is ignored. In other words, the 10000 is a lie :)

Motivated by James' reply, I spent some time to find out how the process could be further optimized. The idea to have the workers cons up the data and then collect all data into the hash table at the end of the operation reduced the run time for my application by some 30%, which is not bad.

A comment from Max Mikhanosha on G+ suggested that I could avoid the queue and directly use a thread safe hash table. It is a little embarrassing that I assumed that I could not write hash tables from multiple threads. Both CCL's and SBCL's hash tables provide for thread safe modes, so there is no point in tunneling the data through a queue.

As I am using CCL, I tried different values for the :SHARED keyword argument to MAKE-HASH-TABLE. The default is :LOCK-FREE, which makes hash tables thread safe without requiring the grabbing of a lock for each operation. Other permitted values are T, which locks the table for each access, or NIL for thread-unsafe hash tables.

I found that when writing to the hash table directly from the file readers, :SHARED T would give me the best performance. With that option, my code ran another 10% faster than with consing up the file results and then processing all the lists at the end.

lparallel is a great example how Common Lisp is an extensible language. The operators provided by it seamlessly integrate parallel programming into the language, and with the help of my readers, the code that I originally posted now became very small:

(defun load-files-in-parallel (pathnames)
  (let ((hash-table (make-hash-table :shared t)))
    (lparallel:pmap nil
                    (lambda (pathname)
                      (read-and-parse-file pathname
                                           (lambda (key data)
                                             (setf (gethash key hash-table)
                                                   data))))
                    pathnames)
    hash-table))

Look, Ma! All I needed was pmap!

Using lparallel to utilize multiple cores [updated]

A while ago, I discovered the lparallel library as a nice set of abstractions to make parallel programming in Common Lisp easier and more pleasant. The use case that I presented in my previous blog post was more about dealing with asynchronicity rather than parallelism, though. As I have seen people ask for lparallel examples, I will present how I used it to actually speed up a long running computation by having multiple cores work on the problem at the same time.

The problem that I have been working on involved reading and parsing a few large data files to extract some data from them. The extracted data needed to be stored in one hash table for later processing. The reading and parsing process is CPU bound, so reading the data files in parallel will speed up the overall process. "Speed up" here means "reduce the run time because I'm waiting for completion", not "make it as fast as possible" or "have it utilize all cores". I'm trying to reduce my own wait times, and I cannot spend time debugging locks. Thus, the high abstraction level of lparallel fits me just right

For the task at hand, I created a set of workers which read the input files and send the extracted data items to a collector process by the way of a queue. The collector process puts them into a hash table for later processing. Once all files are read, the hash table is returned.

(defun load-files-in-parallel (pathnames)
  (let ((queue (lparallel.queue:make-queue 10000)))
    (lparallel:future
      (lparallel:pmap nil
                      (lambda (pathname)
                        (read-and-parse-file
                         pathname
                         (lambda (key data)
                           (lparallel.queue:push-queue (list key data)
                                                       queue)))
                        (lparallel.queue:push-queue :done queue))
                      pathnames))
    (let ((hash-table (make-hash-table))
          (remaining-worker-count (length pathnames)))
      (loop while (plusp remaining-worker-count)
            for item = (lparallel.queue:pop-queue queue)
            if (eq item :done)
              do (decf remaining-worker-count)
            else do (destructuring-bind (key data) item
                       (setf (gethash key hash-table) data)))
      hash-table)))

Some points are worth noting:

• Instead of using a queue to send the data to a separate worker for hash table insertion, the hash table could be locked or a lock-free hash table could be used. This may or may not be faster.
• The scheduling of the workers is done by lparallel and may or may not be efficient. I've looked at CPU utilization and found the process to use 3-6 cores, so it seems that the scheduling is not optimal. I'm guessing that the queue processing and hash table insertion eat more time than I'd hope for, but I have not made a real assessment.
• I've picked the size of the queue randomly. It may be worthwhile to make the queue larger so that file readers are not blocked while hash table rehashing occurs.

Implementing this parallel file loader took me just a few minutes and gave my program considerable speedup. Using memory is already easy, and with lparallel, using multiple cores also becomes much easier than it has been for me.

[Update] James M. Lawrence, the author of lparallel, kindly pointed out that the original version of my example code contained an unneeded forced future to read the data. I have updated the code according to his remark.

Orphaned projects on common-lisp.net

We have lost contact to the maintainers of the following projects on common-lisp.net:

• fret
• ganelon
• innen
• erlisp

If you are the original maintainer of any of these, please tell us your current email address or let us know if we can delete the project.

lparallel - A Parallel Programming Library for Common Lisp

Parallel programming is hard, and as CPUs are getting ever-faster, I usually tend to find optimizing a single thread of control to be less risky than dealing with threads, locks and synchronized data structures. Recently, though, I had to deal with a reporting function which I started from a Hunchentoot handler and that was running long enough to time out the client. I needed to somehow put running the function into the background, and as I heard good things about the relatively new lparallel library, I thought I could give it a try.

lparallel implements an abstraction layer for parallel programming in Common Lisp. In addition to relatively low-level concepts like tasks and channels, it implements mid-level promises and futures as well as high-level parallel mapping, binding and reducing functionality. Futures looked like a suitable mechanism to solve my problem at hand.

The non-parallel version of my HTTP request handler looked something like this:

(hunchentoot:define-easy-handler (something-that-takes-long :uri "/sttl")
    (parameter)
  (if (eql (hunchentoot:request-method*) :post)
      (compute-and-display-response-page parameter)
      (show-job-parameter-form)))

The compute-and-display-response-page function does what the name suggests. As usual with Hunchentoot request handlers, it returns the page to display to the user. Likewise, the show-job-parameter-form returns a HTML form that collects the parameter. Now, compute-and-display-response-page potentially takes long, so instead of just delaying the response to the HTTP request until the function returns, I want to have it execute in the background and respond to the request with a "job has been started" message. The client is then supposed to poll in regular intervals. Once the background job completes, the page that the function has generated is returned to the client.

Using a future, I came up with something like this:

(hunchentoot:define-easy-handler (something-that-takes-long :uri "/sttl")
    (parameter)

  ;; Get or start Hunchentoot session context
  (hunchentoot:start-session)

  (let ((job-running (hunchentoot:session-value :job)))
    (cond

      ;; Previously started job has finished
      ((and job-running
            (lparallel:fulfilledp job-running))
       (hunchentoot:delete-session-value :job)
       (lparallel:force job-running))

      ;; Previously started job still running
      (job-running
       "Previous job still running")

      ;; Start new job
      ((eq (hunchentoot:request-method*) :post)
       (setf (hunchentoot:session-value :job)
             (lparallel:future
               (compute-and-display-response-page parameter)))
       "The job has been started")

      ;; Display job parameter form
      (t
       (show-job-parameter-form)))))

Hunchentoot's session mechanism is used to make the future accessible to subsequent requests. The future is placed in a session value; the completion of the background calculation is determined by calling lparallel:fulfilledp. Once the computation is finished, the return value of the compute-and-display-response-page is determined using lparallel:force.

This is mostly it, and I find parallel programming in this case easy to understand and reason about. Some additional things are worth mentioning:

• To use lparallel, one has to create a kernel, which constitutes the worker thread pool in which futures and other lparallel tasks are executed. Multiple kernels can coexist, the kernel to use is determined by the lparallel:*kernel* special variable. In my application, I am creating a kernel with one worker thread. This provides me with automatic request serialization so that only one long-running function invocation exists at any time, without the need to do any locking or explicit resource management. I like that a lot.
• By default, errors that occur in a worker thread cause the debugger to be entered. This can be changed by setting lparallel:*debug-tasks-p* to nil. Then, errors will be caught and signalled to the caller of lparallel:force, which in my case was the right thing to do (i.e. if the background process triggers an error, I want the backtrace to be sent to the client.
• One thing I find slightly bothersome is the reverse order of the cond clauses in the request handler - During the lifetime of a typical session, the last clause of the cond is true first, then the second to last and so on. This is a pattern that I'm seeing quite often when writing http request handlers. Maybe a cond-reverse macro would be suitable in this case.

lparallel has more to offer of course, and I have only used a very small part of it. So far, I have found it to be very well thought out, documented appropriately and well maintained. If you need parallel programming or even just easy background execution in your Common Lisp programs, I recommend looking at it.

TRACEs of awesomeness

Debugging is hard, and I admit it freely: I am using print statements regularly to debug code that I'm writing, in particular when not writing Lisp code. With Lisp and its interactive development environment, though, my debugging activity often involves calling functions from the repl and tracing individual functions.

The standard TRACE facility that Common Lisp offers often replaces adding debug print statements for me. In order for tracing to help, the code needs to be structured so that a function is called at the place that is interesting in the particular debug situation. In many situations, one is not really interested in all invocations of a function, but only under some conditions. When debugging functions that are called often or that have large argument lists, simple traces can easily grow huge and analyzing trace logs can become a pain. Also, the trace output can slow down processing enough to make the simple approach fail.

Where TRACE's full power lives

The CLHS entry on TRACE suggests that there may be implementation dependent arguments, and in fact most implementations provide additional options that make the TRACE facility much more powerful. In this post, I'm going to show you a few examples of advanced TRACE options in three implementations to show what one may expect to be available. The examples and implementations chosen are pretty arbitrary and are not indicative of what the particular tracing facilities offer. Please look up the details in the respective documentation.

Breaking with CCL

Clozure CL's TRACE supports :BREAK-BEFORE to enter the debugger before a function is called:

Welcome to Clozure Common Lisp Version 1.7-r15160M  (DarwinX8664)!
? (defun foo ())
FOO
? (trace (foo :break-before t))
NIL
? (foo)
0> Calling (FOO) 
> Break: FOO trace entry: NIL
> While executing: (CCL::TRACED FOO), in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Return from BREAK.
> Type :? for other options.
1 > :go
hello
<0 FOO returned NIL
NIL

Notice how the debugger is entered before FOO is called.

Statically selective tracing with Allegro CL

Sometimes, one only wants to trace a function when it is called from a certain other function. Allegro Common Lisp's tracer offers the :INSIDE option:

International Allegro CL Enterprise Edition
8.2 [64-bit Mac OS X (Intel)] (Dec 21, 2011 13:59)
Copyright (C) 1985-2010, Franz Inc., Oakland, CA, USA.  All Rights Reserved.
[...]
cl-user(1): (defun foo (arg) (format t "foo called, arg ~A~%" arg))
foo
cl-user(2): (defun bar () (foo "from bar"))
bar
cl-user(3): (defun qux () (foo "from qux"))
qux
cl-user(4): (defun main () (bar) (qux))
main
cl-user(5): (trace (foo :inside qux))
(foo)
cl-user(6): (main)
foo called, arg from bar
 0[2]: (foo "from qux")
foo called, arg from qux
 0[2]: returned nil
nil

Note how the function FOO is called with no tracing when called from BAR, but with tracing from QUX.

Dynamically selective tracing with SBCL

Sometimes, one wants to trace a function based on some dynamic condition. SBCL's tracing facility offers this, as well as programmatic access to the argument list of the function being called:

This is SBCL 1.0.55.0-abb03f9, an implementation of ANSI Common Lisp.
[...]
* (defun foo (arg) (format t "foo called, arg ~A~%" arg))
FOO
* (defun bar () (foo 1) (foo 2))
BAR
* (trace foo :condition (= (sb-debug:arg 0) 2))
(FOO)
* (bar)
foo called, arg 1
  0: (FOO 2)
foo called, arg 2
  0: FOO returned NIL
NIL

Note how tracing is only enabled when FOO is called with a specific argument value.

Wrapping up

The purpose of this post was to show you some of the non-standard tracing options that Common Lisp implementations offer. As you can see, they are powerful debugging tools that can be used with no source code changes required. As these options are not standardized, make sure that you check out the documentation of your implementation. Happy debugging!

Learning Ruby, and Ruby vs. Lisp

The company I work for has a lot of legacy Ruby code, and as Ruby has become kind of a mainstream language, I decided to get a book about it and learn how it works.

As my learning resource, I chose The Ruby Programming language by David Flanagan and Yukihiro Matsumoto as that receives great customer reviews, covers Ruby 1.8.7 and 1.9 and is authoritative because the language creator is one of the authors.

The book makes a good read in general. There are plenty of code examples, but not too much to obscure the prose. What I found first interesting, later annoying, was the frequent use of words like "complex", "complicated", "confusing", "surprising" or "advanced" to describe features of the language. I'd rather decide myself about using such attributes to describe something that I've just learned.

Having spent so much time with Common Lisp, I almost forgot that programming languages usually evolve over the years. Ruby is no exception, and the fact that there are significant differences between Ruby 1.8.7 and Ruby 1.9 kind of bothers me - I'll probably never write code in Ruby 1.8.7, but the differences between the two versions seem to be rather subtle and I'm curious to see how much that is going to be a bother in the future, working with legacy code.

The common theme for Ruby seems to be succinctness. This comes at the expense of making the syntax rather complex, with several special case rules required to solve ambiguities. I don't have the practice to judge whether this is a problem, but from the book, it seems there are quite some things to remember.

It seems that Ruby started off as a purely object oriented language and only later discovered that function-oriented programming is nice, too. The deep roots of object orientation made it rather hard to actually get free functions (which are not member functions of an object) integrated. Contrary to what I am used to, member functions are not a special case of free functions, but rather something quite different. It requires explicit conversion steps to convert a member function into a free function (called procs in Ruby), and invocation syntax is also different between the two. Again, the description may sound worse than it is in practice.

What I really liked was the generalization of code blocks into fibers. Ruby does not have full coroutines, but the restricted form that is available is generalized well and seems like it could be useful for building pretty wild asynchronous systems. Also, it is nice that the bindings of closures can be accessed.

But then, Ruby is an interpreted language and this fact is re-stated throughout the book. With Just In Time compilation, this could become a non-problem, but I'm not sure how well Ruby can be optimized due to its very dynamic nature. Just to see how fast it is compared to Common Lisp, I implemented the Sudoku solver from chapter 1 of the Ruby book in CL and gave the two implementations a puzzle to solve. It took the Ruby solver 0.890 CPU seconds (Ruby 1.9.2p290), whereas the Lisp solver (Clozure CL 1.7) used 0.087 CPU seconds to solve the puzzle. Ten times slower, whatever you'll make of that.

In the book, it is mentioned how little code the Sudoku solver actually uses. This is true, but then, the Lisp version is not longer. It does not seem as if adding syntax is actually the best way to add the possibility to write succinct programs to a language, and the price of the complex grammar is rather high.

Writing the CL solver, I found myself not writing tests again and then poking around in problems of my implementation without knowing what works and what does not. As I want to practice more TDD, I stepped back and added tests. This led me to solve a problem that I had with my previous attempts to practice TDD in Lisp - I do not want to export all the symbols that the tests exercise from the packages that I use, but I also don't want to import the unit testing library into my own library packages. Thus, I wrote a deftestpackage macro that creates a new package to contain the tests that I write and automatically imports all symbols from the package being tested. That way, I can easily keep tests and library source separate and don't need to qualify internal symbols in the tests.

My overall takeaway on the Ruby is this: Ruby seems to be a language that has grown from being purely object oriented to supporting functional programming. That growth was not completely natural, and it seems that if Ruby is not used as a pure object oriented language, the syntax becomes rather messy and hard to grasp. This is similar to C++, which in its first versions was relatively nice (I hear you "ow"!), but has grown into into an incomprehensible mess once people recognized how templates can be abused for metaprogramming.

I can see the appeal of Ruby, but there seems little it has to offer to me that Common Lisp cannot provide. The lack of a formal specification and the ugly grammar put me off. Then again I'm pretty sure that Ruby is more enjoyable than many other popular languages. I'm looking forward to see my theoretical conceptions be shaken by actual practice.

Global Day of Code Retreat with Lisp

Last Saturday, I attended the Global Day of Code Retreat. I found out about the event in my Twitter stream, and when I signed up for the event, I did not have much of an idea of what it would be about, except code. With some research, I found that it would be a day of using Test Driven Development practices to improve on coding style and quality. Fair enough I thought, got myself a book on TDD (which I have always wanted to do more) and reserved my seat, which was free.

The event was hosted by immobilienscout24 and sponsored by Nokia. Sponsoring included catering, and as the day started at 8:00am, the availability of a breakfast was very welcome. The crowd consisted of some 30 hackers. An informal poll showed that most of them were either using Java or Ruby as their first language, a few were into JavaScript. One guy said that C++ was his primary language, and I was the only Common Lisp hacker.

The format of the all-day event consisted of six sessions of 45 minutes. In every session, people would form new pairs and implement parts of Conway's Game of Life using TDD practices. For each session, the pair chooses a specific challenge or goal to solve. At the end of the session, every pair deletes their code, discusses what they learned and then joins the group to share their thoughts.

The focus of the goals and challenges was on TDD practices. For about a half of the participants, TDD was already part of their daily work routine. The other half had heard of TDD, like myself, and joined to learn about it in the event. For me, this was one of the bigger takeways. I found it very helpful to pair with people that practice TDD in order to learn how they'd go about to write a test before they'd write an implementation. It seems that the TDD school favors a very small test granularity. As was explained by the organizer of the meeting, one would write very small tests that exercise very small aspects of the production code initially, and then build up on that.

In the six sessions, I used Java, JavaScript and Common Lisp. For Java, the organizers had come up with a skeleton project that included a unit testing environment (jUnit?). For JavaScript, I had prepared a simple browser based environment with QUnit, for Common Lisp I used Clozure CL and the simple unit-test library that I have used in the past.

Here are some observations:

The Java dudes travel with heavy baggage

IntelliJ was the most popular IDE among the Java folks that I've talked to. I am split between admiration and disgust when it comes to what seems common practice in the Java world. I'm used to write code by thinking about what I want to write, then typing the stuff that I've thought about. With Java and IntelliJ, the process is more interactive with the IDE, i.e. the programmer types a few characters, the IDE displays some menu to choose from or automatically adds code and so on. This is nice when the IDE automatically recognizes, say, that you are using a name of a class that has not yet been defined or imported and then offers you to either import a package that defines a class of the name that you've typed, or to create a class skeleton (including all the boilerplate and ceremony that Java wants) for you.

While this appears to be convenient, it also does not always work. In both of my Java sessions, the IDE got confused in some way or another, which was not fatal, but still annoyed me. Also, the static nature of Java slowed us down. For one, even though Java does have all the nice data structures (we wanted to use a set of coordinates), we wasted a lot of the short session times converting data types and supplying boilerplate that allowed us to actually put coordinate objects into a set. Also, file based compilation took time - Not minutes, but seconds. I was assured that one would use libraries and code generators in production code and that desktop machines were faster in compiling, but I still can't really relate Java to Agile, in the sense of the word.

Also, the use of code generators in IntelliJ makes me wonder how one maintains such code. How can one actually distinguish between what was carefully crafted and what was pasted, from templates, by the IDE? In my eyes, this is like copy and paste programming with the copy step optimized into templates. I'm not the first to say that and it does not come as a surprise either, but it was an interesting experience for me nevertheless.

Pair programming is fun

Being a remote consultant, I rarely have the chance to interact over code with other programmers. This was something I found enjoyable to practice, even in languages and environments that I'm not familiar with. It is amazing how vastly different the approaches to implementing a relatively simple thing can be, and compared to code reviews - in particular if they're done by email - it is much easier to make suggestions in a constructive way. In that respect, writing code that is supposed to be deleted also helps concentrating on the essence as no sense of code ownership is developed.

I found the pair programming experience significantly impacted by the fact that programmers use different coding environments. Working on an unfamiliar keyboard and with an unfamiliar IDE is a real productivity killer. I'd hope that in a team that pairs regularly, the work environments are more standardized than they have been on this event. I was using Emacs for my Lisp and JavaScript sessions, and my partners had a hard time getting along. What I found rather interesting was that the guys who wrote JavaScript in my Emacs always mimiced what their IDE would do for them. Rather then writing "if (foo) { doSomething()", they'd type "if (foo) {}", then navigate into the braces with the cursor to code along. This is kind of curious because it seems that the balancing of parentheses, brackets and braces is much more of a chore in other languages, to the point where people slow themselves down a lot if not aided by an IDE to the balancing for them. We Lispers, with only the parentheses to keep track of, have a much easier life, in particular with Emacs doing the indentation for us.

Is TDD the kool aid?

In some sense, I am sceptical about TDD. Testing is a great idea, and doing it in a disciplined fashion certainly helps writing better quality software. Also, automated tests are really the best way to prepare software for change. But, and this is where this Saturday could not convince me, I don't believe that spending time on writing very fine grained unit tests for every aspect of of a program helps preparing it better for refactoring and change. I think that testing against external requirements is the real key to writing programs that can be changed facing changing requirements, and that it should be possible to relate every test to some requirement. I must admit that one Saturday of TDD does not give me sufficient experience to judge, though.

Common Lisp and TDD

Some of the TDD discipline probably owes to development cycle in statically compiled languages. Where we Common Lispers have a rather incremental style and do our testing interactively in the repl, developers in languages like Java or C++ write larger chunks of code before they plug it together to do something meaningful. Such environments give the developer less insight into the run-time behavior of a running system, and tests are a way to make sure that more of the interactions of the system components are actually exercised.

I am not claiming that an interactive development environment makes testing less useful. In such an environment, though, it is common to write and test a function in small, iterative cycles. Furthermore, through the use of a tracing facility, the dynamic bahavior of a system can be observed at any time, without the need to touch or recompile the code.

In any case, this was a very nice experience and I'll go to a Code Retreat again, if I can. For that, I'll probably prepare myself for Java tools better in order to get more out of pair programming.

Hunchentoot v1.2.0 released

After over a year of no releases, Hunchentoot v1.2.0 is out. It incorporates a large number of changes from various people and is the first release that I have made as the maintainer of Hunchentoot. There have been a fair number of changes, many of them contributed by Hunchentoot's users. Thank you all for that!

The release tarball is available here: https://github.com/downloads/edicl/hunchentoot/hunchentoot-1.2.0.tar.gz

This is my first release, and as such it is likely that I'm not yet up to the release standards that Edi has set forth. Please report any problems you may have with the new version to the Hunchentoot mailing list. I will try to resolve problems quickly and possible make a new minor release for each major problem found.

The development repository for Hunchentoot is now hosted on github. If you want to contribute, it would be appreciated if you could submit your patches through github, too.

Here is a list of changes:

• Add www/ directory with default file tree that is being served
• Add error template mechanism and improve error reporting in general.
• Improve automatic testing, SBCL kludge to support asdf:test-op
• Allegro CL modern mode fixes
• Fix bugs in serving partial responses
• Limit maximum number of threads that Hunchentoot creates (Dan Weinreb, Scott McKay)
• Export fixes (Gordon Sims, Andrey Moskvitin)
• Factor out easy-handler logic into separate acceptor subclass
• Export two session functions (Nico de Jager)
• Allow no Content-Type header (Chaitanya Gupta)
• Patch for compilation with ECL (Sohail Somani)
• Fix DEFINE-EASY-HANDLER for multiple acceptors (Nicolas Neuss)
• Revived *SHOW-LISP-BACKTRACES-P*
• Made sure "100 Continue" is returned even if the client sends "Expect: 100-continue" twice (reported by Gordon Sims)
• Fixed typo in code which interprets transfer encodings

Thank you for your patience!
Hans

Finding Changes to Quicklisped Software

In my current project, I'm working with Allegro CL in modern mode. Modern mode is a non-standard mode supporting mixed-case symbols in the reader. This means that in modern mode, the two literal symbols :foo and :FOO are different things. This is different from other lisps, that convert names of literal symbols read to uppercase before interning them. This is why :foo, :Foo and :FOO all mean the same :FOO, by default.

I am using the excellent Quicklisp package management utility to pull all dependencies that a certain package needs. With ACL in modern mode, though, I had to fix several libraries that used all-caps symbol literals (like #:FOO, NIL etc.). The fixes were mostly trivial, but when I had all the stuff running that I wanted, I had modified several of the libraries that I had installed with Quicklisp.

I am planning to import all the external libraries into git repositories and not use Quicklisp in production. I still want to be able to easily update to new upstream versions and also send the changes that I made myself upstream. To make the process of finding what I've changed easier, I envisioned a tool that tells me which of the libraries that I had installed with Quicklisp have local changes, and what these changes were. I asked Zach Beane for help, and he promptly came up with a small function called QDIFF that performs a diff between Quicklisp's base version and the locally extracted version of a library. I hacked the tool slightly to make it work with Allegro CL and iterate over all locally installed systems. It is available as a gist.

Zach supplied a SBCL specific way to run the diff command. I added one based on ASDF:RUN-SHELL-COMMAND, which apparently prefixes all output lines with "; ". I don't like that, but I also did not want to pull in a shell compatibility layer into this self-contained thing.

I praised Quicklisp in the past, but there is nothing wrong with doing it again. Open Source Common Lisp has become so much better with it, and I need to thank Zach again for being so passionate about making the Common Lisp world better! Thanks, Zach!

Ediware moving to github

I have been more or less the primary maintainer for the Hunchentoot web server written in Common Lisp for several years now. Its primary Author, Edi Weitz, has been doing Lispworks related work and release engineering, but he now intends to step down from that role, too.

In order to open up development and encourage hacking and extending Hunchentoot, we have decided to move the development from the aging BKNR Subversion repository to github. That will make it easier to fork new versions, integrate patches, discuss code changes and track issues.

Together with Hunchentoot, a bunch of other libraries written by Edi Weitz have been moved to the edicl organization on github. The list is not complete and currently consists of libraries that I have used or hacked in the past. Feel free to let us know if you'd like some other of Edi's libraries to be relocated.

Happy hacking!

Orphaned (?) projects on common-lisp.net

common-lisp.net has moved to a new (virtual) box and in the process, we are cleaning up a little. One particular thing came to my attention: The maintainers of a number of projects can no longer be reached under the email address that they have registered with, so moderation requests bounce (to me). Either these projects have gone stale (even the maintainer lost interest), it is an oversight, or it is just an error that should be fixed. So, please check the list below and let me (hans.huebner@gmail.com) know if you know how to reach the maintainer(s). Projects that we don't get any replies for will eventually have their mailing lists be closed. Thank you for your help!

List of projects:

• cffi
• cl-magick
• cl-match
• cl-mpd
• cl-soap
• cl-stm
• ecl
• elephant
• gecol
• morphologie

Quicklisp - The upcoming solution to Common Lisp's "library problem"

We're all tired of hearing that Common Lisp is all nice, but it has a library problem. True, many of the open source libraries written in Common Lisp are half-baked, incomplete, unmaintained and tasteless. But this is true with libraries in all languages, so that can't be the real problem. The real problem, as I see it, is that until now, there was no universally good way of installing a library and all of its dependencies.

The existing, widely deployed solutions (asdf-install and clbuild) require external tools to work, depend on the availability of diverse servers in the internet, and do not have something like a central maintainer who ensures that things are always in basically working order. Also, these solutions are rather unportable, as they depend on external tools and a Unixish environment which is not available everywhere.

My personal solution to this problem, until now, was to have a Subversion repository that contains all the libraries that any of my projects need. Some of these came from release tarballs, some from other revision control repositories, and maintenance was manual. Whenever I started a new project, I brought some of those libraries up to date, coped with any (new) dependencies, lost backwards compatibility with my older projects etc. This kind of sucked, but it had the beauty that both development and deployment ended up being relatively easy. All that was required was a svn checkout from my repository, and I was all set on a new machine.

Recently, Zach Beane was fed up enough to do something about this: He created Quicklisp, a self-contained, centrally managed, cloud hosted Lisp library system which aims to run everywhere and provide users with a one-stop solution for the "library problem". Quicklisp is in an early alpha stage, but having tried it, I must say that I am pleased and impressed. Finally, Common Lisp can also become a glue language like Python and Perl.

I tried Quicklisp today because Twitter has notified me that basic http authentication for applications will no longer be supported. Instead, one is now supposed to use OAuth to authenticate requests sent to Twitter. This required my reaction.

About two years ago, I have written a small gateway program that forwards new postings to Planet Lisp to Twitter. At the time, I was rather frustrated with Common Lisp and thus took the gateway as an opportunity to try Clojure. Today, I looked at the source of the gateway again to find out what it would take to make it use OAuth instead of basic http authentication. As the gateway was the only program I have written in Clojure so far, and I was not really that eager to extend it to OAuth. I tried for a few minutes, but found that my old program did not run with the current Clojure version right away, so I would have to basically start setting up a Clojure development environment from scratch in order to be able to use an existing, open source OAuth library.

Instead, I thought I'd give Quicklisp a spin. After all, parsing some XML and sending HTTP requests are no big deal in Common Lisp either, and an OAuth library is available, too. With Quicklisp, all it should take was write some glue code to connect DRAKMA, CXML, cl-ppcre and cl-oauth. Installing the bunch should be a matter of loading quicklisp.lisp, then typing (ql:quickload `("drakma" "cxml" "cl-ppcre" "cl-oauth")) and watching the show scroll by.

And hey, it worked right away. I even embedded a web server in the gateway application so that it can be monitored by the system status monitors provided by my hosting provider. All in all, rewriting the thing in Common Lisp and deploying it took no longer than two hours, and the source is not significantly longer than the Clojure version either. Furthermore, the new gateway properly deals with non-ASCII characters. Embarrassingly, the Clojure gateway was buggy in that respect and never properly twittered the titles of my own blog posts. Thanks, Zach!

How to convert your old-style Symbolics keyboard to USB

Peter got in touch with me because he tried to follow my instructions how to convert a Symbolics keyboard to USB with his old-style keyboard and he failed. He assumed, and that was what I would have assumed, that the color coding of the internal cabling of the old and new keyboards would match, but they don't. Here is a table that shows the pinout for both keyboard types:

New Style		Old Style		Function	Teensy Pin
Pin#	Color	Pin#	Color
1	blue	8	white	GND	GND
2	yellow		black	not	connected
3	green	2	red	5V	5V
4	red	3	green	DIN	D4
5	black	1	yellow	CLK	D5
6	white	5	blue	CLR	D6

Clojure Slides from February Berlin Lispers Meetup

Stefan Richter, our speaker at the February Berlin Lispers Meetup, has made the slides of his presentation available on line. Enjoy, and see you next week!

How to convert your Symbolics keyboard to USB

I spent another hour finishing up the Teensy based Symbolics keyboard converter. The software now works as well as the PS/2 version, and uses the same keyboard translation table generation tool. Converting a keyboard should now be doable for anyone who has basic soldering skills.
What you'll need:

• Teensy (The photos show a build with the obsolete Teensy 1.0)
• USB cable (B type)
• 6-pin .1'' header
• Some wire

First, you need to program the Teensy with the Firmware .hex file. Please note that this .HEX file is now for the Teensy 2.0 board, which also looks slightly different than the Teensy 1.0 board shown on the photos. Don't worry about customizing the firmware right now. If you decide that you want it to work differently than my version, you can update it later.
Get five pieces of wire, about 5 inches long. Solder them to the 6 pin header, leaving pin 2 unconnected. Connect the other ends to the Teensy.

pin #	color	teensy pad	function
1	blue	GND
2		(not connected)
3	green	5V
4	red	D4	DIN
5	black	D5	CLK
6	white	D6<	CLR

Yes, 5V and GND were swapped in my original post, sorry and thanks to Gene Diveglia for reporting
When building with a Teensy 2.0, connect the D4, D5, and D6 pins as printed on the Teensy PCB.

Open the Symbolics keyboard by removing the five screws at the bottom. Carefully remove the small PCB that sits on the larger one, unplug the spiral cable.  Using a tool like a large screwdriver, remove the strain relief by pushing it to the outside of the case. Remove the cable, take off the strain relief. Using a pliers, cut out some of the plastic so that the part can be used for the round USB cable.
Using a sharp knife, make the B type connector of the USB cable as small as possible.

Put the strain relief onto the USB cable, about 4 inches away from the scaled down B connector. Using a tool, firmly put it back into the keyboard case. Fixate the cable to the bottom shell of the case like shown. Connect the Teensy to the USB cable, plug the header into the keyboard PCB.
Put some protective plastic around the Teensy so that it does not short circuit anything on the keyboard PCB.
Finally, reassemble the keyboard. It should be easy to put the daughter PCB back onto the main PCB and fit the two shells together. You're done.
Make sure that you've read the README.txt file for the converter firmware. Note that if you want to convert your old-style keyboard, you need to know that the internal color coding of the cables is different with those keyboards. This blog post has a table with the information that you need. Have fun!