BKNR at the Boston Lisp Meeting

After a weekend full of parentheses in Amsterdam, I attended the Boston Lisp Meeting hosted at MIT.  Peter Dillinger gave a presentation on the ACL2s, which is an Eclipse based user interface for the ACL2 theorem prover.  I have no background in mathematics and as such, I had to guess my way through the terminology, but the talk was entertaining and even though it somehow does not feel right to see an advanced Common Lisp program be controlled by a lowly Java application, I see the practical aspects of the approach.  Bottom line:  Common Lisp loses when it comes to programming modern user interfaces in general, and I'm not sure that we can do much about that.

The second speech for the evening was my presentation of the BKNR datastore. I found myself being well received, and the questions from the audience mostly were those that I have been asking myself during the last few years.  I have put the slides online, and I also updated the BKNR website to make the documentation easier to access.  If you have Google Earth installed, you can play with the (alpha quality) GE interface to the BKNR based Samboja Lestari web site.  If you want, you can have a look at the source code, too.  Our use of publish/subscribe for persistent objects may be of interest, as well as the use of Screamer for brute-force determination of the largest rectangle inside an arbitary polygon.  Kilian, who does most of the programming in the BOS project presently, has a background in music composition.  He found it easier to formulate the solution to the problem using constraints rather than using explicit looping.

The socializing part of the meeting was rather short for me as I am mostly staying on Central European Time while being in Boston, so it felt like past midnight to me.  I met several interesting people anyway,  and I hope that I can synchronize my next visits in Cambridge with the meetings.

Chaosradio Express on Lisp

Last week, I have been a guest at Tim Pritlove's Chaosradio Express and he interviewed me about Lisp. The podcast is in german, we talked about how I came to Lisp and what I think it is about.

Feature development and getting carried away on Lisp

Customer Demands

Recently, I have been spending some weekend and late night hours on extending the QuickHoney web system. The artists want to see the system modernized with RSS feeds and more interaction, and we'll also be adding a shop where both digital and real warez can be bought.

Peter in particular wants to get more feedback, so he asked me to add a "quick feedback" mechanism that visitors can use to send a short textual comment on any of the pictures. In this post, I'll describe how I added this feature using Javascript, BKNR, CL-SMTP and CL-MIME.

First off, here is a little overview of the QuickHoney web application. It is a early AJAX style application with one HTML page consisting of a number of layers which are controlled by a Javascript application. Communication between the Javascript application and the backend server is done through Javascript code snippets that are generated on the server and evaluated inside the client application using IFRAMES. The user navigates through categories and subcategories to individual pictures.

The backend for the QuickHoney application is written in Common Lisp using the BKNR framework. It uses the datastore extensively as well as the cl-gd image processing library written by Edi Weitz.

Frontend Functionality

The feedback functionality will work on a per-picture basis. When a picture is displayed, a small "provide feedback" icon will be displayed. When the user clicks it, a form will be displayed in a layered window. The form will consist of "From" and "Text" fields and a "Send" button. The onclick action of the button is connected to a function that packs the contents of the "From" and "Text" fields into a urlencoded form data string and send it to the server using a POST request:

function digg_send()
{
    var d = doXHR("/digg-image/" + current_image.id,
                  { method: 'POST',
                    headers: {"Content-Type":"application/x-www-form-urlencoded"},
                    sendContent: queryString({ from: $('digg_from').value,
                                               text: $('digg_text').value }) })
        .addCallback(function () { alert('sent'); });
    return false;
}

doXHR and $ are functions from the MochiKit Javascript library which I like to use for non-GUI-related things for its conciseness and conceptional soundness.

As can be seen in the Javascript snippet above, there is a /digg-image/ handler on the web server that is used to send the feedback. The URL that is used also consists of the object ID of the image currently displayed. Every persistent object in the BKNR datastore has a unique object ID, and the FIND-STORE-OBJECT function can be used to find an object with a certain object ID in the store.

Implementing a Backend Handler

The BKNR web framework makes it easy for applications to declare new handlers which relate to a certain store object. It provides for a set of handler classes that application handlers can inherit from. These handler base classes implement request URL parsing and automatically call handler methods with with relevant information from the URL and the request body parsed into Lisp objects.

For the feedback feature, we need to subclass the OBJECT-HANDLER base class which extracts the object ID out of the URL, uses FIND-STORE-OBJECT to find the relevant object in the store and then calls the HANDLE-OBJECT method of the handler to actually handle the request. The declaration for this handler class looks like this:

(defclass digg-image-handler (object-handler)
  ()
  (:default-initargs :object-class 'quickhoney-image))

The :OBJECT-CLASS initarg can optionally be specified when creating an OBJECT-HANDLER object to make sure that HANDLE-OBJECT is only called for objects of that class. If the object ID in the URL references an object from a different class, an error page is displayed to the user.

The HANDLE-OBJECT method for the DIGG-IMAGE-HANDLER class is specialized on both the DIGG-IMAGE-HANDLER handler class and on the QUICKHONEY-IMAGE class. It extracts the form parameters from the request and opens a connection to the SMTP server to send a mail to the owner or owners of the QUICKHONEY-IMAGE. The mail itself is a simple HTML mail with a table containing the name of the image, hyperlinked to the online page with the image and the feedback information entered by the user. Also, a thumbnail of the image is included with the email so that the artist immediately sees what picture the user is raving about. Modern mailers (like Apple Mail or Google Mail) display images attached in a multipart/mixed MIME mail inline, so there is no need to come up with a fancy HTML mail that references elements included in the same mail body.

This is the source code of the handler:

(defmethod handle-object ((handler digg-image-handler) (image quickhoney-image))
  (with-query-params (from text)
    (cl-smtp:with-smtp-mail (smtp "localhost"
                                  "webserver@quickhoney.com"
                                  (remove-duplicates (mapcar #'user-email
                                                             (or (owned-object-owners image)
                                                                 (list (find-user "n")
                                                                       (find-user "p"))))))
      (cl-mime:print-mime
       smtp
       (make-instance
        'cl-mime:multipart-mime
        :subtype "mixed"
        :content (list
                  (make-instance
                   'cl-mime:mime
                   :type "text" :subtype "html"
                   :content (with-output-to-string (s)
                              (html-stream s
                                           (:html
                                            (:head
                                             (:title "Picture comment"))
                                            (:body
                                             (:table
                                              (:tbody
                                               (:tr
                                                ((:td :colspan "2")
                                                 "Comment on picture "
                                                 ((:a :href (make-image-link image))
                                                  (:princ-safe (store-image-name image)))))
                                               (:tr
                                                (:td (:b "From"))
                                                (:td (:princ-safe from))))
                                               (:tr
                                                ((:td :valign "top") (:b "Text"))
                                                (:td (:princ-safe text)))))))))
                  (make-instance
                   'cl-mime:mime
                   :type "image"
                   :subtype (string-downcase (symbol-name (blob-type image)))
                   :encoding :base64
                   :content (flexi-streams:with-output-to-sequence (s)
                              (blob-to-stream image s)))))
       t t))))

For completeness, let me also show you how the handler is entered into the list of handlers of the Quickhoney backend server:

(defun publish-quickhoney ()
  (unpublish)
  (make-instance 'website
		 :name "Quickhoney CMS"
		 :handler-definitions `(("/random-image" random-image-handler)
                                 ("/animation" animation-handler)
                                 ("/image-query-js" image-query-js-handler)
                                 ("/login-js" login-js-handler)
                                 ("/clients-js" clients-js-handler)
                                 ("/buttons-js" buttons-js-handler)
                                 ("/edit-image-js" edit-image-js-handler)
                                 ("/upload-image" upload-image-handler)
                                 ("/upload-animation" upload-animation-handler)
                                 ("/upload-button" upload-button-handler)
                                 ("/rss" rss-handler)
                                 ("/admin" admin-handler)
                                 ("/upload-news" upload-news-handler)
                                 ("/digg-image" digg-image-handler)
                                 ("/" template-handler
                                      :default-template "frontpage"
                                      :destination ,(namestring (merge-pathnames "templates/" *website-directory*))
                                      :command-packages (("http://quickhoney.com/" . :quickhoney.tags)
                                                         ("http://bknr.net/" . :bknr.web)))
                                 user
                                 images
                                 ("/static" directory-handler
                                            :destination ,(merge-pathnames #p"static/" *website-directory*))
                                 ("/MochiKit" directory-handler
                                              :destination ,(merge-pathnames #p"static/MochiKit/" *website-directory*))
                                 ("/favicon.ico" file-handler
                                                 :destination ,(merge-pathnames #p"static/favicon.ico" *website-directory*)
                                 :content-type "application/x-icon"))
		 :admin-navigation '(("user" . "/user/")
				     ("images" . "/edit-images")
				     ("import" . "/import")
				     ("logout" . "/logout"))
		 :authorizer (make-instance 'bknr-authorizer)
		 :site-logo-url "/image/quickhoney/color,000000,33ff00"
		 :login-logo-url "/image/quickhoney/color,000000,33ff00/double,3"
		 :style-sheet-urls '("/static/styles.css")
		 :javascript-urls '("/static/javascript.js")))

Getting Carried Away on Common Lisp

As you can see, I had to write little code to implement this functionality. In fact, the whole thing should have taken no longer than one or two hours, but here is how I found myself getting carried away:

For one, I spent substantial time on refactoring CL-SMTP, as you can read in my last blog entry. That took a few hours. For another, I really don't like how mime emails are constructed with MAKE-INSTANCE in the HANDLE-OBJECT method above. I thought that I'd be nice to have a DEFINE-CONSTRUCTOR macro that created a function to create objects of arbitary classes, but that allows for one or more positional arguments in addition to any of the keyword arguments accepted by MAKE-INSTANCE for a particular class.

Thus, instead of just coping with the slight ugliness, I tried myself on a macro. Simple as it looked, it surely became more complicated as I had to deal with defaulted arguments, and I was stopped from investing any more time into this when I was reminded by cmm that INITIALIZE-INSTANCE and SHARED-INITIALIZE can define additional keyword arguments that are accepted by MAKE-INSTANCE. Sure, it would be possible to find all applicable methods and extract more acceptable arguments from their lambda lists. Yet, I felt that I had enough fun for this last weekend and wrapped up the feedback functionality for Quickhoney.

Refactoring CL-SMTP

In my recent refactoring of BKNR, I decided that we do no longer want to use any of Franz' open source libraries if we can avoid it. Even though they work fine in general, hacking them is a pain because they adhere to John Foderaros Common Lisp Coding Standards which basically say that one should use Franz' own non-standard IF* macro for all conditionals. I do think that one should be careful with nesting conditionals deeply, but I do not agree with using a macro and spacing as a cure. If I have code in which conditional nesting exceeds two levels, I refactor to extract conditional code into functions. That way, functions are kept shorter and the use of names instead of literal code usually makes it easier to understand what's going on.

So my easter goal was to replace Franz' NET.POST-OFFICE SMTP client by CL-SMTP. Both clients do not support proper quoting of non-ASCII characters in mail headers, thus the need to hack arose and IF* is nothing I want to get myself used to.

A SMTP client really is not that complicated to begin with, and apart from the basic functionality, CL-SMTP already supported SSL and authentication, which nowadays are two basic requirements. What it was missing was the possibility to send pre-formatted messages, which is something that I require because I usually make up my own messages, including headers and body, using format strings or CL-MIME if I want to send attachments or otherwise need more control over the mail body.

CL-SMTP prove to be a little hackish. Seemingly, only simple mail sending had originally been planned for, and the API had then been extended multiple times with growing user needs. There was no layering between the SMTP protocol aspects and the message formatting functionality, both being freely interleaved. While being simple to use for those use cases that had been planned for, the API was not helpful for my intended use.

In a first round, I simplified the code, collapsed a few common patterns into functions and made the code generally easier to hack on. I then split the SMTP protocol handling into an exported macro, WITH-SMTP-MAIL, that is used to establish an SMTP connection and create the mail envelope. The body of the macro invocation is then invoked with a variable bound to the stream connected to the SMTP server. The existing mail sending functions of CL-SMTP have been converted to use that API, too.

I then incorporated a patch that I found in the CL-SMTP mailing list archive so that raw TLS is supported. The existing TLS functionality worked by connecting to the standard SMTP port, then switching the cleartext connection to encrypted mode by issuing the STARTTLS command. In contrast, raw TLS works by having an SMTP server listen on a separate port for encrypted connections. No initial cleartext handshake is required in this operation mode.

Finally, I implemented automatic quoting of non-ASCII characters in mail headers using a specialized stream class. The Gray Streams Examples in the SBCL manual are my favoured cheat sheet when implementing special purpose stream classes.

The result of my work is available in the BKNR repository at svn://svn.bknr.net/svn/trunk/thirdparty/cl-smtp/ in case you want to give it a try right now.

I developed with Clozure CL on my Powerbook and after I committed, our buildbot went red for SBCL. I had named an accessor for my specialized stream class "STREAM" which triggered a package lock violation error on SBCL. The name was bad, so I changed it to "ENCAPSULATED-STREAM" and saw the buildbot going green for SBCL too. I love that!

I am now waiting for feedback on the refactorings and extensions. There are still bugs which need fixing and support for compilers other than CCL and SBCL needs to be verified. Also, the documentation for CL-SMTP needs to be better, and I think I'll just steal Edis HTML template and write something up myself next week. Also, an automated test for CL-SMTP would be great, but as this requires a SMTP peer to talk to, I have not really had a good idea how to implement that. Maybe later on.

I certainly hope that Jan Idzikowski, who is the author and maintainer of CL-SMTP, will accept my patches and make them part of the main distribution.

BKNR is alive

Recently, there has been quite some activity in the BKNR repository. This is because my company finally seems to become reality, with a real office, customers and all that. I even have an employee now, Kilian, who is working full time on BKNR based projects. I guess we are the only Lisp company in Berlin now!

As I am working for ITA Software full time, the fact that I have some more workforce allowed me to attack the long-outstanding major updates for QuickHoney and Create Rainforest. Quickhoney will be modernized with a Blog and RSS feed, and we will also implement a shopping system so that one can buy digital and real QuickHoney products. Create Rainforest will be extended into Google Earth. Instead of hacking our funny, but close to unusable satellite application, we will use Google Earth to display and visualize our data. The old system will stay, but new features will not be added to that.

These commercial projects have lead to several updates to BKNR in the past weeks:

We have finally converted from portable AllegroServe to Hunchentoot as HTTP server substrate. Even though AllegroServe performs fine, it written in a very peculiar style which makes hacking it a rather unpleasant experience. Also, Hunchentoot is better maintained. In the process, we got away with passing request arguments throughout the web environment. The current request is now (only) carried in a special variable, as applications seldomly require access to it.

SBCL and CCL are our current primary development platforms. I have still not given up on CMUCL completely, but we are waiting for the 19E release before we pick it up again. I am fed up with working around missing Unicode support, which we require in all our current projects.

We moved away from common-lisp.net with our repository and web sites. The primary reason for this move was that we now have a Buildbot running for BKNR, and I could not quickly get the neccessary infrastructure to run on common-lisp.net - Buildbot is Python based and requires substantial library support to run.

BKNR bugs have been fixed. Sure enough, there have been quite some of them. We hope to be better in the future with our continous integration testing and with more unit tests.

Sure enough, I will be moving this Blog to our BKNR based blog system soon. Until that happened, this is my place to practice.

Hardware woes

When I put the SECD project aside, it was because I could not get the RAM controller to run. I felt rather stupid, as the RAM is a very simple asynchronous chip that requires no special timing, yet all my attempts to reliably write the RAM from the 6809 host failed - And as long as that does not work, I can't start any program on the SECD coprocessor.

Today I finally found the source of my problems. It appears that even though there is an address bit ram_a(0) in the constraints file for the Trenz retrocomputing base board, it is not connected to the RAM. Supposedly, it is not connected because the RAM works word-wise.

It took writing a standalone RAM tester in VHDL and a lot of staring at the logic analyzer and Chipscope outputs to find out that I actually had an addressing problem. Also, John Kent was of great help again as he assured me that, prinicipially, the VHDL that accesses the RAM looks right. He also suggested that I might have a problem related to my constraints file, which appeared to be true.

Now, finally, I can turn to actually getting the LispKit compiler to work and see the fixed hardware garbage collection in action. Stay tuned.

Back to SECD

I have finally returned to debugging the SECD implementation on the FPGA. When I stopped with this, I had problems getting the dual ported RAM controller to run. The SECD CPU is designed as a coprocessor and relies on a host CPU to set up the program in its memory and to start itself. Once the program has finished to run, the host CPU reads out the results from the SECD memory.

My host CPU is a 6809, based on System09 written by John Kent. This is a historic coincidence, too, as the original SECD implementation used an 6809 as host CPU, too, at least for the testing. The 6809 software that has originally been used is not available anymore, so I needed to find a way to quickly write some diagnosis and debugging routines. System09 is designed to run the FLEX operating system and there is a BASIC interpreter for Flex available, but somehow I did not like that too much.

My first attempt to create a diagnostic monitor was to write a C program and compile it with gcc6809. This worked, but I did not like the turnaround times too much. Thus, I tried to find a Forth implementation for the 6809 that I could use as a diagnostics monitor. My search led me to Maisforth, which is a Forth implementation for a custom computer called the "mais kastje" which was built and used by a group of dutch Forth enthusiasts. I changed my System09 based 6809 system so that its memory map matches that of the "mais kastje" and hacked the Forth interpreter so that it properly initializes and uses the ACIA serial port in System09.

Getting Maisforth to run was not too hard, but I certainly would not have finished that without the help of the excellent Logicport logic analyzer that I have bought earlier this year. Logicport is a PC based logic analyzer that retails at $389, with 34 channels, complex triggering and 500 Mhz sample rate. The Windows software is very usable, and it comes with interpreters for a few serial protocols (I2C, SPI, RS232) which helped me to track down my ACIA initialization problem in a short timespan. I'm a very happy customer of this and can recommend Logicport to anyone who can't afford a "real" logic analyzer.

Now that I have Forth running on the FPGA, I am back to debugging the memory controller problems. I use Forth as a scriptable debug monitor, so in addition to examine and change memory contents, I can define symbolic addresses, write little test programs and thus automate debugging quite a bit. With Forth and the logic analyzer, I quickly found that my problem lies in how I set up the low and high byte select bits of the RAM chip, which has a 16 bit datapath.