Feature development and getting carried away on Lisp

Customer Demands

Recently, I have been spending some weekend and late night hours on extending the QuickHoney web system. The artists want to see the system modernized with RSS feeds and more interaction, and we'll also be adding a shop where both digital and real warez can be bought. Peter in particular wants to get more feedback, so he asked me to add a "quick feedback" mechanism that visitors can use to send a short textual comment on any of the pictures. In this post, I'll describe how I added this feature using Javascript, BKNR, CL-SMTP and CL-MIME. First off, here is a little overview of the QuickHoney web application. It is a early AJAX style application with one HTML page consisting of a number of layers which are controlled by a Javascript application. Communication between the Javascript application and the backend server is done through Javascript code snippets that are generated on the server and evaluated inside the client application using IFRAMES. The user navigates through categories and subcategories to individual pictures. The backend for the QuickHoney application is written in Common Lisp using the BKNR framework. It uses the datastore extensively as well as the cl-gd image processing library written by Edi Weitz.

Frontend Functionality

The feedback functionality will work on a per-picture basis. When a picture is displayed, a small "provide feedback" icon will be displayed. When the user clicks it, a form will be displayed in a layered window. The form will consist of "From" and "Text" fields and a "Send" button. The onclick action of the button is connected to a function that packs the contents of the "From" and "Text" fields into a urlencoded form data string and send it to the server using a POST request:

function digg_send()
{
    var d = doXHR("/digg-image/" + current_image.id,
                  { method: 'POST',
                    headers: {"Content-Type":"application/x-www-form-urlencoded"},
                    sendContent: queryString({ from: $('digg_from').value,
                                               text: $('digg_text').value }) })
        .addCallback(function () { alert('sent'); });
    return false;
}

doXHR and $ are functions from the MochiKit Javascript library which I like to use for non-GUI-related things for its conciseness and conceptional soundness. As can be seen in the Javascript snippet above, there is a /digg-image/ handler on the web server that is used to send the feedback. The URL that is used also consists of the object ID of the image currently displayed. Every persistent object in the BKNR datastore has a unique object ID, and the FIND-STORE-OBJECT function can be used to find an object with a certain object ID in the store.

Implementing a Backend Handler

The BKNR web framework makes it easy for applications to declare new handlers which relate to a certain store object. It provides for a set of handler classes that application handlers can inherit from. These handler base classes implement request URL parsing and automatically call handler methods with with relevant information from the URL and the request body parsed into Lisp objects. For the feedback feature, we need to subclass the OBJECT-HANDLER base class which extracts the object ID out of the URL, uses FIND-STORE-OBJECT to find the relevant object in the store and then calls the HANDLE-OBJECT method of the handler to actually handle the request. The declaration for this handler class looks like this:

(defclass digg-image-handler (object-handler)
  ()
  (:default-initargs :object-class 'quickhoney-image))

The :OBJECT-CLASS initarg can optionally be specified when creating an OBJECT-HANDLER object to make sure that HANDLE-OBJECT is only called for objects of that class. If the object ID in the URL references an object from a different class, an error page is displayed to the user. The HANDLE-OBJECT method for the DIGG-IMAGE-HANDLER class is specialized on both the DIGG-IMAGE-HANDLER handler class and on the QUICKHONEY-IMAGE class. It extracts the form parameters from the request and opens a connection to the SMTP server to send a mail to the owner or owners of the QUICKHONEY-IMAGE. The mail itself is a simple HTML mail with a table containing the name of the image, hyperlinked to the online page with the image and the feedback information entered by the user. Also, a thumbnail of the image is included with the email so that the artist immediately sees what picture the user is raving about. Modern mailers (like Apple Mail or Google Mail) display images attached in a multipart/mixed MIME mail inline, so there is no need to come up with a fancy HTML mail that references elements included in the same mail body. This is the source code of the handler:

(defmethod handle-object ((handler digg-image-handler) (image quickhoney-image))
  (with-query-params (from text)
    (cl-smtp:with-smtp-mail (smtp "localhost"
                                  "webserver@quickhoney.com"
                                  (remove-duplicates (mapcar #'user-email
                                                             (or (owned-object-owners image)
                                                                 (list (find-user "n")
                                                                       (find-user "p"))))))
      (cl-mime:print-mime
       smtp
       (make-instance
        'cl-mime:multipart-mime
        :subtype "mixed"
        :content (list
                  (make-instance
                   'cl-mime:mime
                   :type "text" :subtype "html"
                   :content (with-output-to-string (s)
                              (html-stream s
                                           (:html
                                            (:head
                                             (:title "Picture comment"))
                                            (:body
                                             (:table
                                              (:tbody
                                               (:tr
                                                ((:td :colspan "2")
                                                 "Comment on picture "
                                                 ((:a :href (make-image-link image))
                                                  (:princ-safe (store-image-name image)))))
                                               (:tr
                                                (:td (:b "From"))
                                                (:td (:princ-safe from))))
                                               (:tr
                                                ((:td :valign "top") (:b "Text"))
                                                (:td (:princ-safe text)))))))))
                  (make-instance
                   'cl-mime:mime
                   :type "image"
                   :subtype (string-downcase (symbol-name (blob-type image)))
                   :encoding :base64
                   :content (flexi-streams:with-output-to-sequence (s)
                              (blob-to-stream image s)))))
       t t))))

For completeness, let me also show you how the handler is entered into the list of handlers of the Quickhoney backend server:

(defun publish-quickhoney ()
  (unpublish)
  (make-instance 'website
		 :name "Quickhoney CMS"
		 :handler-definitions `(("/random-image" random-image-handler)
                                 ("/animation" animation-handler)
                                 ("/image-query-js" image-query-js-handler)
                                 ("/login-js" login-js-handler)
                                 ("/clients-js" clients-js-handler)
                                 ("/buttons-js" buttons-js-handler)
                                 ("/edit-image-js" edit-image-js-handler)
                                 ("/upload-image" upload-image-handler)
                                 ("/upload-animation" upload-animation-handler)
                                 ("/upload-button" upload-button-handler)
                                 ("/rss" rss-handler)
                                 ("/admin" admin-handler)
                                 ("/upload-news" upload-news-handler)
                                 ("/digg-image" digg-image-handler)
                                 ("/" template-handler
                                      :default-template "frontpage"
                                      :destination ,(namestring (merge-pathnames "templates/" *website-directory*))
                                      :command-packages (("http://quickhoney.com/" . :quickhoney.tags)
                                                         ("http://bknr.net/" . :bknr.web)))
                                 user
                                 images
                                 ("/static" directory-handler
                                            :destination ,(merge-pathnames #p"static/" *website-directory*))
                                 ("/MochiKit" directory-handler
                                              :destination ,(merge-pathnames #p"static/MochiKit/" *website-directory*))
                                 ("/favicon.ico" file-handler
                                                 :destination ,(merge-pathnames #p"static/favicon.ico" *website-directory*)
                                 :content-type "application/x-icon"))
		 :admin-navigation '(("user" . "/user/")
				     ("images" . "/edit-images")
				     ("import" . "/import")
				     ("logout" . "/logout"))
		 :authorizer (make-instance 'bknr-authorizer)
		 :site-logo-url "/image/quickhoney/color,000000,33ff00"
		 :login-logo-url "/image/quickhoney/color,000000,33ff00/double,3"
		 :style-sheet-urls '("/static/styles.css")
		 :javascript-urls '("/static/javascript.js")))

Getting Carried Away on Common Lisp

As you can see, I had to write little code to implement this functionality. In fact, the whole thing should have taken no longer than one or two hours, but here is how I found myself getting carried away: For one, I spent substantial time on refactoring CL-SMTP, as you can read in my last blog entry. That took a few hours. For another, I really don't like how mime emails are constructed with MAKE-INSTANCE in the HANDLE-OBJECT method above. I thought that I'd be nice to have a DEFINE-CONSTRUCTOR macro that created a function to create objects of arbitary classes, but that allows for one or more positional arguments in addition to any of the keyword arguments accepted by MAKE-INSTANCE for a particular class. Thus, instead of just coping with the slight ugliness, I tried myself on a macro. Simple as it looked, it surely became more complicated as I had to deal with defaulted arguments, and I was stopped from investing any more time into this when I was reminded by cmm that INITIALIZE-INSTANCE and SHARED-INITIALIZE can define additional keyword arguments that are accepted by MAKE-INSTANCE. Sure, it would be possible to find all applicable methods and extract more acceptable arguments from their lambda lists. Yet, I felt that I had enough fun for this last weekend and wrapped up the feedback functionality for Quickhoney.

Refactoring CL-SMTP

In my recent refactoring of BKNR, I decided that we do no longer want to use any of Franz' open source libraries if we can avoid it. Even though they work fine in general, hacking them is a pain because they adhere to John Foderaros Common Lisp Coding Standards which basically say that one should use Franz' own non-standard IF* macro for all conditionals. I do think that one should be careful with nesting conditionals deeply, but I do not agree with using a macro and spacing as a cure. If I have code in which conditional nesting exceeds two levels, I refactor to extract conditional code into functions. That way, functions are kept shorter and the use of names instead of literal code usually makes it easier to understand what's going on. So my easter goal was to replace Franz' NET.POST-OFFICE SMTP client by CL-SMTP. Both clients do not support proper quoting of non-ASCII characters in mail headers, thus the need to hack arose and IF* is nothing I want to get myself used to. A SMTP client really is not that complicated to begin with, and apart from the basic functionality, CL-SMTP already supported SSL and authentication, which nowadays are two basic requirements. What it was missing was the possibility to send pre-formatted messages, which is something that I require because I usually make up my own messages, including headers and body, using format strings or CL-MIME if I want to send attachments or otherwise need more control over the mail body. CL-SMTP prove to be a little hackish. Seemingly, only simple mail sending had originally been planned for, and the API had then been extended multiple times with growing user needs. There was no layering between the SMTP protocol aspects and the message formatting functionality, both being freely interleaved. While being simple to use for those use cases that had been planned for, the API was not helpful for my intended use. In a first round, I simplified the code, collapsed a few common patterns into functions and made the code generally easier to hack on. I then split the SMTP protocol handling into an exported macro, WITH-SMTP-MAIL, that is used to establish an SMTP connection and create the mail envelope. The body of the macro invocation is then invoked with a variable bound to the stream connected to the SMTP server. The existing mail sending functions of CL-SMTP have been converted to use that API, too. I then incorporated a patch that I found in the CL-SMTP mailing list archive so that raw TLS is supported. The existing TLS functionality worked by connecting to the standard SMTP port, then switching the cleartext connection to encrypted mode by issuing the STARTTLS command. In contrast, raw TLS works by having an SMTP server listen on a separate port for encrypted connections. No initial cleartext handshake is required in this operation mode. Finally, I implemented automatic quoting of non-ASCII characters in mail headers using a specialized stream class. The Gray Streams Examples in the SBCL manual are my favoured cheat sheet when implementing special purpose stream classes. The result of my work is available in the BKNR repository at svn://svn.bknr.net/svn/trunk/thirdparty/cl-smtp/ in case you want to give it a try right now. I developed with Clozure CL on my Powerbook and after I committed, our buildbot went red for SBCL. I had named an accessor for my specialized stream class "STREAM" which triggered a package lock violation error on SBCL. The name was bad, so I changed it to "ENCAPSULATED-STREAM" and saw the buildbot going green for SBCL too. I love that! I am now waiting for feedback on the refactorings and extensions. There are still bugs which need fixing and support for compilers other than CCL and SBCL needs to be verified. Also, the documentation for CL-SMTP needs to be better, and I think I'll just steal Edis HTML template and write something up myself next week. Also, an automated test for CL-SMTP would be great, but as this requires a SMTP peer to talk to, I have not really had a good idea how to implement that. Maybe later on. I certainly hope that Jan Idzikowski, who is the author and maintainer of CL-SMTP, will accept my patches and make them part of the main distribution.

BKNR is alive

Recently, there has been quite some activity in the BKNR repository. This is because my company finally seems to become reality, with a real office, customers and all that. I even have an employee now, Kilian, who is working full time on BKNR based projects. I guess we are the only Lisp company in Berlin now! As I am working for ITA Software full time, the fact that I have some more workforce allowed me to attack the long-outstanding major updates for QuickHoney and Create Rainforest. Quickhoney will be modernized with a Blog and RSS feed, and we will also implement a shopping system so that one can buy digital and real QuickHoney products. Create Rainforest will be extended into Google Earth. Instead of hacking our funny, but close to unusable satellite application, we will use Google Earth to display and visualize our data. The old system will stay, but new features will not be added to that. These commercial projects have lead to several updates to BKNR in the past weeks: We have finally converted from portable AllegroServe to Hunchentoot as HTTP server substrate. Even though AllegroServe performs fine, it written in a very peculiar style which makes hacking it a rather unpleasant experience. Also, Hunchentoot is better maintained. In the process, we got away with passing request arguments throughout the web environment. The current request is now (only) carried in a special variable, as applications seldomly require access to it. SBCL and CCL are our current primary development platforms. I have still not given up on CMUCL completely, but we are waiting for the 19E release before we pick it up again. I am fed up with working around missing Unicode support, which we require in all our current projects. We moved away from common-lisp.net with our repository and web sites. The primary reason for this move was that we now have a Buildbot running for BKNR, and I could not quickly get the neccessary infrastructure to run on common-lisp.net - Buildbot is Python based and requires substantial library support to run. BKNR bugs have been fixed. Sure enough, there have been quite some of them. We hope to be better in the future with our continous integration testing and with more unit tests. Sure enough, I will be moving this Blog to our BKNR based blog system soon. Until that happened, this is my place to practice.