netzhansa

Berlin Lispers Meetup: Tuesday 24 January 8 pm at co.up

2012-01-10T07:55:00.002+01:00

You are kindly invited to the next Berlin Lispers Meetup, an informal gathering for anyone interested in Common Lisp and other languages in the Lisp family.

Berlin Lispers Meetup
Tuesday January 24, 2012
8 pm onwards
co.up (bell: Upstream Agile), Adalbertstraße 7, 10999 Berlin (U-Bahn Kottbusser Tor)

There will be two presentations:

"Further notes on sparql processing" by James Anderson and Arto Bendiken who are affiliated with Datagraph, Inc.
"A suggestion for parentheses representation in a Common Lisp IDE" by Nuno Rocha.

Please join for another evening of parentheses!

Twitter: @BerlinLispers

Learning Ruby, and Ruby vs. Lisp

2011-12-11T12:15:00.001+01:00

The company I work for has a lot of legacy Ruby code, and as Ruby has become kind of a mainstream language, I decided to get a book about it and learn how it works.

As my learning resource, I chose The Ruby Programming language by David Flanagan and Yukihiro Matsumoto as that receives great customer reviews, covers Ruby 1.8.7 and 1.9 and is authoritative because the language creator is one of the authors.

The book makes a good read in general. There are plenty of code examples, but not too much to obscure the prose. What I found first interesting, later annoying, was the frequent use of words like "complex", "complicated", "confusing", "surprising" or "advanced" to describe features of the language. I'd rather decide myself about using such attributes to describe something that I've just learned.

Having spent so much time with Common Lisp, I almost forgot that programming languages usually evolve over the years. Ruby is no exception, and the fact that there are significant differences between Ruby 1.8.7 and Ruby 1.9 kind of bothers me - I'll probably never write code in Ruby 1.8.7, but the differences between the two versions seem to be rather subtle and I'm curious to see how much that is going to be a bother in the future, working with legacy code.

The common theme for Ruby seems to be succinctness. This comes at the expense of making the syntax rather complex, with several special case rules required to solve ambiguities. I don't have the practice to judge whether this is a problem, but from the book, it seems there are quite some things to remember.

It seems that Ruby started off as a purely object oriented language and only later discovered that function-oriented programming is nice, too. The deep roots of object orientation made it rather hard to actually get free functions (which are not member functions of an object) integrated. Contrary to what I am used to, member functions are not a special case of free functions, but rather something quite different. It requires explicit conversion steps to convert a member function into a free function (called procs in Ruby), and invocation syntax is also different between the two. Again, the description may sound worse than it is in practice.

What I really liked was the generalization of code blocks into fibers. Ruby does not have full coroutines, but the restricted form that is available is generalized well and seems like it could be useful for building pretty wild asynchronous systems. Also, it is nice that the bindings of closures can be accessed.

But then, Ruby is an interpreted language and this fact is re-stated throughout the book. With Just In Time compilation, this could become a non-problem, but I'm not sure how well Ruby can be optimized due to its very dynamic nature. Just to see how fast it is compared to Common Lisp, I implemented the Sudoku solver from chapter 1 of the Ruby book in CL and gave the two implementations a puzzle to solve. It took the Ruby solver 0.890 CPU seconds (Ruby 1.9.2p290), whereas the Lisp solver (Clozure CL 1.7) used 0.087 CPU seconds to solve the puzzle. Ten times slower, whatever you'll make of that.

In the book, it is mentioned how little code the Sudoku solver actually uses. This is true, but then, the Lisp version is not longer. It does not seem as if adding syntax is actually the best way to add the possibility to write succinct programs to a language, and the price of the complex grammar is rather high.

Writing the CL solver, I found myself not writing tests again and then poking around in problems of my implementation without knowing what works and what does not. As I want to practice more TDD, I stepped back and added tests. This led me to solve a problem that I had with my previous attempts to practice TDD in Lisp - I do not want to export all the symbols that the tests exercise from the packages that I use, but I also don't want to import the unit testing library into my own library packages. Thus, I wrote a deftestpackage macro that creates a new package to contain the tests that I write and automatically imports all symbols from the package being tested. That way, I can easily keep tests and library source separate and don't need to qualify internal symbols in the tests.

My overall takeaway on the Ruby is this: Ruby seems to be a language that has grown from being purely object oriented to supporting functional programming. That growth was not completely natural, and it seems that if Ruby is not used as a pure object oriented language, the syntax becomes rather messy and hard to grasp. This is similar to C++, which in its first versions was relatively nice (I hear you "ow"!), but has grown into into an incomprehensible mess once people recognized how templates can be abused for metaprogramming.

I can see the appeal of Ruby, but there seems little it has to offer to me that Common Lisp cannot provide. The lack of a formal specification and the ugly grammar put me off. Then again I'm pretty sure that Ruby is more enjoyable than many other popular languages. I'm looking forward to see my theoretical conceptions be shaken by actual practice.

Global Day of Code Retreat with Lisp

2011-12-05T10:28:00.001+01:00

Last Saturday, I attended the Global Day of Code Retreat. I found out about the event in my Twitter stream, and when I signed up for the event, I did not have much of an idea of what it would be about, except code. With some research, I found that it would be a day of using Test Driven Development practices to improve on coding style and quality. Fair enough I thought, got myself a book on TDD (which I have always wanted to do more) and reserved my seat, which was free.

The event was hosted by immobilienscout24 and sponsored by Nokia. Sponsoring included catering, and as the day started at 8:00am, the availability of a breakfast was very welcome. The crowd consisted of some 30 hackers. An informal poll showed that most of them were either using Java or Ruby as their first language, a few were into JavaScript. One guy said that C++ was his primary language, and I was the only Common Lisp hacker.

The format of the all-day event consisted of six sessions of 45 minutes. In every session, people would form new pairs and implement parts of Conway's Game of Life using TDD practices. For each session, the pair chooses a specific challenge or goal to solve. At the end of the session, every pair deletes their code, discusses what they learned and then joins the group to share their thoughts.

The focus of the goals and challenges was on TDD practices. For about a half of the participants, TDD was already part of their daily work routine. The other half had heard of TDD, like myself, and joined to learn about it in the event. For me, this was one of the bigger takeways. I found it very helpful to pair with people that practice TDD in order to learn how they'd go about to write a test before they'd write an implementation. It seems that the TDD school favors a very small test granularity. As was explained by the organizer of the meeting, one would write very small tests that exercise very small aspects of the production code initially, and then build up on that.

In the six sessions, I used Java, JavaScript and Common Lisp. For Java, the organizers had come up with a skeleton project that included a unit testing environment (jUnit?). For JavaScript, I had prepared a simple browser based environment with QUnit, for Common Lisp I used Clozure CL and the simple unit-test library that I have used in the past.

Here are some observations:

The Java dudes travel with heavy baggage

IntelliJ was the most popular IDE among the Java folks that I've talked to. I am split between admiration and disgust when it comes to what seems common practice in the Java world. I'm used to write code by thinking about what I want to write, then typing the stuff that I've thought about. With Java and IntelliJ, the process is more interactive with the IDE, i.e. the programmer types a few characters, the IDE displays some menu to choose from or automatically adds code and so on. This is nice when the IDE automatically recognizes, say, that you are using a name of a class that has not yet been defined or imported and then offers you to either import a package that defines a class of the name that you've typed, or to create a class skeleton (including all the boilerplate and ceremony that Java wants) for you.

While this appears to be convenient, it also does not always work. In both of my Java sessions, the IDE got confused in some way or another, which was not fatal, but still annoyed me. Also, the static nature of Java slowed us down. For one, even though Java does have all the nice data structures (we wanted to use a set of coordinates), we wasted a lot of the short session times converting data types and supplying boilerplate that allowed us to actually put coordinate objects into a set. Also, file based compilation took time - Not minutes, but seconds. I was assured that one would use libraries and code generators in production code and that desktop machines were faster in compiling, but I still can't really relate Java to Agile, in the sense of the word.

Also, the use of code generators in IntelliJ makes me wonder how one maintains such code. How can one actually distinguish between what was carefully crafted and what was pasted, from templates, by the IDE? In my eyes, this is like copy and paste programming with the copy step optimized into templates. I'm not the first to say that and it does not come as a surprise either, but it was an interesting experience for me nevertheless.

Pair programming is fun

Being a remote consultant, I rarely have the chance to interact over code with other programmers. This was something I found enjoyable to practice, even in languages and environments that I'm not familiar with. It is amazing how vastly different the approaches to implementing a relatively simple thing can be, and compared to code reviews - in particular if they're done by email - it is much easier to make suggestions in a constructive way. In that respect, writing code that is supposed to be deleted also helps concentrating on the essence as no sense of code ownership is developed.

I found the pair programming experience significantly impacted by the fact that programmers use different coding environments. Working on an unfamiliar keyboard and with an unfamiliar IDE is a real productivity killer. I'd hope that in a team that pairs regularly, the work environments are more standardized than they have been on this event. I was using Emacs for my Lisp and JavaScript sessions, and my partners had a hard time getting along. What I found rather interesting was that the guys who wrote JavaScript in my Emacs always mimiced what their IDE would do for them. Rather then writing "if (foo) { doSomething()", they'd type "if (foo) {}", then navigate into the braces with the cursor to code along. This is kind of curious because it seems that the balancing of parentheses, brackets and braces is much more of a chore in other languages, to the point where people slow themselves down a lot if not aided by an IDE to the balancing for them. We Lispers, with only the parentheses to keep track of, have a much easier life, in particular with Emacs doing the indentation for us.

Is TDD the kool aid?

In some sense, I am sceptical about TDD. Testing is a great idea, and doing it in a disciplined fashion certainly helps writing better quality software. Also, automated tests are really the best way to prepare software for change. But, and this is where this Saturday could not convince me, I don't believe that spending time on writing very fine grained unit tests for every aspect of of a program helps preparing it better for refactoring and change. I think that testing against external requirements is the real key to writing programs that can be changed facing changing requirements, and that it should be possible to relate every test to some requirement. I must admit that one Saturday of TDD does not give me sufficient experience to judge, though.

Common Lisp and TDD

Some of the TDD discipline probably owes to development cycle in statically compiled languages. Where we Common Lispers have a rather incremental style and do our testing interactively in the repl, developers in languages like Java or C++ write larger chunks of code before they plug it together to do something meaningful. Such environments give the developer less insight into the run-time behavior of a running system, and tests are a way to make sure that more of the interactions of the system components are actually exercised.

I am not claiming that an interactive development environment makes testing less useful. In such an environment, though, it is common to write and test a function in small, iterative cycles. Furthermore, through the use of a tracing facility, the dynamic bahavior of a system can be observed at any time, without the need to touch or recompile the code.

In any case, this was a very nice experience and I'll go to a Code Retreat again, if I can. For that, I'll probably prepare myself for Java tools better in order to get more out of pair programming.

Hunchentoot v1.2.0 released

2011-10-30T21:58:00.000+01:00

After over a year of no releases, Hunchentoot v1.2.0 is out. It incorporates a large number of changes from various people and is the first release that I have made as the maintainer of Hunchentoot. There have been a fair number of changes, many of them contributed by Hunchentoot's users. Thank you all for that!

The release tarball is available here: https://github.com/downloads/edicl/hunchentoot/hunchentoot-1.2.0.tar.gz

This is my first release, and as such it is likely that I'm not yet up to the release standards that Edi has set forth. Please report any problems you may have with the new version to the Hunchentoot mailing list. I will try to resolve problems quickly and possible make a new minor release for each major problem found.

The development repository for Hunchentoot is now hosted on github. If you want to contribute, it would be appreciated if you could submit your patches through github, too.

Here is a list of changes:

Add www/ directory with default file tree that is being served
Add error template mechanism and improve error reporting in general.
Improve automatic testing, SBCL kludge to support asdf:test-op
Allegro CL modern mode fixes
Fix bugs in serving partial responses
Limit maximum number of threads that Hunchentoot creates (Dan Weinreb, Scott McKay)
Export fixes (Gordon Sims, Andrey Moskvitin)
Factor out easy-handler logic into separate acceptor subclass
Export two session functions (Nico de Jager)
Allow no Content-Type header (Chaitanya Gupta)
Patch for compilation with ECL (Sohail Somani)
Fix DEFINE-EASY-HANDLER for multiple acceptors (Nicolas Neuss)
Revived *SHOW-LISP-BACKTRACES-P*
Made sure "100 Continue" is returned even if the client sends "Expect: 100-continue" twice (reported by Gordon Sims)
Fixed typo in code which interprets transfer encodings

Thank you for your patience!
Hans

Berlin Lispers Meetup: Thursday 20 October 8 pm in St Oberholz

2011-10-16T17:01:00.001+02:00

You are kindly invited to the next "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee:

Berlin Lispers Meetup
Thursday October 20, 2011
8 pm onwards
St Oberholz, Rosenthaler Straße 72, 10119 Berlin
U-Bahn Rosenthaler Platz

We will try to occupy a nice table at the floor level, but in case you don't see us please contact Hans: 0177 512 1024.

Please join for another evening of parentheses!

Finding Changes to Quicklisped Software

2011-09-27T16:15:00.003+02:00

In my current project, I'm working with Allegro CL in modern mode. Modern mode is a non-standard mode supporting mixed-case symbols in the reader. This means that in modern mode, the two literal symbols :foo and :FOO are different things. This is different from other lisps, that convert names of literal symbols read to uppercase before interning them. This is why :foo, :Foo and :FOO all mean the same :FOO, by default.

I am using the excellent Quicklisp package management utility to pull all dependencies that a certain package needs. With ACL in modern mode, though, I had to fix several libraries that used all-caps symbol literals (like #:FOO, NIL etc.). The fixes were mostly trivial, but when I had all the stuff running that I wanted, I had modified several of the libraries that I had installed with Quicklisp.

I am planning to import all the external libraries into git repositories and not use Quicklisp in production. I still want to be able to easily update to new upstream versions and also send the changes that I made myself upstream. To make the process of finding what I've changed easier, I envisioned a tool that tells me which of the libraries that I had installed with Quicklisp have local changes, and what these changes were. I asked Zach Beane for help, and he promptly came up with a small function called QDIFF that performs a diff between Quicklisp's base version and the locally extracted version of a library. I hacked the tool slightly to make it work with Allegro CL and iterate over all locally installed systems. It is available as a gist.

Zach supplied a SBCL specific way to run the diff command. I added one based on ASDF:RUN-SHELL-COMMAND, which apparently prefixes all output lines with "; ". I don't like that, but I also did not want to pull in a shell compatibility layer into this self-contained thing.

I praised Quicklisp in the past, but there is nothing wrong with doing it again. Open Source Common Lisp has become so much better with it, and I need to thank Zach again for being so passionate about making the Common Lisp world better! Thanks, Zach!

Ediware moving to github

2011-08-30T11:12:00.004+02:00

I have been more or less the primary maintainer for the Hunchentoot web server written in Common Lisp for several years now. Its primary Author, Edi Weitz, has been doing Lispworks related work and release engineering, but he now intends to step down from that role, too.

In order to open up development and encourage hacking and extending Hunchentoot, we have decided to move the development from the aging BKNR Subversion repository to github. That will make it easier to fork new versions, integrate patches, discuss code changes and track issues.

Together with Hunchentoot, a bunch of other libraries written by Edi Weitz have been moved to the edicl organization on github. The list is not complete and currently consists of libraries that I have used or hacked in the past. Feel free to let us know if you'd like some other of Edi's libraries to be relocated.

Happy hacking!

Orphaned (?) projects on common-lisp.net

2011-05-12T20:59:00.000+02:00

common-lisp.net has moved to a new (virtual) box and in the process, we are cleaning up a little. One particular thing came to my attention: The maintainers of a number of projects can no longer be reached under the email address that they have registered with, so moderation requests bounce (to me). Either these projects have gone stale (even the maintainer lost interest), it is an oversight, or it is just an error that should be fixed. So, please check the list below and let me (hans.huebner@gmail.com) know if you know how to reach the maintainer(s). Projects that we don't get any replies for will eventually have their mailing lists be closed. Thank you for your help!

List of projects:

cffi
cl-magick
cl-match
cl-mpd
cl-soap
cl-stm
ecl
elephant
gecol
morphologie

Berlin Lispers Meetup - Tuesday May 17, 2011: Christian Pape

2011-05-06T11:16:00.001+02:00

Title: Berlin Lispers Meetup
Date: Tuesday May 17, 2011
Time: 7pm onwards
Venue: co.up, Adalbertstr. 7-8, 10999 Berlin
Transport: U-Bahn Kottbusser Tor, Kreuzberg
Admission: Free

You are kindly invited to the next "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee, organized by Hans Hübner and Willem Broekema.

We are glad to have Christian Pape as speaker, talking about:

"A homegrown not yet finished minimal toolkit to build desktop like web interfaces to applications. Focus is on code reusability, simplicity, fallback to non-Ajax, fallback to a more conventional non-desktop like web interface and mainly learning how all this stuff works."

Note the new location: co.up is a coworking space very close to Kottbusser Tor. On the Adalbertstr, enter the space at number 7-8. Then walk across the parking space until you have a row of letter boxes on the right. There is the entrance to the stairs and elevator. In case of problems call Willem: 0151 419 222 72.

Thanks go to Franz Inc., the maker of Allegro Common Lisp and the AllegroGraph Web 3.0 database, for sponsoring the Berlin Lispers meetups.

Please join for another evening of parentheses!

Berlin Lispers Meetup - Tuesday April 19, 2011

2011-04-17T22:14:00.002+02:00

Title: Berlin Lispers Meetup
Date: Tuesday April April 19, 2011
Time: 7pm onwards
Venue: co.up, Adalbertstr. 7-8, 10999 Berlin
Transport: U-Bahn Kottbusser Tor, Kreuzberg
Admission: Free

You are kindly invited to the next "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee, organized by Hans Hübner and Willem Broekema.

There will be no presentations, so this is a good opportunity to catch up with everyone.

Thanks go to Franz Inc., the maker of Allegro Common Lisp and the AllegroGraph Web 3.0 database, for sponsoring the Berlin Lispers meetups.

Please join for another evening of parentheses!

Berlin Lisp Meetup - Today

2011-03-15T12:20:00.003+01:00

Sorry for the short notice. Please note the new location.

Title: Berlin Lispers Meetup
Date: Tuesday March 15, 2011
Time: 7pm onwards
Venue: co.up, Adalbertstr. 7-8, 10999 Berlin
Transport: U-Bahn Kottbusser Tor, Kreuzberg
Admission: Free

You are kindly invited to the next "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee, organized by Hans Hübner and Willem Broekema.

We are happy to announce the presentation by Willem Broekema: "The Lisp reader and printer: from reader macros to pretty printing"

Note the new location: co.up is a coworking space very close to Kottbusser Tor. From the road, walk on until you have a row of briefcases on the right, that's the entrance to the stairs and elevator. In case of problems call Willem: 0151 419 222 72. Thanks go to Franz Inc., the maker of Allegro Common Lisp and the AllegroGraph Web 3.0 database, for sponsoring the Berlin Lispers meetups. Please join for another evening of parentheses!

Berlin Lispers Meetup - Tuesday December 7, 2010

2010-12-01T22:09:00.002+01:00

Title: Berlin Lispers Meetup
Date: Tuesday December 7, 2010
Time: 7pm onwards
Venue: New Thinking Store, Tucholskystr. 48, 10117 Berlin
Transport: U-Bahn Oranienburger Tor, S-Bahn Oranienburger Straße
Admission: Free

You are kindly invited to the next "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee, organized by Hans Hübner and Willem Broekema.

We are happy to announce a presentation by Kilian Sprotte: "Lazy Evaluation for Computer-Aided Composition -- from PWGL to PWCLIM"

Thanks go to Franz Inc., the maker of Allegro Common Lisp and the AllegroGraph Web 3.0 database, for sponsoring the meeting by paying for the venue, so that admission can be free.

Please join for another evening of parentheses!

Berlin Lispers Meetup - Tuesday November 2, 2010

2010-10-31T18:21:00.001+01:00

Title: Berlin Lispers Meetup
Date: Tuesday November 2, 2010
Time: 7pm onwards
Venue: New Thinking Store, Tucholskystr. 48, 10117 Berlin
Transport: U-Bahn Oranienburger Tor, S-Bahn Oranienburger Straße
Admission: Free

You are kindly invited to the next "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee, organized by Hans Hübner and Willem Broekema.

We are happy to announce two presenters:

- James Anderson will talk about "Control"
- Christian Pape, who describes himself as "non-programmer", will talk about his successful use of Common Lisp at work

At the last meetup we had an organizational hiccup which made us end up in a cozy Italian restaurant; this time we'll surely be in the New Thinking Store again.

Thanks go to Franz Inc., the maker of Allegro Common Lisp and the AllegroGraph Web 3.0 database, for sponsoring the meeting by paying for the venue, so that admission can be free.

Please join for another evening of parentheses!

Raumpflege

2010-10-19T14:20:00.002+02:00

Sollte jemand sich in Berlin mit dem Gedanken tragen, das Unternehmen "Bärliner Putzteufel" beauftragen zu wollen, so seid gewarnt: Die Leistungsbereitschaft der Mitarbeiter ist gering, aber im Stellen hoher Rechnungen ist der Inhaber Volker Brunemann sehr beflissen. Von Freundlichkeit und Entgegenkommen bei Kundenunzufriedenheit kann man nur träumen. Ich kann zwar kein anderes Reinigungsunternehmen empfehlen, aber die "Bärliner Putzteufel" sind es sicher nicht. Ja, klar. Schon der Name hätte mich skeptisch machen sollen.

Postbank saugt

2010-10-11T16:40:00.001+02:00

Ich bin Business-Kunde bei der Postbank, und auf meiner letzten Monatsabrechnung fand ich einen Posten "Zinsen und Gebühren" in Höhe von 66 Euro. Ich dachte mir: "Das ist ja viel", und schrieb eine (autorisierte und so weiter) Email an die Postbank mit der Bitte, mir diesen Posten doch mal aufzuschlüsseln. Als ich nach 10 Tagen noch keine Antwort hatte, habe ich mal nachgefragt. Einen Tag später erhielt ich die Antwort, daß meine Frage nicht so ohne weiteres beantwortet werden könne, und sie daher an die "Fachabteilung" zur Bearbeitung weiter geleitet wurde. Ich glaube, ich brauche ganz schnell eine andere Bank.

Berlin Lispers Meetup - Tuesday October 5, 2010 - James Anderson & Marijn Haverbeke

2010-09-30T22:30:00.002+02:00

Title: Berlin Lispers Meetup
Date: Tuesday October 5, 2010
Time: 7pm onwards
Venue: New Thinking Store, Tucholskystr. 48, 10117 Berlin
Transport: U-Bahn Oranienburger Tor, S-Bahn Oranienburger Straße
Admission: Free

You are kindly invited to the next "Berlin Lispers Meetup", an
informal gathering for anyone interested in Lisp, beer or coffee,
organized by Hans Hübner and Willem Broekema.

We are happy to announce two presenters:

- James Anderson: "Control"
- Marijn Haverbeke: "CL-JavaScript, a JS-to-CL compiler"

The attentive reader might note that these are the same speakers as at
the September meeting. Unfortunately there were not many people
attending that meeting, perhaps due to a lack of promotion form our
side. We are grateful the presenters are willing to talk one more time.

Update: This time, Marijn Haverbeke will talk about CL-JavaScript.  

Thanks go to Franz Inc., the maker of Allegro Common Lisp and the
AllegroGraph Web 3.0 database, for sponsoring the meeting by paying
for the venue, so that admission can be free.

Please join for another evening of parentheses!

Quicklisp - The upcoming solution to Common Lisp's "library problem"

2010-09-02T22:01:00.004+02:00

We're all tired of hearing that Common Lisp is all nice, but it has a library problem. True, many of the open source libraries written in Common Lisp are half-baked, incomplete, unmaintained and tasteless. But this is true with libraries in all languages, so that can't be the real problem. The real problem, as I see it, is that until now, there was no universally good way of installing a library and all of its dependencies.

The existing, widely deployed solutions (asdf-install and clbuild) require external tools to work, depend on the availability of diverse servers in the internet, and do not have something like a central maintainer who ensures that things are always in basically working order. Also, these solutions are rather unportable, as they depend on external tools and a Unixish environment which is not available everywhere.

My personal solution to this problem, until now, was to have a Subversion repository that contains all the libraries that any of my projects need. Some of these came from release tarballs, some from other revision control repositories, and maintenance was manual. Whenever I started a new project, I brought some of those libraries up to date, coped with any (new) dependencies, lost backwards compatibility with my older projects etc. This kind of sucked, but it had the beauty that both development and deployment ended up being relatively easy. All that was required was a svn checkout from my repository, and I was all set on a new machine.

Recently, Zach Beane was fed up enough to do something about this: He created Quicklisp, a self-contained, centrally managed, cloud hosted Lisp library system which aims to run everywhere and provide users with a one-stop solution for the "library problem". Quicklisp is in an early alpha stage, but having tried it, I must say that I am pleased and impressed. Finally, Common Lisp can also become a glue language like Python and Perl.

I tried Quicklisp today because Twitter has notified me that basic http authentication for applications will no longer be supported. Instead, one is now supposed to use OAuth to authenticate requests sent to Twitter. This required my reaction.

About two years ago, I have written a small gateway program that forwards new postings to Planet Lisp to Twitter. At the time, I was rather frustrated with Common Lisp and thus took the gateway as an opportunity to try Clojure. Today, I looked at the source of the gateway again to find out what it would take to make it use OAuth instead of basic http authentication. As the gateway was the only program I have written in Clojure so far, and I was not really that eager to extend it to OAuth. I tried for a few minutes, but found that my old program did not run with the current Clojure version right away, so I would have to basically start setting up a Clojure development environment from scratch in order to be able to use an existing, open source OAuth library.

Instead, I thought I'd give Quicklisp a spin. After all, parsing some XML and sending HTTP requests are no big deal in Common Lisp either, and an OAuth library is available, too. With Quicklisp, all it should take was write some glue code to connect DRAKMA, CXML, cl-ppcre and cl-oauth. Installing the bunch should be a matter of loading quicklisp.lisp, then typing (ql:quickload `("drakma" "cxml" "cl-ppcre" "cl-oauth")) and watching the show scroll by.

And hey, it worked right away. I even embedded a web server in the gateway application so that it can be monitored by the system status monitors provided by my hosting provider. All in all, rewriting the thing in Common Lisp and deploying it took no longer than two hours, and the source is not significantly longer than the Clojure version either. Furthermore, the new gateway properly deals with non-ASCII characters. Embarrassingly, the Clojure gateway was buggy in that respect and never properly twittered the titles of my own blog posts. Thanks, Zach!

By all means, choose a distinct name for your program

2010-06-28T14:59:00.003+02:00

"I love screen". "screen?", you ask, "do you love your screen or screen in general, like in sun screen?". And right you are, screen is not a name, it is not even a very distinctive noun as it is so ambiguous. So, when I say "I love screen", I'm talking about the program that Oliver Laumann wrote like 23 years ago and that is probably the open source program that I have used for the longest time (basically, since it was first released).

Now, as much as I love screen, I hate its name. I guess that Oliver and his room mate pondered how to name the program for a while, maybe even for a new hours or days, and then went "Damn fuck it, we're just unimaginative hacker dudes, round corners are not yet invented, creative design folks avoid us like the plague (and we like it that way), let's call the darn thing screen and go on hacking."

DON'T DO THAT, SERIOUSLY

Whatever name you choose for your program, make sure that it is not a word. If you absolutely want to use real words, choose at least two of them that never occur together anywhere else. Better yet, invent a word. Use Google to validate your name. Use something stupid. Get on with it.

I'm writing this because the poor choice of naming for screen bites me every day. screen has a serious bug. Sometimes, one of the screens (sic!) hangs. It then never recovers, and once one of the screens (yaaah!) hangs, it does not take long until the whole session ceases to work. In the end, even trying to reattach the session hangs. I think I do have basic Google skills, but so far I have failed to find anyone having the same problem. I can't imagine that I would be the only person to have this problem, but the name of the program makes it impossible to find that person.

You are making the life of your users much much easier if you give your programs names that are distinctive and unique. Yes, Ryan Gahl and Aycan Gulez, I'm looking at you today while I write this. :)

How to convert your old-style Symbolics keyboard to USB

2010-06-17T10:33:00.009+02:00

Peter got in touch with me because he tried to follow my instructions how to convert a Symbolics keyboard to USB with his old-style keyboard and he failed. He assumed, and that was what I would have assumed, that the color coding of the internal cabling of the old and new keyboards would match, but they don't. Here is a table that shows the pinout for both keyboard types:

New Style		Old Style		Function	Teensy Pin
Pin#	Color	Pin#	Color	Function	Teensy Pin
1	blue	8	white	GND	GND
2	yellow		black	not	connected
3	green	2	red	5V	5V
4	red	3	green	DIN	D4
5	black	1	yellow	CLK	D5
6	white	5	blue	CLR	D6

Embracing the ARM

2010-05-29T10:18:00.003+02:00

Last year, I worked on a project for visomat inc. which involved interfacing to a LED matrix display so that it could display some video. We had some latency problems and at one stage we decided to try out DMX instead of Ethernet as interface techniquce, hoping that we'd get better results that way. The idea was to feed the LED matrix, which consisted of 12 power supplies/controllers, with 12 DMX interfaces to make sure that data arrives at the LEDs synchronously.

DMX is an asynchronous serial protocol running at 250 kbits per second. I decided to try using an AVR microcontroller running at 16 Mhz and have it bit-bang the data out of a parallel port. For each bit, 64 machine cycles would be available which should be enough to set the port bits and also grab incoming data out of the USB interface that we wanted to use to interface to the host.

I built a prototype of the interface and we made some promising experiments with it, but in the end we found that the visual latency problems were not fixed. The problem was really in how the LED hardware worked and could not be repaired on the feeding side. The prototype went onto the shelf and we stuck with Ethernet for that project.

A few weeks ago, I discussed the project with a friend of mine and we thought that it'd be cool to turn the 16 port DMX interface into a product. We found someone who would want to use it in an OEM setting, and I made bringing the hardware to product level my weekend project.

Good bye AVR, hello ARM

The AVR based solution proved to be problematic, though. Even though the CPU has enough processing headroom, the lack of RAM requires that the data is fed into the interface "just in time". Any variance in the USB input data stream could result in visible hickups in the DMX streams. Getting it to work seemed like possible, but require too much work in the end. Thus, I decided to turn to using more capable hardware which would also give the DMX interface network connectivity and more local intelligence.

As new platform, I chose the ARM9 based Eddy-S4M board produced by the Korean manufacturer SystemBase. These boards are distributed by trenz electronic who I've found to be quick and reliable in the past.

Linux is the standard operating system on Eddy-S4M. I'm not a huge Linux fan, so I spent some time to find an alternative operating system, but neither eCos, nor FreeRTOS or RTEMS seemed to be ported to the AT91SAM9260 CPU with a free development system. As I was into application development, I decided not to port or write an operating system myself. Running on the bare hardware was not an option either as I need a working TCP stack in order to interface to the host.

SystemBase calls their Linux port "lemonix". It appears to be a relatively plain linux-2.6.20 with real-time patches, BusyBox and a bunch of C applications to control startup and configuration. The userland and kernel source code is available from SystemBase after registration, but the kernel tree that they make available is incomplete and needs to be augmented by some files that are available from a Google group so that it can be built. Getting the cross compilation environment to run on my Ubuntu 9.10 box worked flawlessly, and after removing the whole Eddy application stuff and most of the SystemBase drivers from the file tree I ended up with a relatively sane Linux that I could cross compile and install on the Eddy hardware through tftp.

Making it fast

Back to the DMX interface, I was now faced with a system that gave me about 400 machine cycles for each DMX bit sent. That should be plenty of time to get out some bits to the ports. The bit timing would be achived by a timer interrupt every 4 microseconds. Even with interrupt handling overhead, this should leave enough headroom for TCP processing.

The ARM architecture specifies two interrupts. The normal, vectored interrupt system is used by Linux and mapped to its own, portable interrupt architecture, ignoring the vectoring facilities that the hardware provides. The second, fast interrupt system (FIQ) that the ARM provides is not used by Linux, and it seemed like a good fit to my requirements. The FIQ is interesting because when the FIQ handler is entered, the CPU automatically switches six registers to a distinct bank. These six registers can then just be used without the need to save or restore them to the stack. Even though the FIQ is not used by Linux itself, ARM-Linux provides for an interface so that drivers can use it. FIQ support was missing in lemonix, but it was trivial to backport.

Writing a FIQ interrupt in ARM assembly was straightforward. The driver bottom half set up the FIQ register set and enabled the timer interrupt, the FIQ handler set the port bits and incremented the pointers accordingly.

Reducing jitter

When looking at the output ports with the logic analyzer, though, I could see variations in bit edges in the order of one microsecond, which is beyond what the DMX receivers would be able to tolerate. These variations were visible even when the FIQ handler just set the port bits without writing any real data. The source for this jitter, as it turned out, was virtual address translation. The ARM9 CPU includes a MMU and all software, including interrupt handlers, uses virtual addressing, even to access I/O ports. The virtual address maps are stored in SDRAM due to their size, and the hardware automatically traverses these maps if an address can not be found in the translation lookaside buffers in the MMU. So in the FIQ handler, when the output port address was not present in the TLB, the MMU would access the SDRAM, and SDRAM random access is rather slow.

Thus, I had to change the FIQ handler so that it never accesses the SDRAM, either directly (by reading the data buffer) or indirectly (by causing TLB misses). The AT91SAM9260 CPU has two internal SRAMs that are accessible in two machine cycles, and I use one of those as a data buffer. In order to prevent TLB misses, indivdual TLB entries can be locked down so that they're never removed by the MMU automatically. Thus, my driver locks down the I/O and the SRAM buffer addresses that the FIQ handler accesses.

Reducing jitter even more

Even with the TLBs locked down, I still saw some jitter in the leading edges of the DMX bytes. The cause for this were variations in interrupt response times, as that depends on the instruction being interrupted. The variations were below one microsecond, but that was long enough to bring the serial decoding routine in my logic analyzer out of sync. Wanting to play it safe, I decided to remove that jitter as well by synchronizing on the timer value inside of the FIQ handler: Instead of just banging out the next bit as fast as possible, the FIQ handler now waits until the free-running timer that triggered the interrupt reaches a certain value. That way, the effective FIQ response time is made mostly constant.

Wow, 2010!

In the last few years, I have been using various AVR CPUs for my embedded projects. It is a great CPU that is easy to use and fast, and the advent of LUFA and Teensy made my life a lot easier, as I was freed from USB serial dongles and driver installation. But then, being able to process serious amounts of data is nice, too, and this is were 8 bits just don't suffice. ARM based boards are cheap nowadays, and I'm looking forward to embedding JavaScript or maybe even Common Lisp in one of my future projects. When speed is needed, I can always fall back to C and assembler.

Berlin Lispers Meetup - Location change

2010-05-27T12:38:00.001+02:00

Unfortunately, the New Thinking Store is not available this month, so we'll have the Lisp Machine workshop in Hans' office at 18:00 hours. Seating is limited, so if you're rather up for beers and chats, we'll meet at "I Due Forni" at about 21:00 hours.

Here is the location info:

Hans' Office
Schönhauser Allee 177
10119 Berlin
Tel. 030 4431 9642
"Sauna" signpost, across the alley, first floor, blue door, knock.

I Due Forni
Schönhauser Allee 12
10119 Berlin

Despite the large difference in house numbers, the two locations are just across the street. Here is a map.

Berlin Lispers Meetup - Tuesday June 1, 2010 - Lisp machines!

2010-05-26T20:56:00.003+02:00

Title: Berlin Lispers Meetup
Date: Tuesday June 1, 2010
Time: 7pm onwards
Venue: New Thinking Store, Tucholskystr. 48, 10117 Berlin
Transport: U-Bahn Oranienburger Tor, S-Bahn Oranienburger Straße

You are kindly invited to the next "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee, organized by Hans Hübner and Willem Broekema.

Continuing our series of meetings with presentations, the upcoming Berlin Lisp Meetup will be in New Thinking Store once more. We are happy to announce the theme:

Lisp machine session - When Lisp was really cool

This month's Lisp meetup is about Lisp machines - Computers, that were made to run Lisp exclusively, with their makers so proud that the handbook started with the sentence "Welcome to Genera - The best software development environment available".

This will be a hands-on session. We'll have Symbolics MacIvory II to show, and we're going to play a little with the Linux based virtual Lisp machine. If you have a 64 bit Linux laptop, you can give it a spin, too.

We will start at 19:00 hours, admission is free.

Thanks go to Franz Inc., the maker of Allegro Common Lisp and the AllegroGraph Web 3.0 database, for sponsoring the meeting by paying for the venue.

Please join for another evening of parentheses!

P.S.: If you are interested in giving a Lisp-related presentation on one of our next meetings, please get in touch!

Hacking a USB Infrared receiver

2010-04-18T22:54:00.008+02:00

Our living room Hi-Fi setup needs a MP3 player that is always available, easy to use and, first and foremost, allows us to import all the CDs that we have easily and without effort. I've always wanted to do that using our FreeBSD based home server, but none of the ripping solutions really worked the way I wanted and triggered my desire to hack them to my liking. Finally, I gave in and pulled out the aging and unused PowerBook of my wife. After all, iTunes matches my requirements easily, and as I can treat is as piece of Stereo equipment, I'm willing to put my dislike for Apple Computers to the side.

The problem with the PowerBook is that it does not have an IR receiver. We have a Logitech Harmony Remote 515 that we use to control the rest of our entertainment stuff, and we'd like to control the MP3 player with it, too. I could not find a cheap, ready-to-go solution easily, so I decided to throw together something myself using a Teensy and a Vishay TSOP1836 IR receiver module that I found on a surplus board that I found in a box on my shelf.

The goal was to create an IR receiver that acts as a USB keyboard. The key codes that are sent should be useable to set up some global keyboard shortcuts in MacOS X that would control iTunes to our liking.

Hardware

The TSOP1836 IR receiver module does the IR carrier decoding and interfaces easily with the AVR using one input port. I hooked up my Logicport logic analyzer (excellent product, I write this as a satisfied customer) to the output of the IR receiver and tried a few remotes that I had to see whether the 36 kHz that that the receiver assumes as the carrier frequency would be common. I was lucky, both the remote of our Onkyo Amp and the remote control that came with the Altium Nanoboard 3000 transmitted at that carrier frequency, and both of them used the NEC protocol.

Teensy is a microcontroller board based on an ATMEL AT90USB162 AVR chip. It comes in a DIL24 form factor, so it is easy to integrate into breadboards. A neat thing about it is the flashing utility that looks good, works very well and is unintrusive. The compile/upload/test cycle with the Teensy loader is very short.

Software

I have used LUFA as USB stack on Teensy in the past to implement a adapter for Symbolics keyboards, and I decided to use it again. I was pleased to find that LUFA has made great progress since I last used it. There are more examples, the class drivers have been separated from the demo code, and the cheapish task scheduler is not longer used by the examples that I used.

I tried to make the USB implementation of the adapter offer two end points: A keyboard endpoint that sends the translated key codes when remote keys are pressed, and a serial or CDC endpoint that would be used to configure the adapter in order to make it possible to configure it to whatever remote is available. Unfortunately, I found that MacOS X does not accept devices with multiple endpoints as standard serial/CDC device, so I abandoned the idea of configurability. Instead, I made the adapter work with the Altium remote and a statically compiled translation table.

To find out the codes that the remote sends, I wrote a version of the adapter that acts as a serial device. Using that information, I set up the translation table in the HID keyboard version of the firmware.

When I tried setting up some global keyboard shortcuts for iTunes, I learned that iTunes does not really support global short cut keys, presumably to "protect the user". Using a the free SizzlingKeys utility, I was able to achieve what I wanted, though.

The source code for the two Teensy firmwares is available in my github repository.

Berlin Lispers Meetup - Tuesday April 6, 2010 - Samir Sekkat

2010-03-29T21:54:00.004+02:00

Title: Berlin Lispers Meetup
Date: Tuesday April 6, 2010
Time: 7pm onwards
Venue: New Thinking Store, Tucholskystr. 48, 10117 Berlin
Map: Google Map
Transport: U-Bahn Oranienburger Tor, S-Bahn Oranienburger Straße

You are kindly invited to the sixth "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee, organized by Willem Broekema and Hans Hübner.

Continuing our series of meetings with presentations, the upcoming Berlin Lisp Meetup will be in New Thinking Store once more. We are happy to announce a talk held (in German language) by Samir Sekkat:

Your Life In Plain Text / Stress-free productivity
The elisp-based org-mode package

How to organize your daily life using Org-Mode (which is an emacs package) as a tool and Getting Things Done as methodology.

Quote: I'm having the same feeling for org-mode that I did when I first learned to really program and use emacs. [Jeffery Travis on his Twitter feed.]

Links:
http://orgmode.org/
http://orgmode.org/worg/org-quotes.php
http://members.optusnet.com.au/~charles57/GTD/gtd_workflow.html

We will start at 19:00 hours, admission is free.

Thanks go to Franz Inc., the maker of Allegro Common Lisp and the Allegrograph Web 3.0 database for sponsoring the meeting by paying for the venue.

Please join for another evening of parentheses!

P.S.: If you are interested in given a Lisp related presentation on one of our next meetings, please get in touch!

Clojure Slides from February Berlin Lispers Meetup

2010-02-22T11:12:00.006+01:00

Stefan Richter, our speaker at the February Berlin Lispers Meetup, has made the slides of his presentation available on line. Enjoy, and see you next week!

Berlin Lispers Meetup - Tuesday March 2, 2010 - Luke Gorrie & Edi Weitz

2010-02-11T15:09:00.004+01:00

Title: Berlin Lispers Meetup
Date: Tuesday March 2, 2010
Time: 7pm onwards
Venue: New Thinking Store, Tucholskystr. 48, 10117 Berlin
Map: Google Map
Transport: U-Bahn Oranienburger Tor, S-Bahn Oranienburger Straße

You are kindly invited to the fifth "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee, organized by Willem Broekema and Hans Hübner.

As our last meeting at the New Thinking Store was a great success (so we think), we will have the March meetup there, too. We have two presentations scheduled:

Luke Gorrie will give a talk titled "a new adventure using Lisp to optimize TCP performance over 3G/HSPA networks for fun and profit".

Edi Weitz will give his talk "Lisp in the Real World[tm] - Using a sophisticated language for mundane tasks".

We will start at 19:00 hours, admission is free.

Thanks go to Franz Inc., the maker of Allegro Common Lisp and the Allegrograph Web 3.0 database for sponsoring the meeting by paying for the venue.

Please join for another evening of parentheses!

P.S.: If you are interested in given a Lisp related presentation on one of our next meetings, please get in touch!

Telekom VDSL mit FreeBSD

2010-02-02T12:54:00.005+01:00

Da ich meinen DSL-Anschluß im Büro mit ein paar anderen Freelancern teile, habe ich kürzlich von vorher Congster ADSL zu Telekom VDSL-25 gewechselt. Als Router verwende ich nun einen Rechner mit FreeBSD-7.2, der VDSL-Anschluß wird mit einem Modem Speedport 300HS realisiert. Da der VDSL-Uplink auf einem gesonderten VLAN aus dem Modem kommt, ist eine zweite Netzwerkkarte nicht zwingend erforderlich. Ich habe das Modem einfach ans LAN gehängt und auf dem FreeBSD-Router ein VLAN-Interface konfiguriert, das für den Internetzugang mit PPPoE verwendet wird.

Konfiguriert ist das ganze wie folgt

Änderungen in /etc/rc.conf

# VLAN-Interface konfigurieren, bge0 ist mein Ethernet-Interface. Ggf. anpassen. cloned_interfaces="vlan0" ifconfig_vlan0="vlan 7 vlandev bge0" # IP-Paketforwarding anschalten gateway_enable="YES" # PPP-Client automatisch starten ppp_enable="YES" ppp_mode="ddial" ppp_nat="YES" ppp_profile="telekom"

Eintrag in /etc/ppp/ppp.conf

telekom: set device PPPoE:vlan0 set authname AAAAAAAAAAAATTTTTTTTTTTT0001@t-online.de set authkey KKKKKKKK set dial set login set log phase tun command set ifaddr 10.0.0.1/0 10.0.0.2/0 add default HISADDR

Wie üblich ist AAAAAAAAAAAA durch die Telekom-Anschlusskennung, TTTTTTTTTTTT durch die T-Online-Kennung und KKKKKKKK durch das Kennwort zu ersetzen. (Könnte die Telekom bitte irgendwann mal die BTX-Vergangenheit tief unter der Erde verbuddeln und die Autorisierung auf ein benutzerfreundliches Verfahren umstellen? Danke!)

Das war's. Reboot und die FreeBSD-Box routet.

Berlin Lispers Meetup: February 2, 2010 - Different time and location - Clojure presentation

2010-01-29T11:53:00.005+01:00

Title: Berlin Lispers Meetup
Date: Tuesday February 2, 2010
Time: 7pm onwards
Venue: New Thinking Store, Tucholskystr. 48, 10117 Berlin
Map: Google Map
Transport: U-Bahn Oranienburger Tor, S-Bahn Oranienburger Straße

You are kindly invited to the third "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee, organized by Willem Broekema and Hans Hübner.

This time, the meeting will take place in the New Thinking Store (see address above), as we have a presentation by Stefan Richter of freiheit.com who will talk about a Cloud-deployed web application written in Clojure. The full title of his presentation is

How a Clojure pet project turned into a full-blown cloud-computing web-app.
Or:
What are the differences between a Clojure web-app and one written in Java or Common Lisp?

Stefan will give his talk in German or English, depending on the preference of the audience. It will start at 19:00h.

Thanks go to Franz Inc., the maker of Allegro Common Lisp and the Allegrograph Web 3.0 database for sponsoring the meeting by paying for the venue.

Please join for another evening of parentheses (and brackets this time)!

P.S.: If you are interested in given a Lisp related presentation on one of our next meetings, please get in touch!

Max for Live, Java, MIDI, Windows and Sysex

2010-01-23T21:04:00.006+01:00

Recently, I have spent some of my free time hacking some Java to provide Max for Live patchers direct access to the MIDI ports of my machine. Usually, one would be restricted to the MIDI data that Live has previously processed, which restricts it in many ways.

I have not spent too much time hacking Java in my life, but the times I have done it for various smaller hacks, it was a nice overall experience. This time, I was particularily impressed by Eclipse for its automated source code formatting facility that is widely configurable. I tweaked the settings a little to match my prefered squirly brace style, and then had Eclipse beautify my (and imported) source code with a single key stroke. Also, I like Eclipse's ability to automatically locate unknown classes and add required import statements with a single click. This makes working with examples from documentation rather easy. I still like Emacs, but I must admit that it is rather baroque when compared to what Eclipse can do.

Getting the Max external to interface to the MIDI subsystem that Java provides was rather easy, and I was mostly done after 2-3 hours of hacking. Sadly, sending Sysex strings to the MIDI controller that I used for testing did not work reliably. Naturally, I thought that the problem was with my lack of MIDI, Java and Max skills, so I spent quite some time trying out various implementation strategies, without success.

Finally, I figured that what did not work was sending Sysex messages, and only sometimes (i.e. it appeared as if the first message came through and then some following messages where not transmitted). Google finally pointed me to a web page by Thorsten Klose where, at the end, he describes the behavior as being a bug in the Java MIDI library. The problem is that when sending Sysex messages, the Java library remembers the size of largest Sysex message sent and uses that for all subsequent Sysex messages, even if they are shorter. My workaround is to close and re-open the MIDI device if a Sysex string to be sent is shorter than the previous one. This is kind of ugly, but it works for me. The workaround that Thorsten suggest has the problem that it does not stop Java from sending long messages, which reduces MIDI throughput to a point that was unacceptable for me.

I'm posting this in the hopes that it saves the next person seeing the problem some time. If you are interested in the Max for Live MIDI external, it is available here.

Berlin Lispers Meetup - Tuesday January 5, 2010

2010-01-03T13:55:00.001+01:00

Title: Berlin Lispers Meetup
Date: Tuesday January 5, 2010
Time: 8pm onwards
Venue: St. Oberholz, Rosenthaler Straße 72a, 10119 Berlin
Transport: U-Bahn Rosenthaler Platz

You are kindly invited to the third "Berlin Lispers Meetup", an
informal gathering for anyone interested in Lisp, beer or coffee,
organized by Willem Broekema and Hans Hübner.

Please join for another evening of parentheses!

Admittedly, I like JavaScript

2009-09-07T23:04:00.001+02:00

Call me pervert, but I find writing JavaScript enjoyable. The aspect that I like most about it is its dynamicism. Certainly, the syntax is ugly, but the js2-mode for Emacs makes writing JavaScript much less painful.

My addiction for JavaScript made me accept a project to create an artists' portfolio website that has no commercial value, but gave me lots of opportunity to play around with client-side coding. I decided that I want to put all data into a JSON encoded data structure and use a client-side content management system to edit and update the database. I originally intended to write the required server-side components for the CMS in JavaScript, too, but that idea went down the drain as it made deployment of the whole thing a lot harder. Thus, the server-side components are now reduced to the bare minimum and are written in perl, so that a cheap web host can be used both for deploying both the web site and the CMS.

One common problem with client-side JavaScript is flow control with long-running operations. JavaScript is single threaded, so in order to maintain UI responsiveness, functions may not block waiting for network responses. Thus, whenever a UI wants to interact with the server and then resume operating, things get tricky. There are libraries like jwacs, which is a preprocessor which implements first-class continuations, and Arrows, which is a "generalization of Monads", but these either make debugging hard through the use of a preprocessor or are incomplete.

My approach to the problem is to use closures to manually establish the continuation, and callbacks to invoke these continuations when appropriate. While being dissatisfying in the intellectual sense, the approach works for the applications that I'm using them in.

Sure, I'd rather have macros instead of having to control evaluation through closures. But then, this is still better than any old static language that requires me to cram everything into classes and objects, forcing me to either make all design changes upfront or to use sophisticated refactoring wizardry to get dynamicism on the source code level. Oh well, or maybe I should rather give up on Emacs :)

Javascript Everywhere - Using Axiom Stack

2009-04-27T08:24:00.001+02:00

A friend of mine asked me to help her with creating a portfolio website for a friend of hers, who is an artist. The site will be somewhat similar to QuickHoney's web site: It will be Javascript based, relatively simple in terms of layout, and self maintainable, i.e. the artist will be able to add and update the website without the need for HTML programming.

The site will be hosted on a cheap, no-frills shared web host. On the server, it will consist of static files. The portfolio data set - Projects, descriptions, dates - will be stored in JSON formatted files. Manipulation of the data set will be done with a special Javascript application for content management.

The site will contain various media data: Photos, MP3 audio tracks, videos hosted by a video hoster.

Some parts of the content management tasks cannot be performed by Javascript in the browser:

Image manipulation, in particular rescaling.
Provide information about MP3 audio tracks
Fetching video information from the video hoster
Uploading data to the web hoster

These tasks require some program that runs outside of the browser; obviously, they are performed by a http server running on the artist's box.

My first impulse was to implement these functions in Common Lisp. Image manipulation is easy, using Edi Weitz' excellent CL-GD library. MP3 information should be easy enough to read using Peter Seibel's code from Practical Common Lisp. Parsing information from the video hoster would be easy, as that information is available both in XML and JSON formats, and Gilbert Baumann's and David Lichteblau's Closure XML Parser which I enjoy to use a lot. Lastly, uploading data to the web host should be possible using CL-FTP which I had not used before.

Exit Common Lisp

I spent a few evenings hacking CL-FTP so that it fits my library set and trying to find a sufficiently complete MP3 parsing library, played with dumping images so that startup times would be short. In between, I played with jQuery, HTML and CSS to get the user visible parts of the site into shape. Sadly, I made much better progress and had a lot more fun hacking Javascript than fooling around with Common Lisp and incomplete libraries. At one point, I decided that I need to make more progress and that I should reconsider my strategy for the CMS server.

I decided that my second choice for the CMS server language would be Javascript. I'd rather write both client and server in Common Lisp, but that is not how the world looks like.

Enter Javascript

I spent a few hours shopping for server-side Javascript solutions. The open source Javascript based web servers are either based on the SpiderMonkey or the Rhino Javascript implementations, both by the Mozilla foundation. SpiderMonkey is written in C, thus extending it requires libraries using the C calling convention. Rhino is written in Java, and it gives Javascript applications direct access to libraries written in Java. As I am developing on Windows, but the artist is using a Mac, Rhino looked a lot more attractive.

After looking at some project web pages picked from the Wikipedia entry on Server-side Javascript, I decided to give Axiom Stack a spin. I chose it because it is a real Javascript framework (using Prototypes, not mimicking classes), has easy-to-follow getting started documentation, provides for persistent objects similar to the BKNR datastore and has active support.

My application is not quite typical for applications implemented with Axiom Stack. Usually, the server would be used by the site's users, too. My server manipulates the static files that make up the web site, and is basically a slave to the CMS application that runs in the browser. Thus, I am not using Axiom Stack's persistent object store in this application.

Enslave Java

I have never developed a very friendly relationship to Java, despite the fact that when I chose to use it for one reason or another, I had no trouble with it. Java is great, as long as someone else writes code in it. Being able to call Java from Javascript is really nice, though, as there are so many libraries out there. In Rhino, Java libraries can be just used. Thus, as Axiom Stack has no "native" MP3 reading facilities, I grabbed a Java MP3 library off the net and used that:

importPackage(Packages.de.vdheide.mp3);

function sound_list () {
    var dir = '../../../files/sound/';

    return MochiKit.Base.map(
        function (filename) {
            var pathname = dir + filename;
            var mp3 = new MP3File(pathname);
            return { filename: filename,
                     title: id3.getTitle(),
                     length: mp3.getLength() }
        },
        (new axiom.SystemFile(dir)).list(/\.mp3$/i) || []);
}

The function importPackage can be used to import a Java package into the current name space. Java objects can be instantiated like Javascript object using the new operator. Strings and numbers returned by Java can be handled like any other Javascript object. I particularily like the fact that what you see above really is all code that is required to use a Java library. No interface generator or mapping layer is needed. At the same time, I can continue to use MochiKit, which is a Javascript library that I like for its nice functional programming abstractions.

Hacking away

Since I have solved my server problem, I have made great progress with the application. Switching between client-side and server side programming is really easy now. No more procastinating because the other side's mindset does not happen to be there. No more accidential typing of "the other" syntax.

Though I must say, I'm glad to have a Lisp job, too! :)

How to convert your Symbolics keyboard to USB

2009-04-06T20:53:00.011+02:00

I spent another hour finishing up the Teensy based Symbolics keyboard converter. The software now works as well as the PS/2 version, and uses the same keyboard translation table generation tool. Converting a keyboard should now be doable for anyone who has basic soldering skills.

What you'll need:

Teensy
USB cable (B type)
6-pin .1'' header
Some wire

First, you need to program the Teensy with the Firmware .hex file. Don't worry about customizing the firmware right now. If you decide that you want it to work differently than my version, you can update it later.

Get five pieces of wire, about 5 inches long. Solder them to the 6 pin header, leaving pin 2 unconnected. Connect the other ends to the teensy.

pin #	color	teensy pad	function
1	blue	GND
2		(not connected)
3	green	5V
4	red	D4	DIN
5	black	D5	CLK
6	white	D6<	CLR

Yes, 5V and GND were swapped in my original post, sorry and thanks to Gene Diveglia for reporting

Open the Symbolics keyboard by removing the five screws at the bottom. Carefully remove the small PCB that sits on the larger one, unplug the spiral cable. Using a tool like a large screwdriver, remove the strain relief by pushing it to the outside of the case. Remove the cable, take off the strain relief. Using a pliers, cut out some of the plastic so that the part can be used for the round USB cable.

Using a sharp knife, make the B type connector of the USB cable as small as possible.

Put the strain relief onto the USB cable, about 4 inches away from the scaled down B connector. Using a tool, firmly put it back into the keyboard case. Fixate the cable to the bottom shell of the case like shown. Connect the Teensy to the USB cable, plug the header into the keyboard PCB.

Put some protective plastic around the Teensy so that it does not short circuit anything on the keyboard PCB.

Finally, reassemble the keyboard. It should be easy to put the daughter PCB back onto the main PCB and fit the two shells together. You're done.

Make sure that you've read the README.txt file for the converter firmware. Note that if you want to convert your old-style keyboard, you need to know that the internal color coding of the cables is different with those keyboards. This blog post has a table with the information that you need. Have fun!

Reimplementing the Symbolics Keyboard Adapter with Teensy

2009-04-05T23:13:00.004+02:00

In a recent post on the AVR Freaks blog, Teensy, a new AVR based development board was announced. At $19, it is rather cheap and it comes with an on-chip USB controller. The neat thing about this is that applications are not restricted to some sort of serial emulation protocol, as Arduino-like boards with a separate, FTDI chip base USB controller, are. Instead, Teensy has a full USB 2.0 controller available to the application, and it comes with libraries that makes implementing custom USB keyboard controllers that require no special drivers very easy.

I was a little unsatisfied with the state of the kbdbabel based Symbolics keyboard adapter. The adapter works so far, but there are still some caps lock related bugs and I don't know 8051 assembler. Thus, I decided to give Teensy a first spin by implementing the protocol adapter in C, utilizing the Teensy HID USB keyboard library. As I did the protocol analysis for the Symbolics keyboard years ago, implementing that part was easy enough. I did make some mistakes with respect to accessing program memory on the AVR which took me several hours to sort out, but in the end, I now have a neat little program that runs on a Teensy and that requires no external circuitry to convert a Symbolics keyboard to a USB keyboard. Just solder on a few wires, flash the Teensy, replace the RJ11 cable by an USB cable and off you go.

I'll still support the kbdbabel based Symbolics adapter, but for users who want to hook up a Symbolics keyboard to a modern, USB equipped system, I now recommend getting a Teensy. I'm willing to do the conversion for those who don't have soldering skills, but the shipping will not be cheap. If you want the Teensy firmware, let me know.

Next on my list: Make Teensy talk MIDI. If you've done it already, please share what you know.

Rekonstrukt Progress: MIDI Drum Machine, Turnkey Application Support

2009-03-15T21:34:00.003+01:00

I made quite some progress on rekonstrukt during the last few weeks, and I can happily report that this whole "build a system from scratch" has not turned into a full time occupation, but is very doable as an after work thing. I'd certainly be interested in lifting the effort to a more, say, professional scale in the future, but for the time being, it is nice to both have something to play with in the evening and still maintain a family and a professional life.

Implementing a MIDI drum machine in hardware

One of the application areas that I am currently interested in is music hardware. I'm into electronic music, and even though I am not much of an artist (that'd be hardly compatible with a family and a professional life), I like playing around with and building gear. For a start, I created a drum machine in VHDL. It is a very simple 8 track, 16 beats drum machine, but it runs fully autonomous and requires no CPU, other than for setting up the pattern, tempo and notes. Suffice to say, the drum machine is clocked tightly. There is no interrupt latency to consider, and the CPU is free for the user interface.

This is an interesting experiment in that what I have basically done is create a special purpose processor that plays rythms. This processor is very restricted, but it performs the task that it has been created for efficiently and without a lot of overhead. Certainly, the same thing can be done in software and development times might have been shorter, yet the real time aspect of the problem makes the details rather hard to get right.

The drum machine is rather limited, though, as it can only play 16th notes. I may be implementing a MIDI file player in VHDL in the future, which will be able to play tracks of arbitary complexity. The 16th structured grid of the drum machine will then only be implemented in software.

Turnkey application support

The drum machine actually works well and makes real noises, but getting the software into the running rekonstrukt system on the FPGA started to bother me. There were two ways of getting Forth words into the FPGA: One could either add them to the code that is cross compiled and placed in ROM, or one could upload the source code through the serial port onto the running FPGA. Putting random code into ROM, in addition to the limited amount of space available, has the down side that variables cannot be created in the usual Forth way, by just defining a word and leaving some space in the word's body free as storage for the variable value. Instead, variables in ROM need to be explictly allocated in RAM, which is at least cumbersome. Uploading through the serial port is not a viable option for deploying applications - After all, the device needs to work when switched on.

The easiest way to load data into the rekonstrukt system is to include it in the FPGA configuration bitstream as initialization vector for the FPGA. Block RAM that is used to implement both the ROM and the RAM of rekonstrukt, and block RAM can be initialized from the configuration bitstream.

The binary image for the ROM is created using cross compilation on the host system. The contents of the RAM blocks that are used for the rekonstrukt system RAM is normally not considered by Maisforth. During system startup, the dictionary pointer is normally initialized so that new words are created in RAM right above the 768 bytes that are reserved for system use.

In order to support loading application code into rekonstrukt with the FPGA bitstream, I modified the COLD word which is called to initialize the Forth system. COLD now calls the new DICT-FROM-RAM word which checks whether the RAM contains application code (consisting of dictionary entries in the Forth vocabulary) and, if so, includes these words in the dictionary of the starting Forth system. In addition, I implemented a new TURNKEY vectored word that is called upon system startup. The default TURNKEY action prints the version and copyright herald and then enters the text interpreter loop. Applications can implement whatever code in their turnkey word.

To create a FPGA configuration bitstream with RAM initialization vectors with application code, I modified the usim mc6809 simulator so that it can dump a core image of the running simulated Forth system to a file on the host. Dumping a core can be initiated from Forth. I have implemented hooks to the SWI2 (Software Interrupt 2) instruction so that the Forth can "call back" into the host, initiating the file dump. This process can be fully automated, so creating preinitialized FPGA bitstreams is possible from the Makefile in the forth/ subdirectory of the rekonstrukt source tree.

To sum up: It is now possible to create FPGA configuration bitstreams containing the hardware description, the Forth core and the application code and have the application run automatically after the bitstream has been loaded. I am still using block RAM for all of rekonstrukt's memory, so a lot of the block RAM resources of the FPGA are currently consumed. My next plan is to port rekonstrukt to the Avnet Spartan-3A Evaluation Kit. At $49, this board is cheap and Avnet also offers a 1 Megabyte SRAM add-on board, costing $20. I will try to implement a small boot loader that fits into 1 block RAM which initializes the external SRAM from either the serial or the parallel Flash ROM. That way, most of the block RAM will be available to the application.

Der Chaos Computer Club feiert seinen großartigen Erfolg!

2009-03-03T11:11:00.003+01:00

Heute hat das Bundesverfassungsgericht seine Entscheidung zu der Klage zweier Bürger gegen den Einsatz von Wahlcomputern bei der Bundestagswahl 2005 bekanntgegeben. Dieser Einsatz war nicht rechtmäßig, und der CCC bejubelt dies nun als großartigen Erfolg seiner Kampagne. Über das Urteil selbst möchte ich mir als Nichtjurist keine größeren Gedanken machen, über das Gejubel des CCC jedoch schon:

Der CCC versteht sich ja als sowas wie das NGO der Computer-Experten. In der Öffentlichkeit wird er als kompetent, engagiert und politsch wachsam wahrgenommen. Wann immer es gilt, die angeblichen bürgerlichen Rechte von Menschen im Netz zu schützen, erklärt sich der CCC für zuständig. So auch hier: Ausgehend von einer Aktion holländischer Hacker wurde eine Kampagne gegen Wahlcomputer losgetreten: Wahlcomputer seien grundsätzlich schlecht, weil nicht transparent, und deswegen gehören sie verboten. In einigen Bundesländern ist das schon passiert, und jetzt scheint es auch im Bund bald so weit zu sein: Wahlen in Deutschland haben mit Stift und Zettel zu geschehen, denn alles andere, wir wissen es dank der fleißigen Hacker, ist "unsicher".

Als Konsequenz dürfen wir also in Zukunft nicht darauf hoffen, mehr direkte Demokratie mit Hilfe elektronischer Verfahren zu realisieren. Diskutieren, ja. Entscheiden? Nur mit Stimmzetteln. Und das haben wir nun ausgerechnet den Leuten zu verdanken, die am Liebsten an Ihren Computern herumhängen, sich ihr Leben mit dem Internet organisieren und sich mehr und mehr über die elektronischen Medien definieren.

Herzlichen Glückwunsch, lieber CCC!

P.S.: Weniger polemisch und fundierter hat Till Westermeyer seine Gedanken zu diesem Thema veröffentlicht. Danke dafür!

Hardware/Software Co-Design - What it means with rekonstrukt

2009-02-27T21:33:00.003+01:00

One of my primary goals with the rekonstrukt project is to create a hardware/software co-design environment that cuts down on turnaround times and gives me, the designer, flexibility to test and try out things at various levels. Here is what this means in practice:

rekonstrukt consists of the MC6809 microprocessor running Maisforth. The microprocessor can be implemented in three ways:

In the usim software emulator, which is written in C++ and runs on my workstation. In this mode, the emulator needs be extended in C++ if I want to emulate more of the hardware that the real implementation provides. This mode has fast turnaround times (the simulated 6809 is way faster than the real hardware) and is useful to try out algorithms, learn Forth and the like. It is not timing accurate.
In the Active-HDL simulator, again running on my workstation. In this mode, the actual VHDL implementation of rekonstrukt is simulated. The simulation is still fast enough to run small pieces of actual Forth code. It is a good mode to pinpoint hardware and timing issues, and Active-HDL has great flexibility with respect to setting breakpoints and inspecting the simulated state, which helps a great deal with diagnosing issues.
On the FPGA. The interactive nature of Forth means that this way is often most productive. By actually trying out things, I can quickly gain confidence in my design or explore ideas, getting immediate feedback. In this mode, the logic analyzer is the crucial tool to verify waveforms and timings, and it is not that suitable for pinpointing problems that are inside the FPGA.

In all three modes, it is possible to recompile the Forth kernel and put it into operation in a few seconds. The usim simulator directly works with the ROM image that the cross compiler generates, the VHDL simulator can load a pre-initialized ROM image in VHDL format, and the new ROM image can also be merged with a completely synthesized FPGA bit stream. Thus, synthesizing the hardware is required only when changes need to be tried out on the real hardware. Trying out firmware changes only requires running the Forth cross compiler and restarting the simulator or downloading the bitstream to the target hardware.

Synthesizing rekonstrukt on my workstation takes about 10 minutes, but as the design is split into partitions, changes to individual parts of the hardware does not require a full synthesis run. Incremental runs take only about 3 minutes. Sure, I'd love to see that be faster, but it is bearable

I love this shit :)

ACIA fixed, S3E SK sucks

2009-02-22T22:37:00.004+01:00

In the last two weeks, I was making good progress with my attempt to fully support the Spartan-3E starter kit by Maisforth. I had to improve my SPI controller slightly so that it works better for programming the serial Flash, worked on implementing the (ancient) standard Block I/O vocabulary so that the serial Flash can be used like a floppy disk in the old days and almost got the beautiful vibe full screen block editor to work.

One thing that was began to bother me a lot was the development cycle for Forth programs: I use Emacs to edit stuff, then copy and paste it to the real hardware. As my programs grow, this begins to take considerable time, but for some reason I could not send the data through the serial port at full speed: Maisforth reproduceably lost characters, and I had to insert a small delay after each character sent which made uploads slow. Too slow.

Thus, I went to pinpoint the reason for this character loss. At first, I thought that what I saw was a problem related to the lack of flow control. The serial port on the Spartan-3E Starter Kit is only equipped with RX and TX signals, so no hardware flow control is possible and software flow control is not implemented. After some investigation and experiments it became clear to me, though, that the problem was not flow control related. Both hardware (RTS/CTS) and software (XON/XOFF) flow control work under the assumption that data is buffered in larger chunks, but my character loss happened even when I sent short bursts at full speed to the FPGA.

The problem really was related to the lack of double buffering in the transmitter of the System09 UART. This meant that the transmitter signalled the "transmit buffer full" condition for the whole duration of the actual send operation, even though the data had already been copied from the transmit buffer to the shift register. As the serial port echo in Maisforth is generated by Forth itself, processing of every character and sending it back out through the serial port introduced a pause between every character echoed, which quickly caused problems when characters to be echoed came in at full serial port speed.

The fix for the problem was to introduce double buffering. As soon as the character to be transmitted has been copied over to the transmitter shift register, the UART module now signals the host that the transmit buffer is available for the next character to be sent. This made the problem disappear as Forth echo processing can now take place while the previous character echo is transmitted. Given the slow serial port and the high clock rate of System09, there now is plenty of time for all processing needed.

The Spartan-3E Starter Kit sucks

Well, not really. It is a nice cheap board with a good load of on-board peripherals, useful documentation and nice example designs, but it also has severe limitations: The on-board FPGA is a XC3S500E, which has 40 kByte of block RAM. In my current Maisforth setup, I use 16k for the Forth kernel ROM, 16k for RAM, and 6k for the VGA controller. This leaves only 2k free, so that makes playing with table based waveform synthesis really hard. What I hoped for was that I could put the Forth kernel and the board support Forth library into the parallel NOR flash of the S3ESK and map the Flash into the 6809 address space.

The show stopper for this plan is the fact that data bit 0 of the NOR flash is shared with the SPI bus data signal. Thus, one can either access the flash or the SPI, but not both at the same time. Seemingly, the idea is to copy the contents of the NOR flash into SDRAM and then run from there, but I'm still reluctant to try getting the SDRAM to work. The easy path is blocked, though, and maybe using SDRAM is the best option.

Rekonstrukt - A "New" Forth Machine

2009-02-15T12:38:00.004+01:00

I have been a Fan of Forth since the 1980ies, and when I picked up my FPGA stuff a few weeks, I decided to put some more effort into getting a complete Forth based FPGA system to run before turning back to the SECD reimplementation project. This new Forth machine is called "Rekonstrukt", and I have created a Google Code project to publish the source code.

My last FPGA productivity rush ended with Maisforth running on the Spartan-3E Starter Kit. Maisforth is a ANS-like 8 bit Forth originally written by Albert Nijhof for the somewhat obscure MC6809 based computer called "Maiskastje" built by a dutch computer club a few years ago. Porting Maisforth to System09 was rather easy; all that was needed was a little tweaking of the serial I/O routines so that the System09 MC6850-lookalike serial port was properly handled.

My next goal is to support most of the Hardware that the Spartan-3E Starter Kit has to offer by Forth. This will require some VHDL hacking in order to implement the low level interfaces as well as implementing Forth libraries.

Implementing a SPI controller

I started by implementing an SPI interface. SPI is a serial bus protocol for moderately high speed connections between devices inside one system. On the Spartan-3E board, the analog capture unit, analog preamplifier, digital to analog converter, serial flash and platform flash chips are all connected to one shared SPI bus. It would be possible to operate this bus by bit-banging in software, but this would be rather slow. In hardware, it is easily possible to operate the SPI bus at a bit rate of 10 Mhz.

While other open source VHDL SPI controllers exist, none of them easily interfaced with the "proprietary" bus protocol of System09. I took this as an opportunity to freshen up my VHDL and FPGA design skills. Again, I learned that it does not make sense to synthesize to the real hardware before simulation has shown that the design basically works, but that successful simulation does not automatically mean that the design works in the real hardware. With the help of some regulars in the #fpga IRC channel on freenode.net, I got the SPI controller to run.

Implementing vectorized I/O in Maisforth

System09 comes with a VGA controller that provides for a 80x25 text console, named vdu8. vdu8 is register based, i.e. to write a character onto the screen, the host CPU needs to write the screen position where the character should go into the cursor address registers, then write the character to be shown into the data registers. In order to make vdu8 useable as I/O device for Forth, a small terminal emulator is needed that interprets standard control sequences like Linefeed, Carriage Return and Backspace properly so that it can be used to handle EMIT calls.

The VGA console should be useable as an alternative to the serial console and switching between the two should be possible at run time. The common technique to achieve that is to make the words that need to be switched between alternative implementations be vectored, or "deferred" words. What this means is that a pointer to the word that actually implements the word is stored in the word that is being called by the system, and this pointer can be modified at run time in order to switch to the desired implementation.

Normally, the Forth compiler performs a name lookup when compiling a word, not when executing it. Thus, once a word has been compiled, the addresses of the words that are being called are fixed, and the implementations of these words cannot be changed in retrospect.

For normal, RAM based operation, implementing deferred words is quite simple:

: alias ( xt -- ) create , does> @ execute ;
: noop ( -- ) ;
: defer (  -- ) ['] noop alias ;
: is ( xt  -- ) ' >body ! ;

alias is used to set up an alias for a word. defer is used to set up an alias for the do-nothing word alias. The implementation of such a word can later be changed using the is word.

This works well for RAM based Forth in which the , word can be used to compile a number into the dictionary and change that later. For a ROM based Forth like Maisforth, the default dictionary is read only and as we are interested in vectorizing words that are used by the system, the vectors need to be put somewhere else. Maisforth provides for a "user space" for this purpose. The "user space" is located in RAM at the beginning of the address space, and the IVEC word can be used to allocate a cell from that space in the cross compiler. When adding new user vectors, the USERBYTES constant needs to be increased so that the cross compiler knows the changed memory layout. I did not find out how I could get the vectors initialized automatically by the cross compiler, so I added a new 'EMIT vector initialized to 0 which would be called in the (EMIT word, and changed the COLD word to intialize the 'EMIT vector when rekonstrukt starts.

More information about vectorized functions can be found in Leo Brodie's excellent book Starting Forth in the Chapter Under the Hood.

Trying Clojure...

2008-10-02T14:54:00.003+02:00

I am becoming increasingly frustrated by Common Lisp's age. On the one hand, history makes it what it is: Mature, well-documented, thoroughly understood and practical. On the other, it fails to keep up with current system designs, lacking convenient native support for rich data structures, infrastructure access and parallel programming. No programming language choice is without tradeoffs and in that respect, and I'll still chose Common Lisp in many situations. Realistically, though, Common Lisp cannot be the only language in my tool chest. For browser work, Javascript is much more practical, and for parallel programming, I'm on the lookout.

On my last visit in Cambridge, I attended the Boston Lisp Meeting. The presentation scheduled was about the new Lisp dialect the Clojure by Rich Hickey, who is the creator of this language.

Parallel programming is one of the areas that Clojure wants to make easier. It does so by making all data structure immutable and by language-level abstractions for concurrent data access. In that respect, it is similar to Erlang as it requires a functional programming style everywhere. Unlike Erlang, which puts almost all operations with side effects behind a common message passing interface, Clojure exposes the full Java API to programs. Thus, side effects can be produced everywhere, except in algorithms implemented in Clojure.

Clojure wants to be a Lisp, but it explicitly does not try to be backwards compatible. This opened a rather large design space to Rich Hickey, and some of the choices he made really do make sense. He specifies a reader, yet his reader does not intern symbols. That is a big win, as it allows the reader to actually work with arbitary Clojure source files. In Common Lisp, one needs to re-implement a full reader which does not intern symbols if one wants to read Common Lisp source files. This is kind of ironic, as the "Code is Data" mantra that we keep repeating does not really reflect what is possible in practice.

Rich managed to make me enthusiastic about Clojure, and I decided to give it a spin with a real project that I wanted to conduct anyway: I am a Twitter user, and I would like to be notified of new posts to Planet Lisp on Twitter. The program would read the Planet's RSS feed using a HTTP request, determine if new items have been posted and update the Twitter status of the Planet Lisp Twitter account that I have set up for this purpose.

Getting the development environment up

My laptop runs Windows, but I do most of my Lisp development in a VMware running FreeBSD. As Clojure is hosted on the Java virtual machine, I decided to avoid the indirection and use Windows as my native platform. Clojure support for Slime is available, so I can stay in my familiar environment. A very recent CVS checkout of Slime is required, which took me a little while to figure out.

Processing XML

It took me a another while to discover what the proper way to process XML in Clojure is. An XML parser is included with the base distribution, but there is no information in the documentation how one would actually work with the data structure that is generated by the parser. For my application, I wanted to iterate over certain elements in the XML and extract a few subelements in a loop. An evaluator for XPath expressions would have suited the job, but obviously that is not the Clojure way to do it.

In Clojure, XML does not receive any special treatment. The parser reads it into a tree, and processing is performed using functions that process trees. As simple as it may sound, I had a hard time finding a practical example of how this was supposed to work. Chris Houser finally got me on the right track when he pointed me to his zip_filter package. zip_filter is a tool for filtering data out of trees, and it can work with trees produced by Clojure's XML parser.

Once I had figured this out, things were very easy. The Planet Lisp RSS feed can be read into an XML tree with

(clojure.zip/xml-zip (clojure.xml/parse "http://planet.lisp.org/rss20.xml"))

and one can extract data out of the parsed tree using path expressions:

(clojure.contrib.zip-filter.xml/xml-> xml :channel :item)

This certainly is concise and elegant, and I like the fact that XML is kept out of the program's way. Note that I've included the namespace prefixes in the examples above so that they can be copied and pasted. Normally, one would import these namespaces so that shorter or no namespace needs to be specified to access the symbols.

Making HTTP requests

In order to update the Twitter status, a HTTP POST request to Twitter must be made. A HTTP client which is based on the java.net.HttpsURLConnection class is available from the Files section of the Clojure Google Group. It is not a polished product and required some minor tweaking, but after that, fetching something using HTTP is as easy as

(http-client/url-do "http://planet.lisp.org/" "GET" {})

Reading and writing files

The Twitter gateway needs to persistently store the list of articles that it has found on Planet Lisp in order to decide which of the current articles are new. Clojure is a Lisp and thus can read and write its own data structures easily, so all it takes is calling a few functions. As in the XML case, the hard part was figuring out what the proper functions are, as the documentation is sparse. I found Stuart Halloway's Blog in which he publishes a number of articles describing how the examples from Peter Seibel's book Practical Common Lisp can be implemented in Clojure. The Simple Database example pretty much does the same thing that I required, so I looked at that. Unfortunately, the "spit" function that is the complement to "slurp" and writes a string to a named file did not appear to exist in the clojure namespace. It took a little grepping to find it in clojure.contrib.duck-streams, and referenced from there, it works as expected.

Functional glue

Having all the required components in place, it was time to come up with the real code. As data structures are immutable in Clojure, the challenge was to express the required loop so that both the new persistent state and the list of new postings would be maintained. I came up with a recursive function with two accumulators:

(defn poll
  "Poll planet lisp, check for new postings, update Twitter status when new postings have appeared"
  []
  (save-data
   (let [old-data (load-data)
         process
         (fn [items new-data new-items]
           (if items
             (let [item (first items)
                   guid (first (xml-> item :guid text)) ]
               (recur (rest items)
                      (conj new-data guid)
                      (if (old-data guid)
                        new-items
                        (conj new-items (first (xml-> item :title text))))))
             (do
               (maybe-post-twit new-items)
               new-data)))]
     (process (xml-> (feed-to-zip "http://planet.lisp.org/rss20.xml")
                     :channel :item)
              #{} []))))

To Scheme programmers, this style should be familiar. I find it not terrible myself, although one could certainly push things around to suit taste.

The Verdict

I was sceptical when I first read about Clojure. I am sceptical of new languages in general, and new Lisp family members always make me think "why?" first. Rich Hickey's presentation got me interested because he gave very good reasoning for his design choices. He also is a very good presenter and could easily withstand an audience of die-hard Lispers that included people who have played an active role in the creation of Common Lisp. But that is the singer, not the song.

The good

The fixed reader, vastly improved data structure support, access to a host of libraries and concurrency support all make a good case for Clojure. It is Lisp in many respects, and in the uncompromised macro facility opens up the extensibility that I am used to from Common Lisp.

The bad

Clojure is not multi-paradigm in the sense that Common Lisp is: A functional programming style is required, and there is no way around that. Making the comma be white space is somewhat of an arbitary decision, that does not match the other design choices that seem to be grounded better.

The ugly

The error messages that the compiler produces are mostly useless. Debugging is hard, as no tracing facility and no breakpoints are available - Or maybe they are, but I could not find them in the documentation. The overall immaturity of the language shows frequently, and I have spent hours looking for code examples and finding my way through something that is very much in flux.

My Conclusion

I like Clojure, as it seems to fill my need for a language that supports concurrency and makes it possible to write modern desktop applications, while still being a Lisp. I will use it for another project I have been planning to do in Erlang. For general exploratory development, Clojure is not yet a good choice as the development environment is too immature.

I am not buying Rich Hickey's claim that classic object oriented programming is bad because it does not support concurrency. I begin thinking of object oriented systems more as active databases that allow modeling of complex, interconnected and persistent data structures. Such structures are managable with a pure sequential execution model, and one should refrain from mixing that with tasks that are inherently concurrent.

Finally: My Twitter gateway for Planet Lisp exists, but I have not yet been able to deploy it. Getting Java to run on FreeBSD/amd64 has proven to be kind of a challenge. I will have this sorted out soon, so feel free to follow planet_lisp.

2008-09-27T00:20:00.007+02:00

Revisiting SECD and the power of Lisp

While visiting the ICFP Conference 2008, I was discussing my SECD microprocessor reimplementation project with my co-worker Kilian. The SECD is a classic virtual machine architecture supporting functional programming languages invented by Peter J. Landin in the early 1960ies. It is a relatively unusual architecture in that it is commonly described by transformations of the machine state represented as four registers containing cons cells. In fact, the virtual machine's memory conceptionally consists of linked cons cells, not of a vector of words that is addressed by integer addresses. Jia-Huai You of University of Alberta has a good description of how the SECD virtual machine works.

Quoting SECD: DESIGN ISSUES, here is the specification of the SECD virtual machine. There is one line for each machine instruction, describing the state of the four machine registers before and after the particular instruction has been executed:

While we were discussing my hardware project, Kilian suggested that we should implement a SECD in Common Lisp in order to bridge some time that we had until the next break. Given that Lisp supports lists nicely, we chose to use Lisp lists to represent the SECD memory. We first fooled around with imperative code that mimicked the required state transformations, but soon realised that it would be much easier to directly transform the specification for each machine instruction into executable Lisp. Following common practice, we started writing down how we wanted to type in the specification for each of the SECD instructions. As an example, the LD instruction would be defined by this:

(define-instr 1 LD
  (s e (LD (m . n) . c) d)
  `((,(locate m n e) . ,s) ,e ,c ,d))

The literal "1" is the op code, the "LD" is the symbolic name of the instruction, the first list is the state before the instruction is executed (with the four list elements representing the state of the four registers), and the rest specifying the code to execute for the instruction. The executed code again returns a four element list representing the new state of the machine registers.

We then wrote a macro to transform this specification syntax into Lisp code. Our DEFINE-INSTR macro uses the first list as an argument to DESTRUCTURING-BIND, binding all the names that we then use in the transformed state:

(defmacro define-instr (code name pre post)
  (let ((name* (intern (format nil "~A-INSTR" name))))
    `(progn
       (setf (gethash ,code *instr-hash*) ',name*)
       (defun ,name* ()
         (multiple-value-setq (s e c d)
           (destructuring-bind ,pre (list s e c d)
             (assert (= ,name ,code))
             (values-list ,post)))))))

We were quite happy with our virtual machine, and we were able to run several of the precompiled SECD "binaries" from the ancient LispKit software collection. Getting the macro right and implementing the 21 instructions took little time, and it was easy to, say, instrument the macro for tracing or augmenting it with checks to make sure that we got the original syntax right. It would certainly be possible to polish the macro more - for example, having to specify the mnemonic of the operation twice is not so pretty. We just meant to create something that we could use to explore the behavior of the SECD architecture, so we did not care polishing.

The next day, I attended an introductory Erlang workshop and as I had some prior knowledge of Erlang (Programming Erlang by Joe Armstrong is a good read), I decided to try implementing a SECD VM in Erlang instead of doing the exercises suggested by the lecturer. Erlang has built-in pattern matching, so I used a transliteration scheme of the VM specification very similar to what I used in Lisp. Also, Erlang has lists, so representing the SECD memory as Erlang lists should be a one to one match, too.

So, here is the CAR instruction implemented in Erlang:

secd(S = [ [ CAR | _ ] | _ ],
     E,
     [ ?CAR | C ],
     D) ->
    secd([ CAR | S ],
         E,
         C,
         D);

In Erlang, function dispatch is based on pattern matching, and there is total flexibility to match structures and bind variables to what has matched. For example, in the implementation of the car operation above, we bind the variable S to the previous stack contents, and at the same time bind the variable CAR to the car of the cons cell that is sitting on top of the stack.

The Erlang version of the VM grew well, and the source code looked as nice as the Lisp version. Yet, as I tried to implement the the RAP instruction, I faced a major stumbling block.

The RAP instruction is used to support recursive function calls and works by replacing a previously created dummy environment on the stack, called OMEGA, by one which contains all the functions that are visible in the recursive scope. The specification uses RPLACA to denote that replacement operation, and that is what we used in our Lisp implementation of SECD, too:

(define-instr 7 RAP
  (((c* . e*) v . s) (om . e) (rap . c) d)
  (progn
    (assert (equal om 'omega))
    `(nil ,(rplaca e* v) ,c* (,s ,e ,c . ,d))))

When trying to implement RAP in Erlang, I got stuck because there are no destructive operations on lists. Not in the standard API, and seemingly not in the system API either. So, the Erlang SECD looks nice, only it does not run. Frankly, our Lisp based SECD has bugs, too, and I am not totally sure whether the problem is not in the RAP instruction.

Curiously, getting the RAP instruction right is harder than one wants to expect: In the SECD hardware implementation that I worked on, I used the original microcode of the SECD chip developed at the University of Calgary as part of a verification project. I found two major bugs in this microcode which, ironically, was part of the file set that compromises the proof that the SECD hardware implementation was correct according to its abstract specification. One bug was the garbage collector which did not work at all, and the other was in the RAP instruction which did not work either. I have trouble not concluding something regarding academia in general from these findings.

Anyway.... this was another time when I realized how much I like Common Lisp and why it will propably not be replaced by another language anytime soon. Common Lisp is teh tool of choice for my exploratory programming, as it does not have a terrible lot of restrictions, yet supports some advanced programming abstractions. There are domains where Common Lisp does not have much to offer, though. Concurrent programming currently is the one area where the lack of support in Common Lisp hurts me most. Erlang may be my cure, and I hope to look into Lisp Flavoured Erlang in order to spare myself the arcane syntax that Erlang provides by default.

Schema evolution made easy

2008-08-01T10:32:00.006+02:00

Applications age, and with age often comes the desire to change things. One of the more painful aspects of changing a deployed application is transforming existing data. This post shows some of the schema evolution techniques that we have used to update applications using the BKNR datastore.

With traditional database-based persistence solutions, converting existing databases is often done by first exporting the data using database tools or specialized exporter software and then importing it into the new database schema, possibly converting data while reading. Using explicit export and import steps can be tricky and time consuming. The export side often comes for free, but the importer will need to reinstantiate the correct objects in the new schema knowing both the export format and the new schema. Making such importers work reliably is an additional task that has no additional value, and after having converted the existing databases, the code written can typically be thrown away.

A common technique for making schema evolution less painful is to make only additive schema changes. Old information is preserved, and the new code does the conversion either on the fly or in a separate conversion run. This scheme leads to both cruft in the database and in the source code, and requires additional data cleanup steps to finalize the schema upgrade once the application has been migrated to the new code base. Substantial structural changes require more thought in this scheme, in particular if database side integrity constraints need to be considered.

All in all, schema evolution is a messy business.

Recently, we have been working on extending and updating two of the applications that we have developed using the BKNR datastore, and the schema evolution problem occured for us to be solved, too. When designing the store, we felt that schema evolution should be easy. After all, we have all our data in main memory, so we can always load the old data using the old application code, reload the application, and implement methods for the relevant CLOS functions that are called by the object system when classes are changed.

In practice, we found the need to load the old code, the data and then the new code in succession to be inconvenient. Keeping the old code around in the right places would be rather burdensome, and we also had some problems with (our usage of) CLOS in the face of large metadata changes.

As we keep snapshots of our data in a file, we decided that our schema evolution process would work by converting the data into the new schema while reading the snapshot file. Our snapshot file format contains serialized versions of all persistent CLOS objects that constitutes a data store. For each class used in the snapshot, a layout record is present in the snapshot file that contains the name of the class and the names of the slots of the class.

When reading a snapshot file, the object system compares the class layout in the snapshot against the class definitions of the running code. Inconsistencies detected are signalled, and restarts are used to select schema evolution actions.

Currently, we support the following schema evolution steps:

Class removal: When a layout specifies a class that does not exist in the running application, instances of this class can be ignored.
Class renames: When a class has been renamed, the new class name can be specified and the reading can be restarted.
Slot removal: If a slot named in the snapshot file does not exist in the running code, the slot values of the instances can be ignored.

These three options are solely controlled through restarts established while reading the snapshot file. No code needs to be written to do these conversions.

Additionally, we have a slot conversion mechanism: If a slot has been renamed or its semantics have been changed, slot values of previous software versions can be converted. In order to do this, a method on the CONVERT-SLOT-VALUE-WHILE-RESTORING generic function must be defined. It receives the object being restored, the name of the slot as specified in the snapshot layout, and the value to set.

The default method of CONVERT-SLOT-VALUE-WHILE-RESTORING simply assigns the value provided to the slot with the given name. A specialized method could convert the value or assign it to a different slot of the same object. When reading a class layout from the snapshot file that contains an unknown slot name, a restart is offered to call CONVERT-SLOT-VALUE-WHILE-RESTORING for values of this slot.

The beauty of this form of schema evolution is that we can easily load snapshots from our production systems using new code, without any additional required conversion steps. We can regularily verify that our new code works with real data, and can incrementally refine our schema evolution process without having to resort to database tools or external formats that require special handling.

One could certainly improve on this, for example by making some of the evolution steps automatic so that one does not remember what restarts need to be chosen when loading older snapshots. For the moment, these tools serve us well, and it did not take a lot of time (and was fun) to implement them.

Building Load Resilient Web Servers

2008-07-03T13:04:00.003+02:00

One of the bigger challenges with deploying dynamic web servers is being able to cope with load peaks that happen when the URL is communicated to larger communities in a short time. I have seen several sites being slashdotted and collapse under the load, and it is common to blame the underlying server technology for crashes; being Lisp lovers, we don't want to see that happen for our sites.

Using a caching frontend to reduce backend load

Following industry practice, we have been using a caching reverse proxy in front of our Lisp based dynamic web servers from the beginning. Our choice was Squid, as I had prior experience configuring it and found it to work reliably once the configuration was in place. I was never quite happy with that choice, though, because Squid's main operation mode is forward proxying. It has gazillions of configuration options that are not needed for a reverse proxy, which made us feel that Squid would not be a perfect match for our demands.

This is not to say that Squid has hurt us in any way. It has served us well during a load peak on the create-rainforest web site, but as reverse proxying has become very common in the last few years, we found it about time to go shopping for a solution that might be a better match to our needs.

We hoped to find a frontend proxy having the following features:

Caching - We want the frontend to cache all content that is not session dependent in order to reduce the load on our Lisp backend.
Scalable - The frontend must be able to handle loads much larger than what we see normally in order to accomodate for peaks.
Request queueing - Ideally, we'd like the frontend to handle all concurrency issues and send requests to the backend one after the other using a single persistent http connection, maybe with pipelining.

This set of requirements reduced the available options dramatically, and we ended up giving varnish serious consideration.

Evaluating varnish

varnish seems to have most of the features that we require: It supports caching, has been tested under very high loads and supports FreeBSD which is our deployment platform. Varnish has been written by Poul-Henning Kamp of FreeBSD fame, so we also found it to be culturally compatible.

Our evaluation revealed that varnish is under active development and support, and we found the developers be very responsive to our bug reports and requests. Its architecture looks well thought out, and the configuration language (even if underdocumented) makes the request handling process very transparent.

There are a number of downsides with varnish, though:

Eager cache cleanup policy: varnish removes objects from its cache as soon as they expire. This means that expired objects are always fetched from the backend, even if the previous cached copy was still up to date. It is possible to work around this by either hacking varnish to revalidate cached objects before expiring them or by not expiring them automatically, but rather by explicit purge requests sent from the backend to varnish. Both options, while doable, require substantial work.

No high load safeguards: varnish offers no control over the number of backend connections established and does not support any queuing or backend resource control mechanisms. This means that it will unconditionally try to establish a connection to the backend if an object can't be served from the cache. Once the backend reaches saturation, varnish will aggravate the situation by sending even more requests to the backend, increasing the time to recover from the backend saturation.

Threading: varnish uses threading to schedule requests. While this in principle should not be something negative, threaded programs are much harder to debug in practice, and many concurrency bugs that threaded code is susceptible to only show up after a long time or under certain load patterns which may be hard to reproduce. Admittedly, I am a threads hater, but I am writing this here because we found a serious threading race condition on the second day of our evaluation which prevented varnish from even starting up. My trust in the code was seriously affected by this.

When talking to the varnish developers, the problems were acknowledged, help to fixable problems was very quick and they told us that more advanced features will be develop after the upcoming 2.0 release of varnish.

We would have liked to switch to varnish because of the very good support and because it is meant to be a web frontend, nothing else. Yet, at the current point in time, it does not seem to be mature enough to serve our needs. After having evaluated it, we turned back to squid as it seemed to be the only other option. We found that squid meets our requirements for cache cleaning and revalidation of cached objects very well.

Making objects cacheable

Chosing a front end software is only part of what needs to be done to make a web system fast and robust enough to withstand high loads. The most important factor is to make the frontend serve a large percentage of the incoming requests from its cache and only consult the backend server for content that is really dynamic. The HTTP/1.1 protocol provides for request and response headers that control how content can be cached, and there is little need for explicit configuration if these headers are used correctly.

Using If-modified-since

One way to limit the traffic to the backend is implement the If-modified-since mechanism. It is supported by Hunchentoot for static files by default, and can also be used for dynamic handlers that can check whether a resource has changed since it had previously been requested. Varnish will set the If-modified-since header when it requests resources that it already has in the cache, so every cacheable resource will normally be transfered from the backend to Varnish only once.

Controlling cache refreshing

Often, resources are dynamic, yet it is not crucial that every client sees the absolutely latest version of the resource. For example, in our square meter sales application, visitors should always see the current number of square meters sold, but it is not vital that this information is accurate to the second. Thus, we want the cache to refresh such pages only at a certain interval. The HTTP/1.1 provides for the cache-control directive and in particular the max-age parameter. It specifies how long a cache may consider a cached resource be valid without revalidating with the originating server. By setting this parameter in responses sent by the backend, we can effectively limit the maximum refresh rate for dynamic resources that do not need completely up to date every time.

Testing realistically

In order to test the performance of a Web system, it needs better tools than the often-used ApacheBench tool, which only tests response times and throughput of a single URL. For meaningful results, one should simulate a user load that reflects the load that real users create. I have been using SIEGE for informal testing often, as it is easy to use, but I have also found it a little flakey and prune to random crashes, which made us look for more reliable solutions.

In the FreeBSD ports collection, we found tsung. Tsung is an open-source multi-protocol distributed load testing tool written in Erlang, and it has good support for HTTP. In addition to running load tests against web servers and generating statistics reports, it supports a session recorder that can be used to create a log of the URLs that a user visits when browsing a web server which can then be directly used to simulate many simultaneous users. As an added bonus, it is possible to capture dynamically generated information from web server responses into variables and use them in subsequents requests in a session. This feature can be used to generate sale transactions or user registrations in a load simulation.

Using tsung and squid, we were able to tune our Lisp backend so that all non-dynamic content is served from the cache, pinpoint a serious performance problem and simulate realistic loads. We are now confident that our setup can withstand the next rush of users without crashing.

Hamburg Lisp Meeting

2008-05-29T10:36:00.003+02:00

Yesterday, there was a Lisp meeting in Hamburg hosted by Stefan Richter at freiheit.com. Some 30 men from all over northern Germany attended. freiheit.com kindly sponsored beverage and barbeque, and the nice weather made staying on the terrace very enjoyable - see Stefan's photos. Two presentations were given: Edi Weitz talked about Piano, an application which was originally developed under MCL used to design airplanes. Edi ported Piano to LispWorks on Windows in 40 man days, which is rather impressive given that this not only meant a compiler change, but also a change of graphic environments with different semantics and mechanisms. We were shown how a Boeing 737-lookalike can be designed with Piano by entering just a dozen parameters. Rainer Joswig showed off two of his Symbolics Lisp Machines, a NXP1000 and a MacIvory III. He walked us through his genealogy of Lisp machines and then demoed Genera and Dynamic Windows, which still impresses me. I had brought my Symbolics keyboard and tried most of the stuff on my laptop when he was doing it on the real hardware. The VLM on Linux is by orders of a magnitude faster, and with a real keyboard, it is a lot easier to use. Several people asked me if I could get them a PS/2 adapter for the Symbolics keyboard, too, so I think I will need to build a few more. The evening concluded with music, beers and food, and it was unfortunate that we had to leave early and sober in order to hit the road back to Berlin.

Symbolics Keyboard on PS/2 port

2008-05-18T22:20:00.006+02:00

I have been busy with hacking Hunchentoot and flexi-streams at daytime during the last few weeks, and wrote a thread-safe version of a simple deterministic profiler for Clozure CL so that I could isolate performance problems. I'm going to blog about that stuff in a few days, but as that is daytime work, I had to find something else to relax on the weekend. Thus, I turned my attention to the kbdbabel hardware that I received last week. A few months ago, I emailed Alexander Kurz, who is the creator of the kbdbabel project, whether he could help me getting a PS/2 adapter for the Symbolics keyboard to work. At the time, he had created PS/2 adapters for various vintage keyboards and he expressed his interest. We could not make much progress, though, as he did not have a keyboard and I lacked the knowledge to adopt the 8051 based assembler code to the needs of the Symbolics protocol. I'm mostly an AVR guy when it comes to microcontrollers. Alexander was intimidated enough by the pictures of the various Symbolics keyboards that he decided to get one and make a kbdbabel for it. I was more than happy when he contacted me out of the blue a few weeks ago and told me that he now has a solution. The hardware he sent did not work right away with my Thinkpad, but that was because the Thinkpad's PS/2 port do not quite work like most other PS/2 ports do. Thus, I had to get a PS/2 to USB adapter for 10 Euros. With that, the adapter worked as advertised. I had to complete the keyboard mapping so that all keys in the default Open Genera map are mapped to a key on the Symbolics keyboard. This means that there are no mappings for several keys (the square, circle and star keys among them). Supporting those keys will require changing the Open Genera X11 keyboard mapping code, which is something I'll put onto my "weekend projects" list now. I have not yet tested the adapter with the "slim" keyboard - I only have a 3600 one right now and the two "slim" keyboards I bought from David K. Schmidt are still in Cambridge and won't be with me until next month. Contrasted to the 3600 keyboard, the "slim" keyboard is microcontroller based and will most likely require different timing than the TTL based old-style keyboard. Other than that, the keyboards should be compatible and the adapter should work with both variants. I'm going to build a few adapters for myself and some friends of mine. If you want one as well and don't like to do the soldering yourself, contact me.

BKNR at the Boston Lisp Meeting

2008-04-23T09:27:00.008+02:00

After a weekend full of parentheses in Amsterdam, I attended the Boston Lisp Meeting hosted at MIT. Peter Dillinger gave a presentation on the ACL2s, which is an Eclipse based user interface for the ACL2 theorem prover. I have no background in mathematics and as such, I had to guess my way through the terminology, but the talk was entertaining and even though it somehow does not feel right to see an advanced Common Lisp program be controlled by a lowly Java application, I see the practical aspects of the approach. Bottom line: Common Lisp loses when it comes to programming modern user interfaces in general, and I'm not sure that we can do much about that. The second speech for the evening was my presentation of the BKNR datastore. I found myself being well received, and the questions from the audience mostly were those that I have been asking myself during the last few years. I have put the slides online, and I also updated the BKNR website to make the documentation easier to access. If you have Google Earth installed, you can play with the (alpha quality) GE interface to the BKNR based Samboja Lestari web site. If you want, you can have a look at the source code, too. Our use of publish/subscribe for persistent objects may be of interest, as well as the use of Screamer for brute-force determination of the largest rectangle inside an arbitary polygon. Kilian, who does most of the programming in the BOS project presently, has a background in music composition. He found it easier to formulate the solution to the problem using constraints rather than using explicit looping.

The socializing part of the meeting was rather short for me as I am mostly staying on Central European Time while being in Boston, so it felt like past midnight to me. I met several interesting people anyway, and I hope that I can synchronize my next visits in Cambridge with the meetings.

Chaosradio Express on Lisp

2008-04-14T12:37:00.003+02:00

Last week, I have been a guest at Tim Pritlove's Chaosradio Express and he interviewed me about Lisp. The podcast is in german, we talked about how I came to Lisp and what I think it is about.

Feature development and getting carried away on Lisp

2008-03-31T23:32:00.001+02:00

Customer Demands

Recently, I have been spending some weekend and late night hours on extending the QuickHoney web system. The artists want to see the system modernized with RSS feeds and more interaction, and we'll also be adding a shop where both digital and real warez can be bought. Peter in particular wants to get more feedback, so he asked me to add a "quick feedback" mechanism that visitors can use to send a short textual comment on any of the pictures. In this post, I'll describe how I added this feature using Javascript, BKNR, CL-SMTP and CL-MIME. First off, here is a little overview of the QuickHoney web application. It is a early AJAX style application with one HTML page consisting of a number of layers which are controlled by a Javascript application. Communication between the Javascript application and the backend server is done through Javascript code snippets that are generated on the server and evaluated inside the client application using IFRAMES. The user navigates through categories and subcategories to individual pictures. The backend for the QuickHoney application is written in Common Lisp using the BKNR framework. It uses the datastore extensively as well as the cl-gd image processing library written by Edi Weitz.

Frontend Functionality

The feedback functionality will work on a per-picture basis. When a picture is displayed, a small "provide feedback" icon will be displayed. When the user clicks it, a form will be displayed in a layered window. The form will consist of "From" and "Text" fields and a "Send" button. The onclick action of the button is connected to a function that packs the contents of the "From" and "Text" fields into a urlencoded form data string and send it to the server using a POST request:

function digg_send()
{
    var d = doXHR("/digg-image/" + current_image.id,
                  { method: 'POST',
                    headers: {"Content-Type":"application/x-www-form-urlencoded"},
                    sendContent: queryString({ from: $('digg_from').value,
                                               text: $('digg_text').value }) })
        .addCallback(function () { alert('sent'); });
    return false;
}

doXHR and $ are functions from the MochiKit Javascript library which I like to use for non-GUI-related things for its conciseness and conceptional soundness. As can be seen in the Javascript snippet above, there is a /digg-image/ handler on the web server that is used to send the feedback. The URL that is used also consists of the object ID of the image currently displayed. Every persistent object in the BKNR datastore has a unique object ID, and the FIND-STORE-OBJECT function can be used to find an object with a certain object ID in the store.

Implementing a Backend Handler

The BKNR web framework makes it easy for applications to declare new handlers which relate to a certain store object. It provides for a set of handler classes that application handlers can inherit from. These handler base classes implement request URL parsing and automatically call handler methods with with relevant information from the URL and the request body parsed into Lisp objects. For the feedback feature, we need to subclass the OBJECT-HANDLER base class which extracts the object ID out of the URL, uses FIND-STORE-OBJECT to find the relevant object in the store and then calls the HANDLE-OBJECT method of the handler to actually handle the request. The declaration for this handler class looks like this:

(defclass digg-image-handler (object-handler)
  ()
  (:default-initargs :object-class 'quickhoney-image))

The :OBJECT-CLASS initarg can optionally be specified when creating an OBJECT-HANDLER object to make sure that HANDLE-OBJECT is only called for objects of that class. If the object ID in the URL references an object from a different class, an error page is displayed to the user. The HANDLE-OBJECT method for the DIGG-IMAGE-HANDLER class is specialized on both the DIGG-IMAGE-HANDLER handler class and on the QUICKHONEY-IMAGE class. It extracts the form parameters from the request and opens a connection to the SMTP server to send a mail to the owner or owners of the QUICKHONEY-IMAGE. The mail itself is a simple HTML mail with a table containing the name of the image, hyperlinked to the online page with the image and the feedback information entered by the user. Also, a thumbnail of the image is included with the email so that the artist immediately sees what picture the user is raving about. Modern mailers (like Apple Mail or Google Mail) display images attached in a multipart/mixed MIME mail inline, so there is no need to come up with a fancy HTML mail that references elements included in the same mail body. This is the source code of the handler:

(defmethod handle-object ((handler digg-image-handler) (image quickhoney-image))
  (with-query-params (from text)
    (cl-smtp:with-smtp-mail (smtp "localhost"
                                  "webserver@quickhoney.com"
                                  (remove-duplicates (mapcar #'user-email
                                                             (or (owned-object-owners image)
                                                                 (list (find-user "n")
                                                                       (find-user "p"))))))
      (cl-mime:print-mime
       smtp
       (make-instance
        'cl-mime:multipart-mime
        :subtype "mixed"
        :content (list
                  (make-instance
                   'cl-mime:mime
                   :type "text" :subtype "html"
                   :content (with-output-to-string (s)
                              (html-stream s
                                           (:html
                                            (:head
                                             (:title "Picture comment"))
                                            (:body
                                             (:table
                                              (:tbody
                                               (:tr
                                                ((:td :colspan "2")
                                                 "Comment on picture "
                                                 ((:a :href (make-image-link image))
                                                  (:princ-safe (store-image-name image)))))
                                               (:tr
                                                (:td (:b "From"))
                                                (:td (:princ-safe from))))
                                               (:tr
                                                ((:td :valign "top") (:b "Text"))
                                                (:td (:princ-safe text)))))))))
                  (make-instance
                   'cl-mime:mime
                   :type "image"
                   :subtype (string-downcase (symbol-name (blob-type image)))
                   :encoding :base64
                   :content (flexi-streams:with-output-to-sequence (s)
                              (blob-to-stream image s)))))
       t t))))

For completeness, let me also show you how the handler is entered into the list of handlers of the Quickhoney backend server:

(defun publish-quickhoney ()
  (unpublish)
  (make-instance 'website
		 :name "Quickhoney CMS"
		 :handler-definitions `(("/random-image" random-image-handler)
                                 ("/animation" animation-handler)
                                 ("/image-query-js" image-query-js-handler)
                                 ("/login-js" login-js-handler)
                                 ("/clients-js" clients-js-handler)
                                 ("/buttons-js" buttons-js-handler)
                                 ("/edit-image-js" edit-image-js-handler)
                                 ("/upload-image" upload-image-handler)
                                 ("/upload-animation" upload-animation-handler)
                                 ("/upload-button" upload-button-handler)
                                 ("/rss" rss-handler)
                                 ("/admin" admin-handler)
                                 ("/upload-news" upload-news-handler)
                                 ("/digg-image" digg-image-handler)
                                 ("/" template-handler
                                      :default-template "frontpage"
                                      :destination ,(namestring (merge-pathnames "templates/" *website-directory*))
                                      :command-packages (("http://quickhoney.com/" . :quickhoney.tags)
                                                         ("http://bknr.net/" . :bknr.web)))
                                 user
                                 images
                                 ("/static" directory-handler
                                            :destination ,(merge-pathnames #p"static/" *website-directory*))
                                 ("/MochiKit" directory-handler
                                              :destination ,(merge-pathnames #p"static/MochiKit/" *website-directory*))
                                 ("/favicon.ico" file-handler
                                                 :destination ,(merge-pathnames #p"static/favicon.ico" *website-directory*)
                                 :content-type "application/x-icon"))
		 :admin-navigation '(("user" . "/user/")
				     ("images" . "/edit-images")
				     ("import" . "/import")
				     ("logout" . "/logout"))
		 :authorizer (make-instance 'bknr-authorizer)
		 :site-logo-url "/image/quickhoney/color,000000,33ff00"
		 :login-logo-url "/image/quickhoney/color,000000,33ff00/double,3"
		 :style-sheet-urls '("/static/styles.css")
		 :javascript-urls '("/static/javascript.js")))

Getting Carried Away on Common Lisp

As you can see, I had to write little code to implement this functionality. In fact, the whole thing should have taken no longer than one or two hours, but here is how I found myself getting carried away: For one, I spent substantial time on refactoring CL-SMTP, as you can read in my last blog entry. That took a few hours. For another, I really don't like how mime emails are constructed with MAKE-INSTANCE in the HANDLE-OBJECT method above. I thought that I'd be nice to have a DEFINE-CONSTRUCTOR macro that created a function to create objects of arbitary classes, but that allows for one or more positional arguments in addition to any of the keyword arguments accepted by MAKE-INSTANCE for a particular class. Thus, instead of just coping with the slight ugliness, I tried myself on a macro. Simple as it looked, it surely became more complicated as I had to deal with defaulted arguments, and I was stopped from investing any more time into this when I was reminded by cmm that INITIALIZE-INSTANCE and SHARED-INITIALIZE can define additional keyword arguments that are accepted by MAKE-INSTANCE. Sure, it would be possible to find all applicable methods and extract more acceptable arguments from their lambda lists. Yet, I felt that I had enough fun for this last weekend and wrapped up the feedback functionality for Quickhoney.

Refactoring CL-SMTP

2008-03-24T11:49:00.002+01:00

In my recent refactoring of BKNR, I decided that we do no longer want to use any of Franz' open source libraries if we can avoid it. Even though they work fine in general, hacking them is a pain because they adhere to John Foderaros Common Lisp Coding Standards which basically say that one should use Franz' own non-standard IF* macro for all conditionals. I do think that one should be careful with nesting conditionals deeply, but I do not agree with using a macro and spacing as a cure. If I have code in which conditional nesting exceeds two levels, I refactor to extract conditional code into functions. That way, functions are kept shorter and the use of names instead of literal code usually makes it easier to understand what's going on. So my easter goal was to replace Franz' NET.POST-OFFICE SMTP client by CL-SMTP. Both clients do not support proper quoting of non-ASCII characters in mail headers, thus the need to hack arose and IF* is nothing I want to get myself used to. A SMTP client really is not that complicated to begin with, and apart from the basic functionality, CL-SMTP already supported SSL and authentication, which nowadays are two basic requirements. What it was missing was the possibility to send pre-formatted messages, which is something that I require because I usually make up my own messages, including headers and body, using format strings or CL-MIME if I want to send attachments or otherwise need more control over the mail body. CL-SMTP prove to be a little hackish. Seemingly, only simple mail sending had originally been planned for, and the API had then been extended multiple times with growing user needs. There was no layering between the SMTP protocol aspects and the message formatting functionality, both being freely interleaved. While being simple to use for those use cases that had been planned for, the API was not helpful for my intended use. In a first round, I simplified the code, collapsed a few common patterns into functions and made the code generally easier to hack on. I then split the SMTP protocol handling into an exported macro, WITH-SMTP-MAIL, that is used to establish an SMTP connection and create the mail envelope. The body of the macro invocation is then invoked with a variable bound to the stream connected to the SMTP server. The existing mail sending functions of CL-SMTP have been converted to use that API, too. I then incorporated a patch that I found in the CL-SMTP mailing list archive so that raw TLS is supported. The existing TLS functionality worked by connecting to the standard SMTP port, then switching the cleartext connection to encrypted mode by issuing the STARTTLS command. In contrast, raw TLS works by having an SMTP server listen on a separate port for encrypted connections. No initial cleartext handshake is required in this operation mode. Finally, I implemented automatic quoting of non-ASCII characters in mail headers using a specialized stream class. The Gray Streams Examples in the SBCL manual are my favoured cheat sheet when implementing special purpose stream classes. The result of my work is available in the BKNR repository at svn://svn.bknr.net/svn/trunk/thirdparty/cl-smtp/ in case you want to give it a try right now. I developed with Clozure CL on my Powerbook and after I committed, our buildbot went red for SBCL. I had named an accessor for my specialized stream class "STREAM" which triggered a package lock violation error on SBCL. The name was bad, so I changed it to "ENCAPSULATED-STREAM" and saw the buildbot going green for SBCL too. I love that! I am now waiting for feedback on the refactorings and extensions. There are still bugs which need fixing and support for compilers other than CCL and SBCL needs to be verified. Also, the documentation for CL-SMTP needs to be better, and I think I'll just steal Edis HTML template and write something up myself next week. Also, an automated test for CL-SMTP would be great, but as this requires a SMTP peer to talk to, I have not really had a good idea how to implement that. Maybe later on. I certainly hope that Jan Idzikowski, who is the author and maintainer of CL-SMTP, will accept my patches and make them part of the main distribution.

BKNR is alive

2008-03-20T07:16:00.001+01:00

Recently, there has been quite some activity in the BKNR repository. This is because my company finally seems to become reality, with a real office, customers and all that. I even have an employee now, Kilian, who is working full time on BKNR based projects. I guess we are the only Lisp company in Berlin now! As I am working for ITA Software full time, the fact that I have some more workforce allowed me to attack the long-outstanding major updates for QuickHoney and Create Rainforest. Quickhoney will be modernized with a Blog and RSS feed, and we will also implement a shopping system so that one can buy digital and real QuickHoney products. Create Rainforest will be extended into Google Earth. Instead of hacking our funny, but close to unusable satellite application, we will use Google Earth to display and visualize our data. The old system will stay, but new features will not be added to that. These commercial projects have lead to several updates to BKNR in the past weeks: We have finally converted from portable AllegroServe to Hunchentoot as HTTP server substrate. Even though AllegroServe performs fine, it written in a very peculiar style which makes hacking it a rather unpleasant experience. Also, Hunchentoot is better maintained. In the process, we got away with passing request arguments throughout the web environment. The current request is now (only) carried in a special variable, as applications seldomly require access to it. SBCL and CCL are our current primary development platforms. I have still not given up on CMUCL completely, but we are waiting for the 19E release before we pick it up again. I am fed up with working around missing Unicode support, which we require in all our current projects. We moved away from common-lisp.net with our repository and web sites. The primary reason for this move was that we now have a Buildbot running for BKNR, and I could not quickly get the neccessary infrastructure to run on common-lisp.net - Buildbot is Python based and requires substantial library support to run. BKNR bugs have been fixed. Sure enough, there have been quite some of them. We hope to be better in the future with our continous integration testing and with more unit tests. Sure enough, I will be moving this Blog to our BKNR based blog system soon. Until that happened, this is my place to practice.

Hardware woes

2007-08-08T14:50:00.000+02:00

When I put the SECD project aside, it was because I could not get the RAM controller to run. I felt rather stupid, as the RAM is a very simple asynchronous chip that requires no special timing, yet all my attempts to reliably write the RAM from the 6809 host failed - And as long as that does not work, I can't start any program on the SECD coprocessor. Today I finally found the source of my problems. It appears that even though there is an address bit ram_a(0) in the constraints file for the Trenz retrocomputing base board, it is not connected to the RAM. Supposedly, it is not connected because the RAM works word-wise. It took writing a standalone RAM tester in VHDL and a lot of staring at the logic analyzer and Chipscope outputs to find out that I actually had an addressing problem. Also, John Kent was of great help again as he assured me that, prinicipially, the VHDL that accesses the RAM looks right. He also suggested that I might have a problem related to my constraints file, which appeared to be true. Now, finally, I can turn to actually getting the LispKit compiler to work and see the fixed hardware garbage collection in action. Stay tuned.

Back to SECD

2007-07-29T21:51:00.001+02:00

I have finally returned to debugging the SECD implementation on the FPGA. When I stopped with this, I had problems getting the dual ported RAM controller to run. The SECD CPU is designed as a coprocessor and relies on a host CPU to set up the program in its memory and to start itself. Once the program has finished to run, the host CPU reads out the results from the SECD memory. My host CPU is a 6809, based on System09 written by John Kent. This is a historic coincidence, too, as the original SECD implementation used an 6809 as host CPU, too, at least for the testing. The 6809 software that has originally been used is not available anymore, so I needed to find a way to quickly write some diagnosis and debugging routines. System09 is designed to run the FLEX operating system and there is a BASIC interpreter for Flex available, but somehow I did not like that too much. My first attempt to create a diagnostic monitor was to write a C program and compile it with gcc6809. This worked, but I did not like the turnaround times too much. Thus, I tried to find a Forth implementation for the 6809 that I could use as a diagnostics monitor. My search led me to Maisforth, which is a Forth implementation for a custom computer called the "mais kastje" which was built and used by a group of dutch Forth enthusiasts. I changed my System09 based 6809 system so that its memory map matches that of the "mais kastje" and hacked the Forth interpreter so that it properly initializes and uses the ACIA serial port in System09. Getting Maisforth to run was not too hard, but I certainly would not have finished that without the help of the excellent Logicport logic analyzer that I have bought earlier this year. Logicport is a PC based logic analyzer that retails at $389, with 34 channels, complex triggering and 500 Mhz sample rate. The Windows software is very usable, and it comes with interpreters for a few serial protocols (I2C, SPI, RS232) which helped me to track down my ACIA initialization problem in a short timespan. I'm a very happy customer of this and can recommend Logicport to anyone who can't afford a "real" logic analyzer. Now that I have Forth running on the FPGA, I am back to debugging the memory controller problems. I use Forth as a scriptable debug monitor, so in addition to examine and change memory contents, I can define symbolic addresses, write little test programs and thus automate debugging quite a bit. With Forth and the logic analyzer, I quickly found that my problem lies in how I set up the low and high byte select bits of the RAM chip, which has a 16 bit datapath.