Berlin Lispers Meetup - Tuesday November 3, 2009

Title: Berlin Lispers Meetup
Date: Tuesday November 3, 2009
Time: 8pm onwards
Venue: St. Oberholz, Rosenthaler Straße 72a, 10119 Berlin
Transport: U-Bahn Rosenthaler Platz

You are kindly invited to the first "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee, organized by Willem Broekema and Hans Hübner.

There are no official presentations planned, but everyone is free to bring a laptop and demonstrate some cool work.

Please join for an evening of parentheses!

Admittedly, I like JavaScript

Call me pervert, but I find writing JavaScript enjoyable. The aspect that I like most about it is its dynamicism. Certainly, the syntax is ugly, but the js2-mode for Emacs makes writing JavaScript much less painful.

My addiction for JavaScript made me accept a project to create an artists' portfolio website that has no commercial value, but gave me lots of opportunity to play around with client-side coding. I decided that I want to put all data into a JSON encoded data structure and use a client-side content management system to edit and update the database. I originally intended to write the required server-side components for the CMS in JavaScript, too, but that idea went down the drain as it made deployment of the whole thing a lot harder. Thus, the server-side components are now reduced to the bare minimum and are written in perl, so that a cheap web host can be used both for deploying both the web site and the CMS.

One common problem with client-side JavaScript is flow control with long-running operations. JavaScript is single threaded, so in order to maintain UI responsiveness, functions may not block waiting for network responses. Thus, whenever a UI wants to interact with the server and then resume operating, things get tricky. There are libraries like jwacs, which is a preprocessor which implements first-class continuations, and Arrows, which is a "generalization of Monads", but these either make debugging hard through the use of a preprocessor or are incomplete.

My approach to the problem is to use closures to manually establish the continuation, and callbacks to invoke these continuations when appropriate. While being dissatisfying in the intellectual sense, the approach works for the applications that I'm using them in.

Sure, I'd rather have macros instead of having to control evaluation through closures. But then, this is still better than any old static language that requires me to cram everything into classes and objects, forcing me to either make all design changes upfront or to use sophisticated refactoring wizardry to get dynamicism on the source code level. Oh well, or maybe I should rather give up on Emacs :)

Javascript Everywhere - Using Axiom Stack

A friend of mine asked me to help her with creating a portfolio website for a friend of hers, who is an artist. The site will be somewhat similar to QuickHoney's web site: It will be Javascript based, relatively simple in terms of layout, and self maintainable, i.e. the artist will be able to add and update the website without the need for HTML programming.

The site will be hosted on a cheap, no-frills shared web host. On the server, it will consist of static files. The portfolio data set - Projects, descriptions, dates - will be stored in JSON formatted files. Manipulation of the data set will be done with a special Javascript application for content management.

The site will contain various media data: Photos, MP3 audio tracks, videos hosted by a video hoster.

Some parts of the content management tasks cannot be performed by Javascript in the browser:

• Image manipulation, in particular rescaling.
• Provide information about MP3 audio tracks
• Fetching video information from the video hoster
• Uploading data to the web hoster

These tasks require some program that runs outside of the browser; obviously, they are performed by a http server running on the artist's box.

My first impulse was to implement these functions in Common Lisp. Image manipulation is easy, using Edi Weitz' excellent CL-GD library. MP3 information should be easy enough to read using Peter Seibel's code from Practical Common Lisp. Parsing information from the video hoster would be easy, as that information is available both in XML and JSON formats, and Gilbert Baumann's and David Lichteblau's Closure XML Parser which I enjoy to use a lot. Lastly, uploading data to the web host should be possible using CL-FTP which I had not used before.

Exit Common Lisp

I spent a few evenings hacking CL-FTP so that it fits my library set and trying to find a sufficiently complete MP3 parsing library, played with dumping images so that startup times would be short. In between, I played with jQuery, HTML and CSS to get the user visible parts of the site into shape. Sadly, I made much better progress and had a lot more fun hacking Javascript than fooling around with Common Lisp and incomplete libraries. At one point, I decided that I need to make more progress and that I should reconsider my strategy for the CMS server.

I decided that my second choice for the CMS server language would be Javascript. I'd rather write both client and server in Common Lisp, but that is not how the world looks like.

Enter Javascript

I spent a few hours shopping for server-side Javascript solutions. The open source Javascript based web servers are either based on the SpiderMonkey or the Rhino Javascript implementations, both by the Mozilla foundation. SpiderMonkey is written in C, thus extending it requires libraries using the C calling convention. Rhino is written in Java, and it gives Javascript applications direct access to libraries written in Java. As I am developing on Windows, but the artist is using a Mac, Rhino looked a lot more attractive.

After looking at some project web pages picked from the Wikipedia entry on Server-side Javascript, I decided to give Axiom Stack a spin. I chose it because it is a real Javascript framework (using Prototypes, not mimicking classes), has easy-to-follow getting started documentation, provides for persistent objects similar to the BKNR datastore and has active support.

My application is not quite typical for applications implemented with Axiom Stack. Usually, the server would be used by the site's users, too. My server manipulates the static files that make up the web site, and is basically a slave to the CMS application that runs in the browser. Thus, I am not using Axiom Stack's persistent object store in this application.

Enslave Java

I have never developed a very friendly relationship to Java, despite the fact that when I chose to use it for one reason or another, I had no trouble with it. Java is great, as long as someone else writes code in it. Being able to call Java from Javascript is really nice, though, as there are so many libraries out there. In Rhino, Java libraries can be just used. Thus, as Axiom Stack has no "native" MP3 reading facilities, I grabbed a Java MP3 library off the net and used that:

importPackage(Packages.de.vdheide.mp3);

function sound_list () {
    var dir = '../../../files/sound/';

    return MochiKit.Base.map(
        function (filename) {
            var pathname = dir + filename;
            var mp3 = new MP3File(pathname);
            return { filename: filename,
                     title: id3.getTitle(),
                     length: mp3.getLength() }
        },
        (new axiom.SystemFile(dir)).list(/\.mp3$/i) || []);
}

The function importPackage can be used to import a Java package into the current name space. Java objects can be instantiated like Javascript object using the new operator. Strings and numbers returned by Java can be handled like any other Javascript object. I particularily like the fact that what you see above really is all code that is required to use a Java library. No interface generator or mapping layer is needed. At the same time, I can continue to use MochiKit, which is a Javascript library that I like for its nice functional programming abstractions.

Hacking away

Since I have solved my server problem, I have made great progress with the application. Switching between client-side and server side programming is really easy now. No more procastinating because the other side's mindset does not happen to be there. No more accidential typing of "the other" syntax.

Though I must say, I'm glad to have a Lisp job, too! :)

How to convert your Symbolics keyboard to USB

I spent another hour finishing up the Teensy based Symbolics keyboard converter. The software now works as well as the PS/2 version, and uses the same keyboard translation table generation tool. Converting a keyboard should now be doable for anyone who has basic soldering skills.

What you'll need:

• Teensy
• USB cable (B type)
• 6-pin .1'' header
• Some wire

First, you need to program the Teensy with the Firmware .hex file. Don't worry about customizing the firmware right now. If you decide that you want it to work differently than my version, you can update it later.

Get five pieces of wire, about 5 inches long. Solder them to the 6 pin header, leaving pin 2 unconnected. Connect the other ends to the teensy.

pin #	color	teensy pad	function
1	blue	GND
2		(not connected)
3	green	5V
4	red	D4	DIN
5	black	D5	CLK
6	white	D6<	CLR

Yes, 5V and GND were swapped in my original post, sorry and thanks to Gene Diveglia for reporting

Open the Symbolics keyboard by removing the five screws at the bottom. Carefully remove the small PCB that sits on the larger one, unplug the spiral cable.  Using a tool like a large screwdriver, remove the strain relief by pushing it to the outside of the case. Remove the cable, take off the strain relief. Using a pliers, cut out some of the plastic so that the part can be used for the round USB cable.

Using a sharp knife, make the B type connector of the USB cable as small as possible.

Put the strain relief onto the USB cable, about 4 inches away from the scaled down B connector. Using a tool, firmly put it back into the keyboard case. Fixate the cable to the bottom shell of the case like shown. Connect the Teensy to the USB cable, plug the header into the keyboard PCB.

Put some protective plastic around the Teensy so that it does not short circuit anything on the keyboard PCB.

Finally, reassemble the keyboard. It should be easy to put the daughter PCB back onto the main PCB and fit the two shells together. You're done.

Make sure that you've read the README.txt file for the converter firmware. Have fun!

Reimplementing the Symbolics Keyboard Adapter with Teensy

In a recent post on the AVR Freaks blog, Teensy, a new AVR based development board was announced. At $19, it is rather cheap and it comes with an on-chip USB controller. The neat thing about this is that applications are not restricted to some sort of serial emulation protocol, as Arduino-like boards with a separate, FTDI chip base USB controller, are. Instead, Teensy has a full USB 2.0 controller available to the application, and it comes with libraries that makes implementing custom USB keyboard controllers that require no special drivers very easy.

I was a little unsatisfied with the state of the kbdbabel based Symbolics keyboard adapter. The adapter works so far, but there are still some caps lock related bugs and I don't know 8051 assembler. Thus, I decided to give Teensy a first spin by implementing the protocol adapter in C, utilizing the Teensy HID USB keyboard library. As I did the protocol analysis for the Symbolics keyboard years ago, implementing that part was easy enough. I did make some mistakes with respect to accessing program memory on the AVR which took me several hours to sort out, but in the end, I now have a neat little program that runs on a Teensy and that requires no external circuitry to convert a Symbolics keyboard to a USB keyboard. Just solder on a few wires, flash the Teensy, replace the RJ11 cable by an USB cable and off you go.

I'll still support the kbdbabel based Symbolics adapter, but for users who want to hook up a Symbolics keyboard to a modern, USB equipped system, I now recommend getting a Teensy. I'm willing to do the conversion for those who don't have soldering skills, but the shipping will not be cheap. If you want the Teensy firmware, let me know.

Next on my list: Make Teensy talk MIDI. If you've done it already, please share what you know.

Rekonstrukt Progress: MIDI Drum Machine, Turnkey Application Support

I made quite some progress on rekonstrukt during the last few weeks, and I can happily report that this whole "build a system from scratch" has not turned into a full time occupation, but is very doable as an after work thing. I'd certainly be interested in lifting the effort to a more, say, professional scale in the future, but for the time being, it is nice to both have something to play with in the evening and still maintain a family and a professional life.

Implementing a MIDI drum machine in hardware

One of the application areas that I am currently interested in is music hardware. I'm into electronic music, and even though I am not much of an artist (that'd be hardly compatible with a family and a professional life), I like playing around with and building gear. For a start, I created a drum machine in VHDL. It is a very simple 8 track, 16 beats drum machine, but it runs fully autonomous and requires no CPU, other than for setting up the pattern, tempo and notes. Suffice to say, the drum machine is clocked tightly. There is no interrupt latency to consider, and the CPU is free for the user interface.

This is an interesting experiment in that what I have basically done is create a special purpose processor that plays rythms. This processor is very restricted, but it performs the task that it has been created for efficiently and without a lot of overhead. Certainly, the same thing can be done in software and development times might have been shorter, yet the real time aspect of the problem makes the details rather hard to get right.

The drum machine is rather limited, though, as it can only play 16th notes. I may be implementing a MIDI file player in VHDL in the future, which will be able to play tracks of arbitary complexity. The 16th structured grid of the drum machine will then only be implemented in software.

Turnkey application support

The drum machine actually works well and makes real noises, but getting the software into the running rekonstrukt system on the FPGA started to bother me. There were two ways of getting Forth words into the FPGA: One could either add them to the code that is cross compiled and placed in ROM, or one could upload the source code through the serial port onto the running FPGA. Putting random code into ROM, in addition to the limited amount of space available, has the down side that variables cannot be created in the usual Forth way, by just defining a word and leaving some space in the word's body free as storage for the variable value. Instead, variables in ROM need to be explictly allocated in RAM, which is at least cumbersome. Uploading through the serial port is not a viable option for deploying applications - After all, the device needs to work when switched on.

The easiest way to load data into the rekonstrukt system is to include it in the FPGA configuration bitstream as initialization vector for the FPGA. Block RAM that is used to implement both the ROM and the RAM of rekonstrukt, and block RAM can be initialized from the configuration bitstream.

The binary image for the ROM is created using cross compilation on the host system. The contents of the RAM blocks that are used for the rekonstrukt system RAM is normally not considered by Maisforth. During system startup, the dictionary pointer is normally initialized so that new words are created in RAM right above the 768 bytes that are reserved for system use.

In order to support loading application code into rekonstrukt with the FPGA bitstream, I modified the COLD word which is called to initialize the Forth system. COLD now calls the new DICT-FROM-RAM word which checks whether the RAM contains application code (consisting of dictionary entries in the Forth vocabulary) and, if so, includes these words in the dictionary of the starting Forth system. In addition, I implemented a new TURNKEY vectored word that is called upon system startup. The default TURNKEY action prints the version and copyright herald and then enters the text interpreter loop. Applications can implement whatever code in their turnkey word.

To create a FPGA configuration bitstream with RAM initialization vectors with application code, I modified the usim mc6809 simulator so that it can dump a core image of the running simulated Forth system to a file on the host. Dumping a core can be initiated from Forth. I have implemented hooks to the SWI2 (Software Interrupt 2) instruction so that the Forth can "call back" into the host, initiating the file dump. This process can be fully automated, so creating preinitialized FPGA bitstreams is possible from the Makefile in the forth/ subdirectory of the rekonstrukt source tree.

To sum up: It is now possible to create FPGA configuration bitstreams containing the hardware description, the Forth core and the application code and have the application run automatically after the bitstream has been loaded. I am still using block RAM for all of rekonstrukt's memory, so a lot of the block RAM resources of the FPGA are currently consumed. My next plan is to port rekonstrukt to the Avnet Spartan-3A Evaluation Kit. At $49, this board is cheap and Avnet also offers a 1 Megabyte SRAM add-on board, costing $20. I will try to implement a small boot loader that fits into 1 block RAM which initializes the external SRAM from either the serial or the parallel Flash ROM. That way, most of the block RAM will be available to the application.

Der Chaos Computer Club feiert seinen großartigen Erfolg!

Heute hat das Bundesverfassungsgericht seine Entscheidung zu der Klage zweier Bürger gegen den Einsatz von Wahlcomputern bei der Bundestagswahl 2005 bekanntgegeben. Dieser Einsatz war nicht rechtmäßig, und der CCC bejubelt dies nun als großartigen Erfolg seiner Kampagne. Über das Urteil selbst möchte ich mir als Nichtjurist keine größeren Gedanken machen, über das Gejubel des CCC jedoch schon:

Der CCC versteht sich ja als sowas wie das NGO der Computer-Experten. In der Öffentlichkeit wird er als kompetent, engagiert und politsch wachsam wahrgenommen. Wann immer es gilt, die angeblichen bürgerlichen Rechte von Menschen im Netz zu schützen, erklärt sich der CCC für zuständig. So auch hier: Ausgehend von einer Aktion holländischer Hacker wurde eine Kampagne gegen Wahlcomputer losgetreten: Wahlcomputer seien grundsätzlich schlecht, weil nicht transparent, und deswegen gehören sie verboten. In einigen Bundesländern ist das schon passiert, und jetzt scheint es auch im Bund bald so weit zu sein: Wahlen in Deutschland haben mit Stift und Zettel zu geschehen, denn alles andere, wir wissen es dank der fleißigen Hacker, ist "unsicher".

Als Konsequenz dürfen wir also in Zukunft nicht darauf hoffen, mehr direkte Demokratie mit Hilfe elektronischer Verfahren zu realisieren. Diskutieren, ja. Entscheiden? Nur mit Stimmzetteln. Und das haben wir nun ausgerechnet den Leuten zu verdanken, die am Liebsten an Ihren Computern herumhängen, sich ihr Leben mit dem Internet organisieren und sich mehr und mehr über die elektronischen Medien definieren.

Herzlichen Glückwunsch, lieber CCC!

P.S.: Weniger polemisch und fundierter hat Till Westermeyer seine Gedanken zu diesem Thema veröffentlicht. Danke dafür!