Javascript Everywhere - Using Axiom Stack

A friend of mine asked me to help her with creating a portfolio website for a friend of hers, who is an artist. The site will be somewhat similar to QuickHoney's web site: It will be Javascript based, relatively simple in terms of layout, and self maintainable, i.e. the artist will be able to add and update the website without the need for HTML programming.

The site will be hosted on a cheap, no-frills shared web host. On the server, it will consist of static files. The portfolio data set - Projects, descriptions, dates - will be stored in JSON formatted files. Manipulation of the data set will be done with a special Javascript application for content management.

The site will contain various media data: Photos, MP3 audio tracks, videos hosted by a video hoster.

Some parts of the content management tasks cannot be performed by Javascript in the browser:

• Image manipulation, in particular rescaling.
• Provide information about MP3 audio tracks
• Fetching video information from the video hoster
• Uploading data to the web hoster

These tasks require some program that runs outside of the browser; obviously, they are performed by a http server running on the artist's box.

My first impulse was to implement these functions in Common Lisp. Image manipulation is easy, using Edi Weitz' excellent CL-GD library. MP3 information should be easy enough to read using Peter Seibel's code from Practical Common Lisp. Parsing information from the video hoster would be easy, as that information is available both in XML and JSON formats, and Gilbert Baumann's and David Lichteblau's Closure XML Parser which I enjoy to use a lot. Lastly, uploading data to the web host should be possible using CL-FTP which I had not used before.

Exit Common Lisp

I spent a few evenings hacking CL-FTP so that it fits my library set and trying to find a sufficiently complete MP3 parsing library, played with dumping images so that startup times would be short. In between, I played with jQuery, HTML and CSS to get the user visible parts of the site into shape. Sadly, I made much better progress and had a lot more fun hacking Javascript than fooling around with Common Lisp and incomplete libraries. At one point, I decided that I need to make more progress and that I should reconsider my strategy for the CMS server.

I decided that my second choice for the CMS server language would be Javascript. I'd rather write both client and server in Common Lisp, but that is not how the world looks like.

Enter Javascript

I spent a few hours shopping for server-side Javascript solutions. The open source Javascript based web servers are either based on the SpiderMonkey or the Rhino Javascript implementations, both by the Mozilla foundation. SpiderMonkey is written in C, thus extending it requires libraries using the C calling convention. Rhino is written in Java, and it gives Javascript applications direct access to libraries written in Java. As I am developing on Windows, but the artist is using a Mac, Rhino looked a lot more attractive.

After looking at some project web pages picked from the Wikipedia entry on Server-side Javascript, I decided to give Axiom Stack a spin. I chose it because it is a real Javascript framework (using Prototypes, not mimicking classes), has easy-to-follow getting started documentation, provides for persistent objects similar to the BKNR datastore and has active support.

My application is not quite typical for applications implemented with Axiom Stack. Usually, the server would be used by the site's users, too. My server manipulates the static files that make up the web site, and is basically a slave to the CMS application that runs in the browser. Thus, I am not using Axiom Stack's persistent object store in this application.

Enslave Java

I have never developed a very friendly relationship to Java, despite the fact that when I chose to use it for one reason or another, I had no trouble with it. Java is great, as long as someone else writes code in it. Being able to call Java from Javascript is really nice, though, as there are so many libraries out there. In Rhino, Java libraries can be just used. Thus, as Axiom Stack has no "native" MP3 reading facilities, I grabbed a Java MP3 library off the net and used that:

importPackage(Packages.de.vdheide.mp3);

function sound_list () {
    var dir = '../../../files/sound/';

    return MochiKit.Base.map(
        function (filename) {
            var pathname = dir + filename;
            var mp3 = new MP3File(pathname);
            return { filename: filename,
                     title: id3.getTitle(),
                     length: mp3.getLength() }
        },
        (new axiom.SystemFile(dir)).list(/\.mp3$/i) || []);
}

The function importPackage can be used to import a Java package into the current name space. Java objects can be instantiated like Javascript object using the new operator. Strings and numbers returned by Java can be handled like any other Javascript object. I particularily like the fact that what you see above really is all code that is required to use a Java library. No interface generator or mapping layer is needed. At the same time, I can continue to use MochiKit, which is a Javascript library that I like for its nice functional programming abstractions.

Hacking away

Since I have solved my server problem, I have made great progress with the application. Switching between client-side and server side programming is really easy now. No more procastinating because the other side's mindset does not happen to be there. No more accidential typing of "the other" syntax.

Though I must say, I'm glad to have a Lisp job, too! :)

How to convert your Symbolics keyboard to USB

I spent another hour finishing up the Teensy based Symbolics keyboard converter. The software now works as well as the PS/2 version, and uses the same keyboard translation table generation tool. Converting a keyboard should now be doable for anyone who has basic soldering skills.

What you'll need:

• Teensy
• USB cable (B type)
• 6-pin .1'' header
• Some wire

First, you need to program the Teensy with the Firmware .hex file. Don't worry about customizing the firmware right now. If you decide that you want it to work differently than my version, you can update it later.

Get five pieces of wire, about 5 inches long. Solder them to the 6 pin header, leaving pin 2 unconnected. Connect the other ends to the teensy.

pin #	color	teensy pad	function
1	blue	GND
2		(not connected)
3	green	5V
4	red	D4	DIN
5	black	D5	CLK
6	white	D6<	CLR

Yes, 5V and GND were swapped in my original post, sorry and thanks to Gene Diveglia for reporting

Open the Symbolics keyboard by removing the five screws at the bottom. Carefully remove the small PCB that sits on the larger one, unplug the spiral cable.  Using a tool like a large screwdriver, remove the strain relief by pushing it to the outside of the case. Remove the cable, take off the strain relief. Using a pliers, cut out some of the plastic so that the part can be used for the round USB cable.

Using a sharp knife, make the B type connector of the USB cable as small as possible.

Put the strain relief onto the USB cable, about 4 inches away from the scaled down B connector. Using a tool, firmly put it back into the keyboard case. Fixate the cable to the bottom shell of the case like shown. Connect the Teensy to the USB cable, plug the header into the keyboard PCB.

Put some protective plastic around the Teensy so that it does not short circuit anything on the keyboard PCB.

Finally, reassemble the keyboard. It should be easy to put the daughter PCB back onto the main PCB and fit the two shells together. You're done.

Make sure that you've read the README.txt file for the converter firmware. Have fun!

Reimplementing the Symbolics Keyboard Adapter with Teensy

In a recent post on the AVR Freaks blog, Teensy, a new AVR based development board was announced. At $19, it is rather cheap and it comes with an on-chip USB controller. The neat thing about this is that applications are not restricted to some sort of serial emulation protocol, as Arduino-like boards with a separate, FTDI chip base USB controller, are. Instead, Teensy has a full USB 2.0 controller available to the application, and it comes with libraries that makes implementing custom USB keyboard controllers that require no special drivers very easy.

I was a little unsatisfied with the state of the kbdbabel based Symbolics keyboard adapter. The adapter works so far, but there are still some caps lock related bugs and I don't know 8051 assembler. Thus, I decided to give Teensy a first spin by implementing the protocol adapter in C, utilizing the Teensy HID USB keyboard library. As I did the protocol analysis for the Symbolics keyboard years ago, implementing that part was easy enough. I did make some mistakes with respect to accessing program memory on the AVR which took me several hours to sort out, but in the end, I now have a neat little program that runs on a Teensy and that requires no external circuitry to convert a Symbolics keyboard to a USB keyboard. Just solder on a few wires, flash the Teensy, replace the RJ11 cable by an USB cable and off you go.

I'll still support the kbdbabel based Symbolics adapter, but for users who want to hook up a Symbolics keyboard to a modern, USB equipped system, I now recommend getting a Teensy. I'm willing to do the conversion for those who don't have soldering skills, but the shipping will not be cheap. If you want the Teensy firmware, let me know.

Next on my list: Make Teensy talk MIDI. If you've done it already, please share what you know.

Rekonstrukt Progress: MIDI Drum Machine, Turnkey Application Support

I made quite some progress on rekonstrukt during the last few weeks, and I can happily report that this whole "build a system from scratch" has not turned into a full time occupation, but is very doable as an after work thing. I'd certainly be interested in lifting the effort to a more, say, professional scale in the future, but for the time being, it is nice to both have something to play with in the evening and still maintain a family and a professional life.

Implementing a MIDI drum machine in hardware

One of the application areas that I am currently interested in is music hardware. I'm into electronic music, and even though I am not much of an artist (that'd be hardly compatible with a family and a professional life), I like playing around with and building gear. For a start, I created a drum machine in VHDL. It is a very simple 8 track, 16 beats drum machine, but it runs fully autonomous and requires no CPU, other than for setting up the pattern, tempo and notes. Suffice to say, the drum machine is clocked tightly. There is no interrupt latency to consider, and the CPU is free for the user interface.

This is an interesting experiment in that what I have basically done is create a special purpose processor that plays rythms. This processor is very restricted, but it performs the task that it has been created for efficiently and without a lot of overhead. Certainly, the same thing can be done in software and development times might have been shorter, yet the real time aspect of the problem makes the details rather hard to get right.

The drum machine is rather limited, though, as it can only play 16th notes. I may be implementing a MIDI file player in VHDL in the future, which will be able to play tracks of arbitary complexity. The 16th structured grid of the drum machine will then only be implemented in software.

Turnkey application support

The drum machine actually works well and makes real noises, but getting the software into the running rekonstrukt system on the FPGA started to bother me. There were two ways of getting Forth words into the FPGA: One could either add them to the code that is cross compiled and placed in ROM, or one could upload the source code through the serial port onto the running FPGA. Putting random code into ROM, in addition to the limited amount of space available, has the down side that variables cannot be created in the usual Forth way, by just defining a word and leaving some space in the word's body free as storage for the variable value. Instead, variables in ROM need to be explictly allocated in RAM, which is at least cumbersome. Uploading through the serial port is not a viable option for deploying applications - After all, the device needs to work when switched on.

The easiest way to load data into the rekonstrukt system is to include it in the FPGA configuration bitstream as initialization vector for the FPGA. Block RAM that is used to implement both the ROM and the RAM of rekonstrukt, and block RAM can be initialized from the configuration bitstream.

The binary image for the ROM is created using cross compilation on the host system. The contents of the RAM blocks that are used for the rekonstrukt system RAM is normally not considered by Maisforth. During system startup, the dictionary pointer is normally initialized so that new words are created in RAM right above the 768 bytes that are reserved for system use.

In order to support loading application code into rekonstrukt with the FPGA bitstream, I modified the COLD word which is called to initialize the Forth system. COLD now calls the new DICT-FROM-RAM word which checks whether the RAM contains application code (consisting of dictionary entries in the Forth vocabulary) and, if so, includes these words in the dictionary of the starting Forth system. In addition, I implemented a new TURNKEY vectored word that is called upon system startup. The default TURNKEY action prints the version and copyright herald and then enters the text interpreter loop. Applications can implement whatever code in their turnkey word.

To create a FPGA configuration bitstream with RAM initialization vectors with application code, I modified the usim mc6809 simulator so that it can dump a core image of the running simulated Forth system to a file on the host. Dumping a core can be initiated from Forth. I have implemented hooks to the SWI2 (Software Interrupt 2) instruction so that the Forth can "call back" into the host, initiating the file dump. This process can be fully automated, so creating preinitialized FPGA bitstreams is possible from the Makefile in the forth/ subdirectory of the rekonstrukt source tree.

To sum up: It is now possible to create FPGA configuration bitstreams containing the hardware description, the Forth core and the application code and have the application run automatically after the bitstream has been loaded. I am still using block RAM for all of rekonstrukt's memory, so a lot of the block RAM resources of the FPGA are currently consumed. My next plan is to port rekonstrukt to the Avnet Spartan-3A Evaluation Kit. At $49, this board is cheap and Avnet also offers a 1 Megabyte SRAM add-on board, costing $20. I will try to implement a small boot loader that fits into 1 block RAM which initializes the external SRAM from either the serial or the parallel Flash ROM. That way, most of the block RAM will be available to the application.

Der Chaos Computer Club feiert seinen großartigen Erfolg!

Heute hat das Bundesverfassungsgericht seine Entscheidung zu der Klage zweier Bürger gegen den Einsatz von Wahlcomputern bei der Bundestagswahl 2005 bekanntgegeben. Dieser Einsatz war nicht rechtmäßig, und der CCC bejubelt dies nun als großartigen Erfolg seiner Kampagne. Über das Urteil selbst möchte ich mir als Nichtjurist keine größeren Gedanken machen, über das Gejubel des CCC jedoch schon:

Der CCC versteht sich ja als sowas wie das NGO der Computer-Experten. In der Öffentlichkeit wird er als kompetent, engagiert und politsch wachsam wahrgenommen. Wann immer es gilt, die angeblichen bürgerlichen Rechte von Menschen im Netz zu schützen, erklärt sich der CCC für zuständig. So auch hier: Ausgehend von einer Aktion holländischer Hacker wurde eine Kampagne gegen Wahlcomputer losgetreten: Wahlcomputer seien grundsätzlich schlecht, weil nicht transparent, und deswegen gehören sie verboten. In einigen Bundesländern ist das schon passiert, und jetzt scheint es auch im Bund bald so weit zu sein: Wahlen in Deutschland haben mit Stift und Zettel zu geschehen, denn alles andere, wir wissen es dank der fleißigen Hacker, ist "unsicher".

Als Konsequenz dürfen wir also in Zukunft nicht darauf hoffen, mehr direkte Demokratie mit Hilfe elektronischer Verfahren zu realisieren. Diskutieren, ja. Entscheiden? Nur mit Stimmzetteln. Und das haben wir nun ausgerechnet den Leuten zu verdanken, die am Liebsten an Ihren Computern herumhängen, sich ihr Leben mit dem Internet organisieren und sich mehr und mehr über die elektronischen Medien definieren.

Herzlichen Glückwunsch, lieber CCC!

P.S.: Weniger polemisch und fundierter hat Till Westermeyer seine Gedanken zu diesem Thema veröffentlicht. Danke dafür!

Hardware/Software Co-Design - What it means with rekonstrukt

One of my primary goals with the rekonstrukt project is to create a hardware/software co-design environment that cuts down on turnaround times and gives me, the designer, flexibility to test and try out things at various levels. Here is what this means in practice:

rekonstrukt consists of the MC6809 microprocessor running Maisforth. The microprocessor can be implemented in three ways:

• In the usim software emulator, which is written in C++ and runs on my workstation. In this mode, the emulator needs be extended in C++ if I want to emulate more of the hardware that the real implementation provides. This mode has fast turnaround times (the simulated 6809 is way faster than the real hardware) and is useful to try out algorithms, learn Forth and the like. It is not timing accurate.
• In the Active-HDL simulator, again running on my workstation. In this mode, the actual VHDL implementation of rekonstrukt is simulated. The simulation is still fast enough to run small pieces of actual Forth code. It is a good mode to pinpoint hardware and timing issues, and Active-HDL has great flexibility with respect to setting breakpoints and inspecting the simulated state, which helps a great deal with diagnosing issues.
• On the FPGA. The interactive nature of Forth means that this way is often most productive. By actually trying out things, I can quickly gain confidence in my design or explore ideas, getting immediate feedback. In this mode, the logic analyzer is the crucial tool to verify waveforms and timings, and it is not that suitable for pinpointing problems that are inside the FPGA.

In all three modes, it is possible to recompile the Forth kernel and put it into operation in a few seconds. The usim simulator directly works with the ROM image that the cross compiler generates, the VHDL simulator can load a pre-initialized ROM image in VHDL format, and the new ROM image can also be merged with a completely synthesized FPGA bit stream. Thus, synthesizing the hardware is required only when changes need to be tried out on the real hardware. Trying out firmware changes only requires running the Forth cross compiler and restarting the simulator or downloading the bitstream to the target hardware.

Synthesizing rekonstrukt on my workstation takes about 10 minutes, but as the design is split into partitions, changes to individual parts of the hardware does not require a full synthesis run. Incremental runs take only about 3 minutes. Sure, I'd love to see that be faster, but it is bearable

I love this shit :)

ACIA fixed, S3E SK sucks

In the last two weeks, I was making good progress with my attempt to fully support the Spartan-3E starter kit by Maisforth. I had to improve my SPI controller slightly so that it works better for programming the serial Flash, worked on implementing the (ancient) standard Block I/O vocabulary so that the serial Flash can be used like a floppy disk in the old days and almost got the beautiful vibe full screen block editor to work.

One thing that was began to bother me a lot was the development cycle for Forth programs: I use Emacs to edit stuff, then copy and paste it to the real hardware. As my programs grow, this begins to take considerable time, but for some reason I could not send the data through the serial port at full speed: Maisforth reproduceably lost characters, and I had to insert a small delay after each character sent which made uploads slow. Too slow.

Thus, I went to pinpoint the reason for this character loss. At first, I thought that what I saw was a problem related to the lack of flow control. The serial port on the Spartan-3E Starter Kit is only equipped with RX and TX signals, so no hardware flow control is possible and software flow control is not implemented. After some investigation and experiments it became clear to me, though, that the problem was not flow control related. Both hardware (RTS/CTS) and software (XON/XOFF) flow control work under the assumption that data is buffered in larger chunks, but my character loss happened even when I sent short bursts at full speed to the FPGA.

The problem really was related to the lack of double buffering in the transmitter of the System09 UART. This meant that the transmitter signalled the "transmit buffer full" condition for the whole duration of the actual send operation, even though the data had already been copied from the transmit buffer to the shift register. As the serial port echo in Maisforth is generated by Forth itself, processing of every character and sending it back out through the serial port introduced a pause between every character echoed, which quickly caused problems when characters to be echoed came in at full serial port speed.

The fix for the problem was to introduce double buffering. As soon as the character to be transmitted has been copied over to the transmitter shift register, the UART module now signals the host that the transmit buffer is available for the next character to be sent. This made the problem disappear as Forth echo processing can now take place while the previous character echo is transmitted. Given the slow serial port and the high clock rate of System09, there now is plenty of time for all processing needed.

The Spartan-3E Starter Kit sucks

Well, not really. It is a nice cheap board with a good load of on-board peripherals, useful documentation and nice example designs, but it also has severe limitations: The on-board FPGA is a XC3S500E, which has 40 kByte of block RAM. In my current Maisforth setup, I use 16k for the Forth kernel ROM, 16k for RAM, and 6k for the VGA controller. This leaves only 2k free, so that makes playing with table based waveform synthesis really hard. What I hoped for was that I could put the Forth kernel and the board support Forth library into the parallel NOR flash of the S3ESK and map the Flash into the 6809 address space.

The show stopper for this plan is the fact that data bit 0 of the NOR flash is shared with the SPI bus data signal. Thus, one can either access the flash or the SPI, but not both at the same time. Seemingly, the idea is to copy the contents of the NOR flash into SDRAM and then run from there, but I'm still reluctant to try getting the SDRAM to work. The easy path is blocked, though, and maybe using SDRAM is the best option.