Monday, April 27, 2009

Javascript Everywhere - Using Axiom Stack

A friend of mine asked me to help her with creating a portfolio website for a friend of hers, who is an artist. The site will be somewhat similar to QuickHoney's web site: It will be Javascript based, relatively simple in terms of layout, and self maintainable, i.e. the artist will be able to add and update the website without the need for HTML programming.

The site will be hosted on a cheap, no-frills shared web host. On the server, it will consist of static files. The portfolio data set - Projects, descriptions, dates - will be stored in JSON formatted files. Manipulation of the data set will be done with a special Javascript application for content management.

The site will contain various media data: Photos, MP3 audio tracks, videos hosted by a video hoster.

Some parts of the content management tasks cannot be performed by Javascript in the browser:

  • Image manipulation, in particular rescaling.
  • Provide information about MP3 audio tracks
  • Fetching video information from the video hoster
  • Uploading data to the web hoster
These tasks require some program that runs outside of the browser; obviously, they are performed by a http server running on the artist's box.

My first impulse was to implement these functions in Common Lisp. Image manipulation is easy, using Edi Weitz' excellent CL-GD library. MP3 information should be easy enough to read using Peter Seibel's code from Practical Common Lisp. Parsing information from the video hoster would be easy, as that information is available both in XML and JSON formats, and Gilbert Baumann's and David Lichteblau's Closure XML Parser which I enjoy to use a lot. Lastly, uploading data to the web host should be possible using CL-FTP which I had not used before.

Exit Common Lisp

I spent a few evenings hacking CL-FTP so that it fits my library set and trying to find a sufficiently complete MP3 parsing library, played with dumping images so that startup times would be short. In between, I played with jQuery, HTML and CSS to get the user visible parts of the site into shape. Sadly, I made much better progress and had a lot more fun hacking Javascript than fooling around with Common Lisp and incomplete libraries. At one point, I decided that I need to make more progress and that I should reconsider my strategy for the CMS server.

I decided that my second choice for the CMS server language would be Javascript. I'd rather write both client and server in Common Lisp, but that is not how the world looks like.

Enter Javascript

I spent a few hours shopping for server-side Javascript solutions. The open source Javascript based web servers are either based on the SpiderMonkey or the Rhino Javascript implementations, both by the Mozilla foundation. SpiderMonkey is written in C, thus extending it requires libraries using the C calling convention. Rhino is written in Java, and it gives Javascript applications direct access to libraries written in Java. As I am developing on Windows, but the artist is using a Mac, Rhino looked a lot more attractive.

After looking at some project web pages picked from the Wikipedia entry on Server-side Javascript, I decided to give Axiom Stack a spin. I chose it because it is a real Javascript framework (using Prototypes, not mimicking classes), has easy-to-follow getting started documentation, provides for persistent objects similar to the BKNR datastore and has active support.

My application is not quite typical for applications implemented with Axiom Stack. Usually, the server would be used by the site's users, too. My server manipulates the static files that make up the web site, and is basically a slave to the CMS application that runs in the browser. Thus, I am not using Axiom Stack's persistent object store in this application.

Enslave Java

I have never developed a very friendly relationship to Java, despite the fact that when I chose to use it for one reason or another, I had no trouble with it. Java is great, as long as someone else writes code in it. Being able to call Java from Javascript is really nice, though, as there are so many libraries out there. In Rhino, Java libraries can be just used. Thus, as Axiom Stack has no "native" MP3 reading facilities, I grabbed a Java MP3 library off the net and used that:

importPackage(Packages.de.vdheide.mp3);

function sound_list () {
    var dir = '../../../files/sound/';

    return MochiKit.Base.map(
        function (filename) {
            var pathname = dir + filename;
            var mp3 = new MP3File(pathname);
            return { filename: filename,
                     title: id3.getTitle(),
                     length: mp3.getLength() }
        },
        (new axiom.SystemFile(dir)).list(/\.mp3$/i) || []);
}
The function importPackage can be used to import a Java package into the current name space. Java objects can be instantiated like Javascript object using the new operator. Strings and numbers returned by Java can be handled like any other Javascript object. I particularily like the fact that what you see above really is all code that is required to use a Java library. No interface generator or mapping layer is needed. At the same time, I can continue to use MochiKit, which is a Javascript library that I like for its nice functional programming abstractions.

Hacking away

Since I have solved my server problem, I have made great progress with the application. Switching between client-side and server side programming is really easy now. No more procastinating because the other side's mindset does not happen to be there. No more accidential typing of "the other" syntax.

Though I must say, I'm glad to have a Lisp job, too! :)

Monday, April 6, 2009

How to convert your Symbolics keyboard to USB

I spent another hour finishing up the Teensy based Symbolics keyboard converter. The software now works as well as the PS/2 version, and uses the same keyboard translation table generation tool. Converting a keyboard should now be doable for anyone who has basic soldering skills.
What you'll need:
  • Teensy (The photos show a build with the obsolete Teensy 1.0)
  • USB cable (B type)
  • 6-pin .1'' header
  • Some wire
First, you need to program the Teensy with the Firmware .hex file. Please note that this .HEX file is now for the Teensy 2.0 board, which also looks slightly different than the Teensy 1.0 board shown on the photos. Don't worry about customizing the firmware right now. If you decide that you want it to work differently than my version, you can update it later.
Get five pieces of wire, about 5 inches long. Solder them to the 6 pin header, leaving pin 2 unconnected. Connect the other ends to the Teensy.
pin # color teensy pad function
1 blue GND
2 (not connected)
3 green 5V
4 red D4 DIN
5 black D5 CLK
6 white D6< CLR

Yes, 5V and GND were swapped in my original post, sorry and thanks to Gene Diveglia for reporting
When building with a Teensy 2.0, connect the D4, D5, and D6 pins as printed on the Teensy PCB.

Open the Symbolics keyboard by removing the five screws at the bottom. Carefully remove the small PCB that sits on the larger one, unplug the spiral cable.  Using a tool like a large screwdriver, remove the strain relief by pushing it to the outside of the case. Remove the cable, take off the strain relief. Using a pliers, cut out some of the plastic so that the part can be used for the round USB cable.
Using a sharp knife, make the B type connector of the USB cable as small as possible.

Put the strain relief onto the USB cable, about 4 inches away from the scaled down B connector. Using a tool, firmly put it back into the keyboard case. Fixate the cable to the bottom shell of the case like shown. Connect the Teensy to the USB cable, plug the header into the keyboard PCB.
Put some protective plastic around the Teensy so that it does not short circuit anything on the keyboard PCB.
Finally, reassemble the keyboard. It should be easy to put the daughter PCB back onto the main PCB and fit the two shells together. You're done.
Make sure that you've read the README.txt file for the converter firmware. Note that if you want to convert your old-style keyboard, you need to know that the internal color coding of the cables is different with those keyboards. This blog post has a table with the information that you need. Have fun!

Sunday, April 5, 2009

Reimplementing the Symbolics Keyboard Adapter with Teensy


In a recent post on the AVR Freaks blog, Teensy, a new AVR based development board was announced. At $19, it is rather cheap and it comes with an on-chip USB controller. The neat thing about this is that applications are not restricted to some sort of serial emulation protocol, as Arduino-like boards with a separate, FTDI chip base USB controller, are. Instead, Teensy has a full USB 2.0 controller available to the application, and it comes with libraries that makes implementing custom USB keyboard controllers that require no special drivers very easy.
I was a little unsatisfied with the state of the kbdbabel based Symbolics keyboard adapter. The adapter works so far, but there are still some caps lock related bugs and I don't know 8051 assembler. Thus, I decided to give Teensy a first spin by implementing the protocol adapter in C, utilizing the Teensy HID USB keyboard library. As I did the protocol analysis for the Symbolics keyboard years ago, implementing that part was easy enough. I did make some mistakes with respect to accessing program memory on the AVR which took me several hours to sort out, but in the end, I now have a neat little program that runs on a Teensy and that requires no external circuitry to convert a Symbolics keyboard to a USB keyboard. Just solder on a few wires, flash the Teensy, replace the RJ11 cable by an USB cable and off you go.
I'll still support the kbdbabel based Symbolics adapter, but for users who want to hook up a Symbolics keyboard to a modern, USB equipped system, I now recommend getting a Teensy. I'm willing to do the conversion for those who don't have soldering skills, but the shipping will not be cheap. If you want the Teensy firmware, see my github repository.
Next on my list: Make Teensy talk MIDI. If you've done it already, please share what you know.