Embracing the ARM

Last year, I worked on a project for visomat inc. which involved interfacing to a LED matrix display so that it could display some video. We had some latency problems and at one stage we decided to try out DMX instead of Ethernet as interface techniquce, hoping that we'd get better results that way. The idea was to feed the LED matrix, which consisted of 12 power supplies/controllers, with 12 DMX interfaces to make sure that data arrives at the LEDs synchronously.

DMX is an asynchronous serial protocol running at 250 kbits per second. I decided to try using an AVR microcontroller running at 16 Mhz and have it bit-bang the data out of a parallel port. For each bit, 64 machine cycles would be available which should be enough to set the port bits and also grab incoming data out of the USB interface that we wanted to use to interface to the host.

I built a prototype of the interface and we made some promising experiments with it, but in the end we found that the visual latency problems were not fixed. The problem was really in how the LED hardware worked and could not be repaired on the feeding side. The prototype went onto the shelf and we stuck with Ethernet for that project.

A few weeks ago, I discussed the project with a friend of mine and we thought that it'd be cool to turn the 16 port DMX interface into a product. We found someone who would want to use it in an OEM setting, and I made bringing the hardware to product level my weekend project.

Good bye AVR, hello ARM

The AVR based solution proved to be problematic, though. Even though the CPU has enough processing headroom, the lack of RAM requires that the data is fed into the interface "just in time". Any variance in the USB input data stream could result in visible hickups in the DMX streams. Getting it to work seemed like possible, but require too much work in the end. Thus, I decided to turn to using more capable hardware which would also give the DMX interface network connectivity and more local intelligence.

As new platform, I chose the ARM9 based Eddy-S4M board produced by the Korean manufacturer SystemBase. These boards are distributed by trenz electronic who I've found to be quick and reliable in the past.

Linux is the standard operating system on Eddy-S4M. I'm not a huge Linux fan, so I spent some time to find an alternative operating system, but neither eCos, nor FreeRTOS or RTEMS seemed to be ported to the AT91SAM9260 CPU with a free development system. As I was into application development, I decided not to port or write an operating system myself. Running on the bare hardware was not an option either as I need a working TCP stack in order to interface to the host.

SystemBase calls their Linux port "lemonix". It appears to be a relatively plain linux-2.6.20 with real-time patches, BusyBox and a bunch of C applications to control startup and configuration. The userland and kernel source code is available from SystemBase after registration, but the kernel tree that they make available is incomplete and needs to be augmented by some files that are available from a Google group so that it can be built. Getting the cross compilation environment to run on my Ubuntu 9.10 box worked flawlessly, and after removing the whole Eddy application stuff and most of the SystemBase drivers from the file tree I ended up with a relatively sane Linux that I could cross compile and install on the Eddy hardware through tftp.

Making it fast

Back to the DMX interface, I was now faced with a system that gave me about 400 machine cycles for each DMX bit sent. That should be plenty of time to get out some bits to the ports. The bit timing would be achived by a timer interrupt every 4 microseconds. Even with interrupt handling overhead, this should leave enough headroom for TCP processing.

The ARM architecture specifies two interrupts. The normal, vectored interrupt system is used by Linux and mapped to its own, portable interrupt architecture, ignoring the vectoring facilities that the hardware provides. The second, fast interrupt system (FIQ) that the ARM provides is not used by Linux, and it seemed like a good fit to my requirements. The FIQ is interesting because when the FIQ handler is entered, the CPU automatically switches six registers to a distinct bank. These six registers can then just be used without the need to save or restore them to the stack. Even though the FIQ is not used by Linux itself, ARM-Linux provides for an interface so that drivers can use it. FIQ support was missing in lemonix, but it was trivial to backport.

Writing a FIQ interrupt in ARM assembly was straightforward. The driver bottom half set up the FIQ register set and enabled the timer interrupt, the FIQ handler set the port bits and incremented the pointers accordingly.

Reducing jitter

When looking at the output ports with the logic analyzer, though, I could see variations in bit edges in the order of one microsecond, which is beyond what the DMX receivers would be able to tolerate. These variations were visible even when the FIQ handler just set the port bits without writing any real data. The source for this jitter, as it turned out, was virtual address translation. The ARM9 CPU includes a MMU and all software, including interrupt handlers, uses virtual addressing, even to access I/O ports. The virtual address maps are stored in SDRAM due to their size, and the hardware automatically traverses these maps if an address can not be found in the translation lookaside buffers in the MMU. So in the FIQ handler, when the output port address was not present in the TLB, the MMU would access the SDRAM, and SDRAM random access is rather slow.

Thus, I had to change the FIQ handler so that it never accesses the SDRAM, either directly (by reading the data buffer) or indirectly (by causing TLB misses). The AT91SAM9260 CPU has two internal SRAMs that are accessible in two machine cycles, and I use one of those as a data buffer. In order to prevent TLB misses, indivdual TLB entries can be locked down so that they're never removed by the MMU automatically. Thus, my driver locks down the I/O and the SRAM buffer addresses that the FIQ handler accesses.

Reducing jitter even more

Even with the TLBs locked down, I still saw some jitter in the leading edges of the DMX bytes. The cause for this were variations in interrupt response times, as that depends on the instruction being interrupted. The variations were below one microsecond, but that was long enough to bring the serial decoding routine in my logic analyzer out of sync. Wanting to play it safe, I decided to remove that jitter as well by synchronizing on the timer value inside of the FIQ handler: Instead of just banging out the next bit as fast as possible, the FIQ handler now waits until the free-running timer that triggered the interrupt reaches a certain value. That way, the effective FIQ response time is made mostly constant.

Wow, 2010!

In the last few years, I have been using various AVR CPUs for my embedded projects. It is a great CPU that is easy to use and fast, and the advent of LUFA and Teensy made my life a lot easier, as I was freed from USB serial dongles and driver installation. But then, being able to process serious amounts of data is nice, too, and this is were 8 bits just don't suffice. ARM based boards are cheap nowadays, and I'm looking forward to embedding JavaScript or maybe even Common Lisp in one of my future projects. When speed is needed, I can always fall back to C and assembler.

Hacking a USB Infrared receiver

Our living room Hi-Fi setup needs a MP3 player that is always available, easy to use and, first and foremost, allows us to import all the CDs that we have easily and without effort. I've always wanted to do that using our FreeBSD based home server, but none of the ripping solutions really worked the way I wanted and triggered my desire to hack them to my liking. Finally, I gave in and pulled out the aging and unused PowerBook of my wife. After all, iTunes matches my requirements easily, and as I can treat is as piece of Stereo equipment, I'm willing to put my dislike for Apple Computers to the side.

The problem with the PowerBook is that it does not have an IR receiver. We have a Logitech Harmony Remote 515 that we use to control the rest of our entertainment stuff, and we'd like to control the MP3 player with it, too. I could not find a cheap, ready-to-go solution easily, so I decided to throw together something myself using a Teensy and a Vishay TSOP1836 IR receiver module that I found on a surplus board that I found in a box on my shelf.

The goal was to create an IR receiver that acts as a USB keyboard. The key codes that are sent should be useable to set up some global keyboard shortcuts in MacOS X that would control iTunes to our liking.

Hardware

The TSOP1836 IR receiver module does the IR carrier decoding and interfaces easily with the AVR using one input port. I hooked up my Logicport logic analyzer (excellent product, I write this as a satisfied customer) to the output of the IR receiver and tried a few remotes that I had to see whether the 36 kHz that that the receiver assumes as the carrier frequency would be common. I was lucky, both the remote of our Onkyo Amp and the remote control that came with the Altium Nanoboard 3000 transmitted at that carrier frequency, and both of them used the NEC protocol.

Teensy is a microcontroller board based on an ATMEL AT90USB162 AVR chip. It comes in a DIL24 form factor, so it is easy to integrate into breadboards. A neat thing about it is the flashing utility that looks good, works very well and is unintrusive. The compile/upload/test cycle with the Teensy loader is very short.

Software

I have used LUFA as USB stack on Teensy in the past to implement a adapter for Symbolics keyboards, and I decided to use it again. I was pleased to find that LUFA has made great progress since I last used it. There are more examples, the class drivers have been separated from the demo code, and the cheapish task scheduler is not longer used by the examples that I used.

I tried to make the USB implementation of the adapter offer two end points: A keyboard endpoint that sends the translated key codes when remote keys are pressed, and a serial or CDC endpoint that would be used to configure the adapter in order to make it possible to configure it to whatever remote is available. Unfortunately, I found that MacOS X does not accept devices with multiple endpoints as standard serial/CDC device, so I abandoned the idea of configurability. Instead, I made the adapter work with the Altium remote and a statically compiled translation table.

To find out the codes that the remote sends, I wrote a version of the adapter that acts as a serial device. Using that information, I set up the translation table in the HID keyboard version of the firmware.

When I tried setting up some global keyboard shortcuts for iTunes, I learned that iTunes does not really support global short cut keys, presumably to "protect the user". Using a the free SizzlingKeys utility, I was able to achieve what I wanted, though.

The source code for the two Teensy firmwares is available in my github repository.

Clojure Slides from February Berlin Lispers Meetup

Stefan Richter, our speaker at the February Berlin Lispers Meetup, has made the slides of his presentation available on line. Enjoy, and see you next week!

Telekom VDSL mit FreeBSD

Da ich meinen DSL-Anschluß im Büro mit ein paar anderen Freelancern teile, habe ich kürzlich von vorher Congster ADSL zu Telekom VDSL-25 gewechselt. Als Router verwende ich nun einen Rechner mit FreeBSD-7.2, der VDSL-Anschluß wird mit einem Modem Speedport 300HS realisiert. Da der VDSL-Uplink auf einem gesonderten VLAN aus dem Modem kommt, ist eine zweite Netzwerkkarte nicht zwingend erforderlich. Ich habe das Modem einfach ans LAN gehängt und auf dem FreeBSD-Router ein VLAN-Interface konfiguriert, das für den Internetzugang mit PPPoE verwendet wird.

Konfiguriert ist das ganze wie folgt

Änderungen in /etc/rc.conf

# VLAN-Interface konfigurieren, bge0 ist mein Ethernet-Interface. Ggf. anpassen.
cloned_interfaces="vlan0"
ifconfig_vlan0="vlan 7 vlandev bge0"
# IP-Paketforwarding anschalten
gateway_enable="YES"
# PPP-Client automatisch starten
ppp_enable="YES"
ppp_mode="ddial"
ppp_nat="YES"
ppp_profile="telekom"

Eintrag in /etc/ppp/ppp.conf

telekom:
        set device PPPoE:vlan0
        set authname AAAAAAAAAAAATTTTTTTTTTTT0001@t-online.de
        set authkey KKKKKKKK
        set dial
        set login
        set log phase tun command
        set ifaddr 10.0.0.1/0 10.0.0.2/0
        add default HISADDR

Wie üblich ist AAAAAAAAAAAA durch die Telekom-Anschlusskennung, TTTTTTTTTTTT durch die T-Online-Kennung und KKKKKKKK durch das Kennwort zu ersetzen. (Könnte die Telekom bitte irgendwann mal die BTX-Vergangenheit tief unter der Erde verbuddeln und die Autorisierung auf ein benutzerfreundliches Verfahren umstellen? Danke!)

Das war's. Reboot und die FreeBSD-Box routet.

Max for Live, Java, MIDI, Windows and Sysex

Recently, I have spent some of my free time hacking some Java to provide Max for Live patchers direct access to the MIDI ports of my machine. Usually, one would be restricted to the MIDI data that Live has previously processed, which restricts it in many ways.

I have not spent too much time hacking Java in my life, but the times I have done it for various smaller hacks, it was a nice overall experience. This time, I was particularily impressed by Eclipse for its automated source code formatting facility that is widely configurable. I tweaked the settings a little to match my prefered squirly brace style, and then had Eclipse beautify my (and imported) source code with a single key stroke. Also, I like Eclipse's ability to automatically locate unknown classes and add required import statements with a single click. This makes working with examples from documentation rather easy. I still like Emacs, but I must admit that it is rather baroque when compared to what Eclipse can do.

Getting the Max external to interface to the MIDI subsystem that Java provides was rather easy, and I was mostly done after 2-3 hours of hacking. Sadly, sending Sysex strings to the MIDI controller that I used for testing did not work reliably. Naturally, I thought that the problem was with my lack of MIDI, Java and Max skills, so I spent quite some time trying out various implementation strategies, without success.

Finally, I figured that what did not work was sending Sysex messages, and only sometimes (i.e. it appeared as if the first message came through and then some following messages where not transmitted). Google finally pointed me to a web page by Thorsten Klose where, at the end, he describes the behavior as being a bug in the Java MIDI library. The problem is that when sending Sysex messages, the Java library remembers the size of largest Sysex message sent and uses that for all subsequent Sysex messages, even if they are shorter. My workaround is to close and re-open the MIDI device if a Sysex string to be sent is shorter than the previous one. This is kind of ugly, but it works for me. The workaround that Thorsten suggest has the problem that it does not stop Java from sending long messages, which reduces MIDI throughput to a point that was unacceptable for me.

I'm posting this in the hopes that it saves the next person seeing the problem some time. If you are interested in the Max for Live MIDI external, it is available here.

Admittedly, I like JavaScript

Call me pervert, but I find writing JavaScript enjoyable. The aspect that I like most about it is its dynamicism. Certainly, the syntax is ugly, but the js2-mode for Emacs makes writing JavaScript much less painful.

My addiction for JavaScript made me accept a project to create an artists' portfolio website that has no commercial value, but gave me lots of opportunity to play around with client-side coding. I decided that I want to put all data into a JSON encoded data structure and use a client-side content management system to edit and update the database. I originally intended to write the required server-side components for the CMS in JavaScript, too, but that idea went down the drain as it made deployment of the whole thing a lot harder. Thus, the server-side components are now reduced to the bare minimum and are written in perl, so that a cheap web host can be used both for deploying both the web site and the CMS.

One common problem with client-side JavaScript is flow control with long-running operations. JavaScript is single threaded, so in order to maintain UI responsiveness, functions may not block waiting for network responses. Thus, whenever a UI wants to interact with the server and then resume operating, things get tricky. There are libraries like jwacs, which is a preprocessor which implements first-class continuations, and Arrows, which is a "generalization of Monads", but these either make debugging hard through the use of a preprocessor or are incomplete.

My approach to the problem is to use closures to manually establish the continuation, and callbacks to invoke these continuations when appropriate. While being dissatisfying in the intellectual sense, the approach works for the applications that I'm using them in.

Sure, I'd rather have macros instead of having to control evaluation through closures. But then, this is still better than any old static language that requires me to cram everything into classes and objects, forcing me to either make all design changes upfront or to use sophisticated refactoring wizardry to get dynamicism on the source code level. Oh well, or maybe I should rather give up on Emacs :)

Javascript Everywhere - Using Axiom Stack

A friend of mine asked me to help her with creating a portfolio website for a friend of hers, who is an artist. The site will be somewhat similar to QuickHoney's web site: It will be Javascript based, relatively simple in terms of layout, and self maintainable, i.e. the artist will be able to add and update the website without the need for HTML programming.

The site will be hosted on a cheap, no-frills shared web host. On the server, it will consist of static files. The portfolio data set - Projects, descriptions, dates - will be stored in JSON formatted files. Manipulation of the data set will be done with a special Javascript application for content management.

The site will contain various media data: Photos, MP3 audio tracks, videos hosted by a video hoster.

Some parts of the content management tasks cannot be performed by Javascript in the browser:

• Image manipulation, in particular rescaling.
• Provide information about MP3 audio tracks
• Fetching video information from the video hoster
• Uploading data to the web hoster

These tasks require some program that runs outside of the browser; obviously, they are performed by a http server running on the artist's box.

My first impulse was to implement these functions in Common Lisp. Image manipulation is easy, using Edi Weitz' excellent CL-GD library. MP3 information should be easy enough to read using Peter Seibel's code from Practical Common Lisp. Parsing information from the video hoster would be easy, as that information is available both in XML and JSON formats, and Gilbert Baumann's and David Lichteblau's Closure XML Parser which I enjoy to use a lot. Lastly, uploading data to the web host should be possible using CL-FTP which I had not used before.

Exit Common Lisp

I spent a few evenings hacking CL-FTP so that it fits my library set and trying to find a sufficiently complete MP3 parsing library, played with dumping images so that startup times would be short. In between, I played with jQuery, HTML and CSS to get the user visible parts of the site into shape. Sadly, I made much better progress and had a lot more fun hacking Javascript than fooling around with Common Lisp and incomplete libraries. At one point, I decided that I need to make more progress and that I should reconsider my strategy for the CMS server.

I decided that my second choice for the CMS server language would be Javascript. I'd rather write both client and server in Common Lisp, but that is not how the world looks like.

Enter Javascript

I spent a few hours shopping for server-side Javascript solutions. The open source Javascript based web servers are either based on the SpiderMonkey or the Rhino Javascript implementations, both by the Mozilla foundation. SpiderMonkey is written in C, thus extending it requires libraries using the C calling convention. Rhino is written in Java, and it gives Javascript applications direct access to libraries written in Java. As I am developing on Windows, but the artist is using a Mac, Rhino looked a lot more attractive.

After looking at some project web pages picked from the Wikipedia entry on Server-side Javascript, I decided to give Axiom Stack a spin. I chose it because it is a real Javascript framework (using Prototypes, not mimicking classes), has easy-to-follow getting started documentation, provides for persistent objects similar to the BKNR datastore and has active support.

My application is not quite typical for applications implemented with Axiom Stack. Usually, the server would be used by the site's users, too. My server manipulates the static files that make up the web site, and is basically a slave to the CMS application that runs in the browser. Thus, I am not using Axiom Stack's persistent object store in this application.

Enslave Java

I have never developed a very friendly relationship to Java, despite the fact that when I chose to use it for one reason or another, I had no trouble with it. Java is great, as long as someone else writes code in it. Being able to call Java from Javascript is really nice, though, as there are so many libraries out there. In Rhino, Java libraries can be just used. Thus, as Axiom Stack has no "native" MP3 reading facilities, I grabbed a Java MP3 library off the net and used that:

importPackage(Packages.de.vdheide.mp3);

function sound_list () {
    var dir = '../../../files/sound/';

    return MochiKit.Base.map(
        function (filename) {
            var pathname = dir + filename;
            var mp3 = new MP3File(pathname);
            return { filename: filename,
                     title: id3.getTitle(),
                     length: mp3.getLength() }
        },
        (new axiom.SystemFile(dir)).list(/\.mp3$/i) || []);
}

The function importPackage can be used to import a Java package into the current name space. Java objects can be instantiated like Javascript object using the new operator. Strings and numbers returned by Java can be handled like any other Javascript object. I particularily like the fact that what you see above really is all code that is required to use a Java library. No interface generator or mapping layer is needed. At the same time, I can continue to use MochiKit, which is a Javascript library that I like for its nice functional programming abstractions.

Hacking away

Since I have solved my server problem, I have made great progress with the application. Switching between client-side and server side programming is really easy now. No more procastinating because the other side's mindset does not happen to be there. No more accidential typing of "the other" syntax.

Though I must say, I'm glad to have a Lisp job, too! :)