Raumpflege

Sollte jemand sich in Berlin mit dem Gedanken tragen, das Unternehmen "Bärliner Putzteufel" beauftragen zu wollen, so seid gewarnt: Die Leistungsbereitschaft der Mitarbeiter ist gering, aber im Stellen hoher Rechnungen ist der Inhaber Volker Brunemann sehr beflissen. Von Freundlichkeit und Entgegenkommen bei Kundenunzufriedenheit kann man nur träumen. Ich kann zwar kein anderes Reinigungsunternehmen empfehlen, aber die "Bärliner Putzteufel" sind es sicher nicht. Ja, klar. Schon der Name hätte mich skeptisch machen sollen.

Postbank saugt

Ich bin Business-Kunde bei der Postbank, und auf meiner letzten Monatsabrechnung fand ich einen Posten "Zinsen und Gebühren" in Höhe von 66 Euro. Ich dachte mir: "Das ist ja viel", und schrieb eine (autorisierte und so weiter) Email an die Postbank mit der Bitte, mir diesen Posten doch mal aufzuschlüsseln. Als ich nach 10 Tagen noch keine Antwort hatte, habe ich mal nachgefragt. Einen Tag später erhielt ich die Antwort, daß meine Frage nicht so ohne weiteres beantwortet werden könne, und sie daher an die "Fachabteilung" zur Bearbeitung weiter geleitet wurde. Ich glaube, ich brauche ganz schnell eine andere Bank.

Quicklisp - The upcoming solution to Common Lisp's "library problem"

We're all tired of hearing that Common Lisp is all nice, but it has a library problem. True, many of the open source libraries written in Common Lisp are half-baked, incomplete, unmaintained and tasteless. But this is true with libraries in all languages, so that can't be the real problem. The real problem, as I see it, is that until now, there was no universally good way of installing a library and all of its dependencies.

The existing, widely deployed solutions (asdf-install and clbuild) require external tools to work, depend on the availability of diverse servers in the internet, and do not have something like a central maintainer who ensures that things are always in basically working order. Also, these solutions are rather unportable, as they depend on external tools and a Unixish environment which is not available everywhere.

My personal solution to this problem, until now, was to have a Subversion repository that contains all the libraries that any of my projects need. Some of these came from release tarballs, some from other revision control repositories, and maintenance was manual. Whenever I started a new project, I brought some of those libraries up to date, coped with any (new) dependencies, lost backwards compatibility with my older projects etc. This kind of sucked, but it had the beauty that both development and deployment ended up being relatively easy. All that was required was a svn checkout from my repository, and I was all set on a new machine.

Recently, Zach Beane was fed up enough to do something about this: He created Quicklisp, a self-contained, centrally managed, cloud hosted Lisp library system which aims to run everywhere and provide users with a one-stop solution for the "library problem". Quicklisp is in an early alpha stage, but having tried it, I must say that I am pleased and impressed. Finally, Common Lisp can also become a glue language like Python and Perl.

I tried Quicklisp today because Twitter has notified me that basic http authentication for applications will no longer be supported. Instead, one is now supposed to use OAuth to authenticate requests sent to Twitter. This required my reaction.

About two years ago, I have written a small gateway program that forwards new postings to Planet Lisp to Twitter. At the time, I was rather frustrated with Common Lisp and thus took the gateway as an opportunity to try Clojure. Today, I looked at the source of the gateway again to find out what it would take to make it use OAuth instead of basic http authentication. As the gateway was the only program I have written in Clojure so far, and I was not really that eager to extend it to OAuth. I tried for a few minutes, but found that my old program did not run with the current Clojure version right away, so I would have to basically start setting up a Clojure development environment from scratch in order to be able to use an existing, open source OAuth library.

Instead, I thought I'd give Quicklisp a spin. After all, parsing some XML and sending HTTP requests are no big deal in Common Lisp either, and an OAuth library is available, too. With Quicklisp, all it should take was write some glue code to connect DRAKMA, CXML, cl-ppcre and cl-oauth. Installing the bunch should be a matter of loading quicklisp.lisp, then typing (ql:quickload `("drakma" "cxml" "cl-ppcre" "cl-oauth")) and watching the show scroll by.

And hey, it worked right away. I even embedded a web server in the gateway application so that it can be monitored by the system status monitors provided by my hosting provider. All in all, rewriting the thing in Common Lisp and deploying it took no longer than two hours, and the source is not significantly longer than the Clojure version either. Furthermore, the new gateway properly deals with non-ASCII characters. Embarrassingly, the Clojure gateway was buggy in that respect and never properly twittered the titles of my own blog posts. Thanks, Zach!

By all means, choose a distinct name for your program

"I love screen". "screen?", you ask, "do you love your screen or screen in general, like in sun screen?". And right you are, screen is not a name, it is not even a very distinctive noun as it is so ambiguous. So, when I say "I love screen", I'm talking about the program that Oliver Laumann wrote like 23 years ago and that is probably the open source program that I have used for the longest time (basically, since it was first released).

Now, as much as I love screen, I hate its name. I guess that Oliver and his room mate pondered how to name the program for a while, maybe even for a new hours or days, and then went "Damn fuck it, we're just unimaginative hacker dudes, round corners are not yet invented, creative design folks avoid us like the plague (and we like it that way), let's call the darn thing screen and go on hacking."

DON'T DO THAT, SERIOUSLY

Whatever name you choose for your program, make sure that it is not a word. If you absolutely want to use real words, choose at least two of them that never occur together anywhere else. Better yet, invent a word. Use Google to validate your name. Use something stupid. Get on with it.

I'm writing this because the poor choice of naming for screen bites me every day. screen has a serious bug. Sometimes, one of the screens (sic!) hangs. It then never recovers, and once one of the screens (yaaah!) hangs, it does not take long until the whole session ceases to work. In the end, even trying to reattach the session hangs. I think I do have basic Google skills, but so far I have failed to find anyone having the same problem. I can't imagine that I would be the only person to have this problem, but the name of the program makes it impossible to find that person.

You are making the life of your users much much easier if you give your programs names that are distinctive and unique. Yes, Ryan Gahl and Aycan Gulez, I'm looking at you today while I write this. :)

How to convert your old-style Symbolics keyboard to USB

Peter got in touch with me because he tried to follow my instructions how to convert a Symbolics keyboard to USB with his old-style keyboard and he failed. He assumed, and that was what I would have assumed, that the color coding of the internal cabling of the old and new keyboards would match, but they don't. Here is a table that shows the pinout for both keyboard types:

New Style		Old Style		Function	Teensy Pin
Pin#	Color	Pin#	Color
1	blue	8	white	GND	GND
2	yellow		black	not	connected
3	green	2	red	5V	5V
4	red	3	green	DIN	D4
5	black	1	yellow	CLK	D5
6	white	5	blue	CLR	D6

Embracing the ARM

Last year, I worked on a project for visomat inc. which involved interfacing to a LED matrix display so that it could display some video. We had some latency problems and at one stage we decided to try out DMX instead of Ethernet as interface techniquce, hoping that we'd get better results that way. The idea was to feed the LED matrix, which consisted of 12 power supplies/controllers, with 12 DMX interfaces to make sure that data arrives at the LEDs synchronously.

DMX is an asynchronous serial protocol running at 250 kbits per second. I decided to try using an AVR microcontroller running at 16 Mhz and have it bit-bang the data out of a parallel port. For each bit, 64 machine cycles would be available which should be enough to set the port bits and also grab incoming data out of the USB interface that we wanted to use to interface to the host.

I built a prototype of the interface and we made some promising experiments with it, but in the end we found that the visual latency problems were not fixed. The problem was really in how the LED hardware worked and could not be repaired on the feeding side. The prototype went onto the shelf and we stuck with Ethernet for that project.

A few weeks ago, I discussed the project with a friend of mine and we thought that it'd be cool to turn the 16 port DMX interface into a product. We found someone who would want to use it in an OEM setting, and I made bringing the hardware to product level my weekend project.

Good bye AVR, hello ARM

The AVR based solution proved to be problematic, though. Even though the CPU has enough processing headroom, the lack of RAM requires that the data is fed into the interface "just in time". Any variance in the USB input data stream could result in visible hickups in the DMX streams. Getting it to work seemed like possible, but require too much work in the end. Thus, I decided to turn to using more capable hardware which would also give the DMX interface network connectivity and more local intelligence.

As new platform, I chose the ARM9 based Eddy-S4M board produced by the Korean manufacturer SystemBase. These boards are distributed by trenz electronic who I've found to be quick and reliable in the past.

Linux is the standard operating system on Eddy-S4M. I'm not a huge Linux fan, so I spent some time to find an alternative operating system, but neither eCos, nor FreeRTOS or RTEMS seemed to be ported to the AT91SAM9260 CPU with a free development system. As I was into application development, I decided not to port or write an operating system myself. Running on the bare hardware was not an option either as I need a working TCP stack in order to interface to the host.

SystemBase calls their Linux port "lemonix". It appears to be a relatively plain linux-2.6.20 with real-time patches, BusyBox and a bunch of C applications to control startup and configuration. The userland and kernel source code is available from SystemBase after registration, but the kernel tree that they make available is incomplete and needs to be augmented by some files that are available from a Google group so that it can be built. Getting the cross compilation environment to run on my Ubuntu 9.10 box worked flawlessly, and after removing the whole Eddy application stuff and most of the SystemBase drivers from the file tree I ended up with a relatively sane Linux that I could cross compile and install on the Eddy hardware through tftp.

Making it fast

Back to the DMX interface, I was now faced with a system that gave me about 400 machine cycles for each DMX bit sent. That should be plenty of time to get out some bits to the ports. The bit timing would be achived by a timer interrupt every 4 microseconds. Even with interrupt handling overhead, this should leave enough headroom for TCP processing.

The ARM architecture specifies two interrupts. The normal, vectored interrupt system is used by Linux and mapped to its own, portable interrupt architecture, ignoring the vectoring facilities that the hardware provides. The second, fast interrupt system (FIQ) that the ARM provides is not used by Linux, and it seemed like a good fit to my requirements. The FIQ is interesting because when the FIQ handler is entered, the CPU automatically switches six registers to a distinct bank. These six registers can then just be used without the need to save or restore them to the stack. Even though the FIQ is not used by Linux itself, ARM-Linux provides for an interface so that drivers can use it. FIQ support was missing in lemonix, but it was trivial to backport.

Writing a FIQ interrupt in ARM assembly was straightforward. The driver bottom half set up the FIQ register set and enabled the timer interrupt, the FIQ handler set the port bits and incremented the pointers accordingly.

Reducing jitter

When looking at the output ports with the logic analyzer, though, I could see variations in bit edges in the order of one microsecond, which is beyond what the DMX receivers would be able to tolerate. These variations were visible even when the FIQ handler just set the port bits without writing any real data. The source for this jitter, as it turned out, was virtual address translation. The ARM9 CPU includes a MMU and all software, including interrupt handlers, uses virtual addressing, even to access I/O ports. The virtual address maps are stored in SDRAM due to their size, and the hardware automatically traverses these maps if an address can not be found in the translation lookaside buffers in the MMU. So in the FIQ handler, when the output port address was not present in the TLB, the MMU would access the SDRAM, and SDRAM random access is rather slow.

Thus, I had to change the FIQ handler so that it never accesses the SDRAM, either directly (by reading the data buffer) or indirectly (by causing TLB misses). The AT91SAM9260 CPU has two internal SRAMs that are accessible in two machine cycles, and I use one of those as a data buffer. In order to prevent TLB misses, indivdual TLB entries can be locked down so that they're never removed by the MMU automatically. Thus, my driver locks down the I/O and the SRAM buffer addresses that the FIQ handler accesses.

Reducing jitter even more

Even with the TLBs locked down, I still saw some jitter in the leading edges of the DMX bytes. The cause for this were variations in interrupt response times, as that depends on the instruction being interrupted. The variations were below one microsecond, but that was long enough to bring the serial decoding routine in my logic analyzer out of sync. Wanting to play it safe, I decided to remove that jitter as well by synchronizing on the timer value inside of the FIQ handler: Instead of just banging out the next bit as fast as possible, the FIQ handler now waits until the free-running timer that triggered the interrupt reaches a certain value. That way, the effective FIQ response time is made mostly constant.

Wow, 2010!

In the last few years, I have been using various AVR CPUs for my embedded projects. It is a great CPU that is easy to use and fast, and the advent of LUFA and Teensy made my life a lot easier, as I was freed from USB serial dongles and driver installation. But then, being able to process serious amounts of data is nice, too, and this is were 8 bits just don't suffice. ARM based boards are cheap nowadays, and I'm looking forward to embedding JavaScript or maybe even Common Lisp in one of my future projects. When speed is needed, I can always fall back to C and assembler.

Hacking a USB Infrared receiver

Our living room Hi-Fi setup needs a MP3 player that is always available, easy to use and, first and foremost, allows us to import all the CDs that we have easily and without effort. I've always wanted to do that using our FreeBSD based home server, but none of the ripping solutions really worked the way I wanted and triggered my desire to hack them to my liking. Finally, I gave in and pulled out the aging and unused PowerBook of my wife. After all, iTunes matches my requirements easily, and as I can treat is as piece of Stereo equipment, I'm willing to put my dislike for Apple Computers to the side.

The problem with the PowerBook is that it does not have an IR receiver. We have a Logitech Harmony Remote 515 that we use to control the rest of our entertainment stuff, and we'd like to control the MP3 player with it, too. I could not find a cheap, ready-to-go solution easily, so I decided to throw together something myself using a Teensy and a Vishay TSOP1836 IR receiver module that I found on a surplus board that I found in a box on my shelf.

The goal was to create an IR receiver that acts as a USB keyboard. The key codes that are sent should be useable to set up some global keyboard shortcuts in MacOS X that would control iTunes to our liking.

Hardware

The TSOP1836 IR receiver module does the IR carrier decoding and interfaces easily with the AVR using one input port. I hooked up my Logicport logic analyzer (excellent product, I write this as a satisfied customer) to the output of the IR receiver and tried a few remotes that I had to see whether the 36 kHz that that the receiver assumes as the carrier frequency would be common. I was lucky, both the remote of our Onkyo Amp and the remote control that came with the Altium Nanoboard 3000 transmitted at that carrier frequency, and both of them used the NEC protocol.

Teensy is a microcontroller board based on an ATMEL AT90USB162 AVR chip. It comes in a DIL24 form factor, so it is easy to integrate into breadboards. A neat thing about it is the flashing utility that looks good, works very well and is unintrusive. The compile/upload/test cycle with the Teensy loader is very short.

Software

I have used LUFA as USB stack on Teensy in the past to implement a adapter for Symbolics keyboards, and I decided to use it again. I was pleased to find that LUFA has made great progress since I last used it. There are more examples, the class drivers have been separated from the demo code, and the cheapish task scheduler is not longer used by the examples that I used.

I tried to make the USB implementation of the adapter offer two end points: A keyboard endpoint that sends the translated key codes when remote keys are pressed, and a serial or CDC endpoint that would be used to configure the adapter in order to make it possible to configure it to whatever remote is available. Unfortunately, I found that MacOS X does not accept devices with multiple endpoints as standard serial/CDC device, so I abandoned the idea of configurability. Instead, I made the adapter work with the Altium remote and a statically compiled translation table.

To find out the codes that the remote sends, I wrote a version of the adapter that acts as a serial device. Using that information, I set up the translation table in the HID keyboard version of the firmware.

When I tried setting up some global keyboard shortcuts for iTunes, I learned that iTunes does not really support global short cut keys, presumably to "protect the user". Using a the free SizzlingKeys utility, I was able to achieve what I wanted, though.

The source code for the two Teensy firmwares is available in my github repository.

Clojure Slides from February Berlin Lispers Meetup

Stefan Richter, our speaker at the February Berlin Lispers Meetup, has made the slides of his presentation available on line. Enjoy, and see you next week!

Telekom VDSL mit FreeBSD

Da ich meinen DSL-Anschluß im Büro mit ein paar anderen Freelancern teile, habe ich kürzlich von vorher Congster ADSL zu Telekom VDSL-25 gewechselt. Als Router verwende ich nun einen Rechner mit FreeBSD-7.2, der VDSL-Anschluß wird mit einem Modem Speedport 300HS realisiert. Da der VDSL-Uplink auf einem gesonderten VLAN aus dem Modem kommt, ist eine zweite Netzwerkkarte nicht zwingend erforderlich. Ich habe das Modem einfach ans LAN gehängt und auf dem FreeBSD-Router ein VLAN-Interface konfiguriert, das für den Internetzugang mit PPPoE verwendet wird.

Konfiguriert ist das ganze wie folgt

Änderungen in /etc/rc.conf

# VLAN-Interface konfigurieren, bge0 ist mein Ethernet-Interface. Ggf. anpassen.
cloned_interfaces="vlan0"
ifconfig_vlan0="vlan 7 vlandev bge0"
# IP-Paketforwarding anschalten
gateway_enable="YES"
# PPP-Client automatisch starten
ppp_enable="YES"
ppp_mode="ddial"
ppp_nat="YES"
ppp_profile="telekom"

Eintrag in /etc/ppp/ppp.conf

telekom:
        set device PPPoE:vlan0
        set authname AAAAAAAAAAAATTTTTTTTTTTT0001@t-online.de
        set authkey KKKKKKKK
        set dial
        set login
        set log phase tun command
        set ifaddr 10.0.0.1/0 10.0.0.2/0
        add default HISADDR

Wie üblich ist AAAAAAAAAAAA durch die Telekom-Anschlusskennung, TTTTTTTTTTTT durch die T-Online-Kennung und KKKKKKKK durch das Kennwort zu ersetzen. (Könnte die Telekom bitte irgendwann mal die BTX-Vergangenheit tief unter der Erde verbuddeln und die Autorisierung auf ein benutzerfreundliches Verfahren umstellen? Danke!)

Das war's. Reboot und die FreeBSD-Box routet.

Max for Live, Java, MIDI, Windows and Sysex

Recently, I have spent some of my free time hacking some Java to provide Max for Live patchers direct access to the MIDI ports of my machine. Usually, one would be restricted to the MIDI data that Live has previously processed, which restricts it in many ways.

I have not spent too much time hacking Java in my life, but the times I have done it for various smaller hacks, it was a nice overall experience. This time, I was particularily impressed by Eclipse for its automated source code formatting facility that is widely configurable. I tweaked the settings a little to match my prefered squirly brace style, and then had Eclipse beautify my (and imported) source code with a single key stroke. Also, I like Eclipse's ability to automatically locate unknown classes and add required import statements with a single click. This makes working with examples from documentation rather easy. I still like Emacs, but I must admit that it is rather baroque when compared to what Eclipse can do.

Getting the Max external to interface to the MIDI subsystem that Java provides was rather easy, and I was mostly done after 2-3 hours of hacking. Sadly, sending Sysex strings to the MIDI controller that I used for testing did not work reliably. Naturally, I thought that the problem was with my lack of MIDI, Java and Max skills, so I spent quite some time trying out various implementation strategies, without success.

Finally, I figured that what did not work was sending Sysex messages, and only sometimes (i.e. it appeared as if the first message came through and then some following messages where not transmitted). Google finally pointed me to a web page by Thorsten Klose where, at the end, he describes the behavior as being a bug in the Java MIDI library. The problem is that when sending Sysex messages, the Java library remembers the size of largest Sysex message sent and uses that for all subsequent Sysex messages, even if they are shorter. My workaround is to close and re-open the MIDI device if a Sysex string to be sent is shorter than the previous one. This is kind of ugly, but it works for me. The workaround that Thorsten suggest has the problem that it does not stop Java from sending long messages, which reduces MIDI throughput to a point that was unacceptable for me.

I'm posting this in the hopes that it saves the next person seeing the problem some time. If you are interested in the Max for Live MIDI external, it is available here.