Using the VMS PATCH utility to fix a VAX LISP issue

In my previous post about VAX LISP, I complained about a misfeature of the editor that prevented binding editor functions to the Ctrl-S and Ctrl-Q keys.  In the manual, it is stated that these keys cannot be bound due to "system restrictions". It appears, though, that it is a measure to prevent users from footing themselves in the foot because Ctrl-S and Ctrl-Q could be sent by terminals to implement software flow control.  Software flow control is optional, though; it can be switched on and off for each terminal line and it is not needed for virtual terminal connections (Telnet, DECNET) as flow control is implemented by the network layer and networked terminal sessions usually have enough buffer space.  Thus, reserving Ctrl-S and Ctrl-Q really only makes sense for asynchronous terminal lines that require software flow control, and there is no reason to reserve these characters for all terminal sessions, regardless of technology.
I am used to having Ctrl-S and Ctrl-Q be bound to certain functions from GNU Emacs, and I want to use the same bindings with VAX LISP as they've become part of my muscle memory.  Thus, I wanted to remove the explicit checks for these keys in the EDITOR::BIND-COMMAND and BIND-KEYBOARD-FUNCTION functions.  Lacking the source code to VAX LISP, I could not just remove the checks and recompile, so I needed a different way.
This is where the VMS PATCH utility comes into play.  It is a standard tool that is used to modify program executables, usually to remove bugs without requiring a full reinstallation of the software package that needs to be fixed.

Finding what to patch

In order to remove the checks for Ctrl-S/Q in EDITOR::BIND-COMMAND and BIND-KEYBOARD-FUNCTION, I disassembled the two function using the standard VAX LISP function DISASSEMBLE.  VAX assembler code is easy to read, so it was rather straightforward to find out how the checks work.  Here is the relevant extract of the disassembly output from the VAX LISP BIND-KEYBOARD-FUNCTION function:

000FC  MOVQ   BIND, -(STACK)
000FF  MOVL   -10(FRAME), -(STACK)
00103  MOVZBL #89, -(STACK)
00107  MOVZBL #99, -(STACK)
0010B  MOVAB  18(STACK), FRAME
0010F  MOVL   'CHAR/=, LEX
00113  MOVL   4(LEX), FUNC
00117  JMP    @4(FUNC)
0011A  CMPL   RES, SV
0011D  BEQL   121
0011F  BRB    145
00121  MOVL   FRAME, -(STACK)
00124  MOVL   #A28, -(STACK)
0012B  MOVQ   BIND, -(STACK)
0012E  MOVL   '"The character argument must be a control character (other than~%~
            CTRL/S or CTRL/Q): ~S", -(STACK)

Apparently, #89 and #99 refer to Ctrl-S and Ctrl-Q, and the CHAR/= function is called to check whether the argument is one of those characters.
The easiest way to get rid of the check seemed to me to replace the BEQL 121 instruction, which jumps to printing the error message and returning from the function, to BRB 145, which is the jump to the rest of the function, which performs the actual binding.

Locating the code

To make the actual modification using the PATCH utility, I now needed to know where in the executable file these instructions can be found - the addresses in the DISASSEMBLE output are only relative to the start of the function, and the LISP.EXE executable file does not include any symbols.  I thus picked out a few instructions that seemed characteristic and assembled them to machine code using the VMS macroassembler.  Then, I searched for this instruction sequence in LISP.EXE using a trivial perl program, which gave me the offset of the function that I needed to update.
Knowing the offsets of the locations in the binary file that I needed to change, I could now go about and use the PATCH utility in interactive mode to devise the actual patch.  The PATCH manual suggests that the VMS standard debugger would be used for patch development, but as I did not have the symbol table for LISP.EXE, using PATCH itself for development was more appropriate as it can work in /ABSOLUTE mode which uses file offsets for loctions.
First, I needed to find and disassemble the code using the locations I had devised.  As I only had the offset of the MOVZBL #89, -(STACK) instruction and VAX instructions have variable lengths, I needed to probe a bunch of addresses before I could come up with this:

PATCH>e/i 00403E68:00403E9A
00403E68:  MOVQ    R10,-(R7)
00403E6B:  MOVL    B^0F0(R8),-(R7)
00403E6F:  MOVZBL  #089,-(R7)
00403E73:  MOVZBL  #099,-(R7)
00403E77:  MOVAB   B^18(R7),R8
00403E7B:  MOVL    B^30(R11),R9
00403E7F:  MOVL    B^04(R9),R11
00403E83:  JMP     @B^04(R11)
00403E86:  CMPL    R6,AP
00403E89:  BEQL    00403E8D
00403E8B:  BRB     00403EB1
00403E8D:  MOVL    R8,-(R7)
00403E90:  MOVL    #00000A28,-(R7)
00403E97:  MOVQ    R10,-(R7)
00403E9A:  MOVL    B^38(R11),-(R7)

This is exactly the same instruction sequence that I disassembled using VAX LISP above. As you can see, it is significantly harder to read the disassembly produced by PATCH as it does not show symbolic register names, and it also does not translate data pointers like in the last instruction - VAX LISP showed the error message where PATCH prints B^38(R11).  The addresses in the disassembly, however, are absolute offsets into the LISP.EXE file, and that was what I needed.

Creating the actual patch

Equipped with this information, I could create the actual patch to change VAX LISP so that it no longer disallows binding of Ctrl-S and Ctrl-Q, which I reproduce below:

$ PATCH/ABSOLUTE LISP$SYSTEM:LISP.EXE/OUTPUT=LISP$SYSTEM:
REPLACE /INSTRUCTION
^X00403E5A
'BLEQ ^X00403E5E'
EXIT
'BLEQ ^X00403EB1'
EXIT
REPLACE /INSTRUCTION
^X002F3246
'BEQL ^X002F324A'
EXIT
'BRB ^X002F3265'
EXIT
UPDATE
EXIT

I used the REPLACE/INSTRUCTION command which requires the specification of an address, the current instruction(s) at the location and the new instruction(s) to put at the location.  PATCH would only perform the replacement if the instructions currently in the file match what is specified, otherwise it would abort with an error.  This is useful to prevent the patch from patching the wrong file or the wrong version of a file.

Better patches

As I did not have the source code or the symbol table for LISP.EXE when I developed this patch, I needed to resort to patching in absolute mode which is kind of brittle.  PATCH can also work in symbolic mode if a symbol table is available for the executable file, which allows patches to work even if the executable has been linked differently than the executable used to develop the patch (i.e. you could write a patch for a library function that'd work for executables that were linked statically against the library).  Also, PATCH allows patches to insert more instructions than were originally present in the executable.  It does this by adding additional disk space to the executable to hold the new, longer code, and then patch the required jump instructions to the patch area into the original location.

Closing thoughts

It is quite nice to be able to fix a bug in or remove a feature from a commercial program without having the source code, just by working with a bit of disassembly and on-board utilities. Back in the day, it was also useful to be able to distribute fixes to customers in the form of tiny patches. These could even be sent as a printed letter and typed in by the customer on a terminal, with reasonable effort and good chances of success.

Evaluating VAX LISP 3.1

The back story

Back in the day, when there still existed a number of different computer architectures and operating systems, I was a huge fan of the VAX/VMS operating system. VMS was designed together with the VAX architecture, and it used specific mechanisms offered by the processor to implement its I/O, security and other services. As it was still common at the end of the 1970ies, portability was not a design objective, and computer companies often had both hardware and software teams working together to implement computing solutions, from hardware to operating systems, development environments and applications.
I became a VMS fan mostly for three reasons: VMS machines were popular in the early packet switched networks that I could access, their default installation often left accounts with default passwords open, and VMS has an excellent online help system which helped me to learn and program VMS without having access to printed manuals.

I also liked the VAX for its orthogonal architecture, which made it somewhat similar to my favorite 8 bit microprocessor at the time, the Zilog Z80.  Back then, being able to program a machine in assembly language was still somewhat desirable, and an orthogonal instruction set with many general purpose registers and fancy addressing modes allowing complex programs to be written in assembly language appealed to me, too.
Frankly, though, I never wrote a lot of assembly language code. Instead, C quickly became my language of choice even on VMS, and when g++ 1.42 became available on VMS, I mostly became a full  time C++ programmer.  This was when object oriented programming just became the fad, with magazines like the C/C++ Users Journal and the Journal of Object Oriented Programming being the prime sources of inspiration. Around that time, I read Grady Booch's excellent book Object-Oriented Analysis and Design with Applications, which explained OO programming using examples in a variety of then-popular OO languages like Smalltalk, C++ and Object Pascal. It also had a chapter on using CLOS as the example language. To me, CLOS was completely out of reach and locked in the ivory tower, requiring utterly expensive machines and software licenses that I'd never hope to have access to, so I read the chapter more as science fiction than as something of practical relevance.
Thus, I stuck with C++ and later Perl on VMS, which was amusing enough, also for being a rather exotic blend.

Around 2001, I was introduced to Lisp by the way of VMS. At the time, my programming gigs were all on Unix or Windows and my affection to VMS became part of my hobby as a collector of vintage computers.

At some point, I brought a MicroVAX with me to a vintage computer convention, and a friend had his MacIvory Symbolics Lisp Machine on display. As the two systems were from the same era, we thought that we should try to bring up some network between them.  My VAX ran a fairly recent version of VMS, though, and when we tried to establish a DECNET connection, the Lisp machine would signal an error and put us into a debugger. What got me hooked on Lisp was that my friend could display the source code of the failing DECNET device driver, figure out that the problem was the too-new DECNET version number that the VAX had reported in the connection setup packet, fix the Lisp code in the DECNET driver of the Lisp Machine and then continue the file transfer operation that had signaled the problem.  I certainly understood that this would theoretically be possible back then, but I had never seen a system that would have that level of integration of operating system and programming language. I was intrigued.
Fortunately, machines had become much cheaper and faster in the decade since I had first read about CLOS, and the open source movement had made Unix and Common Lisp affordable. I went for CMUCL and eventually became a full time Common Lisp developer for several years.

The holy combination

I had been looking for VAX LISP for some time, as LISP and VMS sounded like a blend of very likeable technologies, so I was very happy when someone found the installation tapes and made them available through the comp.os.vms news group in 2017.  I played around with it a little, but unfortunately the manuals that were publicly available at the time were only for VAX LISP V2.1, and V3.1 was supposedly different enough to make me want the updated manual set.  It took a bit of research and emailing until I got in touch with the former architect of VAX LISP at digital equipment, and he was kind enough to borrow me the complete manual set for version 3.0, which I scanned and OCRed so that it is now available on bitsavers.  Thus, i felt equipped to give VAX LISP a test drive as a vacation project.

About VAX LISP

VAX LISP is an implementation of Common Lisp, but it implements it as described in the first version of “Common Lisp the Language” (CLtL1). This means that it feels like Common Lisp as it was standardized nine years later, but with several important features missing. Among these missing features are:

• LOOP, the general purpose, well-hated macro for iteration.  You get simple LOOP, DOLIST, DOTIMES and of course the dreadful DO and DO* couple.
• The Common Lisp Object System (CLOS), although there is a version of PCL coming with VAX LISP in the examples directory.  There also is a version of the Flavors object system included.
• The condition system. VAX LISP allows you to bind the *UNIVERSAL-ERROR-HANDLER* special variable to a function that’d be called when ERROR, CERROR or WARN would be called and that could take a non-local jump using THROW.
• DESTRUCTURING-BIND and lambda lists with &KEY, &REST etc.  Lambda lists like we know them today are only available in DEFUN

Furthermore, the package system is not yet where it is today, there are no generalized sequences and there are probably a several other bits and pieces that are missing.

Notable features of VAX LISP are:

• Integrated, user-extensible editor with both EDT (VMS’s standard editor) and Emacs emulation modes, both with a terminal and a DECwindows backend. There is extensive documentation on programming the editor in LISP and I’ve added several features myself, which was a joy.
• Interface to the standard DECwindows widget set. This required the implementation of a multithreading system so that DECwindows code, written in C, could be called from LISP and DECwindows handlers could be written in LISP and called from C.
• A foreign function interface implementing the VMS procedure calling standard, making it possible to call VMS system services and library routines as well as code written in other VMS languages.

Additionally, it comes with a good bunch of example code that is quite helpful when trying to figure out how to use the language, although the quality of the samples vary.

A Project to Try It Out

I wanted to get a feel of how programming in VAX LISP works, so I decided to write a simple http server in it.  This would give me the opportunity to try out the system interface as I would have to call into the TCP/IP subsystem.  As I wanted to evaluate VAX LISP during a vacation trip, I used SIMH on my MacBook instead of a real VAX, running VMS V5.5-2.

The Development Experience

Initially, I had hoped that I could use the graphical version of VAX LISP for development, but I did not manage to set up a fully functional DECwindows workstation environment and fell back to the terminal version.  The downside of this is that I had to alternate between working in the editor and in the REPL, as the editor does not have a convenient way to evaluate expressions and display the results built-in.  It would probably not be very hard to add that, but I wanted to spend most of my time working on the web server.

The REPL

The terminal REPL of VAX LISP does not have a lot of convenience features.  LISP expressions are entered using the standard VMS line editor, which means that no completion is available and all you get is a way to recall the previously entered line.  Annoyingly, if a line is longer than whatever your current terminal width is, there is no way to edit text before the line break.  Obviously, no work has been put into making the terminal REPL nice.

The only notable convenience is that one can bind keys (ASCII codes, really) to call LISP functions, and one use case suggested in the manual is to bind, say, Ctrl-E to call the 'ED function.  That way, switching to the editor from the REPL is very quick and I use that feature all the time.  Beyond that, the REPL experience is rather bare-bones and only slightly better than, say, a bare SBCL or Clozure CL REPL because of VMS's standard line-editing facility, which gives you a one-line recall buffer.

The Editor

VAX LISP comes with an editor that is written in LISP itself and that is extensively documented so that users can extend and adapt it as desired.  By default, it tries to emulate EDT, which is the original VMS editor, but EDT is rather quirky and pretty much unusable without a DEC keyboard as it relies on certain function keys that are not present on modern keyboard layouts.  The VAX LISP editor also supports an Emacs compatibility mode, because even back in the late 1980ies, most LISP programmers would be used to Emacs and not be willing to use another editor.  In Emacs emulation mode, I found it rather easy to write and edit code and I even did some substantial refactorings of my http server code base with it without too much grief.  I spent a bit of time of adding a few features that I am used to from GNU Emacs.  Writing editor extensions in the same language and environment as the actual project was rather joyful.

The documentation to the editor is extensive, but it leaves a number of questions unanswered.  I also find its organization a bit odd, with a whole chapter on editor concepts that comes after all the nitty gritty details of editor customization explanations as separate "Part II" of the manual.  I'd expect the description of editor concepts to be coming first, before applying the concepts would be explained.  Also, the explanation how editor major and minor modes work is rather fuzzy.

One annoying misfeature of the editor is that it explicitly disallows binding Ctrl-S and Ctrl-Q, and the manual claims that this is due to "system restrictions".  While it is true that Ctrl-S and Ctrl-Q were used to implement software flow control on terminal connections that lacked hardware flow control, software flow control is an optional feature of the VMS terminal driver and can easily be switched off if out-of-band flow control is available for a given terminal connection.  This would be easy to fix if one hard the VAX LISP source code, but that seems to be lost, unfortunately.

Maybe not so unexpectedly for an editor written in the 1980ies, the VAX LISP editor does not have an "undo" facility.  This dampened my enthusiasm when I started writing code, but in practice, it has not hurt me too much.  When I made larger changes, I saved my buffers beforehand, and as the VMS file system automatically keeps old versions of files until they're purged, I could go back to the previous state if I wanted to roll back.

VMS and the Internet Protocols

My romanticized memories of VMS date back from a time when there was no Internet, and the Internet Protocols (IP) were just one set of protocols that were used in global networking.  DEC had its own protocol suite, DECNET, and was committed to making DECNET OSI compatible, as they believed that OSI would become the dominant wide-area networking protocol suite of the future.  DECNET was integrated into VMS quite nicely, and it was easy to implement peer to peer connections from any language, even from command line scripts (which enabled the creation of early networked malware like WANK).  Implementations of other networking protocols like RSCS (for BITNET/EARN) and IP (for Arpanet) were available for VMS, but DEC just treated them as secondary networking systems that did not deserve a lot of attention.

To implement my http server in VAX LISP, I needed an implementation of IP, so I installed "DEC TCP/IP Services for VMS" (traditionally referred to as UCX, "VMS/Ultrix Connection") as an add-on product to my base VMS installation.  UCX implements a standard set of IP clients and servers like FTP, Telnet, NFS etc. and supports two programming interfaces, one similar to the BSD socket calls meant to be used from programs written in C, and another one based on VMS system services, meant for programs written in other languages.  Boldly, I decided that I wanted to use the System Services interface, also in the hope that it would be as nice and convenient as the DECNET programming interface.  I was quite wrong.

Apparently, the System Services interface to UCX is a layer on top of the socket interface, so you get the "best" of both worlds:  BSD sockets with its quirky sequence of required calls to socket(), listen() and accept(), asking you to specify the length of the "listen queue" and to set the "reuse address" option in your server code because, well, because that is how it wants its chicken to be waved to work well.  And then the beautiful VMS $QIO interface with its opaque "function code" and the parameters named P1 to P6 which would be used differently based on the function code supplied.

DEC, in its lack of enthusiasm for supporting IP on VMS, really made sure that the combination of $QIO and sockets would be as ugly as possible.

So, the required sequence to listen to incoming TCP connections and then accept them requires the program to:

• Assign a channel to the UCX network device
• Set the channel to be a TCP/IP socket
• Enable the REUSEADDR option, which is already inconvenient in C with the setsockopt interface, but with $QIO you have to create an additional descriptor to point to the socket option
• Set the local address of the socket, again using an additional descriptor to point to the local address
• Set the socket to listen, using another $QIO invocation, now passing the maximum queue length

These steps directly correspond to what you'd have to do when using the BSD socket interface, only that with the System Services interface, you need to call $QIO all the time.  What is actually done underneath is then determined by which of the P1 to P6 parameters you set and how, so if you call $QIO with the IO$_SETMODE function code and specify a non-zero P5 parameter, you are calling setsockopt() and if you specify a non-zero P3 parameter, you're calling listen() and so on.

Of course, there are few things that can't be syntactically improved with a bunch of LISP macros and in my http server, the whole sequence does not look so bad, but seriously:  This is one of the worst APIs that I have seen in my life and the experience put quite a damper on my romanticized view on VMS and DEC in general.

Serving Files

A http server commonly serves content from files, so I wanted mine to do so as well so that it could deliver a HTML page and some additional resources to the client.  On VMS, files are not the simple byte streams that we think about when talking about files nowadays.  Rather, the Record Managment System (RMS) maintains the file structure for applications and supports various file organizations like line-sequential and indexed with fixed and varying record lengths etc.  It is still possible to read the raw data stored in a file, but that will include the bookkeeping information of RMS.  This means that the http server also needs to know something about the structure of the file being served and possibly perform some conversion instead of just forwarding the file data to the client directly.

Binary Files

Serving binary files like images requires the http server to just read the files in a block-wise fashion, but as VAX LISP does not support block-wise reading, this needs to be done with direct calls to RMS.  This gave me an opportunity to learn working with RMS, as its API is quite different from, say, the POSIX file API with its simple open()/read()/write()/close() interface.

As RMS supports different file organizations, there are many parameters that an application program needs to specify when it wants to operate on a file.  Instead of putting all these different parameters into many functions, RMS uses in-memory structures (blocks) that describe the operations and parameters.  These blocks could be defined using the facilities of the application programming language (i.e. struct definitions and numeric constant macros in C), or they could be described using the File Definition Language (FDL).  FDL is a domain specific language that describes file and record structures.  Applications could invoke the FDL interpreter to turn textual file structure descriptions into binary RMS blocks that it could then use for accessing files.

As a result, the RMS API entry points only require specifying pointers to the blocks that describe the operation, which makes the call sites look rather clean.  As, the entry points themselves clearly describe the operation, this makes the LISP code to serve a binary file using RMS calls rather clean-looking.

Text Files

Serving text files like HTML or CSS files is rather more challenging in VAX LISP, considering that the on-disk data of RMS text files cannot simply be copied to the HTTP client.  This is because, depending on the file's RMS record format, text files are stored with the length of each line stored in the file in binary form before the actual characters in the line.  For that reason, I chose to use the text file I/O mechanism of VAX LISP to read and provide for some explicit buffering to avoid writing each line to the client in a separate $QIO call.  I am not sure whether that strategy really provides performance benefits or whether UCX's own buffering would be as good.  The resulting code does not really look terribly complex, though.

Roundup

In the end, I've now implemented a basic http server in VAX LISP, called rasselbock.  The development experience was quite good, and I found it to be fun to write some code in a Lisp that felt like the Common Lisp I've been using professionally for several years, but that lacks several of the features that I took for granted.  I think that in the end, VAX LISP provides enough tooling for serious programming work.  It was also an interesting experience to write some code in a language for which the Internet offers no support at all.  All I had was the manuals, some sample code and DESCRIBE, and that was sufficient to create something which (kind of) works.

My vacation comes to an end so I'm going to stop at this point for now.  In the future, I'm going to put rasselbock onto a real VAX, and I also want to spend some time working with the DECwindows version of VAX LISP as that will probably be a much nicer development experience.  Furthermore, I would like to try converting rasselbock to run asynchronously, as that one of the cool things in VMS is that all its system services can be used asynchronously.

I need to thank Walter van Roggen who was kind enough to borrow me the VAX LISP V3.0 manual set for scanning, and Michael Kukat who parted with his remaining VAX hardware so that I have a real machine to run VAX LISP and rasselbock on.  I'd also like to thank Richard Wells who pointed me at the vectorized version of the digital logo which led me to become a producer of t-shirts.

If you made it this far, you might also be interested in my follow-on post, in which I describe how I patched the editor so that I could bind Ctrl-S and Ctrl-Q.

Computern kann man nicht vertrauen

In einem vorherigen Beitrag sprach ich mich dagegen aus, die Verschlüsselung von Emails zu empfehlen. Eins meiner Argumente ist dabei die fehlende allgemeine Vertrauenswürdigkeit von Computern, auf die ich in diesem Post etwas detailierter eingehen möchte.

Ziel von Verschlüsselung ist es, Vertraulichkeit herzustellen. Die Nachricht soll nur dem Absender und dem Empfänger bekannt sein, das genau ist, was durch die Verschlüsselung erreicht werden soll. Um dieses Ziel wirksam zu erreichen, ist es jedoch erforderlich, dass die Systeme, auf denen die Nachricht verarbeitet wird, vertrauenswürdig sind. Diese Vertrauenswürdigkeit soll in Deutschland sogar durch ein Grundrecht auf Gewährleistung der Vertraulichkeit und Integrität informationstechnischer Systeme von staatlicher Seite zugesichert und eingeräumt werden. Insofern ist Verschlüsselung die technische Umsetzung dieses Grundrechts.

"Integrität" ist in diesem Zusammenhang wie folgt zu interpretieren: Das System soll genau das und nur das machen, was die Benutzerin oder der Benutzer von ihm erwartet. Erweitert auf die Vertraulichkeit von Kommunikation heisst das: Wenn Alice eine Email an Bob schreibt, dann sollen alle beteiligten Systeme sicher stellen, dass niemand ausser Alice oder Bob den Inhalt der Email sieht.

"Seit Snowden" wissen alle, dass insbesondere US-Amerikanische Firmen nicht um unsere Grundrechte kümmern. Die Integrität von Facebook, Google, Amazon usw. ist also nicht gewährleistet. Hier kommt die Ende-zu-Ende-Verschlüsselung ins Spiel, die von Experten empfohlen wird, um die trotz kompromittierten Transportsystemen die Integrität der Kommunikation gewährleisten zu können.

Es bleiben die Endgeräte, im Idealfall die beiden Computer von Alice und Bob. Wenn die Verschlüsselung wirksam sein soll, um Vertraulichkeit und Integrität zu Gewährleisten, dann müssen zumindest die verbleibenden beiden Computer vertrauenswürdig und integer sein. Das sind sie jedoch nicht.

Die fehlende Vertrauenswürdigkeit beginnt dabei bei der Hardware: Weder die verbauten Chips, noch die in die Rechner werkseitig eingebetette Firmware ist integer im Sinne der Wünsche der Anwenderin. Als Kunde weiss man nicht, ob die Hardware nur dass kann, was man glaubt, dass sie kann. Es ist auch nicht sicher gestellt, dass die eingebettete Systemsoftware fehlerfrei im Sinne der Anwenderwünsche ist. Als Betreiber eines Computers kann man an die Integrität seines neuen Computers glauben, sicher kann man sich jedoch nicht sein.

Schwerwiegender jedoch ist die fehlende Vertrauenswürdigkeit der Software, die auf den Endgeräten zum Einsatz kommt. Die Erkenntnis, dass Software nicht "fehlerfrei" sein kann, ist heutzutage ein Allgemeinplatz. Eine Konsequenz daraus ist, dass das automatische Einspielen von Software-Korrekturen zur Fehlerbehebung eine Notwendigkeit ist, mit der sich Anwender davor schützen, dass Softwarefehler von Kriminellen zur Kompromittierung ihrer Systeme ausgenutzt werden. Durch diese automatischen Updates jedoch wird die Integrität der Endgeräte ständig kompromittiert, da jedes Update an das Potenzial hat, weitere Fehler und Sicherheitslücken nach sich zu ziehen. Es spielt dabei keine Rolle, ob man kommerzielle oder Open Source-Software verwendet: Wer Code aus dem Internet auf seinem Computer laufen lässt, kann nicht sicher sein, dass nicht auch unerwünschte Software zur Ausführung kommt.

Um zu meinem ursprünglichen Szenario zurückzukehren, in dem Alice eine Email an Bob schreibt und in dem alle zwischengelagerten Systeme durch Ende-zu-Ende-Verschlüsselung nicht mehr für die Sicherheitsanalyse in Betracht gezogen werden müssen. Angenommen, Alice hat auf wundersamen Wegen einen vertrauenswürdigen Computer kaufen können betreibt ihn ausschließlich mit selbst durchgelesener und für fehlerfrei befundener Software. Welche Annahmen kann sie denn über Bobs Computer machen?

Frei programmierbare Systeme, die schon werkseitig ohne eine Abgrenzung ihrer Funktionalität und ohne Überprüfungsmöglichkeit für den Benutzer sind, eignen sich nicht im mal Ansatz dazu, als Basis für die vertrauenswürdige Datenverarbeitung zu fungieren. Die Postulation von "Grundrechten" ist kein geeignetes Mittel, um diesen tatsächlichen Zustand zu ändern. Schlimmer noch: Wer behauptet, dass Sicherheit nur eine Frage der Installation der "richtigen" Software ist, nährt nur die Illusion von Sicherheit, die dann in der Zukunft weitere Enttäuschungen erfahren wird.

Vermutlich muss man den ganzen Komplex als Kampagne der heimischen IT-Industrie sehen. Angeblich schlechte Systeme, die von bösartigen ausländischen Mächten absichtlich unsicher gemacht werden, stehen gute Systeme von unserer heimischen Industrie gegenüber, die diese ganzen Nachteile nicht haben. Aber das ist Quatsch, und diese Versprechen werden nicht gehalten werden.

Was wäre denn die Alternative? Angenommen, die Integrität von IT-Systemen würde von staatlicher Seite als zu gewährleistendes Grundrecht interpretiert. Es müsste von staatlicher Seite sicher gestellt werden, dass zumindest bestimmte Datenverarbeitung in einem sicheren Rahmen stattfindet, auf den sich dann auch die Jurisdiktion verbindlich berufen kann. Da das Internet inklusive aller an ihn angeschlossenen Systeme insgesamt nur als kompromittiert angesehen werden kann, wurde im öffentlichen Diskurs die Forderung nach einem "Clean Slate"-Ansatz laut. Dieser Ansatz würde erfordern, dass eine komplett neue Systemwelt aufgebaut wird, in der jede Komponente für sich verifizierbar einem bestimmten Anwendungszweck genügt, und in der sich eine dritte Instanz (neben Alice und Bob) für die Systemintegrität verbürgt. Es muss für den Netzteilnehmer möglich sein, die Integrität der Endgeräte zu überprüfen, damit er oder sie wirksam davon ausgehen kann, dass die Kommunikation im Rahmen der von Dienst gewährleisteten Integritätsgarantien stattfinden kann.

Wer einen "Clean Slate"-Ansatz befürwortet, verabschiedet sich von dem offenen Internet, dass wir als Innovationsraum und als Erweiterung unseres Bewusstseins derzeit zu nutzen lernen. Mich erinnert das an die Zeiten vor der Liberalisierung des Telekommunikationsmarktes: Die Bundespost war damals die Instanz, die mit dem Telefonnetz das Kommunikationssystem betrieb. Die Integrität des Netzes wurde von staatlicher Stelle garantiert, und war - zumindest so lange das Netz analog war - auch faktisch gewährleistet. Die Liberalisierung des TK-Marktes und die Digitalisierung haben dazu geführt, dass die Annahmen über die Integrität der Kommunikationsnetze nicht mehr in der gleichen Form gemacht werden können - Einerseits, weil die staatlichen Garantien dank der Liberalisierung weggefallen sind, anderseits weil die Digitaltechnologie Integrität nicht von Hause mitbringt und es erheblichen Aufwand erfordert, die gleiche faktische Integrität in digitalen Systemen zu implementieren.

In den kommenden Jahren werden wir zunehmend Geräte kaufen können, die nur einem bestimmten Zweck dienen: Telefone, mit denen man nur (sicher) telefonieren kann. Tablets, mit denen man nur (sicher) Bankgeschäfte erledigen kann, Kommunikatoren, mit denen man nur (sicher) Emails senden und empfangen kann, Shopping-Konsolen, mit denen man (sicher) Produkte ansehen und einkaufen kann, Fernseher, mit denen man (sicher) fernsehen kann, das Staatsterminal, mit dem man (sicher) wählen und seine Behördengänge erledigen kann. Alle diese Systeme wird auszeichnen, dass sie komplett geschlossen sind. Es wird nicht möglich sein, auf ihnen eigene Programme laufen zu lassen, und ihre Funktionen werden vollständig von den Firmen und Institutionen bestimmt werden, die die entsprechenden Systeme betreiben.

In dieser Welt wird das Internet, wie wir es heute kennen, zunehmend an den Rand gedrängt werden. Wir werden etwas dafür tun müssen, wenn wir Freiräume er- und behalten wollen. Freiräume und Sicherheit stehen in grundsätzlichem Widerspruch, und es sollte nicht so getan werden, als liesse sich das eine durch das andere erreichen.

Warum ich meine Emails nicht verschlüssele

Das Verschlüsseln von Email ist ja derzeit wieder in Mode - Es wird als Beitrag zum Kampf gegen staatliche und nichtstaatliche Überwachung empfohlen, und es gibt Leute, die sich nur noch mit verschlüsselten Emails austauschen wollen.  Ich mache das nicht, und habe dafür meine Gründe.
Zunächst:  In jeder Sicherheitsdiskussion muss abgewogen werden, gegen welches Risiko man sich schützen möchte.  Es gibt - insbesondere in technischen Systemen - keine allumfassende Sicherheit gegen alle Risiken.  Wirksame  Sicherheitsmassnahmen erfordern stets die Identifikation des Risikos, und sie lassen sich in ihrer Wirksamkeit auch überprüfen.
Ein Beispiel für eine spezifische Sicherheitsmaßnahme ist der Sicherheitsgurt im Auto:  Er kann verhindern, dass man bei einer Frontalkollision mit dem Auto durch die Windschutzscheibe fliegt.  Bei einem Seitenaufprall jedoch kann man dennoch schwer verletzt werden, und der Gurt kann schlimmstenfalls sogar verhindern, dass man sich selbst befreien kann.  Die Wirksamkeit von Sicherheitsgurten lässt sich durch Statistik be- oder widerlegen.  Entweder, die Schwere der Verletzungen bei Frontalkollisionen ist seit der Einführung der Gurtpflicht zurückgegangen, oder eben nicht.
Allgemeine Sicherheitsmassnahmen hingegen sind Sicherheitstheater.  Sie dienen nicht dazu, einen bestimmten Schadenfall zu verhindern, sondern sie dienen der Beruhigung von Menschen, die Angst haben.  Das typische Beispiel für Sicherheitstheater ist das seit einigen Jahren erforderliche Ausziehen der Schuhe bei der Sicherheitskontrolle in Flughäfen.  Abgesehen davon, dass die Anzahl der Terrorschläge auf Flugzeuge gemessen an der Anzahl der Flugbewegungen extrem gering ist, ist auch die Möglichkeit, Sprengstoff in Schuhen zu verstecken, nur durch einen erfolglosen Versuch verbürgt.
Das private Verschlüsseln von Emails, wie es derzeit propagiert wird, fällt für mich in den Bereich von Sicherheitstheater.

Hier ein paar Argumente:

• Die Rechner, auf denen wir im Internet arbeiten und auf denen wir verschlüsseln, sind frei programmierbar.  Virenschutzsoftware und Open-Source-Programme bieten keinerlei Gewährleistung für Fehlerfreiheit.  Gezielte Angriffe auf jeden Computer sind jederzeit möglich, und sie werden auch durchgeführt.  Nach einem erfolgreichen Angriff muss im Prinzip das gesamte System als kompromittiert gelten, und wirksame Verifikationsmöglichkeiten existieren nicht.
• Der Glaube, durch Verschlüsselung könne man "den Geheimdiensten" "ihre Arbeit" "erschweren", ist irrig.  Verschlüsselung ist ein esoterisches Thema.  Zu glauben, dass das globale Internet-Email-System sich vollständig oder auch nur zu nennenswerten Teilen auf Ende-zu-Ende-Verschlüsselung mit bewussten Anwendern umstellen liesse, ist komplett unrealistisch.  Mit privater Verschlüsselung begibt sich der Einzelne einfach in die Gruppe derjenigen, die Verschlüsseln.  Das allein macht vielleicht noch nicht verdächtig, und im Zweifelsfall werden die Dienste bei einem Verdacht dann eben andere Massnahmen zur Aufklärung aufbringen.
• Private Verschlüsselung wiegt diejenigen, die es tun, in falscher Sicherheit.  Vertrauenswürdige Computer gibt es nicht, und was man ein mal in einen Computer, der mit einem öffentlichen Netz verbunden hat, eingegeben hat, ist potenziell öffentlich und sollte im Zweifelsfall auch öffentlich sein können.  Das gilt für private Kommunikation, und das gilt erst recht in konspirativen Strukturen, in denen sehr sorgfältige und individuelle Planung der Nutzung öffentlicher Netze notwendig ist.
• Auch, wenn Krypto-Befürworter stets die Wirksamkeit Ihrer Verschlüsselungstheorien behaupten, und auf die Endgültigkeit der Mathematik verweisen, so bleibe ich skeptisch.  Zum Einen, weil ich nicht überprüfen kann, ob das Verschlüsselungsprogramm wirklich das tut, was ich von der Mathematik verstanden habe (was ohnehin begrenzt ist), zum Anderen, weil immer wieder kryptographische Verfahren gebrochen werden.  Dabei ist für mich die fehlende praktische Verfikationsmöglichkeit ausschlaggebend.

Meine Konsequenz ist, dass ich keinen gesonderten Aufwand treibe, um mich gegen einen Feind zu schützen, den ich nicht kenne.  Wer mit mir kommunizieren will, kann mir eine Klartext-Email schreiben, mich anrufen. mir einen Brief schreiben oder mich treffen.  Mir ist wichtig, dass ich erreichbar bin und dass ich Leute erreichen kann.

TL;DR
Das Internet ist öffentlich und somit nicht der richtige Ort für Konspiration.

Das Internet ist die Erweiterung Deines Computers, aber dadurch ist Dein Computer Teil des Internets.

No ECLM in 2014

As many of you may know, there was the plan to have the European Common Lisp Meeting happen in Berlin in October 2014. I was part of the organizing team, and we've tried our best to find high-quality speakers to make the event be at least as good as the previous ECLMs.
Unfortunately, and you've determined that already from the title and the previous wording, we've failed. We simply did not find enough interesting commercial projects that have not been presented on one of the previous ECLMs. As we do not want to compromise on the content, we thus have decided that there will be no ECLM this year. We hope to learn about new and exciting developments in the course of this year so that we can have an ECLM in 2015.

Ein paar Sätze zum Thema "Alle sollen verschlüsseln"

Das Hauptproblem bei der Empfehlung des Einsatzes von Verschlüsselung ist die fehlende Möglichkeit für den Einzelnen, die Vorgänge in den digitalen Systemen zu überprüfen.  Ein erfolgreicher Angriff sowohl auf die Endgeräte als auch auf die beteiligten Software-Systeme ist jederzeit möglich, ohne verwertbare Spuren zu hinterlassen.  Es kommt dabei nur auf den Aufwand an, den der Angreifer treibt.  Hinzu kommt, das keinerlei Systemtechnik zur Verfügung steht, die das Etikett "vertrauenswürdig" auch nur im Ansatz verdient, und dabei ist es noch nicht einmal notwendig, auf die bösen Chinesen zu verweisen, die ja alle unsere Chips herstellen und beliebiges Zeug in die schwarzen Plastikdinger einbauen können, ohne dass wir es je merken würden.

Ich finde es daher unklug, den Einsatz von Verschlüsselungstechnik als so grundlegend sinnvoll zu bezeichnen, wie das bei weiten Teilen der Diskursteilnehmer derzeit gemacht wird.  Allenfalls handelt es sich um Sicherheits-Theater, mit dem der oder die Einzelne ausdrücken kann, dass ihm oder ihr die Entwicklung "nicht gefällt".  Tatsache ist jedoch, dass digitale Systeme aufgrund der mangelnden Verifikationsmöglichkeit inhärent unsicher sind.  Wer das verschweigt und Verschlüsselung empfiehlt, sorgt dafür, dass die nächste Empörungswelle nur noch größer, und was schlimmer ist, nur noch machtloser werden lässt.

Vermutlich sind noch ein paar GAUs notwendig, bis es in der Öffentlichkeit ankommt, dass die Digitalisierung nicht nur ein netter Layer von Information ist, sondern die Abhängigkeit von Ihr die Gesellschaft insgesamt erheblichen Risiken aussetzt.  Zu befürchten ist jedoch, dass die Reaktion dann gleich die positiven Seiten, die die Informationsgesellschaft für den Einzelnen zweifelsohne mit sich bringt, auch gleich einstampft.  Erste Schritte in diese Richtung sind ja derzeit schon zu sehen, siehe "Schlandnet" und ähnlicher Unsinn.

Aus politischer Perspektive muss man sich natürlich fragen, inwiefern die positiven Aspekte der Informationsgesellschaft insgesamt relevant genug sind, um für Ihre Bewahrung zu kämpfen.  Wenn man das jedoch dann tun möchte, sollten die inhärenten Risiken nicht in der Art und Weise ausgeblendet werden, wie das derzeit von vermeintlich progressiver Seite aus geschieht.  Der Einsatz nicht vertrauenswürdiger Verschlüsselungssoftware auf nicht vertrauenswürdigen Systemen jedenfalls wird das Problem nicht nur nicht lösen, sondern aufgrund der zu erwartenden Enttäuschungen eher noch verschlimmern.

"Drosselkom" - All your IP networks are belong to us

Seid Wochen entrüstet sich die Internet-Szene über die Pläne der Telekom, zukünftig Flatrates mit einer Volumenbegrenzung anzubieten. Nachdem es zunächst als Ungerechtigkeit dargestellt wurde, dass man für einen schnellen Anschluß bezahlt, den dann aber nicht voll auslasten können soll, und die Telekom erklärt hat, dass es für Vielnutzer eine kostenpflichtige Upgrade-Möglichkeit geben wird, konzentriert sich der Protest nun auf die durch die Telekom bedrohte "Netzneutralität". Stein des Anstosses ist, dass die Netzbetreiber durch technische Massnahmen Daten mit unterschiedlicher Priorität behandeln können und wollen. Das sei, so wird erklärt, eine Verletzung der "Netzneutralität", die von den Protestiernden quasi als gegebenes Recht angenommen wird.

Unabhängig davon, dass die Proteste nun eigentlich mit der ursprünglichen Geschichte nicht mehr so besonders viel zu tun haben, lohnt es sich doch, die Angelegenheit mal sachlich zu durchleuchten:

Warum will die Telekom ein gemanagtes Netz?

Für die "Netzgemeinde" ist das Internet ihr eigener, unregulierter Raum: Eine Sphäre, in der man beliebige Datenpakete zwischen Servern austauschen kann. Das kann auch mal nicht so gut funktionieren, dann dauert das Laden einer Webseite eben lange oder es ruckelt im Video oder der Ping ist schlecht, so dass man im Spiel ständig abgeschlachtet wird, aber so ist das Internet eben. Funktioniert meistens, und wenn es nicht funktioniert, dann ärgert man sich eben ein bisschen und guckt ein bisschen fern oder spielt lokal.

Für die Carrier ist die Infrastruktur, auf der auch dieses Internet läuft, in der Zukunft aber wesentlich mehr: Traditionell haben die Netzbetreiber über dem Internet, für uns Benutzer unsichtbar, das eigentliche Carrier-Netz betrieben. Dieses Netz basiert nicht auf der IP-Protokollwelt, sondern auf den ITU-Standards, die aus dem Telefonnetz hervorgegangen sind. Die Carrier-Netze waren die Basis für die Telefon- und sonstigen Datendienste, die man neben dem Internet "früher" verwendet hat. Nun ist es aber zunehmend so, dass es kaum noch Dienste gibt, die nicht auf der IP-Protokollfamilie basieren, und die Infrastruktur für IP-Protokollwelt aufgrund der massenhaften Verbeitung des Internet wesentlich preiswerter als klassische Carrier-Netztechnologie ist. Die alten Carrier bekommen das natürlich zu spüren, denn sie muss noch die alte, teure Technik unterhalten, während IP-only-Anbieter das komplett nachgefragte Spektrum von Netzwerkdienstleistungen preiswerter anbieten können.

Um marktfähig zu bleiben, stellen die großen Traditionscarrier nun also Ihre Infrastruktur komplett auf IP um. Das heisst auch, dass höherwertige Dienste - wie beispielsweise virtuelle Standleitungen mit fester Bitrate und Latenz - nicht mehr im Overlay-Carriernetz realisiert werden, sondern innerhalb der allumfassenden IP-Infrastruktur, in der auch das normale Internet angeboten wird. Es macht einfach Sinn, das so zu tun, und dazu braucht man eben ein gemanagtes Netz, in dem sich unterschiedliche Verkehrsklassen realisieren lassen, und darin unterscheidet sich ein Carrier-Grade-IP-Netz vom "Internet". Im Internet nämlich, dessen Freiheit derzeit verteidigt wird, werden traditionell alle Daten als Best-Effort-Verkehr gleich behandelt. Es macht im Internet keinen Unterschied, ob ein Datenpaket einen Teil einer Website oder ein Stück Ton aus einem Telefonat enthält. Wenn es zu Engpässen kommt, dann wird das Netz für alle Verkehrsarten langsamer. Beim Laden einer Website muss man länger warten, bei einem IP-Telefonat versteht man seinen Gesprächspartner nicht oder die Latenz wird größer.

Wenn nun also behauptet wird, dass so ein gemanagtes Netz der Netzneutralität entgegen steht und nach Regulierungen gerufen wird, dann stellt sich die Frage, welches Ziel damit verfolgt werden soll. Sollen die Carrier keine höherwertigen Dienste anbieten dürfen? Sollen sie gezwungen werden, höherwertige Dienste nur auf Leitungen anzubieten, die nicht auch best-Effort-Verkehr transportieren? Wozu soll das denn eigentlich gut sein?

Mir kommt es so vor, als sei die Diskussion im wesentlichen durch Unkenntnis und den Unwillen geprägt, sich mit der Technik und den Skalierungsnotwendigkeiten auseinanderzusetzen. Es soll alles immer schneller werden und fast nichts kosten, und wenn es denn was kostet, dann soll es doch die Allgemeinheit bezahlen. Wenn dieses Thema jetzt im Bundestag diskutiert werden sollte, dann kann ich mir nicht vorstellen, dass dabei irgendwas Sinnvolles herauskommt. Wenn schon die elitäre Netzgemeinde nicht versteht, wie Datennetze funktionieren, und auf einem Niveau von "Bei mir kommt das Netz aus der Telefondose" argumentiert, wie sollen denn dann die Internet-Ausdrucker aus dem Bundestag einen sinvollen Beitrag leisten?

Liebe Netzgemeinde, fordert doch bitte mal was Präzises, technisch Realisierbares, und nicht von technischen Dienstleistern, dass sie Euch bitte für 30 Euro im Monat die Illusion von "Freiheit" erzeugen sollen. Damit überfordert Ihr sie. Für die meisten Menschen ist die Diskussion ohnehin unverständlich, da sie überall und immer und billig Netz haben und das auch so bleiben wird.

FPGA based tabletop Pengo console

As a kid in the 1980ies, one of my favoured arcade games was Pengo, a maze action / puzzle game. I spent most of my pocket money playing it, and when I was introduced to dialup bulletin board systems in the same era, I chose Pengo as my online nick name, which stuck for some 10 years.

The adaptions of the game for home computers never sparked any fire in me, though, mostly because of the inferior graphics. The arcade version only had a display resolution of 224 by 288 pixels, but sprites and colors made it much prettier than what Sinclair ZX Spectrum, Amstrad CPC 464 or Commodore 64 could do.

At one point, a friend gave me an original Pengo arcade game logic board, which I have been carrying around for years, always wanting to make it playable at some point. Due to the non-standard video output generated by the board, I've never actually done it. Recently, I wanted to do something electronics related in my free time, and I picked the Pengo project up.

Arcade game emulation

Emulation is a common way nowadays to run ancient software. The older the software is, the harder it usually gets to find original hardware to run it on. Emulators provide a way to run original ancient software on current hardware, using software to impersonate the original hardware that the program expects to run on. Emulators exist for all kinds of systems and they are also useful when developing for embedded or other non-desktop environments.

Arcade game emulation is rather popular, with the open source MAME emulator being ubiquitous. MAME emulates all sorts of game hardware, starting in the 1970ies up to rather recently. Versions for most popular desktop operating systems are available, and there is MAME4ALL for the Raspberry Pi, too. MAME emulates the hardware for a huge number of games, but it is not distributed together with the ROM images for any of these games, for licensing reasons. ROM images are distributed through the familiar channels, though.

Another way to run older software is to use an FPGA to actually implement the original hardware. In the FPGA, almost arbitrary digital hardware can be created using configuration, so to run arcade software, one needs to recreate the original hardware in a hardware description language. Several arcade hardware systems have been implemented using FPGAs, and Pengo is one of them.

Compared to the software emulation approach that MAME uses, hardware emulation is less flexible as only one typically only one game fits into the configuration memory of one of the cheap FPGA boards. Also, fewer hardware emulations are available. FPGA based emulation boots quicker and requires less power, though.

FPGA based Pengo build

As I wanted to play with FPGAs again, I opted to use a Paplio Pro FPGA board for which a port of Pengo exists. The port includes a video scan doubler so that a standard VGA output signal is generated. All I/O is done through an Arcade MegaWing, which conveniently makes all I/O needed for emulating Arcade hardware accessible on standard connectors.

Wanting to finish up the project once and for all, I decided to get a cheap, used Dell 1504FP 15'' TFT monitor, a Zippyy Arcade and Fight Stick, a bunch of arcade buttons from eBay and talked my brother, who is a carpenter, into helping me build a proper desktop cabinet in his workshop.

To complete the hardware, I found myself a nice industrial 5V switched mode power supply to power the FPGA board and a small Kemo M031N audio amplifier module connected to a speaker.

The port of Pengo to the FPGA required some tweaking in the top level VHDL module to adapt it to the wiring that I used inside of the cabinet - The first joystick port is used for the coin and start signals, the second port is connected to the joystick. I also tweaked the DIP switch settings to suit my taste, which is something that must be done in VHDL as the Arcade MegaWing does not have switches that one could use for run-time reconfiguration.

More games!

At this point, my tabletop console only works for Pengo, and sure enough there are other games that one would want to play on it. Using a more flexible MAME based engine would get me there, and I experimentally swapped the FPGA board for a Raspberry Pi with MAME4ALL. While Pengo works fine on the Pi, the video display quality is much lower because MAME4ALL unconditionally uses antialiasing. This makes the resulting image blurry, no matter what physical resolution is chosen for the Pi. The FPGA based emulation uses simple scan doubling for video scaling which results in super-crisp display quality even though the TFT's physical resolution is not matched by the FPGA video output. So for now, I'm going to stick with the one game solution until MAME4ALL is fixed or I can find another way to run MAME.

Dealing with Excel files from Common Lisp - Using ABCL and Apache POI

In my day job, I mostly program in Common Lisp. Most of what I do is file and database work which Common Lisp is pretty well-suited for. Sometimes, though, I have to deal with data that comes in Excel files, and in the past that meant loading the files into Excel, exporting it into some plain text format and then working with those plain text files from Common Lisp.

While this works, it is a manual and error prone process. Also, Excels plain text export mechanism often mangles the data in undesired ways, which requires yet more additional manual steps (or VB scripting), which I'd rather like to avoid for processes that need to be automatic. Thus, I was on the looks for a way to access the Excel files directly with a Common Lisp program.

Writing a new Excel file parser was out of the question - I have real customer needs to fulfill, and implementing a capable Excel file reader is a large infrastructure project. So I looked into using an existing Excel file reading library instead. There are numerous options, commercial and non-commercial, and I've looked into one of the libraries written in C, but the requirement to more or less manually create FFI stubs and foreign structure layouts for a library that in itself was not documented very well and also did not look like being very accessible made me look for other options.

Apache POI

One of the more prominent open source libraries for accessing Microsoft Office files is Apache POI. It has been around for over 10 years and supports most MS Office formats, including the old OLE2 Excel format as well as the newer OpenXML format (that, despite using XML as the base format, is a horribly complex mess that I hope to never have to deal with directly). Apache POI is a Java library, so it can't directly be used from SBCL, which is the Common Lisp implementation that I normally use.

Armed Bear Common Lisp (ABCL)

In the recent months, I have noticed that there was quite some activity around Armed Bear Common Lisp (ABCL). I had tried an earlier release of it, and while it somewhat worked, it seemed to have a fair number of restrictions that made it unsuitable for me at the time. In particular, ABCL lacked support for the Metaobject Protocol which is something that I often use, either directly or as a library dependency. Also, the older version that I tried could not load the Postmodern library that we use to access our Postgres database, which was the final show stopper. But all that was before the recent 1.1.1 release of ABCL.

ABCL is hosted on the Java Virtual Machine (JVM), and maybe the biggest advantage of that is that access to other JVM-hosted code is straightforward and easy from Common Lisp programs running in ABCL. Thus, using Apache POI should be a snap. Also, as ABCL is becoming a reasonably complete implementation of the Common Lisp standard now, I had hopes to be able to use some of my existing infrastructure code in the program that dealt with Excel files.

To make it short: ABCL works great now. It took me very little time to translate the calls that I found in some Apache POI example program to Common Lisp, and I could also use all of the Common Lisp libraries that I needed for the task. There are some important libraries that don't work on ABCL yet (i.e. CXML-STP, CL+SSL), but I don't need these right now. And ABCL, during development, behaves like any other reasonable Common Lisp implementation in that it supports SLIME.

The Excel reading was a snap and the read process is reasonably fast, but ABCL's startup times are a bit annoying. There currently is no way to do the equivalent of "saving the world" on ABCL, so one has to load all required software at startup time. We're using ASDF for that, and it seems that some of the slow startup times need to be attributed to it. The Excel file reader will run as a batch job, so the startup times don't matter for our production uses, but testing the scripts was a tad tardy.

To illustrate how easy accessing Excel files from Common Lisp is, here is some example code that dumps the first worksheet of an Excel file to the standard output in a Tab separated values format:

;; -*- Lisp -*-

(defpackage :export-tsv
  (:use :cl))

(in-package :export-tsv)

(defun init-classpath (&optional (poi-directory "~/poi-3.9/"))
  (let ((*default-pathname-defaults* poi-directory))
    (dolist (jar-pathname (or (directory "**/*.jar")
                              (error "no jars found in ~S - expected Apache POI binary ~
                                        installation there"
                                     (merge-pathnames poi-directory))))
      (java:add-to-classpath (namestring jar-pathname)))))

(defun process-file (pathname)
  (let* ((file-input-stream (java:jnew "java.io.FileInputStream"
                                       (namestring pathname)))
         (workbook (java:jstatic "create"
                                 "org.apache.poi.ss.usermodel.WorkbookFactory"
                                 file-input-stream))
         (sheet (java:jcall "getSheetAt" workbook 0))
         (formatter (java:jnew "org.apache.poi.ss.usermodel.DataFormatter" java:+true+))
         (total-row-count (java:jcall "getLastRowNum" sheet)))
    (dotimes (row-number total-row-count)
      (let* ((row (java:jcall "getRow" sheet row-number))
             (column-count (java:jcall "getLastCellNum" row)))
        (dotimes (column-number column-count)
          (unless (zerop column-number)
            (write-char #\Tab))
          (write-string (java:jcall "formatCellValue"
                                    formatter
                                    (java:jcall "getCell" row column-number))))
        (terpri)))
    (java:jcall "close" file-input-stream)))

Before the process-file function can be used, init-classpath must be called to add the Apache POI jars to the Java class path.

As you can see, the program is rather short, and even if all comments are stripped, the Java version contains a lot more ceremony. No surprise here, maybe you wanted to have your prejudice confirmed :).

ABCL will now have a firm place in my toolkit. Big shouts go to the maintainers who did a great job lifting ABCL up to a level where it will be very useful to me.

Using Pogoplug as a OpenVPN server - with FreeBSD and VLAN

Recently, I needed to establish a VPN between my office, a colocated server and my home office. I generally prefer building networks on my own rather than buying cheap customer grade appliances, so here is what I've used.

Router

As router, I use a Pogoplug E02. It is marketed as a "private cloud device" and basically is a Linux based NAS with Internet sharing built in. Hardware wise, it is an ARM box with 256 MB RAM, four USB ports and one Gigabit Ethernet port in a funny looking plastic enclosure, with a built-in power supply. Kind of an industrialized Raspberry Pi, if you will. The E02 is from a previous generation and no longer marketed by Pogoplug, but it is available from a number of sources for prices below $50 - Mine cost 35 Euro a piece. I spent another 11 Euros for a high-speed USB Flash stick with 4GB and 10 Euros for a Nokia CA-42 cable that I installed to access the serial console port (which I actually did not need yet, but just in case).

Switch

In order to provide me with Ethernet ports which are part of the VPN, I use the Netgear GS108E 8-port managed Gigabit Ethernet switch. At some earlier point in my life, I swore that I will never buy a Netgear device again, but at a price tag of €37, I was willing to take a chance. Sure enough, the experience with this device is not totally pleasant. Configuration requires a Windows computer (or an obscure Linux utility that I could not get to run), the Windows configuration program uses Adobe AIR (wth?), and it never just works. But, with some patience, it does.

FreeBSD

The first step to transform the Pogoplug into a useable computer is to replace the Busybox based Linux with a real operating system. This requires replacing the boot loader with a new u-boot version which can boot from USB media, a procedure that is very well documented on the platform support page for running Arch Linux on the Pogoplug E02.

As a long-time FreeBSD user, I was very happy to discover that my favorite OS would run on the device, too. Detailed and very accurate installation instructions, together with quite some pre-built packages, are available on the excellent FreeBSD for Kirkwood page. I followed these instructions and got going very quickly. I had to build a new kernel in order to add the tun device driver and make some minor adjustments for the Pogoplug hardware, but again, the instructions on Nicole's page were accurate. I tried building a newer FreeBSD release with the same patches, but I did not succeed with that and thus sticked with FreeBSD-8.1.

Router configuration

In addition to the FreeBSD base installation, the openvpn and isc-dhcp41-server packages are required. I built them on the Pogoplug myself.

The Pogoplug only has one Ethernet port, so in order to talk to two separate Ethernet segments, I use a tagged VLAN. On the untagged VLAN, the router is connected to my "normal" home network. The VPN is in a VLAN with tag 1, and the Ethernet switch is used to make some ports into members of the VPN and some members of my home network.

The relevant configuration section from my /etc/rc.conf looks like this:

ifconfig_mge0="Home-LAN-IP/24"
static_routes="vpn_router"
route_vpn_router="VPN-GW-Public-IP Home-Router-IP"
defaultrouter="VPN-GW-VPN-IP"
cloned_interfaces="vlan0"
ifconfig_vlan0="vlan 1 vlandev mge0 Home-VPN-IP"
gateway_enable="YES"

Some explanation is in order: Home-LAN-IP is the (static) IP address of the router in my home LAN. DHCP can be used as well, but I prefer having my infrastructure devices on fixed addresses. A static route named "vpn_router" is established so that packets to the VPN router, on IP address VPN-GW-Public-IP, are always routed through my home LAN gateway with IP address Home-Router-IP. For all other packets, a default route to the VPN gateway with the address VPN-GW-VPN-IP is established. A VLAN interface named "vlan0" is created and the address Home-VPN-IP is assigned to that virtual interface. Finally, IP packet forwarding is enabled.

The OpenVPN router is started from /etc/rc.conf:

openvpn_enable="YES"
openvpn_configfile="/usr/local/etc/openvpn/client.conf"

On the VLAN segment, a DHCP server is run by the way of these lines in /etc/rc.conf:

dhcpd_enable="YES"
dhcpd_flags="-q"
dhcpd_conf="/usr/local/etc/dhcpd.conf"
dhcpd_ifaces="vlan0"
dhcpd_withumask="022"

DHCP server configuration

The DHCP server configuration for the vlan0 segment is placed in /usr/local/etc/dhcpd.conf and reproduced for completeness:

option domain-name-servers VPN-GW-VPN-IP;

default-lease-time 600;
max-lease-time 7200;

log-facility local7;

subnet Home-VPN-LAN netmask 255.255.255.0 {
  range 192.168.21.100 192.168.21.120;
  option routers VPN-GW-VPN-IP;
}

Obviously, the range needs to be adapted to the network in use.

OpenVPN configuration

OpenVPN uses SSL certificates to authenticate clients and servers. Creating and maintaining a certificate authority using the standard OpenSSL command line tools is too cumbersome for me. I am using the excellent and free XCA tool which makes managing a private certficate authority (CA) rather easy. I used XCA to create a CA, a server certificate for my central VPN router as well as a client certificate for my Pogoplug. Certificates and keys must be exported to PEM files for OpenVPN to use them.

I use a fairly standard LAN-to-LAN configuration for OpenVPN which is based on the server.conf and client.conf example configuration files which are located in /usr/local/share/examples/openvpn/sample-config-files. The client configuration used on the Pogoplug looks like this:

client
dev tun
proto udp
remote VPN-GW-Public-IP 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
ns-cert-type server
comp-lzo
verb 3

Again, VPN-GW-Public-IP is the public IP address of the central VPN router. The client certificate and key need to be placed in the client.crt and client.key files in /usr/local/etc/openvpn/ on the Pogoplug. The root CA certificate that you've created needs to be placed in the ca.crt file in the same directory.

On the server, the OpenVPN configuration is slightly more complex. Here is the server.conf file:

port 1194
proto udp
dev tun
ca ca.crt
cert server.crt
key server.key
dh dh1024.pem
server VPN-GW-VPN-IP 255.255.255.0
ifconfig-pool-persist ipp.txt

client-to-client

push "route 0.0.0.0 0.0.0.0"
push "route Home-VPN-LAN 255.255.255.0"

client-config-dir ccd
route Home-VPN-IP 255.255.255.0

keepalive 10 120
comp-lzo
persist-key
persist-tun
status openvpn-status.log
verb 3

Again, Home-VPN-IP is the IP address of the vlan0 interface on the Pogoplug. The Home-VPN-LAN is the network address of that interface, and the "push" directive is there to push a route to that network to other clients connecting, which may choose to ignore the default route. On the server, a ccd subdirectory needs to be created in /usr/local/etc/openvpn/ to contain client-specific configuration options. Each file needs to have the name that is put into the "Common Name" attribute of the certificate used by the client. For example, I have a file named strelitzer in that directory which is the Common Name of the certificate of my Pogoplug. That file contains this:

iroute 192.168.21.0 255.255.255.0

This directive announces my VPN LAN at home to the OpenVPN router.

Switch configuration

The GS108E switch must be configured for 802.1Q VLANs in "Advanced" mode. In that mode, it is possible to individually set up the VLAN membership for each port. The default VLAN for untagged packets can also be configured on a per-port fashion.

Here is how I configured my switch so that ports 1-4 are members of VLAN 1 (VPN) and ports 4-8 are member of VLAN 2 (Home Network):

Ports 1-3 use VLAN 1 for untagged packets. Port 4 uses VLAN 1 with tagged packets and VLAN 2 with untagged packets. Ports 5-8 use VLAN 2 with untagged packets:

Finally, I configured the PVIDs for each port like so:

Done

That is basically all that is to it. I'm sure I've left out some information that might be useful. Send me email if you have trouble with any of this. It should be possible to adapt this setup to FreeBSD running on other hardware, like the Raspberry Pi. Let me know if you get something like that to run, too!

DRAKMA documentation updates

The DRAKMA documentation has received a big overhaul to bring the docstrings and the HTML documentation in sync again. I also restructured the large example section in the beginning so that the individual examples are indexed. I also added changed two examples so that they deal with JSON data as JSON is so popular nowadays.

DRAKMA and JSON

A user suggested that DRAKMA should handle JSON data like text data, i.e. that the content type "application/json" should, by default, be on the list of text content types. The issue with that suggestion is that JSON data is not following the encoding rules for text content type. By default, JSON data is encoded in UTF-8. It is possible to use UTF-16 and UTF-32 by the way of a byte order mark transmitted in the beginning of the file. For textual content types, the character set is determined by the charset parameter in the Content-Type header, which is not used with "application/json".

I have thought about adding a DRAKMA:HTTP-JSON-REQUEST function that would deal with JSON encoding properly, but I think such a function would rather belong into a JSON library than into DRAKMA, so it is left out for now.

How useful are docstrings, really?

Getting the docstrings and the documentation into sync again was pretty laborious, and I fear that I have been missing quite a few things that were in one version of the documentation, but not in the other. So far, Edi and I have used a forward-annotation strategy where the initial version of the documentation was created off the docstrings in the source code, and then manually edited to contain links and other niceties that real documentation deserves. This, of course, is a pretty bad strategy, as edits must be repeated or the docstrings need to contain some form of markup so that no edits are needed.

Going forward, I will probably start to generate docstrings from the XML documentation source. That way, the documentation source can contain links and other, more advanced markup that is not useful in docstrings, and the two representations will not go out of sync. The docstrings will no longer be inline with the source code, but go into a separate file that is generated as part of the release process.

Any thoughts? Use comments or send email.

lparallel and how I was thinking too complicated

Yesterday, I posted a short article that showed how I used lparallel to speed up a lengthy computation by reading input data files in parallel. A few readers responded, and the response from James M. Lawrence, who is the author of lparallel, is reproduced here because it contains source code that can't be properly formatted in blogspot comments (any suggestions regarding something better?).

[citing James M. Lawrence]
Here is a map-reduce-style version:

(defun slurp-alist (hash-table alist)
  (loop
    :for (key . data) :in alist
    :do (setf (gethash key hash-table) data))
  hash-table)

(defun load-files-in-parallel (pathnames)
  (flet ((parse (pathname)
           (let ((result nil))
             (read-and-parse-file
              pathname
              (lambda (key data)
                (push (cons key data) result)))
             result)))
    (reduce #'slurp-alist
            (lparallel:pmap 'vector #'parse pathnames)
            :initial-value (make-hash-table))))

The benefit here is that worker threads do not have to contend for a queue lock. A possible drawback is that the key-data pairs survive until the final reduce, whereas the pairs have a shorter lifetime when passed through a queue. Of course performance will depend upon the scale and shape of the data.

Benchmarks with early versions of lparallel showed better performance with a vector-based queue, however since then I found a simple cons-based queue to be generally faster, which is currently the default. There is still a compile-time option to get the vector-based queue, and unless that is enabled the number passed to make-queue is ignored. In other words, the 10000 is a lie :)

Motivated by James' reply, I spent some time to find out how the process could be further optimized. The idea to have the workers cons up the data and then collect all data into the hash table at the end of the operation reduced the run time for my application by some 30%, which is not bad.

A comment from Max Mikhanosha on G+ suggested that I could avoid the queue and directly use a thread safe hash table. It is a little embarrassing that I assumed that I could not write hash tables from multiple threads. Both CCL's and SBCL's hash tables provide for thread safe modes, so there is no point in tunneling the data through a queue.

As I am using CCL, I tried different values for the :SHARED keyword argument to MAKE-HASH-TABLE. The default is :LOCK-FREE, which makes hash tables thread safe without requiring the grabbing of a lock for each operation. Other permitted values are T, which locks the table for each access, or NIL for thread-unsafe hash tables.

I found that when writing to the hash table directly from the file readers, :SHARED T would give me the best performance. With that option, my code ran another 10% faster than with consing up the file results and then processing all the lists at the end.

lparallel is a great example how Common Lisp is an extensible language. The operators provided by it seamlessly integrate parallel programming into the language, and with the help of my readers, the code that I originally posted now became very small:

(defun load-files-in-parallel (pathnames)
  (let ((hash-table (make-hash-table :shared t)))
    (lparallel:pmap nil
                    (lambda (pathname)
                      (read-and-parse-file pathname
                                           (lambda (key data)
                                             (setf (gethash key hash-table)
                                                   data))))
                    pathnames)
    hash-table))

Look, Ma! All I needed was pmap!

Using lparallel to utilize multiple cores [updated]

A while ago, I discovered the lparallel library as a nice set of abstractions to make parallel programming in Common Lisp easier and more pleasant. The use case that I presented in my previous blog post was more about dealing with asynchronicity rather than parallelism, though. As I have seen people ask for lparallel examples, I will present how I used it to actually speed up a long running computation by having multiple cores work on the problem at the same time.

The problem that I have been working on involved reading and parsing a few large data files to extract some data from them. The extracted data needed to be stored in one hash table for later processing. The reading and parsing process is CPU bound, so reading the data files in parallel will speed up the overall process. "Speed up" here means "reduce the run time because I'm waiting for completion", not "make it as fast as possible" or "have it utilize all cores". I'm trying to reduce my own wait times, and I cannot spend time debugging locks. Thus, the high abstraction level of lparallel fits me just right

For the task at hand, I created a set of workers which read the input files and send the extracted data items to a collector process by the way of a queue. The collector process puts them into a hash table for later processing. Once all files are read, the hash table is returned.

(defun load-files-in-parallel (pathnames)
  (let ((queue (lparallel.queue:make-queue 10000)))
    (lparallel:future
      (lparallel:pmap nil
                      (lambda (pathname)
                        (read-and-parse-file
                         pathname
                         (lambda (key data)
                           (lparallel.queue:push-queue (list key data)
                                                       queue)))
                        (lparallel.queue:push-queue :done queue))
                      pathnames))
    (let ((hash-table (make-hash-table))
          (remaining-worker-count (length pathnames)))
      (loop while (plusp remaining-worker-count)
            for item = (lparallel.queue:pop-queue queue)
            if (eq item :done)
              do (decf remaining-worker-count)
            else do (destructuring-bind (key data) item
                       (setf (gethash key hash-table) data)))
      hash-table)))

Some points are worth noting:

• Instead of using a queue to send the data to a separate worker for hash table insertion, the hash table could be locked or a lock-free hash table could be used. This may or may not be faster.
• The scheduling of the workers is done by lparallel and may or may not be efficient. I've looked at CPU utilization and found the process to use 3-6 cores, so it seems that the scheduling is not optimal. I'm guessing that the queue processing and hash table insertion eat more time than I'd hope for, but I have not made a real assessment.
• I've picked the size of the queue randomly. It may be worthwhile to make the queue larger so that file readers are not blocked while hash table rehashing occurs.

Implementing this parallel file loader took me just a few minutes and gave my program considerable speedup. Using memory is already easy, and with lparallel, using multiple cores also becomes much easier than it has been for me.

[Update] James M. Lawrence, the author of lparallel, kindly pointed out that the original version of my example code contained an unneeded forced future to read the data. I have updated the code according to his remark.

Orphaned projects on common-lisp.net

We have lost contact to the maintainers of the following projects on common-lisp.net:

• fret
• ganelon
• innen
• erlisp

If you are the original maintainer of any of these, please tell us your current email address or let us know if we can delete the project.

lparallel - A Parallel Programming Library for Common Lisp

Parallel programming is hard, and as CPUs are getting ever-faster, I usually tend to find optimizing a single thread of control to be less risky than dealing with threads, locks and synchronized data structures. Recently, though, I had to deal with a reporting function which I started from a Hunchentoot handler and that was running long enough to time out the client. I needed to somehow put running the function into the background, and as I heard good things about the relatively new lparallel library, I thought I could give it a try.

lparallel implements an abstraction layer for parallel programming in Common Lisp. In addition to relatively low-level concepts like tasks and channels, it implements mid-level promises and futures as well as high-level parallel mapping, binding and reducing functionality. Futures looked like a suitable mechanism to solve my problem at hand.

The non-parallel version of my HTTP request handler looked something like this:

(hunchentoot:define-easy-handler (something-that-takes-long :uri "/sttl")
    (parameter)
  (if (eql (hunchentoot:request-method*) :post)
      (compute-and-display-response-page parameter)
      (show-job-parameter-form)))

The compute-and-display-response-page function does what the name suggests. As usual with Hunchentoot request handlers, it returns the page to display to the user. Likewise, the show-job-parameter-form returns a HTML form that collects the parameter. Now, compute-and-display-response-page potentially takes long, so instead of just delaying the response to the HTTP request until the function returns, I want to have it execute in the background and respond to the request with a "job has been started" message. The client is then supposed to poll in regular intervals. Once the background job completes, the page that the function has generated is returned to the client.

Using a future, I came up with something like this:

(hunchentoot:define-easy-handler (something-that-takes-long :uri "/sttl")
    (parameter)

  ;; Get or start Hunchentoot session context
  (hunchentoot:start-session)

  (let ((job-running (hunchentoot:session-value :job)))
    (cond

      ;; Previously started job has finished
      ((and job-running
            (lparallel:fulfilledp job-running))
       (hunchentoot:delete-session-value :job)
       (lparallel:force job-running))

      ;; Previously started job still running
      (job-running
       "Previous job still running")

      ;; Start new job
      ((eq (hunchentoot:request-method*) :post)
       (setf (hunchentoot:session-value :job)
             (lparallel:future
               (compute-and-display-response-page parameter)))
       "The job has been started")

      ;; Display job parameter form
      (t
       (show-job-parameter-form)))))

Hunchentoot's session mechanism is used to make the future accessible to subsequent requests. The future is placed in a session value; the completion of the background calculation is determined by calling lparallel:fulfilledp. Once the computation is finished, the return value of the compute-and-display-response-page is determined using lparallel:force.

This is mostly it, and I find parallel programming in this case easy to understand and reason about. Some additional things are worth mentioning:

• To use lparallel, one has to create a kernel, which constitutes the worker thread pool in which futures and other lparallel tasks are executed. Multiple kernels can coexist, the kernel to use is determined by the lparallel:*kernel* special variable. In my application, I am creating a kernel with one worker thread. This provides me with automatic request serialization so that only one long-running function invocation exists at any time, without the need to do any locking or explicit resource management. I like that a lot.
• By default, errors that occur in a worker thread cause the debugger to be entered. This can be changed by setting lparallel:*debug-tasks-p* to nil. Then, errors will be caught and signalled to the caller of lparallel:force, which in my case was the right thing to do (i.e. if the background process triggers an error, I want the backtrace to be sent to the client.
• One thing I find slightly bothersome is the reverse order of the cond clauses in the request handler - During the lifetime of a typical session, the last clause of the cond is true first, then the second to last and so on. This is a pattern that I'm seeing quite often when writing http request handlers. Maybe a cond-reverse macro would be suitable in this case.

lparallel has more to offer of course, and I have only used a very small part of it. So far, I have found it to be very well thought out, documented appropriately and well maintained. If you need parallel programming or even just easy background execution in your Common Lisp programs, I recommend looking at it.

BVG - Es ist Zeit, dass Ihr die kaputten U-Bahn-Fahrkartenautomaten ENDLICH repariert

Ich zeige Euch mal, wie Fahrkarten-Automaten in DDR-Straßenbahnen funktioniert haben, vor 25 Jahren:

Durch Ziehen des Hebels wurde das eingeworfene Geld - hinter einer Glasscheibe sichtbar - nach rechts gedreht und ein Fahrschein ausgegeben. Durch die Scheibe konnte man also sehen, was die beiden vorherigen Fahrgäste bezahlt hatten.

Man konnte den Hebel bedienen, ohne Geld eingeworfen zu haben, und erhielt trotzdem ein Ticket. Gedacht war es so, dass die Bezahlung "sozial" kontrolliert wurde, d.h. jeder im Wagen konnte sehen, ob gerade jemand bezahlt hatte und ob der richtige Betrag eingeworfen wurde. Zeitkarten-Inhaber hatten ihre Zeitkarte in die Höhe zu halten, um so zu erklären, dass sie nicht zur Zahlbox gehen mussten.

So weit, so gut. Man mag jetzt darüber philosophieren, ob das nur in einem Spitzelstaat funktionieren kann oder einfach locker und normal war, mich hat das als westberliner Kind auf Besuch jedenfalls beeindruckt. Simpel, einfach, unkompliziert.

Warum ich das schreibe?

deshalb.

Erstmal: Wenn ich eine Fahrkarte kaufen möchte, dann, weil ich fahren will. Ich bin nicht an "anderen Angeboten" interessiert, und ich fände es auch total unhöflich gegenüber folgenden Fahrgästen, wenn ich mir noch die interessanten "anderen Angebote" ansehen würde. Die könnten es ja auch eilig haben. Das scheint bei der BVG aber niemanden zu interessieren, und deswegen wird auf dem Fahrkartenautomaten erst mal eine Werbe-Seite angezeigt mit irgendwelchen irrelevanten Informationen, die noch nie jemand gelesen hat, ausser, wenn er eine Fahrkarte kaufen wollte und herausfinden wollte, wie das geht. Irgendwo in diesem Werbemüll steht "Berühren sie den Bildschirm", und wenn man das macht, dann wird auch tatsächlich die Auswahlseite für die Fahrscheine angezeigt.

Diese Werbeseite ist komplett sinnlos und muss weg. Alle, die sich diesem Automaten nähern, wollen Fahrkarten kaufen. Wozu zur Hölle müssen diejenigen, die es noch nicht wissen, diesen Bildschirm angucken und herausfinden, dass sie ihn berühren müssen? Es gibt da auch nichts zu "saven" oder so. Es kostet einfach nur zusätzliche Zeit, diesen Bildschirm zu verstehen. Er fügt der Dienstleistung "öffentlicher Nahverkehr" keinerlei Mehrwert hinzu. Wenn es noch zusätzliche tolle Sachen gibt, dann sollten die bitteschön auf Werbeflächen beworben werden, nicht auf einem Fahrkartenautomaten, den die Fahrgäste dazu benutzen, sich eine Fahrkarte zu kaufen.

Okay fein, ich habe also den Bildschirm berührt und der Automat zeigt mir an, welche Fahrkarten er mir verkaufen kann. Ich muss zugeben, dass sich hier in den vergangenen Jahren, seit die BVG die grandiose Idee hatte, die Automaten, welche für jeden Fahrkartentyp eine eigene Taste haben und die in der Straßenbahn auch nach wie vor im Dienst sind, durch diese Touchscreen-Windows-Gurken zu ersetzen, einiges getan hat. Früher hatte diese Schrottsoftware auch noch eine Latenz im Sekundenbereich, heutzutage werden immerhin die wichtigsten Fahrscheine direkt auf der Einstiegsseite zum Kauf angeboten. Wenn man Glück hatte und gerade vorher jemand anderes einen Fahrschein gekauft hat, muss man also nur den Fahrschein auswählen und darf dann gleich bezahlen.

Aber:

Das Bezahlen dauert immer noch so lange, dass ich mich dem Automaten inzwischen stets mit dem ausdrücklichen Bewusstsein nähere, den gleich kommenden Zug nicht erreichen zu können. Nur so kann ich Frustration vermeiden und gelegentlich komme ich sogar in die Situation, einen Fahrschein erworben zu haben, bevor ein Zug ein- und ausgefahren ist, was mir dann kleine Glücksgefühle beschert.

Das Prüfen meines Geldes benötigt in diesem Wunderwerk der Technik nämlich tatsächlich Sekunden. Wenn ich mehrere Fahrscheine kaufe, dann wird zwischen dem Ausdruck der Fahrscheine jeweils mehrere Sekunden gewartet, und das Auszahlen des Restgeldes muss der Automat sich auch sekundenlang überlegen. Ich weiss, dass das nicht sein muss. Ich weiss, das das schneller gehen kann, wenn man die Zeit des Fahrgastes wertschätzt und nichts unversucht lässt, den eigentlich unnötigen Vorgang des Fahrscheinerwerbs so schnell wie möglich durchzuführen.

Ganz bizarr wird es, wenn man mit seiner EC-Karte bezahlt. Das man bei uns immer eine PIN eingeben muss, und nicht, wie in vielen anderen Ländern der Welt, einfach seine Karte durchzieht und damit in einer oder zwei Sekunden bezahlt hat, das prangere ich in diesem Zusammenhang noch nicht mal an. Dass ich aber, wenn ich einen Kurzstrecken-Fahrschein für 1,30 Euro gekauft habe, noch warten muss, bis mir ein komplett sinnloser und mit total überflüssigen Fakten bedruckter "EC-Beleg" produziert wurde, das kann ich einfach überhaupt nicht begreifen. Auf dem Zettel steht, ich soll ihn aufbewahren, aber daran denke ich doch gar nicht. Wie kommen die eigentlich dazu, mir dieses Papierstück aufzudrängen? Nur aus Höflichkeit gegenüber meinen Mitreisenden lasse ich den Müll nicht direkt im Automaten liegen, sondern schmeisse ihn in den nächsten Mülleimer, aber das ist einfach der Gipfel des Schwachsinns. Auf dem Fahrschein ist genug Platz, um das Zahlungsmittel zu dokumentieren, wenn das tatsächlich von der Buchhaltung so verlangt wird.

Um bei dieser Kurzstrecke zu bleiben: Den Fahrschein hab ich nun also, der Zug steht bereit und ich will ihn nehmen, aber jetzt muss ich das Ding auch noch entwerten? Auf dem Bahnsteig? Wie bescheuert ist das denn eigentlich? Der Vorverkauf ist ja wohl der Sonderfall, und der normale ist, dass man hier seinen Fahrschein für den sofortigen Fahrtantritt kauft. Wenn dann, wie am U-Bahnhof Rosenthaler Platz, nur ein Entwerter für zwei Automaten zur Verfügung steht, dann ist der Zug eben doch wieder weg, bevor man den Entwerter gefunden und das kostbare Dokument validiert hat. Ich hab's ja so geplant und deswegen ist es auch keine grosse Enttäuschung, aber

Wer auch immer bei der BVG für diesen Kack verantwortlich ist: Sie sind ein Amateur. Sie gehören nicht auf diese Stelle. Machen Sie Platz für jemanden, der seine Aufgabe ernst nimmt und für den das Wichtigste nicht das sorgfältige Zählen des Fahrgeldes, sondern die bequeme und zügige Beförderung der Fahrgäste ist.