Wednesday, February 8, 2012

TRACEs of awesomeness

Debugging is hard, and I admit it freely: I am using print statements regularly to debug code that I'm writing, in particular when not writing Lisp code. With Lisp and its interactive development environment, though, my debugging activity often involves calling functions from the repl and tracing individual functions.

The standard TRACE facility that Common Lisp offers often replaces adding debug print statements for me. In order for tracing to help, the code needs to be structured so that a function is called at the place that is interesting in the particular debug situation. In many situations, one is not really interested in all invocations of a function, but only under some conditions. When debugging functions that are called often or that have large argument lists, simple traces can easily grow huge and analyzing trace logs can become a pain. Also, the trace output can slow down processing enough to make the simple approach fail.

Where TRACE's full power lives

The CLHS entry on TRACE suggests that there may be implementation dependent arguments, and in fact most implementations provide additional options that make the TRACE facility much more powerful. In this post, I'm going to show you a few examples of advanced TRACE options in three implementations to show what one may expect to be available. The examples and implementations chosen are pretty arbitrary and are not indicative of what the particular tracing facilities offer. Please look up the details in the respective documentation.

Breaking with CCL

Clozure CL's TRACE supports :BREAK-BEFORE to enter the debugger before a function is called:

Welcome to Clozure Common Lisp Version 1.7-r15160M  (DarwinX8664)!
? (defun foo ())
FOO
? (trace (foo :break-before t))
NIL
? (foo)
0> Calling (FOO) 
> Break: FOO trace entry: NIL
> While executing: (CCL::TRACED FOO), in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Return from BREAK.
> Type :? for other options.
1 > :go
hello
<0 FOO returned NIL
NIL
Notice how the debugger is entered before FOO is called.

Statically selective tracing with Allegro CL

Sometimes, one only wants to trace a function when it is called from a certain other function. Allegro Common Lisp's tracer offers the :INSIDE option:

International Allegro CL Enterprise Edition
8.2 [64-bit Mac OS X (Intel)] (Dec 21, 2011 13:59)
Copyright (C) 1985-2010, Franz Inc., Oakland, CA, USA.  All Rights Reserved.
[...]
cl-user(1): (defun foo (arg) (format t "foo called, arg ~A~%" arg))
foo
cl-user(2): (defun bar () (foo "from bar"))
bar
cl-user(3): (defun qux () (foo "from qux"))
qux
cl-user(4): (defun main () (bar) (qux))
main
cl-user(5): (trace (foo :inside qux))
(foo)
cl-user(6): (main)
foo called, arg from bar
 0[2]: (foo "from qux")
foo called, arg from qux
 0[2]: returned nil
nil
Note how the function FOO is called with no tracing when called from BAR, but with tracing from QUX.

Dynamically selective tracing with SBCL

Sometimes, one wants to trace a function based on some dynamic condition. SBCL's tracing facility offers this, as well as programmatic access to the argument list of the function being called:

This is SBCL 1.0.55.0-abb03f9, an implementation of ANSI Common Lisp.
[...]
* (defun foo (arg) (format t "foo called, arg ~A~%" arg))
FOO
* (defun bar () (foo 1) (foo 2))
BAR
* (trace foo :condition (= (sb-debug:arg 0) 2))
(FOO)
* (bar)
foo called, arg 1
  0: (FOO 2)
foo called, arg 2
  0: FOO returned NIL
NIL
Note how tracing is only enabled when FOO is called with a specific argument value.

Wrapping up

The purpose of this post was to show you some of the non-standard tracing options that Common Lisp implementations offer. As you can see, they are powerful debugging tools that can be used with no source code changes required. As these options are not standardized, make sure that you check out the documentation of your implementation. Happy debugging!

Tuesday, January 10, 2012

Berlin Lispers Meetup: Tuesday 24 January 8 pm at co.up

You are kindly invited to the next Berlin Lispers Meetup, an informal gathering for anyone interested in Common Lisp and other languages in the Lisp family.

Berlin Lispers Meetup
Tuesday January 24, 2012
8 pm onwards
co.up (bell: Upstream Agile), Adalbertstraße 7, 10999 Berlin (U-Bahn Kottbusser Tor)

There will be two presentations:

  • "Further notes on sparql processing" by James Anderson and Arto Bendiken who are affiliated with Datagraph, Inc.
  • "A suggestion for parentheses representation in a Common Lisp IDE" by Nuno Rocha.

Please join for another evening of parentheses!

Twitter: @BerlinLispers

Sunday, December 11, 2011

Learning Ruby, and Ruby vs. Lisp

The company I work for has a lot of legacy Ruby code, and as Ruby has become kind of a mainstream language, I decided to get a book about it and learn how it works.

As my learning resource, I chose The Ruby Programming language by David Flanagan and Yukihiro Matsumoto as that receives great customer reviews, covers Ruby 1.8.7 and 1.9 and is authoritative because the language creator is one of the authors.

The book makes a good read in general. There are plenty of code examples, but not too much to obscure the prose. What I found first interesting, later annoying, was the frequent use of words like "complex", "complicated", "confusing", "surprising" or "advanced" to describe features of the language. I'd rather decide myself about using such attributes to describe something that I've just learned.

Having spent so much time with Common Lisp, I almost forgot that programming languages usually evolve over the years. Ruby is no exception, and the fact that there are significant differences between Ruby 1.8.7 and Ruby 1.9 kind of bothers me - I'll probably never write code in Ruby 1.8.7, but the differences between the two versions seem to be rather subtle and I'm curious to see how much that is going to be a bother in the future, working with legacy code.

The common theme for Ruby seems to be succinctness. This comes at the expense of making the syntax rather complex, with several special case rules required to solve ambiguities. I don't have the practice to judge whether this is a problem, but from the book, it seems there are quite some things to remember.

It seems that Ruby started off as a purely object oriented language and only later discovered that function-oriented programming is nice, too. The deep roots of object orientation made it rather hard to actually get free functions (which are not member functions of an object) integrated. Contrary to what I am used to, member functions are not a special case of free functions, but rather something quite different. It requires explicit conversion steps to convert a member function into a free function (called procs in Ruby), and invocation syntax is also different between the two. Again, the description may sound worse than it is in practice.

What I really liked was the generalization of code blocks into fibers. Ruby does not have full coroutines, but the restricted form that is available is generalized well and seems like it could be useful for building pretty wild asynchronous systems. Also, it is nice that the bindings of closures can be accessed.

But then, Ruby is an interpreted language and this fact is re-stated throughout the book. With Just In Time compilation, this could become a non-problem, but I'm not sure how well Ruby can be optimized due to its very dynamic nature. Just to see how fast it is compared to Common Lisp, I implemented the Sudoku solver from chapter 1 of the Ruby book in CL and gave the two implementations a puzzle to solve. It took the Ruby solver 0.890 CPU seconds (Ruby 1.9.2p290), whereas the Lisp solver (Clozure CL 1.7) used 0.087 CPU seconds to solve the puzzle. Ten times slower, whatever you'll make of that.

In the book, it is mentioned how little code the Sudoku solver actually uses. This is true, but then, the Lisp version is not longer. It does not seem as if adding syntax is actually the best way to add the possibility to write succinct programs to a language, and the price of the complex grammar is rather high.

Writing the CL solver, I found myself not writing tests again and then poking around in problems of my implementation without knowing what works and what does not. As I want to practice more TDD, I stepped back and added tests. This led me to solve a problem that I had with my previous attempts to practice TDD in Lisp - I do not want to export all the symbols that the tests exercise from the packages that I use, but I also don't want to import the unit testing library into my own library packages. Thus, I wrote a deftestpackage macro that creates a new package to contain the tests that I write and automatically imports all symbols from the package being tested. That way, I can easily keep tests and library source separate and don't need to qualify internal symbols in the tests.

My overall takeaway on the Ruby is this: Ruby seems to be a language that has grown from being purely object oriented to supporting functional programming. That growth was not completely natural, and it seems that if Ruby is not used as a pure object oriented language, the syntax becomes rather messy and hard to grasp. This is similar to C++, which in its first versions was relatively nice (I hear you "ow"!), but has grown into into an incomprehensible mess once people recognized how templates can be abused for metaprogramming.

I can see the appeal of Ruby, but there seems little it has to offer to me that Common Lisp cannot provide. The lack of a formal specification and the ugly grammar put me off. Then again I'm pretty sure that Ruby is more enjoyable than many other popular languages. I'm looking forward to see my theoretical conceptions be shaken by actual practice.

Monday, December 5, 2011

Global Day of Code Retreat with Lisp

Last Saturday, I attended the Global Day of Code Retreat. I found out about the event in my Twitter stream, and when I signed up for the event, I did not have much of an idea of what it would be about, except code. With some research, I found that it would be a day of using Test Driven Development practices to improve on coding style and quality. Fair enough I thought, got myself a book on TDD (which I have always wanted to do more) and reserved my seat, which was free.

The event was hosted by immobilienscout24 and sponsored by Nokia. Sponsoring included catering, and as the day started at 8:00am, the availability of a breakfast was very welcome. The crowd consisted of some 30 hackers. An informal poll showed that most of them were either using Java or Ruby as their first language, a few were into JavaScript. One guy said that C++ was his primary language, and I was the only Common Lisp hacker.

The format of the all-day event consisted of six sessions of 45 minutes. In every session, people would form new pairs and implement parts of Conway's Game of Life using TDD practices. For each session, the pair chooses a specific challenge or goal to solve. At the end of the session, every pair deletes their code, discusses what they learned and then joins the group to share their thoughts.

The focus of the goals and challenges was on TDD practices. For about a half of the participants, TDD was already part of their daily work routine. The other half had heard of TDD, like myself, and joined to learn about it in the event. For me, this was one of the bigger takeways. I found it very helpful to pair with people that practice TDD in order to learn how they'd go about to write a test before they'd write an implementation. It seems that the TDD school favors a very small test granularity. As was explained by the organizer of the meeting, one would write very small tests that exercise very small aspects of the production code initially, and then build up on that.

In the six sessions, I used Java, JavaScript and Common Lisp. For Java, the organizers had come up with a skeleton project that included a unit testing environment (jUnit?). For JavaScript, I had prepared a simple browser based environment with QUnit, for Common Lisp I used Clozure CL and the simple unit-test library that I have used in the past.

Here are some observations:

The Java dudes travel with heavy baggage

IntelliJ was the most popular IDE among the Java folks that I've talked to. I am split between admiration and disgust when it comes to what seems common practice in the Java world. I'm used to write code by thinking about what I want to write, then typing the stuff that I've thought about. With Java and IntelliJ, the process is more interactive with the IDE, i.e. the programmer types a few characters, the IDE displays some menu to choose from or automatically adds code and so on. This is nice when the IDE automatically recognizes, say, that you are using a name of a class that has not yet been defined or imported and then offers you to either import a package that defines a class of the name that you've typed, or to create a class skeleton (including all the boilerplate and ceremony that Java wants) for you.

While this appears to be convenient, it also does not always work. In both of my Java sessions, the IDE got confused in some way or another, which was not fatal, but still annoyed me. Also, the static nature of Java slowed us down. For one, even though Java does have all the nice data structures (we wanted to use a set of coordinates), we wasted a lot of the short session times converting data types and supplying boilerplate that allowed us to actually put coordinate objects into a set. Also, file based compilation took time - Not minutes, but seconds. I was assured that one would use libraries and code generators in production code and that desktop machines were faster in compiling, but I still can't really relate Java to Agile, in the sense of the word.

Also, the use of code generators in IntelliJ makes me wonder how one maintains such code. How can one actually distinguish between what was carefully crafted and what was pasted, from templates, by the IDE? In my eyes, this is like copy and paste programming with the copy step optimized into templates. I'm not the first to say that and it does not come as a surprise either, but it was an interesting experience for me nevertheless.

Pair programming is fun

Being a remote consultant, I rarely have the chance to interact over code with other programmers. This was something I found enjoyable to practice, even in languages and environments that I'm not familiar with. It is amazing how vastly different the approaches to implementing a relatively simple thing can be, and compared to code reviews - in particular if they're done by email - it is much easier to make suggestions in a constructive way. In that respect, writing code that is supposed to be deleted also helps concentrating on the essence as no sense of code ownership is developed.

I found the pair programming experience significantly impacted by the fact that programmers use different coding environments. Working on an unfamiliar keyboard and with an unfamiliar IDE is a real productivity killer. I'd hope that in a team that pairs regularly, the work environments are more standardized than they have been on this event. I was using Emacs for my Lisp and JavaScript sessions, and my partners had a hard time getting along. What I found rather interesting was that the guys who wrote JavaScript in my Emacs always mimiced what their IDE would do for them. Rather then writing "if (foo) { doSomething()", they'd type "if (foo) {}", then navigate into the braces with the cursor to code along. This is kind of curious because it seems that the balancing of parentheses, brackets and braces is much more of a chore in other languages, to the point where people slow themselves down a lot if not aided by an IDE to the balancing for them. We Lispers, with only the parentheses to keep track of, have a much easier life, in particular with Emacs doing the indentation for us.

Is TDD the kool aid?

In some sense, I am sceptical about TDD. Testing is a great idea, and doing it in a disciplined fashion certainly helps writing better quality software. Also, automated tests are really the best way to prepare software for change. But, and this is where this Saturday could not convince me, I don't believe that spending time on writing very fine grained unit tests for every aspect of of a program helps preparing it better for refactoring and change. I think that testing against external requirements is the real key to writing programs that can be changed facing changing requirements, and that it should be possible to relate every test to some requirement. I must admit that one Saturday of TDD does not give me sufficient experience to judge, though.

Common Lisp and TDD

Some of the TDD discipline probably owes to development cycle in statically compiled languages. Where we Common Lispers have a rather incremental style and do our testing interactively in the repl, developers in languages like Java or C++ write larger chunks of code before they plug it together to do something meaningful. Such environments give the developer less insight into the run-time behavior of a running system, and tests are a way to make sure that more of the interactions of the system components are actually exercised.

I am not claiming that an interactive development environment makes testing less useful. In such an environment, though, it is common to write and test a function in small, iterative cycles. Furthermore, through the use of a tracing facility, the dynamic bahavior of a system can be observed at any time, without the need to touch or recompile the code.

In any case, this was a very nice experience and I'll go to a Code Retreat again, if I can. For that, I'll probably prepare myself for Java tools better in order to get more out of pair programming.

Sunday, October 30, 2011

Hunchentoot v1.2.0 released

After over a year of no releases, Hunchentoot v1.2.0 is out. It incorporates a large number of changes from various people and is the first release that I have made as the maintainer of Hunchentoot. There have been a fair number of changes, many of them contributed by Hunchentoot's users. Thank you all for that!

The release tarball is available here: https://github.com/downloads/edicl/hunchentoot/hunchentoot-1.2.0.tar.gz

This is my first release, and as such it is likely that I'm not yet up to the release standards that Edi has set forth. Please report any problems you may have with the new version to the Hunchentoot mailing list. I will try to resolve problems quickly and possible make a new minor release for each major problem found.

The development repository for Hunchentoot is now hosted on github. If you want to contribute, it would be appreciated if you could submit your patches through github, too.

Here is a list of changes:

  • Add www/ directory with default file tree that is being served
  • Add error template mechanism and improve error reporting in general.
  • Improve automatic testing, SBCL kludge to support asdf:test-op
  • Allegro CL modern mode fixes
  • Fix bugs in serving partial responses
  • Limit maximum number of threads that Hunchentoot creates (Dan Weinreb, Scott McKay)
  • Export fixes (Gordon Sims, Andrey Moskvitin)
  • Factor out easy-handler logic into separate acceptor subclass
  • Export two session functions (Nico de Jager)
  • Allow no Content-Type header (Chaitanya Gupta)
  • Patch for compilation with ECL (Sohail Somani)
  • Fix DEFINE-EASY-HANDLER for multiple acceptors (Nicolas Neuss)
  • Revived *SHOW-LISP-BACKTRACES-P*
  • Made sure "100 Continue" is returned even if the client sends "Expect: 100-continue" twice (reported by Gordon Sims)
  • Fixed typo in code which interprets transfer encodings

Thank you for your patience!
Hans

Sunday, October 16, 2011

Berlin Lispers Meetup: Thursday 20 October 8 pm in St Oberholz

You are kindly invited to the next "Berlin Lispers Meetup", an informal gathering for anyone interested in Lisp, beer or coffee:

Berlin Lispers Meetup
Thursday October 20, 2011
8 pm onwards
St Oberholz, Rosenthaler Straße 72, 10119 Berlin
U-Bahn Rosenthaler Platz

We will try to occupy a nice table at the floor level, but in case you don't see us please contact Hans: 0177 512 1024.

Please join for another evening of parentheses!

Tuesday, September 27, 2011

Finding Changes to Quicklisped Software

In my current project, I'm working with Allegro CL in modern mode. Modern mode is a non-standard mode supporting mixed-case symbols in the reader. This means that in modern mode, the two literal symbols :foo and :FOO are different things. This is different from other lisps, that convert names of literal symbols read to uppercase before interning them. This is why :foo, :Foo and :FOO all mean the same :FOO, by default.

I am using the excellent Quicklisp package management utility to pull all dependencies that a certain package needs. With ACL in modern mode, though, I had to fix several libraries that used all-caps symbol literals (like #:FOO, NIL etc.). The fixes were mostly trivial, but when I had all the stuff running that I wanted, I had modified several of the libraries that I had installed with Quicklisp.

I am planning to import all the external libraries into git repositories and not use Quicklisp in production. I still want to be able to easily update to new upstream versions and also send the changes that I made myself upstream. To make the process of finding what I've changed easier, I envisioned a tool that tells me which of the libraries that I had installed with Quicklisp have local changes, and what these changes were. I asked Zach Beane for help, and he promptly came up with a small function called QDIFF that performs a diff between Quicklisp's base version and the locally extracted version of a library. I hacked the tool slightly to make it work with Allegro CL and iterate over all locally installed systems. It is available as a gist.

Zach supplied a SBCL specific way to run the diff command. I added one based on ASDF:RUN-SHELL-COMMAND, which apparently prefixes all output lines with "; ". I don't like that, but I also did not want to pull in a shell compatibility layer into this self-contained thing.

I praised Quicklisp in the past, but there is nothing wrong with doing it again. Open Source Common Lisp has become so much better with it, and I need to thank Zach again for being so passionate about making the Common Lisp world better! Thanks, Zach!