Refactoring CL-SMTP

In my recent refactoring of BKNR, I decided that we do no longer want to use any of Franz' open source libraries if we can avoid it. Even though they work fine in general, hacking them is a pain because they adhere to John Foderaros Common Lisp Coding Standards which basically say that one should use Franz' own non-standard IF* macro for all conditionals. I do think that one should be careful with nesting conditionals deeply, but I do not agree with using a macro and spacing as a cure. If I have code in which conditional nesting exceeds two levels, I refactor to extract conditional code into functions. That way, functions are kept shorter and the use of names instead of literal code usually makes it easier to understand what's going on. So my easter goal was to replace Franz' NET.POST-OFFICE SMTP client by CL-SMTP. Both clients do not support proper quoting of non-ASCII characters in mail headers, thus the need to hack arose and IF* is nothing I want to get myself used to. A SMTP client really is not that complicated to begin with, and apart from the basic functionality, CL-SMTP already supported SSL and authentication, which nowadays are two basic requirements. What it was missing was the possibility to send pre-formatted messages, which is something that I require because I usually make up my own messages, including headers and body, using format strings or CL-MIME if I want to send attachments or otherwise need more control over the mail body. CL-SMTP prove to be a little hackish. Seemingly, only simple mail sending had originally been planned for, and the API had then been extended multiple times with growing user needs. There was no layering between the SMTP protocol aspects and the message formatting functionality, both being freely interleaved. While being simple to use for those use cases that had been planned for, the API was not helpful for my intended use. In a first round, I simplified the code, collapsed a few common patterns into functions and made the code generally easier to hack on. I then split the SMTP protocol handling into an exported macro, WITH-SMTP-MAIL, that is used to establish an SMTP connection and create the mail envelope. The body of the macro invocation is then invoked with a variable bound to the stream connected to the SMTP server. The existing mail sending functions of CL-SMTP have been converted to use that API, too. I then incorporated a patch that I found in the CL-SMTP mailing list archive so that raw TLS is supported. The existing TLS functionality worked by connecting to the standard SMTP port, then switching the cleartext connection to encrypted mode by issuing the STARTTLS command. In contrast, raw TLS works by having an SMTP server listen on a separate port for encrypted connections. No initial cleartext handshake is required in this operation mode. Finally, I implemented automatic quoting of non-ASCII characters in mail headers using a specialized stream class. The Gray Streams Examples in the SBCL manual are my favoured cheat sheet when implementing special purpose stream classes. The result of my work is available in the BKNR repository at svn://svn.bknr.net/svn/trunk/thirdparty/cl-smtp/ in case you want to give it a try right now. I developed with Clozure CL on my Powerbook and after I committed, our buildbot went red for SBCL. I had named an accessor for my specialized stream class "STREAM" which triggered a package lock violation error on SBCL. The name was bad, so I changed it to "ENCAPSULATED-STREAM" and saw the buildbot going green for SBCL too. I love that! I am now waiting for feedback on the refactorings and extensions. There are still bugs which need fixing and support for compilers other than CCL and SBCL needs to be verified. Also, the documentation for CL-SMTP needs to be better, and I think I'll just steal Edis HTML template and write something up myself next week. Also, an automated test for CL-SMTP would be great, but as this requires a SMTP peer to talk to, I have not really had a good idea how to implement that. Maybe later on. I certainly hope that Jan Idzikowski, who is the author and maintainer of CL-SMTP, will accept my patches and make them part of the main distribution.