A partial explanation of why non-commercial Common Lisp distributions don’t have enough libraries

Enough has already been written about how while Lisp, in its Platonic ideal form (as opposed to particular dialects and implementations), is pretty fantastic, a specification that hasn’t been touched for 20-odd years is pretty difficult to work with for many things. (It so happens that it’s still fucking awesome for creating active code generation and model-driven development systems.)

I think there’s a reason why CLISP doesn’t feel as “batteries included” as Python and Ruby do. It’s because “Common Lisp” actually has several languages in it, as well as a large standard library. Consider:

• First there is the base Common Lisp langage; the special forms like lambda and if.
• Then on top of this there is the Common Lisp library. This adds tons of syntactic sugar and all different manner of libraries for working with lists, symbols, functions, macros, and so on. This creates a domain language for working with Common Lisp objects.
• Then there are also multiple embedded domain-specific languages that come with any reasonable Common Lisp implementation.
• There’s CLOS, an awesomely powerful multiple dispatch object system with an unmatched metaobject protocol. Methods, slots (fields), subclasses, and superclasses can all be dynamically inspected, modified, and recompiled all at runtime using simple standard Common Lisp functions. It goes down to even being able to inspect and dynamically modify method signatures. Furthermore, CLOS is metacircular; CLOS is implemented in CLOS.
• There’s loop. It’s called like a function, but actually it’s an embedded psuedocode/imperative language for iteration. In other words, Common Lisp has an entire little language in it for iteration. Practical Common Lisp has an entire chapter on it.
• Then there’s format. See the chapter in Practical Common Lisp for more information and examples.

The Common Lisp specification is 1000 pages, and all of it comes from things that were found to be extremely handy when working on large, difficult software problems. Common Lisp deals with stuff that is timeless. Thus, it’s pretty good if you’re working on a timeless, textbook software problem.

Lisp lends itself very well to declarative metadata. That is why it is so much better than XML for code generation. You get dynamic typing, keyword arguments, optional arguments, closures, a highly interactive environment, macros, a powerful object system, and a powerful condition system instead of just angle brackets and lots of quotation marks. The homoiconic syntax and the ability to pass around code as data is remarkably powerful; if you want to keep an ad hoc block of code that you just banged out together, you can just quote it and then pass it around as a first class citizen.

I am not a Lisp zealot. But to write Lisp off completely is a mistake.

What I’ve been up to

I’ve been trying to make SQL and templates in web development not suck ass. I am going about this by creating a template language and a database tier generator where you declare templates and model information in Lisp and then a lot of PHP code is written to disk.

The database tier generator is fairly unremarkable at this point, but I’m kind of proud of the template language. The Lisp code on the left generates the PHP code on the right:

(#content .left.column (h2.welcome 'Welcome to our site!') (p print_information) (p more_information param1 (get username)) .right.column (include sidebar))

<div id='content'> <div class='left column'> <h2 class='welcome'>Welcome to our site!</h2> <p> <?php print_information(); ?> </p> <p> <?php more_information($param1, $_GET['username']); ?> </p> </div> <div class="right column"> <?php include 'sidebar.php'; ?> </div> </div>

It’s inspired by Haml. This stuff should be dropping pretty soon, stay tuned.

Interesting books worth reading

I made a trip to the library recently and here is what I came back with. I’ve read subsets of most of these thus far.

Putting Metaclasses to Work, by Ira Forman and Scott Dansforth

Hopefully this will make my brain not leak out of my ear when I read about metaclasses in Python and when I hack on CLOS. As an example of how powerful metaclasses are, AllegroCache’s interface to CLOS code is implemented as a giant metaclass hack which interfaces to backend object storage stuff.

Lisp in Small Pieces, by Christian Quiennec

Includes a helpful introduction to denotational semantics. Discusses interpretation and compilation of Lisps in depth, including both bytecode and to C. Includes material on Lisp object systems. Essential for advanced Lisp hackers. It expounds on the idea of having a partially embedded compiler, using Lisp data structures to store and transform abstract syntax trees more than any book I have ever found.

Essentials of Programming Languages by Friedman et al.

Takes the viewpoint that writing interpreters in Scheme is the bee’s knees. Has valuable information on mechanical transformation to continuation-passing style which On Lisp discusses a bit but not as in depth as possible due to scope and space limitations.

PAIP, by Norvig

This book is amazing. Ever wondered how a master Lisp user writes and structures his code? There should be more books like this. Don’t be fooled by the “AI” stuff, it has lots of material on efficiency, macrology, and compilation. Essential.

These books are enough to rock your software world. And if you’re bored with class-based OOP, why not pick up a book on prototype-based programming?

There remains much to learn.

My fears about RIA usability are coming true

I’m having a look at Apollo. I commend Adobe for doing a lot of things right: favoring PNG for icons, releasing early to get community feedback, etc. However:

This looks to be creating a new wave of unusable applications. Jakob Nielsen had this to say about Flash in 2000, before Macromedia made serious improvements to Flash’s usability:

… [M]any Flash designers introduce their own nonstandard GUI controls. How many scrollbar designs do we need? … [T]he specification of a new GUI widget is a major human-factors exercise. The current Macintosh and Windows scrollbars emerged after the world’s best interaction designers worked for years testing numerous design alternatives. A new scrollbar designed over the weekend is likely to get many details wrong. And, even if the new design was workable, it would still reduce a site’s overall usability because users would have to figure out how it worked. They know how to operate the standard widget. When you use standards, users can focus on content and their reasons for visiting your site. Deviate, and you reduce their feeling of environmental mastery.

None of these usability problems are inherent in Flash. You can design usable multimedia objects that comply with the guidelines and are easy to use. The problem is simply that current Flash design tends to encourage abuse.

My middle mouse button doesn’t work in Adobe’s sample Apollo applications. Perhaps there are other issues such as this lurking.

Then again, maybe Apollo, which requires basic programming ability to use, will scare away the kind of people who used Flash improperly. However, Adobe is certainly not setting a good standard by creating fucked up, non-standard widgets.

Overall I think Apollo is pretty cool. There are some things about it that are no doubt due to the fact that it’s only an alpha: the applications feel slow and unresponsive, and the only place you can install packaged applications is to a specific pre-chosen directory, for instance. To anyone who’s also playing around with Apollo, realize that possibly the best stuff is yet to come:

• Drag and drop and cut and paste support.
• Additional HTML and Ajax support.
• Window transparency for HTML-based applications.
• Flash within HTML.
• PDF support.
• Native dialog support for opening and saving files.
• …

To Adobe: I personally find this stuff just fine. I’m thinking of people like my grandma, is all. Could you make some tutorials on how to make Apollo/Flex apps look fuckin’ dope, like Finetune?

The state of computer hardware

It’s trite to observe that starting an Internet company is much cheaper in part because hardware is much cheaper.

To give a concrete example, look at what Google launched with. Here are the stats, care of Jeff Atwood at the excellent Coding Horror blog that you should read on a regular basis if you don’t already:

• 1792 megabytes of memory
• 366 gigabytes of disk storage
• 2933 megahertz in 10 CPUs

In other words, you could find a used IBM workstation on eBay with a gig of RAM for $400 the way I did, slap another gig of RAM and a couple 300 gig hard drives in it, and you would have more power than what Google launched with.

Let me repeat that: for $400-700 you can have the same kind of hardware resources than Google originally had. Combine that with advances in software like DTrace, the much-matured PostgreSQL, and of course excellent web development stacks like Rails, Seaside and Lift, and it’s no wonder why everyone is talking about building products and companies instead of talking about tools and processes.

Email is serious business

Based on a revelatory email from a staff member of Attorney General Gonzales, Chuck Schumer and Dianne Feinstein are suggesting the drastic step of rolling back the heinous PATRIOT Act to a prior version.

When are Democrats going to grow a pair?

Stooping to posting YouTube videos

I have nothing substantial to say that I haven’t already said, so here’s a link to a video of Joe Rogan expertly handling a heckler. I’ve always envied people who can think on their feet like that. NSFW language, of course.

Props to Zeldman and to MyPunchbowl.com

I don’t have anything really substantial to write about, I just wanted to say a few things that need to be said.

On the unreasonable effectiveness of the web standards movement

It was a long time ago, but there was a time when browsers had terrible support for web standards. All the usability experts, especially Jakob Nielsen, eager to preserve the status quo said we’d be stuck with this shit for a long time to come. But like Martin Luther King disregarding those who said blacks had to wait longer for civil rights, Jeffrey Zeldman and the others that made up the Web Standards project disregarded the naysayers and the defenders of the status quo, successfully lobbying major browser vendors to build in basic support for web standards. Today, most excellent web applications make heavy use of the DOM and CSS. That we don’t even really consider this to be unusual is indisputable proof of how thoroughly successful the web standards movement was. It is as far as I know one of the most successful instances of a small, determined community getting huge corporations like Microsoft to listen and to change in the short history of the Internet. And all the while as they were succeeding many intelligent people continued a chorus of negativity. Well, screw the snot-nosed Fortune 500-kowtowing Jakob Nielsens of the world, web standards won big.

Anyone who cares about the Internet owes an immense debt to people like Jeffrey Zeldman, Eric Meyer, Mark Pilgrim and everyone else who has strongly advocated for standards and accessibility on the web. My hat’s off to all of you. Thank you.

MyPunchbowl’s cool greenfield dashboard

Upon the recommendation of Scoble, I signed up for MyPunchbowl, an event planning app. Ignoring the fact that I almost never go to parties or engage in any real-world social activity, it gets all the Getting Real stuff right: it asks for an email address when you sign up, but you don’t have to go through the banal “type in email address/go check email/click URL in email/click superfluous ‘I actually want to register’ button” rigamarole found so commonly in, e.g., PHP-based bulletin boards. Instead, you can start using the app as soon as you give it valid information. The interface has a light, festive (although not cartoonish/unprofessional) feel consistent with its reference to events as “parties” and includes links to a blog, light-hearted illustrations, a list of upcoming holidays and events, and a sample event all to help get you started and keep you from turning away from the app without ever coming back. It has a link to the site’s blog so that you can see that there are real, live people behind it and that the application is alive and well, addressing the concern every user has about whether the app (and your data) will still be around in six months.

In short, it is very carefully designed to minimize the barriers between you and the act of using the application to host events. And plus it has neat-o Google Maps integration, which can be used to give helpful driving directions, etc., but the important thing is that they’ve clearly thought very carefully about how they should go about getting people to sign up and actually use the thing they have worked so hard on.

A Haskell study plan

As I mentioned, I’ve been learning Haskell. It is definitely the most challenging language I have ever learned. It is also probably the coolest.

Here is a collection of things I found to be useful.

• Yet Another Haskell Tutorial is good for getting started. I kind of fizzled out near the end.
• Good Math, Bad Math has a series of articles that take you from “Why Haskell?” to monads. I thought his explanation of monads was kind of poor if you haven’t already thought about them a bit.
• Speaking of monads. Monads have been extremely challenging for me.
  • Part I of all about monads was helpful.
  • After reading that and the Haskell Wikibook article on understanding monads, I started to feel more comfortable.
  • For quality of exposition, I haven’t found anything better than Philip Wadler’s famous 1992 paper, The Essence of Functional Programming.
  • For me, the most enlightening example of monads as a general structure for chaining computations was the Maybe monad. It’s also simple to understand.
• I am currently trying to understand monad transformers, which allow you to use multiple monads together. All About Monads claims to explain them in part III, but he tries to explain them using continuations, another obscure topic people have trouble with! I have not finished digesting/disregarding the following articles.
• A Neighborhood of Infinity has a helpful expository article written as literate Haskell that you can load into your text editor.
• Cale Gibbard, #haskell denizen and all around wizardly Haskell hacker, has an article on how to use monad transformers.
• Some guy who, judging by what appears to be a pro-illegal immigration rally photo as the header of his blog, apparently doesn’t believe America has the right to defend its borders or its sovereignty, has an interesting, more realistic example of using monad transformers which is, as he points out, more concrete.

The flow for reading this non-linear. Jump around, find different sources, look at the material on the Haskell homepage (including Don Stewart’s article on making your own simple IRC bot, for motivation), take a break, come back to it. I have been iterating on monads for about a week and a half now. If you get stuck, the Freenode #haskell channel has been extremely helpful and kind. If you aren’t asking them to do your homework for you, they’re quite willing to help out. They have a Haskell pasting system, hpaste, which is integrated with the chat room, as well as an IRC bot that can evaluate Haskell expressions and do other helpful things. Try it: irc.freenode.net #haskell! It’s full of helpful, smart people. I have never seen it with less than 300 people in it.

From what I can tell, monads and using monads for real stuff are challenging even to experienced, bright people like the Good Math, Bad Math guy (he said he decided to blog about Haskell because he had trouble with combining monads and monad transformers).

This is the first time I have ever really put my mind to something and failed to understand it. I can understand abstract algebra and real analysis, continuation-based web programming, pointers in C, recursion, dynamic typing and higher-order functions, but Haskell has made me its bitch.

Running out of steam?

I’ve been learning Haskell.

I thought it would be an interesting exercise to try to create a metadata language that could describe 95% of simple to medium-complexity web applications (up to, say, Basecamp or Reddit) in a much briefer manner than currently exists, embedded in Haskell.

Well it turns out that doing that is pretty hard. Even when it seems that you’re doing boilerplate stuff, if you’re making an excellent user interface and making something that really fits a niche, it’ll take nearly as much mental effort to start banging out lines of working code as it would to write down a declarative specification of it.

You could create a language that would let you make a to-do list application in about 200 LOC (including the database schema, all view templates, and all the controller code, plus whitespace), but it wouldn’t have a good user interface and it wouldn’t really be flexible. Code generation and generative programming are not a basis for creating products.

---

Some random thoughts. Paul Graham has a list of languages compared by what they fix. To his list I would like to add the following.

Haskell

There are too many experimental lazy, pure functional programming languages.

Scala

There’s no reason not to support both functional and object-oriented programming. And we need better support for components. Plus functional languages have no libraries.

F#

OCaml has no libraries and it has too many needless “wouldn’t it be cool if” features.