On The Lisp Curse

I have a what I call a Links Page which has a set of links to things that have caught my fancy at one point or another.  I created that page, and others like it, because I realized that emailing myself links didn't really work because they got lost in the email before my interest in them returned.  Putting them in a page works and is the perfect thing to avoid the problem of links belonging to only one browser or only one machine and suchlike.

In fact, I've achieved some mild fame with those links pages, which isn't bad for content that I use mostly to help me goof off when I'm at work.

Today, I've been thinking about a couple of programming links on that page.  Those pages make more sense if you understand what the "AI Winter" is.  Basically, in the 1970's and early 1980's everyone knew that AI was coming and it would be implemented in Lisp.  Of course, AI has been on the verge of a breakthrough, for as long as I've been a programmer, but it never seems to actually get here.  The overhyping and subsequent collapse of AI techniques in general (neural networks, expert systems, etc.) and the Lisp language in particular, along with the rather large resource requirements of the 1980's Lisp systems compared to the 1980's microcomputers, is basically why there are few Lisp users now.  I have made several attempts at learning Lisp deeply, which is a far different thing from learning the syntax and writing a few toy programs in it, (something I have done,) but I never got far enough in it to get past the strangeness of it.  I think that's a shame, too, because I believe in the truism that programming languages that do not change the way you think about programming aren't worth learning, and the corollary you should learn every programming language that changes the way you think, and Lisp has the potential to change everything.

And yet it doesn't.

So, why not?  Well, this article attempts to explain it in terms used by this article.  I think that what those articles have to say is interesting, if not for the reasons put as their conclusions.  Briefly summarized, the idea they present is this:  Lisp's biggest problem is that it is too powerful.  While adding a feature to C is so immensely complicated that only a few attempt it and those of us among the great unwashed adopt the solutions of the few, adding similar features to Lisp is so easy that it's easier to add your own to scratch your particular itch than it is to try to figure out how to use someone else's solution.  That means that the "second order" political effect of trying to get people to adopt someone else's GUI library dominates which means the "first order" technical superiority is preventing from coming to the fore because everybody keeps rewriting the same stuff.  Over and over.

The first problem with this line of argument is that it gets the order of the problems backwards.  One way to view programming languages is primarily as a means for programmers to communicate with each other.  This position has been held by a lot of really bright people, and there is a lot to be said for this view.  Indeed, if you're trying to get people to adopt your code, making sure they understand it is of first importance because until your target programmers achieve some sort of understanding with it, it'll always be viewed with suspicion.  Not only that, but technical capability of a programming language is of secondary importance at best.

That second bit there, by the way, completely explains the "Lisp Curse" and is not a new idea.  That was the whole point behind Fred Books's "There Is No Silver Bullet" and "Silver Bullet Refired" articles.  Summarizing that goes something like this:  Programmer productivity (what technical capability is supposed to enhance) is dependent upon the complexity that those programmers have to deal with.  The complexity comes from two places, insufficient expressive power in the programming language and "essential complexity" which is complexity that is inherent in the problem that the programmer is attempting to solve.  The problem being that it doesn't take very much expressive power for a programming language to be expressive enough to eliminate nearly all of the complexity associated with the tools, and that threshold is passed even by lowly C.  That means that changing the programming language can have only the most marginal of effects on programmer productivity.  Lisp isn't used because it is simply impossible for switching to Lisp to have the sort of benefits that would offset the cost.

The problem with not having a generally accepted solution for various problems, and here it's more appropriate to think of things like GUI libraries, Web libraries, threading libraries, and so forth than it is lazy evaluation or object orientation is that what is available tends to be crap.  You see, when people write something for their own use it tends to only implement the parts that whoever wrote it is interested in, and anything difficult or uninteresting is left for a future time.  The fact of the matter is that different projects need different feature sets so what may solve 90+% of one project's problems might solve less than half of another's.  That tendency to implement only part of a solution is a main reason why it's easy to implement libraries from scratch in Lisp.  Of course it's easy, if you leave the hard parts out.

The thing is, partial solutions are very nearly as easy in C++ or Python or whatever as it is in Lisp.  It's not the programming language that makes that task so much easier, it's the leaving of the hard parts out.  Well, that and the blind spot most people have for estimating the effort of a task that is unfamiliar to them.  Never fall for the line that this time a task will be much faster because you'll use better tools this time because the tools never make that much of a difference.  It's just not going to happen.

Anyway, a fully-formed library can make a difference, but it needs to be well-suited for whatever problem you're trying to solve.  So, the fact that people happened to have been writing in C when many of these libraries were created, and the fact that those libraries have been around a long time and solve a wide variety of common problems, makes a C programmer de facto more productive than a Lisp programmer because much of their solution is created for them, and people experienced with paying money for solutions tend to be skeptical of the time estimates for software written from scratch while third-party libraries have a known cost.

Of course the problem is political.  It's always easier to imagine your own project from scratch than it is to attempt to understand someone else's.  However, in the long term that understanding is what makes for a tool that will actually get used instead of one where people write blog posts about.  Posts where someone imagines what might have been.  If only.