Lisp Hackers: Pascal Costanza

Pascal Costanza is a researcher, and an active Common Lisp programmer and community enthusiast:

  • he's the maintainer of Closer to Mop library, that provides a common facade to the MOP implementation in different Lisps, and is the basis of some of his more advanced libraries like: ContextL and FilteredFunctions;

  • the originator of Common Lisp Document Repository (CDR) project, that collects proposals for improving the language (a la JCP for Java or PEP for Python);

  • and the author of a Highly Opinionated Guide to Lisp, which can serve as introductory text for those, who come from other languages. (It was quite a useful text for me, when I started studying Lisp.)

In the interview Pascal shares a lot if insight into his main topic of interest — programming language design — grounded in his experience with Lisp, Java, C++ and other languages.

Tell us something interesting about yourself.

I share a birthday with Sylvester Stallone and George W. Bush. I have been a DJ for goth and industrial music in the Bonn/Cologne area in Germany in the past. I once played a gay Roman emperor in comedic theatre play. I played a few live shows with a band called "Donner über Bonn" ("Thunder over Bonn"). My first rock concert I ever attended was Propaganda in Cologne in 1985. The first programming language I really liked was Oberon. I often try to hide pop culture references in my scientific work, and I wonder if anybody ever notices. My first 7" single was "Major Tom" by Peter Schilling, my first 12" single was "IOU" by Freeez, my first vinyl album was "Die Mensch-Maschine" by Kraftwerk, and my first CD album was "Slave to the Rhythm" by Grace Jones. I don't remember what my first CD single was.

What's your job? Tell us about your company.

I currently work for Intel, a company whose primary focus is on producing CPUs, but that also does business in a lot of other hardware and software areas. (Unfortunately, Intel's legal department requires me to mention that the views expressed in this interview are my own, and not those of my employer.)

I work in a project that focuses on exascale computing, that is, high-performance computers with millions of cores that will be on the market by the end of the decade, if everything goes well. I am particularly involved in developing a scheduler for parallel programs that can survive hardware failures, which due to the enormous scale of such machines cannot be solved by hardware alone anymore, but also need to be dealt with at the software level. The scheduler is based on Charlotte Herzeel's PhD thesis, and you can find more information about it in a paper about her work and at http://www.exascience.com/cobra/.

Do you use Lisp at work? If yes, how you've made it happen? If not, why?

At Intel, I do all software prototyping in Lisp. The scheduler I mentioned above is completely developed and tested in Lisp, before we port it to C++, so that other people in the same project and outside can use it as well. It didn't require a major effort to convince anybody to do this in Lisp. It is actually quite common in the high-performance computing world that solutions are first prototyped in a more dynamic and flexible language, before they are ported to what is considered a "production" language. Other languages that are used in our project are, for example, MATLAB, Python and Lua. (Convincing people to use Lisp beyond prototyping would probably be much harder, though.)

The implementation we use for prototyping is LispWorks, which is really excellent. It provides a really complete, well-designed and efficient API for parallel programming, which turns LispWorks into one of the best systems for parallel programming of any language, not just in the Lisp world. The only other system that is more complete that I am aware of is Intel's Threading Building Blocks for C++.

What brought you to Lisp? What holds you?

I have participated in one of the first Feyerabend workshops, organized by Richard Gabriel, one of the main drivers behind the original Common Lisp effort. I have also read his book Patterns of Software around that time. Later we had a small discussion in the patterns discussion mailing list. He tried to promote Lisp as a language that has the "quality without a name", and I made some cursory remarks about Lisp's unnecessarily complicated syntax, just like anybody else who doesn't get it yet.

To me, the most important comment he made in that discussion was: "True, only the creatively intelligent can prosper in the Lisp world." The arrogance I perceived in that comment annoyed me so much that it made me want to learn Lisp seriously, just to prove him wrong and show him that Lisp is not as great as he thought it is. As they say, the rest is history.

I actually dabbled a little bit in Lisp much earlier, trying out a dialect called XLisp on an Atari XL computer at the end of the 80's. Unfortunately, it took too long to start up XLisp, and there was not enough RAM left to do anything interesting beyond toy examples, plus I was probably not smart enough yet to really get it. I was just generally curious about programming languages. For example, I also remember trying out some Prolog dialect on my Atari XL.

In the end, Lisp won me over because it turns out that it is the mother of all languages. You can bend it and turn it into whatever language you want, from the most flexible and reflective interpreted scripting language to the most efficient and static compiled production system. For example, the scheduler mentioned above easily gets in the range of equivalent C/C++-based schedulers (like Cilk+, TBB, or OpenMP, for example), typically only a factor of 1.5 away for typical benchmarks, sometimes even better. On the other hand, ContextL uses the reflective features of the CLOS Metaobject Protocol to bend the generic function dispatch in really extreme ways. I am not aware of any other programming language that covers such a broad spectrum of potential uses.

What's the most exciting use of Lisp you had?

When I decided to make a serious attempt at learning Common Lisp, I was looking for a project that would be large enough to prove to myself that it is actually possible to use it for serious projects, but that would also be manageable in a reasonable amount of time. At that time, I was intimately familiar with the Java Virtual Machine architecture, because I had developed compilers for Java language extensions as part of my Diploma and PhD theses. So I decided to implement a Java Virtual Machine in Common Lisp - under normal circumstances, I wouldn't have dared to do this, because this is quite a complex undertaking, but I had read in several places that Lisp would be suitable for projects that you would normally not dare to do otherwise, so I thought I would give it a try. Over the course of 8 weeks, with something like 2 hours per day, or so (because I was still doing other stuff during the day), I was able to get a first prototype that would execute a simple "Hello, World!" program. On top of that, it was a portable (!) just-in-time compiler: It loaded the bytecode from a classfile, translated it into s-expressions that resemble the bytecodes, and then just called Common Lisp's compile function to compile those s-expressions, relying on macro and function definitions for realizing these "bytecodes as s-expressions." I was really impressed that this was all so easy to do.

The real moment of revelation was this: to make sure to reuse as many of the built-in Common Lisp features as possible, I actually translated Java classes into CLOS classes, and Java methods into CLOS methods. Java's super calls posed a problem, because it was not straightforward to handle super calls with plain call-next-method calls. Then I discovered user-defined method combinations, which seemed like the right way to solve this issue, but I was still stuck for a while. Until I discovered that moving a backquote and a corresponding unquote around actually finally fixed everything. That was a true Eureka moment: In every other programming language that I am aware of, all the problems I encountered until that stage would have required a dozen redesigns, and several attempts to start the implementation completely from scratch, until I would have found the right way to get everything in the right places. But Common Lisp is so flexible that at every stage in your development, you can tweak and twist things left and right, but in the end you still get a convincing, clean, and efficient design. As far as I can tell, this is not possible in any other language (not even Scheme).

What you dislike the most about Lisp?

There is not much to dislike about Lisp itself. There are some technical details here and there, some minor inconsistencies, but nothing that cannot be fixed in easy and straightforward ways. From a purely conceptual point of view, Common Lisp is one, if not the most complete and best integrated programming language that covers a lot of ground. Some rough edges are just to be expected, because nothing is ever perfect.

What concerns me a lot more is that there is too much unwarranted arrogance in the Lisp community. I don't know exactly where this comes from, but some Lispers seem to believe, just because they understand some Lisp concepts, that they are much smarter than anybody else in the universe. What they are forgetting is that computer science as a discipline is very, very young. In one or two hundred years from now, Lisp concepts will be common knowledge, just like elementary algebra. There is no reason to be arrogant just because you know that the earth is round, even if most other people still believe that it is flat.

Describe your workflow, give some productivity tips to fellow programmers.

I strongly believe that the one thing that made me most productive as a programmer is my interest in doing some form of art. I used to spend a lot of time making my own music, both by myself with synthesizers and computers, as well as in bands. I also was an actor in an amateur theater group. Art gives you a sense of making parts (notes, chords, melodies, rhythms, or acts, characters, plot lines) relate to each other and form a coherent whole. It also makes you aware that there is an inner view on a piece of music or a play, as seen by the artist, but also an outer view, as seen or heard by an audience, and if you want to make good art, you need to be able to build a bridge between those two parts.

Programming is exactly the same: you need to make parts (functions, data structures, algorithms) relate to each other, and you need to bridge the inner view (as seen by the designer and implementer) and the outer view (as seen by the user of a library or the end user of the final software).

The important aspect here is that you need to be able to change perspectives a lot, and shift between the local, detailed view, the global, architectural view, and the many different levels of a layered design. This is especially important when it comes to designs that incorporate meta-programming techniques and reflective approaches, but also already for simpler designs.

Like in art, the concrete workflow and the concrete tools that work best vary a lot for different people. It's also a good idea to just play around with ideas, expecting that most of them will turn out useless and need to be thrown away. Artists do this all the time. Artificial, seemingly nonsensical rules and restrictions can be especially enlightening (use only effect-free functions; use only functions that receive exactly one argument, not more, not less; make every function pass around an additional environment; use only classes with exactly two slots; etc., etc.), and then try to build larger programs strictly following such rules - this will make your mind a lot more flexible and train you to see new potential solutions that you wouldn't see otherwise.

(I actually believe that this is what makes fans of static typing so excited: Static type systems always impose some artificial restrictions on your programs, and enforce them to the extent that programs that violate these rules are rejected. If you then program in such a statically typed programming language, you will indeed have some interesting insights and see new solutions that you would otherwise miss. However, the category error that fans of static typing often seem to make is that they ascribe the results to the static type system, and therefore usually get stuck with one particular set or kinds of rules.)

Apart from that, I believe that LispWorks is a really good development environment.

Among software projects you've participated in what's your favorite?

I don't know how to answer that. At any point in time, I'm always most excited by the one I'm currently working on. So far, I am quite proud of ContextL and the ClassFilters project for Java, because they are or were both used in one way or the other in "real" applications. Closer to MOP is a favorite project of mine, because it makes me feel like I can give something back to the Lisp community from which I otherwise benefit so much. But this doesn't mean I dislike anything else I have done in the past.

One of your papers, "Reflection for the Masses", researches the ideas behind 3-Lisp, "a procedurally reflective dialect of LISP which uses an infinite tower of interpreters". Can you summarize them here?

That paper was actually mostly the work of Charlotte Herzeel, and I was only her sounding board for detailing the ideas in that paper. Reflection is one of the essential concepts that was born out of Lisp. Every program is about something, for example a financial application is about bank accounts, money and interest rates, and a shopping application is about shopping items, shopping carts and payment methods. A program can also be about programs, which turns it into a meta-program. Lisp macros are meta-programs, because they transform pieces of code in the form of s-expressions into other pieces of code. C++ templates are also meta-programs in the same sense. Some meta-programs are about "themselves," and thus become reflective programs.

3-Lisp is "procedurally reflective" in two senses: On the one hand, it allows you to inspect and change the body of procedures (functions). Common Lisp, for example, also gives you that, in the form of function-lambda-expression and compile/eval, among others. On the other hand, 3-Lisp also allows you to inspect and alter the control flow. Scheme, for example, gives you that in the form of call/cc, but in 3-Lisp this is actually part of the eval interface. 3-Lisp goes further than Common Lisp and Scheme combined, in that it provides not only first-class access to function bodies and continuations, but also to lexical environments, always both with facilities to inspect and modify them, and provides a clean and integrated interface to all of these features. Unfortunately, because 3-Lisp goes that far, it cannot be fully compiled but always needs to be able to resort to interpretation, if necessary.

The reason for the requirement to always have an interpreter around is because the eval interface in 3-Lisp is so flexible that you can run programs inside an eval invocation that in turn can inspect and change (!) the environment in which eval runs, and can then invoke further evals in those changed environments, to arbitrary levels of recursive invocations of eval. These recursive invocations of eval build what is called a reflective tower, where every level in the tower is conceptually an interpreter being executed by an interpreter one level up in the tower of interpreters. The amazing thing about 3-Lisp is that an implementation of 3-Lisp can actually collapse the tower into one level of interpretation, and arrange that one interpreter in such a way that the several different levels of interpretations are only simulated, so the tower is actually just an "illusion" created by the 3-Lisp implementation.

This may all sound quite esoteric, but is actually practically relevant, because there is strong evidence that all meta-programming approaches eventually need such towers, and some ad-hoc way to collapse the towers. For example, you can find the tower in Racket's and R6RS's macro systems, where they are explicitly mentioned; in Common Lisp's macros, where eval-when is used to control which part of the tower sees which definitions; in the CLOS Metaobject Protocol, where the generic function dispatch can be influenced by other generic functions, which can in turn be modified by some meta-classes at a higher level; in the template metaprogramming system of C++, where "concepts" were devised for C++11 (and rejected) to introduce a type system for the template interpreter; and so on, and so on. If you understand the concept of a reflective tower better, you can also better understand what is behind these other meta-programming approaches, and how some of their sometimes confusing semantic difficulties can be resolved.

If you had all the time in the world for a Lisp project, what would it be?

I have some ideas how to design reflection differently for a Lisp dialect, which I believe has not been tried before. If I had all the time in the world, I would try to do that. I also have many other ideas, so I'm not sure if I would be able to stick to a single one.

Anything else I forgot to ask?

I think one of the most underrated and most underused Lisp dialects is ISLISP. I think people should take a much closer look at it, and should consider to use it more often. Especially, it would be an excellent basis for a good teaching language.

Discussion on HackerNews submit


Lisp Hackers: Marijn Haverbeke

The next person in our series is Marijn Haverbeke, who's not only the author of several pretty useful Common Lisp libraries (some of which he touches in the interview), but also a succesful JavaScript hacker, winning JS1K contest, and writing a profound book about the language — "Eloquent JavaScript". Besides, he hacks on Mozilla Rust and his own language Hob, not to mention co-authoring a JavaScript-to-Common-Lisp transpiler, writing some games in Scheme, and experimenting with Haskell. In the interview he shares his perspective, based in knowledge and experience with so many different languages.

Tell us something interesting about yourself.

I once thought I wanted to be a Sociologist. Before that, I also once thought that I wanted to become a treasure hunter. And a dinosaur researcher. I ended up a programmer, largely self-taught, and I'm very happy with how things turned out.

What's your job? Tell us about your company.

I'm working free-lance. Currently, I'm doing a lot of work for Mozilla, hacking on their Rust compiler. Rust is a programming language that's supposed to fit the C++ niche, without the insanity of actual C++. If the project works out, and it's going swimmingly so far, Mozilla will try to apply it to writing a next-generation browser, without the constant danger of buffer overrun exploits and threading bugs that comes with C++. True to Mozilla's spirit, all development on Rust is happening in the open, on github.

Another things I'm busy with is my own CodeMirror project, a code editor written in JavaScript. It's not a 'job', really, since it's open-source and no one is paying me to maintain it, but it's grown popular enough to generate a decent amount free-lancing gigs and a small income stream from support contracts.

Do you use Lisp at work? If yes, how you've made it happen? If not, why?

Not currently, beyond that I always have a REPL open in emacs, and switch to it to do quick computations and experiments. I've spent several years doing almost exclusively Lisp (mostly working on AllegroGraph, Franz' RDF database), and it has definitely been somewhat painful to find myself working with non-interactive, macro-less languages again, after being used to the luxuries of Common Lisp and Slime.

What brought you to Lisp? What holds you?

Ten years ago, during my sociology study, I randomly chose a Lisp course in university. I was spending so much time in emacs that I figured I might as well learn more about its scripting language. The course worked through Structure and Interpretation of Computer Programs, using the video lectures from Abelson and Sussman. Having been exposed mostly to Java, C++, and Python before that, the elegance and depth of this material was something of a revelation to me.

My professor for that course, Eduard Hoenkamp, was a real Lisp chauvinist with an MIT background, and properly indoctrinated me with regards to Lisp's glorious superiority. I wrote a text adventure game in Scheme for the end project of the course, my first non-trivial Lisp endeavor. I still have the code. I ended up starting a PhD with this very professor, and though I never finished it, it did provide me with ample opportunity to learn Common Lisp.

What's the most exciting use of Lisp you had?

I think that'd be CL-JavaScript, a JavaScript-to-Common-Lisp transpiler that I wrote together with a friend. It actually handles the whole ECMAScript 3 language and produces programs that run reasonably fast—about on par with the 2009 generation of browser JavaScript engines. This is quite amazing, given that it's less than 4000 lines of code, at least an order of magnitude smaller than those engines. The Lisp compiler handles most of the optimization for us. (I should point out that the browser JS engines have a much higher speed of compilation, which is a necessity for handling web content.)

What you dislike the most about Lisp?

I must say that the lack of static typing really gets in the way on larger projects. Being able to confidently change datatypes and function signatures, knowing that the compiler will point out most inconsistencies that you introduce, is something that I've really come to appreciate after working with Rust and Haskell. Test coverage helps, of course, but I'm not a very diligent test writer, and tend to feel that type signatures are easier to maintain than an exhaustive test suite.

Describe your workflow, give some productivity tips to fellow programmers.

My approach to writing software is embarrassingly unstructured. I'm not sure I can claim to have something that merits the term 'workflow'. One thing that is very important for me, and that, had I figured it out earlier, would probably have saved some past projects from dying, is to never have a system in a half-refactored, non-functional state for longer than an hour or so. A lot of momentum and motivation comes from having a working program in front of me. This sometimes means I have to split up big changes in a number of more indirect, trivial steps, and resist the temptation to start hammering out my big change in one go (which, even when seems straightforward in my head, will always hit some complications during implementation).

Also, really learning git in depth has had a big influence on how I code. Once you really understand that changes (if you bother to commit often and with care) can be reverted and picked at will, and that branches are cheap and easy to create and maintain, you start to see a codebase differently. I often take multiple shots at making a tricky change — if it feels like my first approach is not working out, I simply commit it, back up, and start again, knowing that if the new approach doesn't work out, I can simply flip back to the old one again. Knowing that this is theoretically possible is not enough—you need to have actually done it, often, to start applying it properly.

Among software projects you've participated in what's your favorite?

Postmodern, the PostgreSQL client, was my first non-trivial open-source project, and the first piece of actually useful software that I designed and executed on my own. I was extremely proud when I got it to work the way I wanted (modular, fast), and meticulously wrote out a long set of docs (copying Edi Weitz's style). Seeing people actually use it (and even write good things about it!) really drove home for me the fact that it is worthwhile and rewarding to package up the stuff I write, and share it.

The community involvement in Postmodern, though always low-volume, has also been awesome. Most bug reports came with a patch, or at least an informed view of the problem and useful diagnostic information. This in sharp contrast with the JavaScript community, where of course great contributors also occur, but one has to deal with a lot of noise and lazy bug reporting.

If you had all the time in the world for a Lisp project, what would it be?

I am working on my own programming language, Hob, which tries to combine the virtues of Lisp and ML. This is a huge project, and unfortunately it's moving forward only slowly. If I didn't have financial constraints, I'd spend a year (or two) finishing it. Its 'bootstrap compiler' (the compiler that will compile the first self-hosting compiler) is in the process of being written in Common Lisp. The writing that I currently have online about it is quite old and out of date. I just spent a week rewriting the compiler from scratch, moving the language to a completely regular syntax—think s-expressions, though Hob looks somewhat different. My work on Rust has convinced me that, even with all the good will in the world, you can't bolt macros onto a classical language with a non-regular syntax without making them very painful to write (and use!). They have to be part of the design from the start.

You've been actively using Common Lisp and JavaScript. When you use JavaScript, what do you miss the most from CL? What do you miss from JavaScript, when you're in Lisp land?

The greatest thing about Common Lisp is that the standardization process drew from years and years of experience and organic growth. This makes the language feel singularly solid and practical. Most languages feel immature compared to CL. I suspect that this is not just age, but also the fact that Lisp allows users to extend the language to an unprecedented degree, which allows for much more experimentation, and thus faster growth and invention.

The great thing about JavaScript is that it has so few concepts. It manages to strike a balance between being dumbed-down enough to be easy to grasp, which has allowed it to become as widespread as it currently is, and still being a more-or-less decent lambda language, which makes it scale up to complex programs relatively well. It is on its way to become a lingua franca of scripting, which Common Lisp will never be, because it is much more demanding of its programmers—they need to understand a lot more concepts.

And here is the Hacker News discussion. submit