My bet is it won't. Lisp success stories have been around for ages. PG has articulated its advantages well, as have others. There's some psychological reason why it has never achieved massive popularity. If it ever becomes massively popular, it will be in a generation that is somehow psychologically different.

Quoting Guy Steele, on Java, "we were not out to win over the Lisp programmers; we were after the C++ programmers. We managed to drag a lot of them about halfway to Lisp. Aren't you happy?"

I believe languages (both hot and widely used) will continue to move in the general direction of Lisp. As for why more people don't just leapfrog that process, I think the best answer is in Worse-Is-Better (http://www.jwz.org/doc/worse-is-better.html).

But if either simplicity of implementation or speed were necessary preconditions of popularity, how would Java have become so popular? Marketing? If marketing is what matters, how did Java (in its C# incarnation) displace Visual Basic?

Any time I hear someone say "Lisp would be popular except for problem X" I think of some other computer language that became popular despite problem X. Even its supposedly problematic syntax. Look at XML; it's popular, and has a syntax pretty much like s-expressions only worse.

Complexity of implementation? Slowness? Lack of marketing? Unfamiliar syntax? No, none of these explain the lack of popularity. Someday somebody will hit on what the real roadblock is and remove it, probably by accident.

A lot of languages however have become popular due to large PR budgets (Java and the .NET family).

XML became popular because it addressed the need to have a self-documenting data inter-change format so that governments and companies can be reassured that even if a vendor goes out of business they can reuse the data (similar reasons for the push behind ODF).

Lisp hasn't yet addressed the needs of the masses (develop X blog in 15 minutes with a human readable syntax), or be pushed into the enterprise by large companies, or have a large pool of qualified developers, so it hasn't become popular.

Personally I don't think it ever will become a mainstream language. It requires a different way of thinking that I'm not sure a lot of folks can grasp. Ruby might be as close to Lisp as a lot of folks are willing to get.

In the absence of corporate backing (java , c#) , to become popular, a language needs a BDFL leading an open source implementation effort, who knows how to enthuse, manage and coordinate contributors.

The whole "standardization by comittee" is an anachronism (in the absence of corportae backing,, and imho). Not to mention the misanthropic lisp "community" that hangs out on comp.lang.lisp.

And libraries. Lots of them. And/Or a "killer app" that drives adoption.

Also XML didn't succeed as a programming language. Lisp Vs Xml isn't a proper comparison.

Complexity of implementation? Slowness? Lack of marketing? Unfamiliar syntax?

The answer is more like 'all of the above,' as well as not anticipating the rise of Windows and Unix (which is not necessarily a bad thing).

Java had outstanding marketing. It caught the .com wind in its sails and managed to get the industry press all talking about how this platform independent language was going to replace all the platform specific development environments.

http://programming.reddit.com/info/1z8b8/comments/c1zbhh

I think you dismiss these factors too quickly. We need to remember that even seemingly little things can prevent adoption.

C and C++ have been running fast on cheap hardware longer than Lisp has. The reason Lisp has caught up in performance is because of (relatively) better CPU architecture.

Marketing played a part in Java becoming popular. I can point to a few reasons:

- The language designers modeled it after C++, a language many developers were familiar with.

- Sun sold it as being more multi-platform than C++.

- It was "internet ready", unlike C++. This was a HUGE factor. No other language that people knew of made this claim. If developers used C++ on the web it was in combination with Perl, and only on the server.

- They touted its garbage collection capabilities. No more having to free memory--something C and C++ developers were concerned about. It looked and kind of felt like C++, but without the hassles. That's how it was billed.

- It ran in browsers as an applet, and was securely sandboxed. This was something no other language could do (initially). They caught the "internet fever" at just the right time with this message. Lots of people deployed applets. Eventually Microsoft came out with its own solution: ActiveX. Along with Flash, it effectively killed Java applets. Why? Java applets were problematic. They didn't run well across browsers (partly by design--Microsoft and IE). Flash integrated multimedia more easily than Java did. Nevertheless, people found that Java was more cross-platform than C or C++ on servers (though you had to be careful about your brand of JVM).

On the minus side, Java had its detractors regarding performance. I think this became less of a problem on the server. People "just threw hardware at it", until it ran fast enough.

I listened to the Java evangelists for years. So I know what their developers were excited about. The main thing for them was the slogan "Write once, run anywhere". They loved it. What sold Java to the enterprise was J2EE, an enterprise framework specification, which Sun implemented for Java. They built upon their installed base over time.

Sun showed corporations how using Java allowed customers to choose their hardware based on capability and price more easily, since "you don't have to rewrite the app." That was a key phrase--keep your software investment no matter what hardware and operating system you choose. They differentiated this against the Microsoft ecosystem. Nevermind that for many customers this didn't always work out. There were JVMs that had incompatibilities with each other. Despite Sun's claims that Java programs don't crash the machine, I saw it do just that in its early days.

What mattered was the idea, nevermind the reality: perfect portability, and internet-ready. Corporate customers liked that. Programmers liked that it was "a better C++". It took a language they were already familiar with and made it "better". That's what customers were after, and they thought only Java could deliver it.

One might ask if this is the case why didn't Objective C take off? That's a bit of a hard one for me. It was a favorite among NeXT developers. Ever since the early to mid-1990s it's been implemented in GCC. My guess is there was no corporate support behind it by then. Objective C was initially marketed by a single company in the mid-1980s. It was never recognized as a standard. You had to buy it from the company that created it. Eventually it went out of business, I think, but it did effectively become open source. Today it comes on the Apple Mac, so it has corporate sponsorship now. The Mac's popularity is growing, so perhaps this'll translate into success for Objective C someday.

Corporate sponsorship of languages carries risks, but it can also encourage adoption. If the company that owns the language implodes, it takes the whole community down with it--I refer you to what happened to Smalltalk. Microsoft was just more successful with VB. I could tell you a story about the beginnings of BASIC's popularity, but that would be getting off-topic.

I haven't been keeping track of market share numbers, but if it's the case that Java has overtaken VB, I can give a couple plausible reasons for it, the gist of it being Microsoft screwed VB developers. When MS initially came out with VB.Net it was a major departure from VB 6, the previous version. For most people VB.Net could not translate or run older VB programs. It would try to translate them, but major parts of applications had to be rewritten, because they took out support for some things the VB API used to do. They implemented things differently. They made programming in VB more complicated, so there was a significant learning curve to get up to speed.

C# initially got all of the advantages. Most of the early .Net documentation was written only for C#, not VB.Net. Most of the early .Net code demonstrations were done in C#, not VB.Net. C# had some cool features VB.Net didn't get initially. If you were a VB developer you felt left out in the cold by MS. Some VB developers became C# developers, just to make the transition easier. There were endless debates on MS forums: "Should I learn VB.Net or C#?" There was uncertainty in the VB community about its future. Some migrated over to Java, figuring it was a skill they could invest in and not have to change like they did with VB.Net. So MS in effect "anti-marketed" VB.

You may remember there was a lot of buzz about XML some years ago. The main reason being that it was an "internet ready" meta-language for data. Are you noticing a theme here? "Internet ready" is key. If it's not "designed for the internet" people will turn their nose up at it. Note this is a marketing message.

Re: XML

Microsoft actually did something slightly innovative: They invented the technology that enabled AJAX. They put the XMLHttpRequest() call into IE 10 years ago. They added built-in support for XSLT as well. OWA (Outlook Web Access) was one of the first major AJAX apps., before Google Maps. As you may remember, anything MS pushed got adopted by a lot of people back then.

Why did XML work? It was initially sold strictly as a data description format. I think, like Java, it kind of looked familiar to developers. Getting people to adopt HTML in the 1990s was a learning curve. What was the "killer app" for that? The web. You couldn't be on the web unless you dealt with HTML. That was enough motivation for people to learn it. Once they learned HTML, they only had to learn a design construct to get XML. The same tagging syntax was used.

Using XML to do actual programming didn't come along until later. People looked for a way to end the tedious repetition of code. Some people found a way to get XML to act as a meta-language. Like XML it was descriptive. Everything was labeled. It was "just text" so it was "internet ready" as well. It didn't require a lot of thought either. None of the XML tools I've heard of have a sense of recursion, which is hard for a lot of people to understand.

Why has Ruby become popular? Two answers: Rails, and DHH, who knows how to market a product. Before Rails came along Ruby was as obscure as APL. Rails not only made Ruby "internet ready", it added an extra dimension that web developers have been craving: instant gratification. The idea that you could design your web app. from your database--basically data-driven development--was like a dream come true. The same way that the internet drove people to learn HTML (an unfamiliar syntax), Rails has driven some developers to learn Ruby.

In order for a technology to coax people to learn it, despite it being unfamiliar, it has to have a "killer app", which is something that's compelling (answers an immediate desire or need), and is the exclusive province of that particular technology, at least for a little while. We've seen this again and again.

Perception is a big part of whether a technology takes off or not as well. How has Lisp been "marketed"? For generations, it seems, Lisp has been billed as an AI language. So if you're interested in exploring how people think, or how a machine could think, this language was for you. How are people supposed to go from this to thinking, "Oh yeah, I could use that for my next web app."?? In order for marketing to work you have to show people how the tool or system answers a need they have. It needs to be sold to them as something they can relate to. One of the problems with the Lisp community, so I hear, is this is a real challenge for them. Making Lisp something exciting and relevant seems to be as hard for them as trying to explain market economics or politics to a 4-year-old. The truth is it's not too hard. As you point out, syntactically XML is like Lisp. I tried pointing this out in a blog post once, and I got a bunch of negative responses: "I HATE XML!! Why would I like Lisp any better if it's like what I hate?!" Oh well... :(

Something RMS talked about is years ago when Emacs was first being developed, someone tried introducing Lisp to non-programmers by making it part of Emacs, and not using the word "programming" at all. They merely said, "If you'd like to add convenient features to the editor, you need to describe your feature using these features and syntax." It was successful. The key was, "Teach programming while people are doing something else." Don't teach programming as an end in itself. Teach it as a means to an end that they want.

Here's a hint if anyone wants to take it up. There's an open source project out there called "Vista Smalltalk" that works with WPF, WPF/E, and Flash. It's a way to create RIAs (Rich Internet Applications). It runs both Smalltalk and Lisp code. If anyone's interested in evangelizing Lisp in a way that people can relate to, this is an opportunity to do it.

Lisp culture contains the attitude that this is the right way to do things, and other ways are worse, and choosing them is a mistake. Thus, for anyone to take up Lisp, they have to admit to themselves their past decisions were mistakes. It's a lot easier to say "screw you, I'm right" and not switch over. If Lisp is right, they wasted a lot of time and effort, so they are invested in it being wrong.

The same thing happens in other fields.

What probably also did damage to Lisp's popularity was a perception problem. I talked about this in another comment in this discussion. In the schools Lisp was taught almost exclusively as an AI language. Scheme was different. I remember when I was in college the engineering school taught their students Scheme, and had them do all of their programming assignments in it. But within the CS schools Lisp was seen almost strictly as an AI language. I think this limits its appeal. If students took AI, they learned more about it, if they specialized in something else they learned Fortran, Pascal, and maybe C. Thats was it. AI was such a specialized and problematic field to begin with as well. It seems that hardly anyone saw that Lisp was just as good of a general computing system as C and Unix is on traditional processors.

Smalltalk is the same way. I remember I was taught it (briefly) as a programming language, even though it started out as the operating system of the machine it ran on. It has a rich history, yet most people who have even heard of it think of it as just a language that runs on a VM, which is actually a screwed up way of looking at it.

Another aspect to it is Lisp represents a very different view of programming. Most kids/students are taught imperative languages as their first language. This is encouraged all through school. By the time they meet Lisp it's a bridge too far. That's how it was for me years ago when I first encountered it at school. I had a terrible teacher, so I got no help there, and Lisp was so alien it made my head hurt. I couldn't figure it out. It wasn't until I found a good teacher, and then some good reading material on it in the years that followed that I finally began to grok it a little.

I see some of this same attitude in the Squeak (Smalltalk) community, though it's not as bad. People are more willing to share it with others.