To Web Developers: Think Outside The Thread

A few people have asked me this very good question (I'm paraphrasing): "If I'm just trying to build a webapp, and I don't need telcom-grade scalability, why should I use Erlang? Yes, Erlang may be the best language for concurrent programming, but I will never have to use Erlang's concurrency features directly, just as I don't use any threading libraries when I'm building a webapp with other languages."

Here's my answer: you can certainly build a webapp without using concurrency features directly. Most web developers don't. Just keep in mind that most framework designers use languages that support concurrency so poorly, that they have resorted to fooling you into thinking that the only webapps you can realistically build must follow the good ol' request-response, single-threaded, CRUD-based paradigm.

This is clearly a fallacy.

Many high-quality, innovative webapps don't follow this paradigm.

The Google search engine doesn't follow this paradigm. Neither does Meebo.

Many of the great webapps of tomorrow won't follow this paradigm.

Why should your webapp follow this paradigm?

The paradigm is shifting to a large degree because of one main innovation, which somebody has named "Comet" (it's a silly name if you ask me, but it has stuck). Comet boils down to having the browser keep an open XMLHttpRequest connection to the server, waiting for the server to send it messages. This opens the door to full bidirectional communication between the browser and the server (this is essentially what Meebo does).

If you want to build a Comet app with today's Java, Ruby or Python frameworks -- good luck. Not only will the message-passing -- not to mention clustering -- logic be painful to code (especially if you've tasted the power of Erlang :) ), your servers will also croak when your app reaches a few thousand (or fewer) simultaneous users.

Partly for these reasons, nobody uses Ruby or Python to build commercial phone switches (not that I know of, anyway :) ). However, Ericsson builds them quite successfully with Erlang. Erlang powers highly scalable, fault-tolerant phone switches with nine nines availability. Building scalable clusters is what Erlang is made for, and I dare say Erlang makes such programming tasks (relatively) easy.

If you want to build an app that uses Comet, you will need concurrency and scalability, so use Erlang.

However, if your app doesn't use Comet, should you still care about Erlang's unmatched capabilities for concurrency, clustering and message passing?

Yes.

Another good reason to use Erlang is that your web application may benefit from -- or even require, as in Google's case -- distributing a calculation over multiple processors or even multiple nodes in a cluster prior to sending a response back to the browser.

Erlang is designed for distributed programming. In Erlang, sending a message to a remote node is as easy as sending it to another process in the same box. As Joe Armstrong has shown, writing a parallel version of lists:map is almost laughably easy in Erlang. This is all it takes:


pmap(F, L) ->
S = self(),
Pids = map(fun(I) ->
spawn(fun() -> do_f(S, F, I) end)
end, L),
gather(Pids).

gather([H|T]) ->
receive
{H, Ret} -> [Ret|gather(T)]
end;
gather([]) ->
[].

do_f(Parent, F, I) ->
Parent ! {self(), (catch F(I))}.


Distributed computation will grow in importance as server manufacturers race to add more cores to their boxes. (The next release of Sun Niagara will have 64 cores, for instance.) Even if you don't need distributed computation right now, you'll benefit from adding Erlang to your toolbox. Multi-core computation is where things are going, and knowing Erlang will help you stay on top of the game.

By the way, I'm not suggesting that Erlang has a monopoly on distributed programming -- Erlang is just years ahead of any other language in this area. Erlang has 20 years of development behind it for this specific purpose, with a long track record of success in large-scale production systems. Sure, you always have the option of using, say, Starfish for Ruby. You'd just have to cross your fingers and hope your competition doesn't use Erlang :)

To summarize, if you are 100% confident that you will never attempt to push the envelope of what your webapp's backend can do, you probably won't need Erlang's concurrency features. IMO, you'll have fun by using Erlang, but you certainly don't need it. However, if you start venturing into areas that are beyond the single-threaded CRUD view of the world, Erlang's capabilities will help you greatly.

So why use Erlang?

Think outside the thread.