Favorite Lisps Poll Result

With 78 responses so far, here are the breakdowns for the responders' favorite Lisps:

Scheme: 41%
Common Lisp: 27%
LFE: 12%
Arc: 10%
Emacs Lisp: 6%
Clojure: 4%

You can see the full report here.

I thought that Common Lisp would get more votes Scheme (isn't CL more widely used?), but apparently I was wrong. Arc and LFE have made an impressive showing for their young age (but this is probably due to the bias of my blog readers).

Introducing Presidential Vimagi, Where Anyone Can Paint Political Cartoons for US Presidents

With elections around the corner and politics everywhere these days, I couldn't resist throwing some Erlang into the mix. Don't worry, I'm not about to start discussing my political opinions on this blog. You can find plenty of political commentary on other sites. Instead, I decided to create a sub-Vimagi for political cartoons: Presidential Vimagi.

On Presidential Vimagi, you don't write or talk about your political views. You paint them. If a picture is worth a thousand words, a painting is worth, well, you do the math.

To give you inspiration, I assembled an all star cast of American presidents, vice presidents and presidential candidates: George Bush, Dick Cheney, Barak Obama, Hillary Clinton, Mike Huckabee, John McCain, Ron Paul, Bill Clinton and Al Gore. They each have a profile and you can paint on their profile's v.boards. You can also become friends with them if you want to show your support.

Presidential Vimagi is still a bit rough around the edges, so please let me know if you find any bugs or issues. Feature suggestions are also welcome, of course.

Update: I decided to take the site down for now. It's too prone to vandalism and I don't have the time to constantly monitor it to remove the offensive stuff. It was a fun experiment while it lasted. Oh well :)

Lisp Flavored Erlang!

Erlang has a Lisp syntax! Robert Virding just released LFE, his Lisp syntax compiler for Erlang. Finally, Erlang hackers enjoy the full power Lisp-style macros. I suspect it won't be long before you'll be able to hack ErlyWeb apps in LFE :)

While you're here, please answer this Wufoo poll:


Powered by Wufoo

Your Daily Dose of Erlang Evangelism

This is a good one I fished from the mailing list:

Using Erlang continually makes me both smile and cry at the same time. I smile because of the overall simplicity it brings to solving all those hard issues I mentioned above, but I also cry knowing how many hours, days, weeks, and months my former colleagues and I spent trying
to solve all those really hard issues.

http://www.nabble.com/Steve-Vinosky-interview-to15709698.html#a15720750

Seaside-Style Programming in ErlyWeb

The Arc Challenge started an interesting thread in the ErlyWeb mailing list about continuations-driven web frameworks. ErlyWeb doesn't have built-in support for continuations, but Arc does and so does Seaside. I haven't paid much attention to the use of continuations in web frameworks before the Arc challenge, but I became especially interested in experimenting with them after seeing Seaside solution.

In case you haven't read it, this is the requirement of the Arc challenge:

Write a program that causes the url said (e.g. http://localhost:port/said) to produce a page with an input field and a submit button. When the submit button is pressed, that should produce a second page with a single link saying "click here." When that is clicked it should lead to a third page that says "you said: ..." where ... is whatever the user typed in the original input field. The third page must only show what the user actually typed. I.e. the value entered in the input field must not be passed in the url, or it would be possible to change the behavior of the final page by editing the url.

This is the original Arc solution:


(defop said req
(aform [w/link (pr "you said: " (arg _ "foo"))
(pr "click here")]
(input "foo")
(submit)))


This is how you would write the same logic in Erlang, if it had an Arc-like web framework:


said(A) ->
form(
fun(A1) ->
link(fun(A2) -> ["you said ", get_var(A1, "foo")] end,
"click here")
end,
[input("foo"),
submit()]).


The Erlang code is a bit more verbose, mostly because Erlang macros don't allow you to hide the "fun() -> ... end" syntax the way Lisp macros let you hide the (lambda ..) keyword.

This is the Seaside solution:


| something |
something := self request: 'Say something'.
self inform: 'Click here'.
self inform: something


IMO, this solution cheats a bit by using high-level functions for generating the HTML tags whereas the Arc version generates them explicitly. However, putting minor complaints aside, I think the Seaside version is the winner in readability. As a reddit comment said, it reads like prose. It doesn't even declare any closures explicitly -- it says exactly what it does and nothing more.

Wouldn't it be cool if we could use this style of programming in ErlyWeb applications?

Fortunately, we can! I hacked a continuations plugin for ErlyWeb that lets you write Seaside-style code so fans of this programming style would feel at home with ErlyWeb. (This is all done in 105 lines of code :) ) Before I explain how the plugin works, I'll show you how to create an ErlyWeb controller that implements the the Arc challenge using this plugin:


-module(said_controller).
-compile(export_all).
-import(continuations, [ask/2, confirm/2, pr/1]).

index(A) ->
continuations:do(
A, fun(K) ->
Name = ask(K, "name"),
confirm(K, "click here"),
pr(["your name: ", Name])
end).


This may seem more verbose than the Seaside code because of the module declarations at the top, but the "meat" is about the same. (I could make this code even smaller by integrating continations.erl deep into ErlyWeb, which would remove the explicit call to continuations:do(). However, I didn't want to go too far with this proof of concept.)

How does this work? Using concurrency, of course! For each "continuation", the plugin spawns a process and registers it in a Mnesia table according to a randomly generated key. The key (K) is encoded in the URLs to which the <form> and <a> tags point. When a request arrives, continuations:do() looks up the process in Mnesia and sends it a message of the type {A, self()}. The process does some work and sends back in reply the HTML to be rendered, and waits for the next message. The web server process receives the rendered HTML and sends it to the client using the normal ErlyWeb mechanisms.

If a process doesn't receive a message in 10 seconds, it dies and removes itself from the Mnesia table, which provides automatic garbage collection to stale sessions.

You can get the code for continuations.erl here. Just remember it's a proof of concept and I don't recommend using it in a production environment.

(Before you use it in your application, make sure you call continuations:start().)

Final word: IMO, although continuations help write more natural code in certain multi-page interactions, most of the logic in web applications involves rendering dynamic pages for RESTful URLs. So, if your web framework doesn't support continuations, don't worry about it too much. It's likely the code for your application wouldn't be dramatically smaller if you could write it with the use of continuations. (That said, take my advice with a grain of salt. I haven't used a continuations based framework to build a real application, so I may be missing something.)

More Erlang Fun: Distributed, Fault Tolerant MapReduce

I the Erlang Challenge posting, I showed how to implement a simple distributed parallel map function. You can pass it a list of nodes, and it performs each application of F1 on a random node ("node" means a connected Erlang VM that may exist on a different machine) from the list. Finally, it collects the results in the order of the arguments.

What Erlang nailed beside concurrent and distributed programming is fault tolerance. When processes die or nodes go down, the VM notifies their supervisors so they can perform error recovery. My previous example didn't demonstrate this capability, so in this article I'll show how to implement a similar solution but also taking into account node failures. In addition, instead of doing just a simple parallel map, this example will also fold the results, making it a basic MapReduce implementation.

(The reason I'm adding the 'reduce' phase is because it allows me to not worry about the order at which the results are accumulated. This implies that the reduce operation must be commutative.)

The function mapreduce() takes a function F1 and applies to elements of list L. Each application is performed on a remote node from Nodes. As opposed to the pmap() example, this implementation chooses the next node from the list in a rotating fashion instead of randomly (i.e., if Nodes is [A, B, C], then F1 is applied on remote nodes with the following order: A, B, C, A, B, C, A ...). The results are collected, and if any nodes went down, their computations are retried on the remaining nodes until all computations succeed or no nodes are left. If the computations all succeed, we call lists:foldl(F2, Acc, Vals), where Vals contains the results of the computations.

Implementation details:

- do_map() returns a list of {Pid, X} tuples. Pid is the process on the remote node in which F1 was applied to X. The reason we want to remember X is that if Pid's node goes down, we can later retry to evaluate F1(X) on a different node.
- For each {Pid, X} tuple, collect() receives either an {ok, Pid, Val} message, indicating the computation was performed successfully, or {nodedown, Node}, indicating the node on which Pid lived went down. collect() folds the result into a tuple of type {Successes, Failures, RemainingNodes}. If Failures is not empty, collect() calls do_map() on the list of Failures and the remaining nodes, and then recursively calls itself on the result and appends the outcome to Successes.

The code is below. (Note that I didn't test it, so it may have bugs :) )


-module(mapreduce).
-export([mapreduce/5]).

mapreduce(F1, F2, Acc, L, Nodes) ->
%% map F1 to the elements of L on nodes from Nodes
Results = do_map(F1, L, Nodes),

%% collect the results, retrying in the event of failures
Vals = collect(F1, Results, Nodes),

%% perform the reduce operation
lists:foldl(F2, Acc, Vals).


%% exit if we ran out of good nodes
do_map(_F1, _L, []) -> exit(no_nodes);
do_map(F1, L, Nodes) ->
Parent = self(),

%% apply F1 to values of L on remote nodes in a rotating fashion
{Results, _Nodes1} =
lists:foldl(
fun(X, {Acc, [Node | Rest]}) ->
erlang:monitor_node(Node),
Fun = fun() -> Parent ! {ok, self(), (catch F1(X))} end,
Pid = spawn(Node, Fun),
{[{Pid, X} | Acc], Rest ++ [Node]}
end, {[], Nodes}, L),
Results.

collect(F1, Results, Nodes) ->
collect(F1, Results, Nodes, []).
collect(F1, Results, Nodes, Acc) ->
{Successes, Failures, RemainingNodes}=
lists:foldl(
fun({Pid, X}, {Successes1, Failures1, Nodes1}) ->
Node = node(Pid),
receive
{ok, Pid, Val} ->
{[Val | Successes1], Failures1, Nodes1};

%% we may receive this message because of call to
%% monitor_node()
{nodedown, Node} ->
{Successes1, [X | Failures1], Nodes1 -- Node}
end
end, {[], [], Nodes}, Results),

if Failures =/= [] ->
%% retry the failed computations on the remaining nodes
%% and add the results to the current list of successes
Results2 = do_map(F1, Failures, RemainingNodes),
collect(F1, Results2, RemainingNodes, Successes ++ Acc);
true ->
Successes ++ Acc
end.


This is more complex than the pmap() example, but I think it packs a lot of functionality into a relatively small amount of code

Erlang Challenge Followup

After I published the Erlang Challenge, I saw a few comments on Hacker News and Reddit repudiating it with the following arguments, to which I'd like to respond:

- It was a snide response to the Arc challenge.

Maybe I should have read the article a couple of times over before I published it late last night, because if it came off as snide, I didn't communicate it as I had intended. I've found the Arc challenge fun and interesting (I even submitted my own Erlang solution), and I wouldn't intentionally respond to it in a snide way. The reason I said the Erlang challenge is in the spirit of the Arc challenge is that the Arc challenge highlights Arc's strengths, whereas the Erlang challenge highlights Erlang's strengths.

- It was rigged in favor of Erlang.

That was basically the point. I wanted the Erlang challenge to demonstrate the kinds of problems that Erlang excels at solving. Maybe I should have made this clearer when I wrote the article -- it was more of an illustration of Erlang's strengths than a "real" challenge. I wasn't really expecting other languages to compete with Erlang on its home turf, the same way I wouldn't expect Erlang to compete with, say, C or C++ at low-level systems programming or with ActionScript at Flash programming.

- It showed that Erlang is good at solving a narrow scope of problems, but I didn't show that Erlang is a good general purpose tool.

I don't think it's possible to show in 10 lines of code how good (or bad) a language is at solving anything but problems that are very similar to those in the example. The only true way to judge whether a language is good at building certain types of systems is to build them using the language and evaluate the results.

I know that people have used Erlang successfully to build phone switches, a MMORPG, a massively scalable DBMS, web applications, a collaboration application, high performance messaging servers, ad servers, web servers, a scalable poker server, and a 3D modeling tool. When I say that Erlang is a good language for building certain applications, that's what I base it on.

Edit: I forgot to list CouchDB, an open source document storage system, and Triggit, a widget server unlike any other.

The Erlang Challenge

In the spirit of Paul Graham's Arc challenge, and in spite of the "controversy" around it, I devised a programming challenge of my own. It's called the Erlang Challenge.

The goal is to implement an advanced "parallel map": a function (F1) that applies another function (F2) to every element (E) of a list (L1) concurrently and then gathers the results in a new list (L2). The order of results in L2 corresponds to the order of elements in L1, and the elements of L1 are guaranteed to remain unchanged no matter what function F2 does. To make things more interesting, the F1 should also accept a list of cluster nodes (NL) onto which the computations can be distributed. Each application of F2 to an element of L1 should be done on a random node from NL, and the result should be send back to the node on which F1 was called (N). If NL is empty, all computations should occur locally on N.

This challenge isn't just about succinctness, but, like most real-world systems, also about scalability and performance. Therefore, it has the following requirements:

- The implementation must be able to spawn at least 1 million concurrent processes on a modern server machine.
- On multi-core machines, the application's performance must increase (almost) linearly with the number of available CPUs.
- The application as a whole must exhibit soft real-time performance. Specifically, this means that processes won't freeze for a long time during global garbage collection sweeps, and that no process will be able to block the entire VM to do IO or call native code, regardless of what F2 does. In other words, it is impossible to create a function F2 that could block the scheduler while F2 executes.

Here's how I would implement it in Erlang (this is loosely based on Joe Armstrong's pmap implementation):


pmap(Fun, List, Nodes) ->
SpawnFun =
case length(Nodes) of
0 -> fun spawn/1;
Length ->
NextNode = fun() -> nth(random:uniform(Length), Nodes) end,
fun(F) -> spawn(NextNode(), F) end
end,
Parent = self(),
Pids = [SpawnFun(fun() -> Parent ! {self(), (catch Fun(Elem))} end)
|| Elem <- List],
[receive {Pid, Val} -> Val end || Pid <- Pids].


How would you implement it in your favorite language?

Update: please read the followup.

Debugging with Distel

Until recently, I've done my Erlang debugging either using the built-in debugger or not using a debugger at all and instead relying on the good old io:format() function. The reason I haven't been keen on using the Erlang debugger is that it's not integrated into my editor (I use Emacs). To debug a file, I'd have to open it separately in the debugger UI, where I could set breakpoints and step through the code. For simple bugs, this is more effort than adding a few io:format() statements, reloading the page, and figuring out the bug from the output.

I'm not an IDE fanatic (well, duh, I use Emacs) -- I can do fine without most of the features in advanced IDEs -- but I do expect any decent editor to have at least 3 features: syntax highlighting, auto-indentation, and integrated debugging.

(I'm ambivalent about IDEs with code completion. It adds convenience, but it also it lets you get by without really remembering your APIs. Without code completion, I remember most of the my application's function names and their parameters simply because not remembering them slows me down too much. This is akin to the cellphone effect on remembering numbers: before I had a cellphone, I remembered all my close friends and family members' numbers. Looking them up in an address book was too much work. Now, I remember just a fraction of them. My cellphone made me lazy.)

The Emacs mode for Erlang provides syntax highlighting and auto-indentation, but no integrated debugging. AFAIK, neither Erlide nor ErlyBird have integrated debuggers. I knew Distel was supposed to add debugging support, and I even tried setting it up a while ago but gave up after running into problems I couldn't figure out. Last weekend, though, I finally got it working after following Bill Clemenson's Distel tutorial. I must say that having a debugger integrated into Emacs is a huge improvement. I recommend it to every Erlang hacker.

My only peeve with this debugger is that you must mark each file you want to debug as interpreted (the Emacs shortcut is "C-c C-d i") before you can set breakpoints in it and step into its functions. This seems unnecessary -- Erlang .beam files usually contain metadata including the location of their source files. I wish the debugger used this information to auto-interpret all the files you're trying to debug.

This is a relatively small peeve, though. The Distel debugger is definitely a step up from io:format().

I'm still hoping someone will create a more user-friendly Erlang IDE with a slick UI and an integrated debugger. I like the TextMate interface, but TextMate doesn't do auto-indentation for Erlang or debugging. I'll stick to Emacs+Distel for now.

Triggit: Why Didn't I think of That?

Triggit received some great publicity last week. Triggit, a small startup in San Francisco, recently released a product makes it easy to add widgets and ads to any site by dragging and dropping -- no HTML editing required. I think it can really catch on among bloggers who are disinclined to muck around in mounds of PHP code in order to add some widgets to their blogs. I can also see this kind of technology fitting very well in sites like MySpace, where non tech-savvy users could use Triggit to effortlessly soup up their profiles.

Ryan Tecco, Triggit's CTO, told me Triggit uses Erlang for all the heavy lifting in the backend. You probably won't find this shocking, but I think using Erlang is a smart choice. Low latency, scalability and high availability are all essential for this kind of product: it's one thing to have performance/availability issues on *your* site, but it's much worse to let those issues affect other people's sites as well.

Finally, Triggit is an interesting data point on how very small teams can successfully use Erlang to achieve impressive results in a short time frame.

Update: I added this video with Triggit :)

How to Use Concurrency to Improve Response Time in ErlyWeb Facebook Apps

When I was building the Vimagi Facebook app, I came across a common scenario where using concurrency can make your application more responsive for its users.

The typical flow of responding to requests coming from Facebook looks like this:

1) request arrives
2) do some stuff (mostly DB CRUD operations)
3) call Facebook API to send notifications / update newsfeeds and profile FBML
4) send response

When you're building a facebook app with ErlyWeb, you can instead do the following:

1) request arrives
2) do some stuff (mostly DB CRUD operations)

spawn(fun() ->
  3) call Facebook API to send notifications / update newsfeeds and profile FBML
end)

4) send response


The Facebook API calls in step 3 are much more expensive than the typical ErlyWeb controller operations because these calls involve synchronous round trips to the Facebook servers plus XML processing for the responses. By performing the Facebook API calls in a new process we can return the rendered response to the browser immediately and let the Erlang VM schedule the Facebook API calls to happen leisurely in the background.

The only gotcha is that if an error occurs in the spawned process we can't notify the user right away -- but this isn't really problem because we can log the errors and retry later, which is arguably a better approach anyway from a usability standpoint.

It's really that easy! A simple call to spawn() makes our app *feel* much faster. This puts the debate around language performance comparisons in a new light: how do you take into account the observation that some languages make "cheating" so much easier? :)

Vimagi: The Facebook App

I finally finished the Vimagi Facebook app! Check out the screen shots:





The app works similar to the vimagi.com website: you can create paintings with titles, tags, and descriptions and share them with anyone. Other Facebook users can comment on the paintings and rate them. Paintings created in Facebook are automatically posted to vimagi.com, where vimagi.com users can also add comments and ratings. All paintings have embed codes you can use to embed them on any site (with playback). In Facebook, you can create paintings for your Facebook friends and those paintings will appear on their (and your) profile. Similar to vimagi.com, the Facebook app has a gallery and a tags page, which contains content from both vimagi.com and the Facebook app.

I created the Facebook app mostly as a learning exercise, but I also believe that the Vimagi features would appeal to more people by being embedded into people's their existing social network rather than in a standalone site.

Unfortunately, I underestimated how tricky it would be to port the vimagi.com features and code into Facebook without breaking the site and while seamlessly blending the paintings and data created on Facebook and on vimagi.com. (One of the pesky issues I encountered is that Facebook applications have root URLs of the form http://apps.facebook.com/[app name], which was incompatible with the vimagi.com relative URLs. The existing URLs all start with a forward slash, assuming they follow immediately after the domain name. My life would have been much better if Facebook used the url scheme "http://[app name].apps.facebook.com/" for Facebook apps as it would have allowed existing URLs to remain unmodified.)

If you like to paint, I hope you enjoy the applications, and if you have friends who like to paint, please send them an invitation. Facebook needs some more Erlang-generated pageviews :)

If you like Vimagi and you're on Facebook, please join the Vimagi fan club. Also, I'll appreciate it if you let me know of any feedback or suggestions you may have.

P.S. Thanks I to Bryan Fink for the excellent erlang2facebook library.

Join the ErlyWeb Fan Club on Facebook

Are you on Facebook? Then join the ErlyWeb Fan Club!

If all goes according to plan, some day this fan club will have more members than the If this group reaches 4,294,967,296 it might cause an integer overflow group :)

Amazon SimpleDB Runs on Erlang

I just came across this blog post: http://www.satine.org/archives/2007/12/13/amazon-simpledb/. The author got the scoop that Amazon's just-released SimpleDB runs on Erlang. This is actually the second Amazon Web Service that runs on Erlang -- at least, if the rumors I've heard that Amazon SQS was built with Erlang as well are true (update: I found the reference in this article).

It's pretty cool knowing that you can use the same language that powers some of the massive systems that Amazon and Ericsson have built to whip up a blog app just as easily as in Ruby or Python. (But if you built it in Erlang/ErlyWeb, you could also use Comet to make the blog update itself in real time, resting assured that horizontal scalability is just a matter of reconfiguring your Mnesia schema :) ).

Amazon SimpleDB is a game changer. It fills the last hole in making Amazon Web Services a complete environment for elastic, scalable web application hosting: the need for a fast, reliable data store. Together with S3 and EC2, SimpleDB makes it feasible for small startups to scale like the big companies but without the operational overhead. What a great set of products from Amazon.

ErlyWeb vs. Ruby on Rails EC2 Benchmarking Strangeness

I've been running some more benchmarks for the ErlyWeb vs. Ruby on Rails EC2 Performance Showdown. The results I've gotten are very strange -- so strange, in fact, that I'm not going to "officially" publish anything before I run all the tests one more time.

What's strange about the results? I'll give you a quick glimpse, but please keep in mind that none of these observations are the official benchmarks results and they may all be invalidated in the next run.

- Recompiling the Erlang files *without* HiPE seemed to *increase* max performance (in requests/sec) by ~10%.
- Running Yaws with kernel poll enabled ("+K true" passed to erl) *decreased* the max ErlyWeb performance to 410 (~41% decrease).
- No matter how many Mongrels I started (I tried 1,3,5, and 10) behind Pen (running on the same server), the max performance of Rails was ~18% lower than when it was running a on single Mongrel without Pen (111 requests/sec).

This across-the-board strangeness calls for another run. I'll try to have the results sometime in the next few days. In the meantime, if anybody else wants to take a stab at the these benchmarks, it would be interesting to see how our results compare.

By the way, although the effects of the changes I made were different from my expectations, when I ran the same tests the results were similar to the results I got last time.

ErlyWeb vs. Ruby on Rails EC2 Performance Showdown

A few people have asked me to run benchmarks comparing ErlyWeb and other frameworks, especially Ruby on Rails. Benchmarking is pretty boring, which is why I haven't benchmarked ErlyWeb until now. Plus, although I was pretty confident that ErlyWeb outperforms Rails, I didn't think proving it mattered that much. People who love Rails generally don't care much about raw performance, and people who love Erlang and functional programming don't need much convincing to use ErlyWeb over Rails.

However, as ErlyWeb is becoming more mature, my curiosity has been growing. With the recent releases of Rails 2.0 and Erlang/OTP R12B, I finally decided to sacrifice the better part of my weekend to give both frameworks some serious stress testing and see how they compare.

Before I could run the benchmarks, I had to figure out the physical setup. I needed at least two powerful servers with a fast link between them, and I didn't have this kind of hardware lying around. All I have is my MacBook, but running the benchmarks on my MacBook wouldn't prove much because it would be impossible to isolate the impact the clients would have on the servers by running on the same machine.

Thankfully, Amazon EC2 made this easy to solve. I could just fire up two EC2 instances and have one of them stress test the other over EC2's fast internal network.

Benchmarking can be complex. How do you benchmark a web app? There are so many moving pieces and possible user interactions. I decided to keep it simple: I would test the performance of rendering a single page displaying a simple dynamically generated table. There would be no database queries -- the table's contents would be hardcoded in the controllers and the views would programatically render the contents for each request.

I decided against using a database because I wanted to isolate the performance difference between ErlyWeb and Rails. 3rd party dependencies that may affect the results by introducing bottlenecks that would make one or both frameworks appear artificially slow. In addition, in many, if not most, webapps, the majority of requests can be served from the cache -- which means they don't trigger database queries -- so this test scenario is actually quite realistic.

The Test App

Here's a screenshot of the awesome test page I created:



You can see the generated page here (the output is identical for both apps except for minor whitespace differences).

This is the Rails controller code (songs_controller.rb):


class SongsController < ApplicationController
def index
@songs =
["Sgt. Pepper's Lonely Hearts Club Band",
"With a Little Help from My Friends",
"Lucy in the Sky with Diamonds",
"Getting Better",
"Fixing a Hole",
"She's Leaving Home",
"Being for the Benefit of Mr. Kite!",
"Within You Without You",
"When I'm Sixty-Four",
"Lovely Rita",
"Good Morning Good Morning",
"Sgt. Pepper's Lonely Hearts Club Band (Reprise)",
"A Day in the Life"]
end
end


This is the Rails view code (songs/index.html.erb):

Songs

<% for song in @songs %>



<% end %>

<%= song %>

This is the ErlyWeb controller code (songs_controller.erl):


-module(songs_controller).
-export([index/1]).

index(_A) ->
{data,
[<<"Sgt. Pepper's Lonely Hearts Club Band">>,
<<"With a Little Help from My Friends">>,
<<"Lucy in the Sky with Diamonds">>,
<<"Getting Better">>,
<<"Fixing a Hole">>,
<<"She's Leaving Home">>,
<<"Being for the Benefit of Mr. Kite!">>,
<<"Within You Without You">>,
<<"When I'm Sixty-Four">>,
<<"Lovely Rita">>,
<<"Good Morning Good Morning">>,
<<"Sgt. Pepper's Lonely Hearts Club Band (Reprise)">>,
<<"A Day in the Life">>]}.


This is the ErlyWeb view code (songs_view.et):


<%@ index(Songs) %>

Songs

<% [song(S) || S <- Songs] %>


<%@ song(S) %><% S %>

Setup

This is what I installed on the EC2 image (I started with the basic public Fedora Core 4 image):

- Erlang/OTP R12B.
- Yaws 1.73 (compiled with HiPE).
- ErlyWeb from trunk (compiled with HiPE).
- Rails 2.0.1
- Mongrel 1.1.1
- MySQL 4.1 (unused)
- Tsung 1.2.1.

Notes:

- Rails insists on connecting to a database even if you don't want or need one. There must be some hacky way of running Rails without a database, but it was easier for me to just setup MySQL so Rails stops complaining.
- I also compiled the ErlyWeb app with HiPE.

Tsung (http://tsung.erlang-projects.org/) is the tool I used for the stress testing. It's pretty simple to use -- you specify in an XML file where the server, what page(s) to request and the frequency of initiating new requests and then it runs the test and produces a report. You can read about it more on its website.

One interesting fact about Tsung is that it is written in Erlang to take advantage of Erlang's support for massive concurrency (there will be more on that later :) ).

I installed Tsung on the same machine image as the web servers because although during the test one instance runs the servers and the other runs Tsung, I wanted to have two identical EC2 machine images to ease setting things up.

During the test, I had both servers running on the server instance listening on different ports (Mongrel on port 3000, Yaws on port 3001). I then ran Tsung twice: first against Mongrel and then against Yaws. The tests did not overlap.

For the test, I configured Tsung to go through a sequence of 11 10-second phases during which Tsung would send requests to the server at increasing frequencies. In the first phase, Tsung would send 2 requests per second. It would then progress to 10, 20, 40, 100, 200, 400, 1000, 2000, 4000, and finally 10000 requests per second.

Caveat Emptor

Before I show the results, I would like to say that you should take them with a grain of salt. Many benchmarks have flaws. Maybe I messed something up during the installation. Maybe my methodology was wrong. Maybe I didn't perform some Mongrel optimization trick that boosts its performance. Maybe your application would behave quite differently from my test app. You have been warned.

Results

UPDATE: the initial results were inaccurate because I didn't run Mongrel in production mode. I updated the results and the graphs to reflect Rails's performance in production mode.

Below is the summary of the most important measurements.

Peak Performance
Rails: 15.1 112.8 requests/sec
ErlyWeb: 699.8 requests/sec



Peak Send Rate
Rails: 181.11 1360.78 Kb/sec Kb/sec
ErlyWeb: 5997.50 Kb/sec (hmm... did we hit EC2's limit?)



Total Size Sent
Rails: 1.65 5.83 MB
ErlyWeb: 28.47 MB



Peak Receive Rate
Rails: 84.87 175.79 Kb/sec
ErlyWeb: 941.84 Kb/sec



Total Size Received
Rails: 0.21 0.64 MB
ErlyWeb: 4.43 MB



Performance Degradation Point
(The phase at which the framework started sharply dropping the response rate)

Rails: 100 400 requests/sec phase.
ErlyWeb: 4000 requests/sec phase.



Note: these are the updated reports after running Mongrel in production mode

You can view the full Tsung reports here: ErlyWeb Rails

These reports are from the second test (30 second sustaining at 10 requests/sec):
ErlyWeb Rails

You can also download the reports, zipped: reports.tar.gz

Conclusion

ErlyWeb's peak response rate was 47x 6x Rails's on similar hardware.

I'm actually surprised by the results. I expected ErlyWeb to win, but I didn't expect ErlyWeb to outperform Rails by such a wide margin.

The results probably reflect differences in language and runtime more so than implementation details in the frameworks. After 20 years of refinement driven by needs of high performance telcom applications, Erlang delivers.

Update:
Looking at the peak send rate for ErlyWeb -- an almost round 6 MB/s, I wonder if ErlyWeb just saturated EC2's internal bandwidth, and whether ErlyWeb could perform even better on a faster link.

Update 2:
I ran another test, this time hitting each server at 10 requests/sec sustaining for 30 seconds to measure average response time. Rails's response time was in the 120-180 5-7 msecs range, and ErlyWeb's was in the 1.4-1.8 msec range. This means that under medium to light load, ErlyWeb serves the 'songs' page roughly 100x 4x faster than Rails.

Vimagi Speedups

Vimagi was much slower than it should have been.

I had foolishly compiled Vimagi in production with {auto_compile, true}. This option tells ErlyWeb to scan the app's source files and recompile all the ones that have changed since the last request. This feature greatly speeds up development because you can edit your file, reload the page, and immediately see the effects of your changes. It was also convenient for me in production because after checking in some changes from my dev box I would just 'svn up' on the production server and the changes will be deployed automatically. Unfortunately, I didn't realize what a negative impact it has on performance. Thankfully, David King brought this to my attention on the ErlyWeb mailing list and I've since disabled auto_compile on the production server.

Vimagi performs much better now. Clicking around the site, most pages now load in 0.3 - 0.5 secs according to YSlow. It feels much faster. In fact, Vimagi's performance is now similar to BeerRiot's, which is no small feat.

By the way, I'm not sure about BeerRiot, but Vimagi's pages are all dynamically generated -- I haven't implemented any caching (my VPS can easily handle the current traffic levels, so implmeneting caching right now would be a premature optimization). For an entirely dynamic site, I think this is very good performance.

Erlang Quote of the Day

Who cares if Erlang starts slowly - it was designed to start once and never stop - we have systems that have run for 5 years - a two seconds start-up time amortized over 5 years is not *too* bad.

- Joe Armstrong

(From http://www.nabble.com/idea%3A-service-pack-one-tf4800752.html#a13735173)

ErlyWeb Presentation, Dec. 6th at Berkeley, CA

The BayFP group is on a roll: 3 presentations on functional web development frameworks in 3 months. Alex Jacobson presented HAppS, David Pollak presented Lift, and in the next meeting I'll be presenting ErlyWeb.

The meeting will take place on Dec. 6th in Berkeley. You can find directions here.

There will be a $300 attendance fee.

Just kidding -- it's free. Please attend.

BayFP Meeting Tomorrow

I'm going to the BayFP meeting tomorrow. David Pollack will present lift, the web framework he created for Scala. Anyone who's interested is welcome to attend.

Wide Finder Project Summary

WF II: Erlang Blues (9/22)

Dear Erlang I: · I like you. Really, I do. But until you can read lines of text out of a file and do basic pattern-matching against them acceptably fast (which most people would say is faster than Ruby), you’re stuck in a niche; you’re a thought experiment and a consciousness-raiser and an engineering showpiece, but you’re not a general-purpose tool. Sorry.

WF XI: Results (10/30)

Erlang 6.46
Ruby 50.16
...
Analysis · Are you kidding me!?!? Getouttahere. Maybe someday.

Questioning Questioning Functional Programming

I spotted today on reddit an article from O'Reilly Radar titled Questioning Functional Programming. It's based on a programming contest in which more teams used non-functional languages than functional ones.

[The competition is] notable for two things: the fiendish complexity of the problem and the absence of functional programming languages in the final results. Top place went to a C++ program, with a Perl team right behind in second. In third place was "Team Celestial Badger" using OCaml and C++. In all there were 81 teams using C++, 67 using C, 66 using Haskell, 64 using Python, 52 using OCaml, 48 with Java, and 35 with Perl. Only 4 tried Erlang (the same number as used Delphi).

According to the report from the competition:

...As far as functional programming is concerned, we must conclude that functional languages didn’t fare too well this year (although in the Top 15 there were five users of OCaml and three of Haskell). Better luck next year!

There were 421 teams, of which 122 used functional languages -- about 29% (which ought to be much higher than their rate of adoption by the industry). Still, over 50% of the top 15 places were won by the teams that used functional languages. It looks like good performance to me.

It says to me that we're still in the early days of widespread adoption of functional programming.

I don't think this is what it says. Functional programming languages are obviously less popular than imperative ones. Maybe functional languages will never be as widespread as imperative languages. However, this competition is just a competition. I doubt it says much about language adoption by the industry now or in the future. (By the way, has anyone heard of Ruby? :) ) If anything, this competition suggests that functional programming languages are quite widespread, given their high rate of adoption by the contestants.

Like Ruby, which had tiny uptake until Rails and 37 Signals boosted it into the stratosphere, functional programming languages lack a reason to use them.

If Ruby lacked a reason to be used before 37 Signals, why did DHH use it to create Rails? He could have stuck with PHP. The reason to use Ruby did exist, even if fewer people knew about it.

"Functional programming languages lack a reason to use them." I don't think that great concurrency, scalability, fault tolerance, distributed programming, hot code swapping, Mnesia, Yaws, and the rest are useless.

(In fact, Amazon has made Erlang much less useless today by announcing the new 4-core and 8-core options for EC2 instance. I can't wait to get my hands on those :) )

Even those who advocate them [functional languages] aren't comfortable enough with them to push on with them in the face of hard problems.

The teams that were comfortable with functional languages did push on them in the face of hard problems: those teams used functional languages in the contest. They took 8 of the top 15 places, too.

I must say that I disagree with the conclusions of the article.

I'll end this posting with a relevant message from the Erlang-questions mailing list yesterday:

> Hi.
> Recently, I have been trying to learn a language other than Java.
> Erlang seems to be ready for the prime time (used in the commercial world).
> On the other side, OCaml seems to be a cool and capable language.
> I want to learn something useful and fun.
> Tell what do you think ?
> (if you are Erlang die-hard, please put this aside ;))
>

When I was evaluating Python alternatives for building the core
technology behind MochiAds I tried out a bunch of languages and Erlang
was the only one that was easy for me to learn and had the right
balance of features, performance, and reliability. A year later we
have about 16 machines running 80 Erlang nodes powering about 16
different "components" of our infrastructure and 4 people working on
it at the moment (originally it was just me). It worked out so well
that we rewrote the server component of our MochiBot service in Erlang
and we've been using it to build lots of internal tools such as our
monitoring software, our single sign-on service, etc. as well. None of
us had previous Erlang experience, but we're all very comfortable with
it now.

After about a year with Erlang, I'm not sure I could part with hot
code loading, light-weight processes, and multiplexed socket IO for
writing servers. Also, Mnesia has been really useful to us to
temporarily store "real-time" data (ram_copies) so that we don't have
to make users wait for it to get batched into the SQL databases. The
distribution stuff mostly Just Works once you figure out how to set it
up (though we did have one bad experience with a network partition due
to a switch acting up, it was recoverable manually).

O'Caml is a useful language too, but for writing a network app I can't
really imagine going with anything but Erlang if you're looking for
redundancy and scale. Unless you want to write your own half-baked
Erlang-like system before even trying to solve something a little
closer to your actual problem domain.

-bob

It looks like functional programming must be good for something.

Blog Maintenance

I moved my blog to a different hosting provider. Please let me know if you notice any URLs that the move may have broken.

I also removed that annoying CAPTCHA plugin so you shouldn't have problems leaving comments from now on. If the plugin prevented you from leaving comments in the past you can now go ahead and leave them now. (I may have to put CAPTCHA back if Akismet flakes out on me but I hope I won't have to resort to that.)

I apologize for any inconvenience.

More Vimagi Goodies

- You can now undo strokes.
- There's a better tools interface.
- You can embed paintings on any site, and even play them back!

For example, here's a great painting by Rivka. It's titled "horror flicks".



Enjoy!

If I Were Building Amazon.com...

I would use ErlyWeb :)

From Amazon's Dynamo by Werner Vogel:

"In Amazon’s decentralized service oriented infrastructure, SLAs play an important role. For example a page request to one of the e-commerce sites typically requires the rendering engine to construct its response by sending requests to over 150 services. "

Unless all 150 service requests are executed sequentially, which is hard to believe, Amazon's page rendering ought to be able to benefit from Erlang's lightweight concurrency -- even more so than Vimagi :)

This makes me think about adding the following return value to ErlyWeb controller function:


{concurrent, [{ewc,...}, {ewc, ...}, {ewc, ...}]}


This would tell ErlyWeb to render each component in a different process. It may not make a big performance difference in rendering simple pages (it might even slow it down a bit), but for some applications (the future Erlmazon? :) ), it could be useful.

What do you think?

Vimagi Updates

Vimagi news:

- Alpha channel support. You can now pick the transparency level from the color box.
- Clearing a painting erases all previous stroke data. From now on, the stuff you do before clearing the painting doesn't get played back.
- Performance improvements.
- Nightly backups to S3: your precious paintings are safe :)

Enjoy!

ErlyWeb is Facebook-Ready

In case you didn't know, Bryan Fink of BeerRiot fame has created a Facebook library for Erlang called erlang2facebook. You can get it from http://code.google.com/p/erlang2facebook/. It is bundled with a complete Hello World Facebook app that is built with ErlyWeb, so you can hit the ground running with your Erlang Facebook development.

I can think of some fun projects to build with this library. If you're going to use it to build an app, please let me know!

Embedding Vimagi

Dear reader, I have good news. The internet is finished. The missing link was finally implemented.

You can now embed your Vimagi paintstream in any web page :)

All you have to do is add the following to your HTML, replacing [USERNAME] with your Vimagi username:

My Vimagi PaintStream

Enjoy!

Great Concurrency -> Better Webapps

I've been asked a few times: Is concurrent programming really useful for web developers? Web servers already handle client requests in different threads/processes, so application code doesn't need to use concurrency explicitly, right?

My opinion is that althought it's possible to write many database-driven web apps without using concurrency, concurrency can still be useful.

Vimagi has some real-time features in that rely heavily on concurrency, but it also helps in a more common scenario: speeding up page loads.

In Vimagi, every time someone views a painting or a user profile, a SQL query is executed to increment the view counter for that page. This is more-or-less what the controller code does:


show(A, Id) ->
painting:increment([num_views], {id,'=',Id}),
Painting = find_id(Id),
{data, Painting}.


This caused a problem. When traffic increased, the 'increment' function, which executes an UPDATE statement in the database, started performing poorly, thereby noticeably slowing down page loads. Occasionally, it even timed out after 5 seconds, showing an error screen to the user.

I increased the timeout to 20 seconds. This helped avoid the error screen, but the response time was still too long.

Erlang made this almost too easy to fix. All I had to do was change


painting:increment([num_views], {id,'=',Id})


to


spawn(
fun()->
painting:increment([num_views], {id,'=',Id})
end)


It's darn simple, and it works. Instead of executing the UPDATE statement in the process that's responsible for rendering the page, I spawn a new process and execute the transaction there. Spawning processes in Erlang is dirt-cheap, so I don't have to worry about performance or memory issues. The user gets a quick response, the view counter is incremented, and the problem goes away.

Introducing Vimagi.com

I just launched Vimagi, my first ErlyWeb app: vimagi.com.

Painting is fun. Sharing paintings with your friends is also fun. Painting together with your friends is a new kind of fun you can have on Vimagi.

Give it a try and let me know what you think -- I'll appreciate the feedback.

Note: Vimagi is in early beta. It probably has many bugs. Please email me if you find any.