However, as ErlyWeb is becoming more mature, my curiosity has been growing. With the recent releases of Rails 2.0 and Erlang/OTP R12B, I finally decided to sacrifice the better part of my weekend to give both frameworks some serious stress testing and see how they compare.
Before I could run the benchmarks, I had to figure out the physical setup. I needed at least two powerful servers with a fast link between them, and I didn't have this kind of hardware lying around. All I have is my MacBook, but running the benchmarks on my MacBook wouldn't prove much because it would be impossible to isolate the impact the clients would have on the servers by running on the same machine.
Thankfully, Amazon EC2 made this easy to solve. I could just fire up two EC2 instances and have one of them stress test the other over EC2's fast internal network.
Benchmarking can be complex. How do you benchmark a web app? There are so many moving pieces and possible user interactions. I decided to keep it simple: I would test the performance of rendering a single page displaying a simple dynamically generated table. There would be no database queries -- the table's contents would be hardcoded in the controllers and the views would programatically render the contents for each request.
I decided against using a database because I wanted to isolate the performance difference between ErlyWeb and Rails. 3rd party dependencies that may affect the results by introducing bottlenecks that would make one or both frameworks appear artificially slow. In addition, in many, if not most, webapps, the majority of requests can be served from the cache -- which means they don't trigger database queries -- so this test scenario is actually quite realistic.
The Test App
Here's a screenshot of the awesome test page I created:
You can see the generated page here (the output is identical for both apps except for minor whitespace differences).
This is the Rails controller code (songs_controller.rb):
class SongsController < ApplicationController
["Sgt. Pepper's Lonely Hearts Club Band",
"With a Little Help from My Friends",
"Lucy in the Sky with Diamonds",
"Fixing a Hole",
"She's Leaving Home",
"Being for the Benefit of Mr. Kite!",
"Within You Without You",
"When I'm Sixty-Four",
"Good Morning Good Morning",
"Sgt. Pepper's Lonely Hearts Club Band (Reprise)",
"A Day in the Life"]
This is the Rails view code (songs/index.html.erb):
<% for song in @songs %>
<%= song %>
<% end %>
This is the ErlyWeb controller code (songs_controller.erl):
[<<"Sgt. Pepper's Lonely Hearts Club Band">>,
<<"With a Little Help from My Friends">>,
<<"Lucy in the Sky with Diamonds">>,
<<"Fixing a Hole">>,
<<"She's Leaving Home">>,
<<"Being for the Benefit of Mr. Kite!">>,
<<"Within You Without You">>,
<<"When I'm Sixty-Four">>,
<<"Good Morning Good Morning">>,
<<"Sgt. Pepper's Lonely Hearts Club Band (Reprise)">>,
<<"A Day in the Life">>]}.
This is the ErlyWeb view code (songs_view.et):
<%@ index(Songs) %>
<% [song(S) || S <- Songs] %>
<%@ song(S) %>
<% S %>
This is what I installed on the EC2 image (I started with the basic public Fedora Core 4 image):
- Erlang/OTP R12B.
- Yaws 1.73 (compiled with HiPE).
- ErlyWeb from trunk (compiled with HiPE).
- Rails 2.0.1
- Mongrel 1.1.1
- MySQL 4.1 (unused)
- Tsung 1.2.1.
- Rails insists on connecting to a database even if you don't want or need one. There must be some hacky way of running Rails without a database, but it was easier for me to just setup MySQL so Rails stops complaining.
- I also compiled the ErlyWeb app with HiPE.
Tsung (http://tsung.erlang-projects.org/) is the tool I used for the stress testing. It's pretty simple to use -- you specify in an XML file where the server, what page(s) to request and the frequency of initiating new requests and then it runs the test and produces a report. You can read about it more on its website.
One interesting fact about Tsung is that it is written in Erlang to take advantage of Erlang's support for massive concurrency (there will be more on that later :) ).
I installed Tsung on the same machine image as the web servers because although during the test one instance runs the servers and the other runs Tsung, I wanted to have two identical EC2 machine images to ease setting things up.
During the test, I had both servers running on the server instance listening on different ports (Mongrel on port 3000, Yaws on port 3001). I then ran Tsung twice: first against Mongrel and then against Yaws. The tests did not overlap.
For the test, I configured Tsung to go through a sequence of 11 10-second phases during which Tsung would send requests to the server at increasing frequencies. In the first phase, Tsung would send 2 requests per second. It would then progress to 10, 20, 40, 100, 200, 400, 1000, 2000, 4000, and finally 10000 requests per second.
Before I show the results, I would like to say that you should take them with a grain of salt. Many benchmarks have flaws. Maybe I messed something up during the installation. Maybe my methodology was wrong. Maybe I didn't perform some Mongrel optimization trick that boosts its performance. Maybe your application would behave quite differently from my test app. You have been warned.
UPDATE: the initial results were inaccurate because I didn't run Mongrel in production mode. I updated the results and the graphs to reflect Rails's performance in production mode.
Below is the summary of the most important measurements.
ErlyWeb: 699.8 requests/sec
Peak Send Rate
ErlyWeb: 5997.50 Kb/sec (hmm... did we hit EC2's limit?)
Total Size Sent
ErlyWeb: 28.47 MB
Peak Receive Rate
ErlyWeb: 941.84 Kb/sec
Total Size Received
ErlyWeb: 4.43 MB
Performance Degradation Point
(The phase at which the framework started sharply dropping the response rate)
ErlyWeb: 4000 requests/sec phase.
Note: these are the updated reports after running Mongrel in production mode
You can view the full Tsung reports here: ErlyWeb Rails
These reports are from the second test (30 second sustaining at 10 requests/sec):
You can also download the reports, zipped: reports.tar.gz
ErlyWeb's peak response rate was
I'm actually surprised by the results. I expected ErlyWeb to win, but I didn't expect ErlyWeb to outperform Rails by such a wide margin.
The results probably reflect differences in language and runtime more so than implementation details in the frameworks. After 20 years of refinement driven by needs of high performance telcom applications, Erlang delivers.
Looking at the peak send rate for ErlyWeb -- an almost round 6 MB/s, I wonder if ErlyWeb just saturated EC2's internal bandwidth, and whether ErlyWeb could perform even better on a faster link.
I ran another test, this time hitting each server at 10 requests/sec sustaining for 30 seconds to measure average response time. Rails's response time was in the