Wednesday, August 30, 2006

Many-to-Many Relations Are Now at an ErlyDB Near You

The title says it all :)



Not only that, I also added more powerful SELECT query capabilities to many-to-one relations.



Here's how it works:



Add these tables to erlydb.sql



create table developer(
id integer auto_increment primary key,
name varchar(30),
country varchar(20));

create table developer_project(
developer_id integer,
project_id integer,
primary key(developer_id, project_id));


This is our modified project.erl file:



-module(project).
-compile(export_all).

relations() ->
[{many_to_one, [language]},
{many_to_many, [developer]}].


This is our new developer.erl files:



-module(developer).
-export([relations/0]).

relations() ->
[{many_to_many, [project]}].


Following is some code demonstrating the new capabilities. The full module is in the distribution, in test/erlydb_test.erl.



%% fancier project queries
{ok, [Yaws4]} = language:projects_where(
Erlang, "name='Yaws'"),
true = Yaws4 == Yaws1,

{ok, [Yaws5]} = language:projects_with(
Erlang, "limit 1"),
true = Yaws4 == Yaws5,

{ok, [Ejabberd4]} = language:projects(
Erlang, "name like '%e%'", "limit 1"),
true = Ejabberd4 == Ejabberd3,


%% Let's show off some many-to-many features

%% First, add some more projects
[OTP, Mnesia] =
lists:map(
fun(Proj) ->
{ok, P1} = project:save(Proj),
P1
end,
[project:new("OTP", "The Open Telephony Platform", Erlang),
project:new("Mnesia", "A distributed database " ++
"engine written in Erlang", Erlang)]),

%% Next, add some developers
[Joe, Ulf, Klacke] =
lists:map(
fun(Developer) ->
{ok, D1} =
developer:save(Developer),
D1
end,
[developer:new("Joe Armstrong", "Sweden"),
developer:new("Ulf Wiger", "Sweden"),
developer:new("Claes (Klacke) Wikstrom", "Sweden")]),

%% Add some developers to our projects
ok = project:add_developer(OTP, Joe),
ok = project:add_developer(OTP, Klacke),
ok = project:add_developer(OTP, Ulf),

%% Add some projects to our developers
ok = developer:add_project(Klacke, Yaws1),
ok = developer:add_project(Klacke, Mnesia),
ok = developer:add_project(Ulf, Mnesia),


%% basic SELECT query
{ok, [Joe, Ulf, Klacke]} =
project:developers(OTP),

%% fancier SELECT queries
{ok, [Ulf, Klacke]} =
project:developers(Mnesia),
{ok, [Ulf]} = project:developers_with(Mnesia,
"limit 1"),
{ok, [Klacke]} =
project:developers_where(Mnesia,
"name like 'Claes%'"),

%% SELECT query, from the other direction
{ok, [Yaws1, OTP, Mnesia]} =
developer:projects(Klacke),

%% Klacke, nothing personal here :)
ok = developer:remove_project(Klacke, Yaws1),
{ok, [OTP, Mnesia]} = developer:projects(Klacke),
ok = developer:remove_project(Klacke, OTP),
{ok, [Mnesia]} = developer:projects(Klacke),
ok = developer:remove_project(Klacke, Mnesia),
{ok, []} = developer:projects(Klacke),

%% I forgot to show you can delete records, too :)
ok = language:delete(Java).


ErlyDB is still in alpha version, so you may have problems. Please let me know if you find any bugs :)



(FYI, I'm moving to Boston tomorrow, so I will probably not be of much help in the next few days.)



Enjoy!

Tuesday, August 29, 2006

Introducing ErlyDB: The Erlang Twist on Database Abstraction

Every web developer craves a database abstraction layer. Ruby, PHP, Java and Python all have them. Many developers have adopted web frameworks just for their database abstraction capabilities. The lack of such a framework has even caused some developers to hesitate before using the best language of them all for building world-class backends: Erlang.



That's why I created ErlyDB: The Erlang Twist on Database Abstraction :)



ErlyDB isn't a database abstraction layer, but a database abstraction layer generator. ErlyDB taps into Erlang's runtime metaprogramming powers to generate an abstraction layer for your database on the fly. This layer is as flexible as it is paper-thin.



Unlike ORM frameworks, ErlyDB works directly with Erlang tuples, which are simple immutable arrays that map very easily to database records. This incurs less overhead than object-relational mapping solutions, which rely on more heavyheight objects to represent database rows.



With ErlyDB, productivity doesn't come at the expense of performance :)



ErlyDB is in an alpha stage right now. It only supports MySQL; it hasn't been thoroughly tested and optimized; and it's not feature-complete by any means. However, I'll be adding many more features and improvements over time. If you strongly need a feature before I implement it, I encourage you to implement it yourself and share it with the rest of us. I will be accepting any useful, high quality contributions from other developers.



Let's do a short tutorial.




  • Get ErlyDB from the subversion repository and install it in your code path (on my Mac, it's /usr/local/lib/erlang/lib).
  • Get the MySQL driver from process-one.net and install it in your code path.
  • Create a MySQL database called 'test'

  • Run the following SQL script:



create table language (
id integer auto_increment primary key,
name varchar(50),
paradigm varchar(30),
creation_year integer);

create table project (
id integer auto_increment primary key,
name varchar(50),
description varchar(50),
language_id integer,
index(language_id));


  • Create the file "language.erl" with the follwing code and compile it in your code path:




-module(language).
-export([relations/0]).

relations() ->
[{one_to_many, [project]}].


  • Create the file "project.erl" with the following code and compile it in your code path:





-module(project).
-export([relations/0]).

relations() ->
[{many_to_one, [language]}].


  • Implement the following module, which illustrates some of the common idioms of ErlyDB.






-module(erlydb_test).
-export(
[init/0,
test/0]).

init() ->
%% connect to the database
erlydb:connect(mysql, "localhost", "username", "password",
"test"),

%% generate the abstraction layer modules
erlydb:code_gen([language, project]).

test() ->
init(),


%% clean up old records
erlydb:q("delete from language;"),
erlydb:q("delete from project;"),

%% Create some new records
Languages =
[language:new("Erlang",
"A fun and productive functional language with " ++
"top-notch concurrency features designed for "++
"building scalable, fault-tolerant systems. " ++
"Erlang is an excellent choice for building "++
"anything from websites like hamsterster.com " ++
"to commercial phone switches.",
"functional/dynamic/concurrent", 1981),
language:new("Java",
"An OO language designed to make the lives of " ++
"C++ programmers less painful.",
"OO/static", 1992),
language:new("Ruby",
"An OO language designed to make the lives of " ++
"Java and Perl programmers less painful.",
"OO/script/dynamic", 1995)],

%% Save the records in the database and collect the updated
%% tuples.
[Erlang, Java, Ruby] =
lists:map(
fun(Language) ->
%% executes an INSERT statement
{ok, Lang} = language:save(Language),
Lang
end, Languages),

%% demonstrate getters
"Erlang" = language:name(Erlang),
"functional/dynamic/concurrent" =
language:paradigm(Erlang),
1981 = language:creation_year(Erlang),

%% demonstrate setter
J1 = language:creation_year(Java, 1993),

%% executes an UPDATE statement
{ok, J2} = language:save(J1),

1993 = language:creation_year(J2),


%% Lets run some queries
{ok, E1} = language:find_id(language:id(Erlang)),
true = E1 == Erlang,

{ok, [E2]} = language:find_where("name = 'Erlang'"),
true = E2 == Erlang,

{ok, [E3]} = language:find(
"WHERE paradigm = 'functional/dynamic/concurrent' " ++
"LIMIT 1"),
true = E3 == Erlang,
{ok, [E4, J4, R4]} = language:find_all(),
true =
E4 == Erlang andalso
J4 == J1 andalso
R4 == Ruby,

%% Let's make some projects

Yaws = project:new(
"Yaws", "A beautiful web server written in Erlang",
Erlang),
Ejabberd = project:new("ejabberd",
"The best Jabber server, hands down", Ruby),
OpenPoker =
project:new("OpenPoker",
"A scalable, fault-tolerant poker server", Erlang),


%% We call language:id just to demonstrate that constructors
%% accept both related tuples or, alternatively, their id's.
%% This example would behave identically if we used the
%% Java variable directly.
Tomact =
project:new("Tomcat",
"A Java Server with XML config files",
language:id(Java)),

JBoss =
project:new("JBoss",
"A Java Application Server with more XML files",
Java),
Spring =
project:new("Spring Framework",
"A Java IoC framework oozing XML love", Java),


Mongrel =
project:new("Mongerl",
"A web server with a funny name", Ruby),
Rails =
project:new("Ruby on Rails",
"A nice integrated web framework for building " ++
"CRUD websites.", Ruby),
Ferret = project:new("Ferret",
"A Ruby port of Apache Lucene. It would be nice " ++
"if someone ported it to Erlang", Ruby),
Gruff = project:new("Gruff",
"A Ruby library for easy graph generation. " ++
"An Erlang implementation would be nice as well",
Ruby),

Projects = [Yaws, Ejabberd, OpenPoker, Tomact, JBoss,
Spring, Mongrel, Rails, Ferret, Gruff],

%% Insert our projects into the database
[Yaws1, Ejabberd1, OpenPoker1 | _Rest] =
lists:map(
fun(Project) ->
{ok, P1} = project:save(Project),
P1
end, Projects),

%% let's get the language associated with Yaws
{ok, Erlang2} = project:language(Yaws1),
true = (Erlang2 == Erlang),

%% now let's correct a grave error
{ok, Ejabberd2} = project:save(
project:language(Ejabberd1, Erlang)
),
true = language:id(Erlang) ==
project:language_id(Ejabberd2),


%% let's get all the projects for a language
{ok, [Yaws3, Ejabberd3, OpenPoker3]} =
language:projects(Erlang),
true =
Yaws3 == Yaws1
andalso Ejabberd3 == Ejabberd2
andalso OpenPoker3 == OpenPoker1,

ok.

(This code is included in the 'test' directory of the ErlyDB distribution.)




  • Read the code and experiment.




As you've probably guessed, ErlyDB uses Smerl to do its magic. (Confession: when I created Smerl, ErlyDB was just what I had in mind :) ) ErlyDB takes advantage of all of Smerl's advanced features, such as metacurrying, parameter embedding and module extension. You may find it interesting to look at the code and see how ErlyDB works under the hood.



Adhering to the Erlang philosophy of zero downtime, the ErlyDB abstraction layer can change in runtime without taking the system offline. Simply call erlydb:code_gen with the module names to regenerate and everything will work as expected.



Please get your hands on ErlyDB and give it a test drive. Let me know if you find any bugs or if you have any suggestions.



Coming very soon:



  • Many-to-many relations

  • Transactions

  • Event handlers (before_save, after_save, etc.)



Coming soon:



  • Prepared statements

  • Support for additional database engines (probably when edbc comes out, unless other people make temporary drivers beforehand.)

  • Better connection pooling (also probably tied to edbc)

  • Real documentation :)

  • Performance optimizations

  • More customizations

  • Fine-grained control on field visibility

  • Multiple models per table

  • More versatile SQL query generation

  • Support for complex queries

  • and more



Enjoy!





Appendix



Erlang users have actually had a nice alternative to database abstraction: Mnesia. Mnesia is an industrial-strength, distributed database engine written in pure Erlang. Although Mnesia is quite beautiful, it isn't the best fit for all applications. Mnesia's biggest drawback (at least as I see it) is that Mnesia isn't designed to scale to very large (many gigs) data volumes. I'm hoping the OTP team will improve this aspect of Mnesia, but until it does, we have to live with this limitation.



Even if Mnesia didn't have this shortcoming, many developers would still prefer to use other database engines for their various capabilities. MySQL, Postgres and Oracle aren't going away any time soon.



Having said that, Mnesia can very useful even if you're using another database engine. For instance, a killer application for Mnesia is a distributed session store shared between instances of a Yaws cluster. Mnesia could also be used as a distributed cache for recently-accessed database records.



Update: I just realized that there's no reason ErlyDB shouldn't have a Mnesia driver. Any takers? :)

Smerl Updates

2 Quick updates:




  • I made source discovery in Smerl more robust by using the module_info function.

  • I changed the Smerl license from the BSD to the MIT license.

Sunday, August 27, 2006

Museum of Natural History

Dear readers,



I'd like to take a short break from my Erlang postings and share a few pictures I took on my trip earlier today to the Museum of Natural History in New York.



Note: I did some retouching with GIMP (actually, GimpShop for OS X) to enhance the dynamic range of these images, as the originals suffered from faded colors due to the poor lighting conditions.



A Velociraptor


IMG_1433.JPG



A green lizard


IMG_1507.JPG



This dino must have been coding in Java. Its neck looks like this stack trace :)


IMG_1447.JPG



A chameleon


IMG_1515.JPG



You can view the rest of the set here

To Web Developers: Think Outside The Thread

A few people have asked me this very good question (I'm paraphrasing): "If I'm just trying to build a webapp, and I don't need telcom-grade scalability, why should I use Erlang? Yes, Erlang may be the best language for concurrent programming, but I will never have to use Erlang's concurrency features directly, just as I don't use any threading libraries when I'm building a webapp with other languages."



Here's my answer: you can certainly build a webapp without using concurrency features directly. Most web developers don't. Just keep in mind that most framework designers use languages that support concurrency so poorly, that they have resorted to fooling you into thinking that the only webapps you can realistically build must follow the good ol' request-response, single-threaded, CRUD-based paradigm.



This is clearly a fallacy.



Many high-quality, innovative webapps don't follow this paradigm.



The Google search engine doesn't follow this paradigm. Neither does Meebo.



Many of the great webapps of tomorrow won't follow this paradigm.



Why should your webapp follow this paradigm?



The paradigm is shifting to a large degree because of one main innovation, which somebody has named "Comet" (it's a silly name if you ask me, but it has stuck). Comet boils down to having the browser keep an open XMLHttpRequest connection to the server, waiting for the server to send it messages. This opens the door to full bidirectional communication between the browser and the server (this is essentially what Meebo does).



If you want to build a Comet app with today's Java, Ruby or Python frameworks -- good luck. Not only will the message-passing -- not to mention clustering -- logic be painful to code (especially if you've tasted the power of Erlang :) ), your servers will also croak when your app reaches a few thousand (or fewer) simultaneous users.



Partly for these reasons, nobody uses Ruby or Python to build commercial phone switches (not that I know of, anyway :) ). However, Ericsson builds them quite successfully with Erlang. Erlang powers highly scalable, fault-tolerant phone switches with nine nines availability. Building scalable clusters is what Erlang is made for, and I dare say Erlang makes such programming tasks (relatively) easy.



If you want to build an app that uses Comet, you will need concurrency and scalability, so use Erlang.



However, if your app doesn't use Comet, should you still care about Erlang's unmatched capabilities for concurrency, clustering and message passing?



Yes.



Another good reason to use Erlang is that your web application may benefit from -- or even require, as in Google's case -- distributing a calculation over multiple processors or even multiple nodes in a cluster prior to sending a response back to the browser.



Erlang is designed for distributed programming. In Erlang, sending a message to a remote node is as easy as sending it to another process in the same box. As Joe Armstrong has shown, writing a parallel version of lists:map is almost laughably easy in Erlang. This is all it takes:




pmap(F, L) ->
S = self(),
Pids = map(fun(I) ->
spawn(fun() -> do_f(S, F, I) end)
end, L),
gather(Pids).

gather([H|T]) ->
receive
{H, Ret} -> [Ret|gather(T)]
end;
gather([]) ->
[].

do_f(Parent, F, I) ->
Parent ! {self(), (catch F(I))}.


Distributed computation will grow in importance as server manufacturers race to add more cores to their boxes. (The next release of Sun Niagara will have 64 cores, for instance.) Even if you don't need distributed computation right now, you'll benefit from adding Erlang to your toolbox. Multi-core computation is where things are going, and knowing Erlang will help you stay on top of the game.



By the way, I'm not suggesting that Erlang has a monopoly on distributed programming -- Erlang is just years ahead of any other language in this area. Erlang has 20 years of development behind it for this specific purpose, with a long track record of success in large-scale production systems. Sure, you always have the option of using, say, Starfish for Ruby. You'd just have to cross your fingers and hope your competition doesn't use Erlang :)



To summarize, if you are 100% confident that you will never attempt to push the envelope of what your webapp's backend can do, you probably won't need Erlang's concurrency features. IMO, you'll have fun by using Erlang, but you certainly don't need it. However, if you start venturing into areas that are beyond the single-threaded CRUD view of the world, Erlang's capabilities will help you greatly.



So why use Erlang?



Think outside the thread.

Friday, August 25, 2006

Who's Going To The Erlang Workshop?

Hi,



I'm curious as to who's planning on going to the Fifth ACM SIGPLAN Erlang Workshop in Portlang on the 16th (the link is on the front page of http://erlang.org).



I haven't made up my mind yet if I should go. It would be fun to meet some Erlangers face to face but I have no idea how popular this workshop is.



I'd appreciate it if somebody could clue me in.



Thanks in advance!

Thursday, August 24, 2006

New Traffic Records

The picture says it all.






This is weird.



When I started blogging about Erlang, I enjoyed being in a small niche of bloggers that write about "exotic" languages. I've never thought my blog would get all this attention. Sometimes, I don't like all this attention :) However, you don't always get in life what you've expected, so I'll do my best to enjoy the ride.



Erlang is just getting started. Erlang will make huge strides into the web development world. We will build frameworks that make Erlang developer's lives easy, if not easier, than those of Ruby on Rails developers. Ruby on Rails didn't exist two years ago, and Erlang (with Yaws) is years ahead of where Ruby was when Rails was born. In a matter of months, Erlang will be a serious contender in the web development world -- not just for developers who want insane scalability with 3 milliseconds downtime per year -- but also for developers who just want to get things done.



Real-time interactivity on the web is a relatively unexploited land full of opportunities for adventerous developers. These days, building real-time interactive applications with Comet technology in any language but Erlang just makes no sense. Erlang has 20 years of development behind it with the specific design goal of building scalable servers. No other language comes close. It also makes development fun.



As Joe Armstrong has said, Erlangers have a head start.



Libraries are coming.

Ericsson's Biggest PR Blunder: Forgetting to Tell Us That Erlang Programming Is FUN

As you may have noticed, Ericsson hasn't done the best PR for Erlang. The closest things resembling sales pitches I see from that part of the world are either academic-looking papers or PowerPoint presentations that require downloading heavy PDF files. There are also the mailing list postings that are often intelligent and hilarious but they are ultimately preaching to the choir.



When people talk about Erlang, they usually mention how Erlang is great for building concurrent, distributed, highly-available, fault-tolerant, systems; it powers phone switches with nine nines availability; the AXD 301 switch has 850K lines of Erlang code; Yaws scales with large numbers of connections; ejabberd is the most scalable XMPP server; and all sorts of such heavy-sounding accomplishments.



If I hadn't read Joel Reymont's wagerlabs.com blog, where he wrote about his experiences writing a poker server in Erlang, I probably would have never thought I would actually like Erlang as a language.



What nobody from Ericsson talks about is that Erlang makes coding FUN.



It may be possible to arrive at this realization through deduction: to build large-scale, stable systems you need a productive language. If the language got in your way and prevented you from doing things in a straightforward manner, your code would start growing in complexity. Complexity leads to bugs. Bugs lead to downtime. Systems written in Erlang must have close to zero downtime. Therefore, Erlang must make developer's lives easy.



For developers, ease often equates with fun. Ease means that they can express their thoughts succinctly, test their solutions quickly, and then move on to the next problem.



Programmers love garbage collection, for instance, because it makes their lives easy. Similarly, some programmers are Ruby fanatics because they think Ruby is the epitome of ease (I used to think so too in the old days -- about 3 months ago :) ). Other programmers believe ease comes from using functional languages, because they are the most powerful.



You can read Paul Graham's essays on Lisp about this. You can also read Peter Norvig's comparison between Python and Lisp. (As you're reading this comparison, mentally add Erlang as well as 2 more categories: concurrency and fault tolerance, and picture Erlang as being the top dog :) )



So what exactly makes Erlang coding fun? To me, these are the main factors:




  • Erlang is a functional language. It fits well in my brain.

  • Erlang is dynamically typed, which means that I can write less code (read Haskell vs. Erlang, Reloaded) and also test my code easily in the Erlang shell. (Erlang's dynamic nature also allowed me to create Smerl :) )

  • Erlang has pattern matching, which lets me write a single line of code to express a thought that would in other languages translate into a horrendous structure of nested if-else statements.

  • Erlang has single-assignment semantics, which means that functions have no side effects (on bound variables). This makes Erlang code very readable. It also makes debugging easy.

  • Concurrency, concurrency, concurrency.

  • With Erlang's hot code swapping, I can deploy my changes unto a system while it's running. When I made the haXe remoting adapter for Yaws, I didn't have to take Yaws offline (try that with Apache or Lighttpd :) ).

  • The compiler checks the validity of my code without locking me into a type system.

  • Good runtime metaprogramming capabilities (with Smerl).

  • I used to like Eclipse, but Erlang made me an Emacs fiend :)

  • I'm learning a lot and interacting with very smart people.



Some people have asked me, "If all I want is to build a stateless, CRUD-style webapp, and I don't need telcom-grade scalability, etc, why should I bother learning Erlang? Ruby on Rails handles it just fine. It also has more web development libraries."



I hope I was able to give a reasonable answer to this question. However, as much as I write about Erlang, reading this blog won't take you all the way there. You have to experiment with Erlang and see how it feels for yourself. I can't guarantee you'll like it -- all I can do is tell you why I do.

I Can Feel It in the Ether: A Big Erlang Renaissance Is Coming

A few weeks ago, I submitted to digg.com an article I had written comparing web development with Erlang + Yaws to Ruby on Rails. Why? Because it seemed like Ruby on Rails was getting all the buzz, whereas the developer community was overlooking an arguably much more powerful alternative: Erlang + Yaws.



By "more powerful," I don't mean to suggest that Yaws has more web development libraries or that it's as accessible to most web developers as Ruby on Rails.



It isn't.



(Yaws does have many useful APIs -- just not as many as Rails).



I mean "powerful" in the sense that Ericsson has been developing Erlang for 20 years with the goal of creating the ultimate language for powering its distributed, fault-tolerant, massively scalable telephone switches -- the stuff that operates with %99.9999999 (nine nines) availability.



Erlang has delivered.



Yaws is a web server written in pure Erlang, which means that it automatically inhertis the traits that have made Erlang-powered telcom switches so robust. When you're building a webapp with Yaws, you have the full power of Erlang at your fingertips.



The cost? $0.



The source code? Open.



All this stuff struck me as a very appealing package, especially in contrast to the popular array of web development tools (last I checked, nobody was programming commercial phone switches in Python or Ruby :) ) but despite my semi-serious attempt to promote it, my article got only 18 diggs.



Possibly overreacting, I thought that my Digg experience has proven that the larger developer community will probably never be interested in Erlang.



"It's a functional language in a Java dominated world," was my favorite rationalization.



Apparently, I made a mistake by thinking of digg as "the" destination for serious developers. A couple of weeks ago, somebody submitted my Erlang + Yaws vs. Ruby on Rails article was to reddit.com, where it made the front page. Then another article I wrote, Why Erlang Is a Great Language for Concurrent Programming was featured on reddit.com as well.



"Wow," I thought, "it looks like reddit.com is where less mainstream but more interesting programming blogs get real attention."



After these events, I started following reddit.com -- especially programming.reddit.com. I was mostly curious as to what kind of programming languages are featured there.



Today, I had a very pleasant -- almost shocking -- surprise: programming.reddit.com has 6 Erlang-related links, the top of which is also on the front page of reddit.com.



Here they are:



  • 1) New Blog: Joe Armstrong (one of the fathers of Erlang) (armstrongonsoftware.blogspot.com).

  • 3) Secret Weapons for Startups (yarivsblog.com)

  • 5) The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software (gotw.ca)

  • 9) Haskell vs. Erlang (wagerlabs.com)

  • 19) HiPE (High Performance Erlang compiler) presentation (erlang.se).

  • 22) The DIALYZER: a DIscrepancy AnaLYZer for ERlang programs (eg, an Erlang debugger) (ii.uu.se)



This is nothing short of amazing. I've never seen so many links specific to one language on programming.reddit.com -- and Erlang is far from one would call "popular."



Maybe I'm just seeing trends where I'm subconsciously trying to look for them, but maybe this really does indicate real shifts are happening and that developers are starting to recognize that for building scalable, high-available, fault tolerant systems, today's array of popular programming languages are simply nowhere close to where Erlang was 20 years ago.



Interesting times are ahead for Erlang.




Side note: DIALYZER isn't a debugger, but a static analysis tool for finding software defects.

Wednesday, August 23, 2006

New Erlang Blog by Joe Armstrong

There's big news in the Erlang blogsphere.



Today, Joe Armstrong, one of Erlang's original creators ("The" creator of Erlang?), started a new blog in which he'll write about the events and people that have shaped Erlang's history, as well as the stories that will shape its future.



This is very exciting. Until today, the most comprehesive source for Erlang history I've read has been Joe's PhD thesis. However, as you might expect, although this paper is very rich in technical material, it doesn't dwell too much on the human side. Now, we'll be able to know more about the Erlang story from the perspective of a true insider.



Judging by the substance and humor of Joe's mailing list postings, which I sometimes quote on this blog, I'd bet that Joe's blog will be a great read.



I hope more people follow in Joe's footsteps and start blogging about Erlang. Many people on the Erlang mailing list are far more knowledgable about Erlang -- and other programming languages (well, maybe except for Ruby and haXe :) ) -- than I am. Many of them are also great writers. If they started blogging, it would make the Erlang blogsphere much richer.



I get much of my knowledge about Erlang from reading mailing list postings, but this kind of knowledge isn't very accessible to non-Erlangers. Blogging is a better tool for spreading the word to the outside world. Blogs also can be less technical and more open-ended than mailing list postings, in which one follows a stricter etiquette in trying to stick to the point and not waste too much of people's time. A blog, on the other hand, is what you make of it. If a blog is good, people discover it and its importance grows. If a blog sucks, at least people have the option of ignoring it :)



The best part of Joe's announcement, at least for me, is that he credited me with inspiring him to start blogging. Reading these words made me feel very honored.




Yariv's blog http://yarivsblog.com/ with his frequent Erlang postings
has inspired me.



I think he needs a little competition :-)



So I've started writing ...



In this blog I'll try to tell you some of the things that are happening in the Erlang world. A lot of stuff is happening behind the scenes, I'll try to tell you about some of this.



The first post is:



http://armstrongonsoftware.blogspot.com/2006/08/making-money-from-erlang.html



Cheers



/Joe




When I stared this blog, I really didn't have very high expectations, so it's thrilling to see this blog make a real impact on the Erlang world. (I hope that the world outside has taken notice as well :) )



Getting credit is great, but giving it is even better: it was Joel Reymont's wagerlabs.com blog which has turned a mild curiousity about Erlang into a strong interest for me and also inspired me to start blogging about Erlang.




Oh, I almost forgot to mention -- Joe will also blog about cats. I must not forget the cats! :)



Happy RSSing! :)

A Couple of Smerl Updates

I added a few new features to Smerl earlier today.




  • smerl:extend/2 now injects the parent/0 function, which returns the parent module's name, into the child module. This allows explicitly calling parent functions from the child module. (Using smerl:for_module/1, the child can also mutate its own parent in runtime, which puts an interesting twist on family relationships :-) )
  • smerl:to_src/1 returns the pretty-printed source code for the MetaMod object.

  • smerl:to_src/2 writes the pretty-printed source code for the MetaMod object into a file.



Enjoy!

Tuesday, August 22, 2006

New: Module Extension in Erlang with Smerl

Have you ever wished you could easily add all exports from a "parent" module to a "child" module in the form of remote function calls to the parent, without overwriting the child module's exported functions?



If you have, today is your lucky day! With Smerl, you can now do it in a single line of code by calling smerl:extend/2 :)



I know words like 'extension' and 'inheritance' are generatlly shunned in the functional programming world, and for a good reason: in OO languages, these concepts have caused endless complexity, crufty code and bug-ridden horrors.



However, I should stress that the Smerl extension mechanism doesn't violate any functional programming tenets. Smerl's extend/2 is basically a smart function generator. It does not allow you to extend data fields in any way, nor does it create any artifical bindings between data and code.



(I believe Haskell has a similar, statically typed approach to module inheritance, but I only have rudimentary knowledge of Haskell so I may be wrong on that one.)



Here's a quick example:



P1 = smerl:new(parent),
{ok, P2} = smerl:add_func(P1, "add(A, B) -> A+B."),

%% 'false' indicates to not export the function
{ok, P3} = smerl:add_func(P2, "subtract(A, B) -> A-B.", false),

{ok, P4} = smerl:add_func(P3, "multiply(A,B) -> A*B."),
smerl:compile(P4),

C1 = smerl:new(child),
{ok, C2} = smerl:add_func(C1,
"multiply(A,B) -> throw(not_allowed)."),

C3 = smerl:extend(P4, C2),
smerl:compile(C3),

8 = parent:add(3,5),
{'EXIT', {undef, _}} = begin catch parent:subtract(4,3) end,
6 = parent:multiply(3, 2),
11 = child:add(2,9),
not_allowed = begin catch child:multiply(5,3) end,
{'EXIT', {undef, _}} = begin catch child:subtract(4,2) end.


Seems exotic? Maybe. However, I think this feature is very powerful. I have actually seriously needed this feature in my own project. If you have needed it too, I hope I just made your life a bit easier :)



Please let me know if you have any problems or suggestions.



Enjoy.



Update (8/23/06): The extend/2 function now also embeds the parent/0 function in the child module. parent/0 returns an atom with the name of the parent's module. This allows child functions to explicitly invoke parent functions (among other things).

New Smerl Capability: Embedding Values in Place of Arbitrary Function Parameters

I added a couple of new functions to Smerl to make Erlang metaprogramming with Smerl even more powerful. They provide a capability similar to metacurrying, but more flexible.



smerl:embed_params/2 takes a function form and a list of {Name, Value} pairs. For all clauses of the function, it iterates through their parameters, replacing each parameter that matches an item in the list with a variable in the body of the function that is bound to the item's value.



Reading the above sentence may feel like getting hit by a truck, so let's take a look at an example:




M1 = smerl:new(foo),
{ok, M2} = smerl:add_func(M1, "bar(A,B,C) ->
io:format(\"~s ~s ~s\", [A,B,C])."),
{ok, F1} = smerl:get_func(M2, bar, 3),
F2 = smerl:embed_params(F1, [{'A', "Lucy"}, {'C', "Sky"}]),
{ok, M3} = smerl:add_func(M2, F2),
smerl:compile(M3),
foo:bar("in the"). %% outputs "Lucy in the Sky"


The other function I added is smerl:embed_all/2. This function takes a MetaMod object (the thing you get back when calling smerl:new/1, smerl:for_module/1, or smerl:for_file/1), and applies smerl:embed_params/2 to all functions in the module. Sounds bizzare? It may be, but I have found it useful in a project I got cooking :)



Enjoy!

Sunday, August 20, 2006

Smerl Has A New Home

Smerl started as an rough idea, then became an experiment, and now it has grown into pretty powerful library. I think Smerl only scratches the surface of what one can do with Erlang metaprogramming, but I want to keep Smerl simple ('Smerl' is, after all, an acronym for Simple Metaprogramming for Erlang), so I'm not planning on adding many feature to Smerl at the moment.



(I will implement fixes and tweaks to make Smerl's current features easier to use but I'm not planning on adding new features unless they have a compelling value, e.g. metacurrying.)



To make it easier for others to follow Smerl's evolution (and to contribute to Smerl, if they wish ;) ), I have decided to move Smerl to code.google.com, where it will live in a subversion repository and enjoy other amenities for "real" projects, such as an issue tracking system.



Without further ado, here's Smerl's new home: http://code.google.com/p/smerl/.



I will keep blogging about Smerl. To see only Smerl-related articles, you can visit http://yarivsblog.com/tag/smerl (which also has a corresponding RSS/Atom feed).

Saturday, August 19, 2006

Erlang Metaprogramming Breakthrough: Metacurrying!

Quick download: smerl.erl

This is pretty big.



As I've been hacking away with Smerl last night, I had the following realization: in most cases, the form for a metafunction is mostly known in compile-time, and it only requires a small injection of runtime data to attain its final form.



This is conceptually very similar to currying.



Once these thoughts have fully materialized in my mind, I again sat down by my MacBook and frantically hacked into Smerl the very capability of runtime metacurrying!



What is metacurrying? In regular currying, you bind a value to a function's parameter(s), which gives you a new, simpler, function. In metacurrying, you bind a value to a function form, which gives you a new, simpler, function form!



This is a huge step forward for Smerl. Why? Because it finally frees the programmer from having to know the Erlang abstract format. With metacurrying, Erlang programmers finally get full compile-time checking for their metafunctions, which finally truly blurs the line between compile-time and runtime metaprogramming in Erlang.



I added a few new functions to Smerl:




  • smerl:curry/2 takes a function's abstract form and a parameter value (or list of paramter values) and returns the new, curried form. This is great for debugging and understanding what happens behind the scenes in metacurrying.

  • smerl:curry/4 takes a MetaCtx object or a module name, a function name, arity and parameter value (or list of parameter values) and returns the curried for for the function. This is essentially a smerl:get_func followed by a smerl:curry/2.
  • smerl:curry_replace/3 takes a MetaCtx object, a function form and a parameter value (or list of parameter values) and replaces the old function with the function's curried form

  • smerl:curry_replace/4 takes a MetaCtx object, a function name, arity and parameter value (or list of parameter values) and replaces the old function form with the function's curried form.

  • smerl:rename/2 is a convenience function that takes a function form and returns its renamed form.

  • smerl:curry/5 take a MetaCtx object or a module name, a function name, arity, value and new function name, and returns the curried form after renaming it.



You can read the source for the full documentation.



Ok, enough talk. Let's show an example. Remember my last demo of Smerl's capabilities? Here's the new version, which uses metacurrying:




-module(func_recs).
-export([generate/1]).

generate(Filename) ->
{ok, Forms} = epp:parse_file(Filename, [], []),
lists:foreach(
fun({attribute, _Line, record, {RecName, RecFields}}) ->
case smerl:for_module(RecName) of
{ok, C1} ->
process_module(C1, RecFields);
_Err ->
process_module(smerl:new(RecName), RecFields)
end;
(_Other) -> undefined
end, Forms).

process_module(MetaCtx, RecFields) ->
{_, C2} = lists:foldl(
fun({record_field, _Line,
{atom, _Line2, FieldName}},
{Idx, MetaCtx1}) ->
{Idx+1, process_field(MetaCtx1,
FieldName, Idx)};
({record_field, _Line,
{atom, _Line2, FieldName}, _Def},
{Idx, MetaCtx1}) ->
{Idx+1, process_field(MetaCtx1,
FieldName, Idx)}
end, {2, MetaCtx}, RecFields),
smerl:compile(C2).

get(Idx, Obj) ->
element(Idx, Obj).

set(Idx, Obj, Val) ->
setelement(Idx, Obj, Val).

process_field(MetaCtx, FieldName, Idx) ->
{ok, Getter} =
smerl:curry(func_recs, get, 2, Idx, FieldName),
{ok, Setter} =
smerl:curry(func_recs, set, 3, Idx, FieldName),
{ok, MetaCtx1} = smerl:add_func(MetaCtx, Getter),
{ok, MetaCtx2} = smerl:add_func(MetaCtx1, Setter),
MetaCtx2.


As you can see, there's not a single abstract format for a function in sight :)



What's happening here? The get/2 and set/3 functions are mere skeletons. They are not used directly anywhere in the code. (In fact, when you compile this module, the compiler complains that these functions are unused. Little does it know... :) ) When we process the file person.erl, -- in runtime -- we finally have data we need to embed get/2 and set/3 with the index parameters as well as give them their final names. We achieve this in 1 line of code by calling smerl:curry/5. Not bad, huh?



With metacurrying, I feel that Erlang metaprogramming is finally where it should be. I had thought salvation has arrived in the form (no pun intended : ) ) of fun expressions, but I was wrong (fun expressions are very convenient, but their abstract form is not available in runtime unless you're in the Erlang shell). Metacurrying has finally given us freedom from knowing the Erlang abstract format.



Now Erlang code is just the way I like it: like clay :)



As always, please get the code, give it a test drive, and tell me if you find any problems.



Enjoy!

Thursday, August 17, 2006

Smerl Demo: Easy Function-Based Record Access in Erlang

A few people have told me that they thought Smerl was neat, but they didn't really know what it could be used for. So, I thought I'd share a short demo app that goes a little further than the code snippets in my last few postings about Smerl.



I've heard some people complain about Erlang' record access syntax. I agree with them that the notation Var#RecType.fieldname is not very pleasant to look at sometimes. With Smerl, we can make things better.



Here's a short module I wrote, called func_recs, which generates getters and setters for all records in a given source file:




-module(func_recs).
-export([generate/1]).

generate(Filename) ->
{ok, Forms} = epp:parse_file(Filename, [], []),
lists:foreach(
fun({attribute, _Line, record, {RecName, RecFields}}) ->
case smerl:for_module(RecName) of
{ok, C1} ->
process_module(C1, RecFields);
_Err ->
process_module(smerl:new(RecName), RecFields)
end;
(_Other) -> undefined
end, Forms).

process_module(MetaCtx, RecFields) ->
{_, C2} = lists:foldl(
fun({record_field, _Line,
{atom, _Line2, FieldName}},
{Idx, MetaCtx1}) ->
{Idx+1, process_field(MetaCtx1,
FieldName, Idx)};
({record_field, _Line,
{atom, _Line2, FieldName}, _Def},
{Idx, MetaCtx1}) ->
{Idx+1, process_field(MetaCtx1,
FieldName, Idx)}
end, {2, MetaCtx}, RecFields),
smerl:compile(C2).

process_field(MetaCtx, FieldName, Idx) ->
L = 999,
Getter = {function,L,FieldName,1,
[{clause,L,
[{var,L,'Obj'}],
[],
[{call,L,
{atom,L,element},
[{integer,L,Idx},{var,L,'Obj'}]}]}]},

{ok, MetaCtx1} = smerl:add_func(MetaCtx, Getter),
Setter =
{function,L,
FieldName,
2,
[{clause,1,
[{var,L,'Obj'},{var,L,'Val'}],
[],
[{call,L,
{atom,L,setelement},
[{integer,L,Idx},{var,L,'Obj'},
{var,L,'Val'}]}]}]},
{ok, MetaCtx2} = smerl:add_func(MetaCtx1, Setter),
MetaCtx2.


If the code path contains a module that has the same name as a record name in the source file, Smerl adds a getter and setter to the module for each field in the record. Otherwise, Smerl creates a new module purely in runtime.



Using func_recs is easy. Let's say we have the following source file called person.erl:




-module(person).
-export([new/0]).

-record(person, {name, city}).

new() ->
#person{}.


We can now write the following code:




func_recs:generate("person.erl"),
P = person:new(),
P1 = person:name(P, "Yannick"),
P2 = person:city(P1, "Barcelona"),
io:format("~w is from ~w", [person:name(P2), person:city(P2)])


This example doesn't exactly show you how to build a Google Killer in Erlang :) -- but I hope it illustrates some of the fun things you can do with Erlang metaprogramming using Smerl.



New tip (8/18/06): Are you looking an easy way of knowing what the abstract form for a function is? Fire up the erlang shell and type




io:parse_erl_form(". ").


This will give you a one-line mini shell, in which you can type your function and get back its abstract form representation.

Smerl Attains Erlang Metaprogramming Nirvana (or maybe not)

Update (8/17): Today may not have been my lucky day after all. Apparently, erlang:fun_info doesn't behave identically in the shell and in a compiled module. I'm trying to figure out a way around this but this may be a big roadblock. We may even have to forget about those nice fun expressions and go back to abstract forms :(



...



What does erlang:fun_info do? erlang:fun_info returns a list of tuples that contain metadata about a fun expression -- including the fun expression's abstract form representation.



What did I use it for? To enhance Smerl so that Smerl users never have to to write source fragments or know a single thing about Erlang abstract forms!



Here's the new approach to Erlang (runtime) metaprogramming:




A = 17,
C1 = smerl:new(foo),
{ok, C2} = smerl:add_func(C1, bar,
fun(B, C) -> D = 5, A + B + C + D end),
ok = smerl:compile(C2),
foo:bar(3, 7). %% returns 17 + 3 + 7 + 5 = 32


The astute reader would notice that Smerl supports closures. How? Smerl gets information about closure variables from erlang:fun_info, and expands them in the beginning of the new function. Cool, huh? :)



Another great benefit to this approach is that the compiler now verifies the syntactic correctness of your runtime-compiled code. Indeed, this capability blurs the line between compile-time and run-time Erlang metaprogramming.



I can't wait to see what Erlang developers are going to do with this. I have some ideas in mind but I can't say I know all the possibilities Smerl opens. If you have some thoughts, please share :) Also, please try this out and let me know if you have any problems.



Note: smerl:replace_func now supports fun expressions as well.

Options for Erlang Webapp Hosting

If you're writing a webapp in PHP or Ruby, your have plenty of hosting options. By contrast, not a single company offers Erlang hosting AFAIK . This is a legitimate complaint I've heard from people who are new to Erlang and who want to use it for building a webapp.



Indeed, the web development community has largely overlooked Erlang -- which I'm trying to fix, btw :) -- but this doesn't mean that you're out of hosting options.



Here are the two good options for an Erlang/Yaws app I know of that you may find appealing:




  • The cheap option, which is great especially for the development phase, is to run your app on a VPS (Virtual Private Server). This is actually a nice option for any kind of web development. I'm running this blog on a $20/month VPS server that I'm renting from JaguarPC. I think this is a pretty good package, at least from some research I've done on webhostingtalk.com a while ago, but you might find something even better.

  • When your app is ready for production, you can run it on a rented dedicated server. Such servers are pretty cheap these days ($200-$400 a month IIRC). One interesting hosting provider you should know about is SoftLayer, which lets you set up a 1Gbps private network between multiple dedicated servers. This is ideal for running an Erlang cluster without all the headaches of setting up your own hardware. Softlayer has also been getting good buzz, at least when I've done my research a couple of months ago.



Having your own (virtual) server requires more work than a shared hosting plan, but I think it's great because it gives you full control. For people who are interested in Erlang, I think this is a good fit.



I hope you find this useful! Let me know if you know of any other alternatives.

#1 on programming.reddit.com

Wow, now this is very cool: my article from yesterday, Why Erlang Is A Great Language for Concurrenct Programming, is now #1 on programming.reddit.com. Woot! :)







When I started blogging about Erlang, I never thought it would generate much interest among the larger developer community. I seriously thought, "Erlang so is great, but (relatively) few people use it. I guess people are just not interested in Erlang's benefits, or maybe they are just too attached to OO or procedural languages and they will never be interested in a functional language like Erlang, no matter how great it is."



It looks like I was wrong. I've been getting some great feedback about my Erlang postings, and much of it came from non-Erlangers who've become interested in Erlang just from reading about it in my blog. And now, since my article has reached the #1 spot on programming.reddit.com, I know that Erlang strikes a chord among many developers who are looking for a better -- and more fun :) -- way of creating software.



Side note: it was a strange yet pleasant surprise to see that my article is 5 spots ahead of an article on Paul Graham's website, History of T (Paul Graham didn't write it, though). I'm a big fan of Paul Graham's essays. I've read his book, Hackers and Painters, which is a collection of his essays, from cover to cover. I link to Paul Graham's essays in almost every "long" article I write. I recommend Hackers and Painters to everybody who thinks programming is more than the implementation of specs handed down to you by a suit :)

Wednesday, August 16, 2006

Closing the Loop on Erlang Metaprogramming with Smerl

Earlier today after work, I went jogging in the Hudson River Park. Jogging must do something postivie for the flow of mental juices, because as I was making my way back home, it suddenly struck me that I hadn't fully closed the loop on Erlang metaprogramming with Smerl! Smerl let you add functions to and remove functions from modules, but it didn't let you modify existing functions. Duh!



After I got back home, I sat by my MacBook and frantically hacked away until I fixed this unfortunate omission. I added two new functions: smerl:get_func and smerl:replace_func. You can use smerl:get_func to get the abstract form for a function. smerl:replace_func is a convenience function you can use instead of called smerl:remove_func and then smerl:add_func.



Finally, Smerl has all the capabilites for transforming Erlang modules from solid to liquid and then back to solid.



Does your language let you do that? If not, I think you should seriously forget everything you know and break on through to the other side :)

haXe Now Supports Flash 9!

I haven't blogged about haXe in a while, but this one is certainly worth mentioning: thanks to the amazing work of Nicolas Cannasse, haXe 1.05 was just released today, with support for Flash 9! What does this mean? You can now develop the most cutting-edge Flash apps without shelling any $$$ out to Adobe by using this wonderful open source language that also comes with a blazing-fast compiler (it's written in OCaml ;) ).



For more information, visit http://haxe.org/tutos/flash9.



Also... in case you don't know, I have written a haXe remoting adapter for Yaws, so your haXe clients can easily talk to your Yaws-powered backend. See? You can get the best of both worlds! :)



So what are you waiting for? Go grab that good stuff and start hacking the next great killer app! :)

Why Erlang Is a Great Language for Concurrent Programming

Most programming languages implement concurrency poorly, if at all. The most common approach, which involves multiple OS threads or processes sharing common data structures that are guarded by locks or semaphores, has proven itself to be too complex for most programmers to "get right." Problems such as deadlocks, unguarded access, and coarse locking tend to creep up even in code created by skilled programmers.



To help programmers tackle some of the difficult challenges involved in shared memory-based concurrency, some language designers have created numerous-additional-APIs to supplement and replace some of the existing ones. This has probably made programmers even more confused about concurrency, not less.

Concurrency has gained such reputation for complexity that many programmers have learned to regard it as a snakepit of trouble that they best avoid altogether.



Update (8/16/06, 18:46PM EST): Ruby on Rails, for instance, is entirely single threaded, relying on a pool of OS processes to handle simultaneous requests (at least when deployed with FastCGI). As you will soon see, this is practically guaranteed to be less scalable than Yaws under high load.



Erlang has taken a different approach to concurrency than today's popular programming languages. Instead of implementing an ad-hoc concurrency API, Erlang's designers have built concurrency into the language. They also taught programmers to embrace concurrency rather than run away from it. That's why Erlang's Getting Started guide has a section about concurrency. It's also why every Erlang application and numerous libraries I know of -- including the ones created by the open source community -- use concurrency extensively. Check out Yaws (internals), ejabberd, OpenPoker and (parts of) Jungerl for some great examples.



What makes Erlang's approach to concurrency different? In Erlang, processes have no shared memory. To communicate, Erlang processes send messages to each other. A message can be any Erlang term -- tuples, lists, records, even functions. Sending a message to a process is simple:




Pid ! Message


where Pid is the process's ID as returned from the spawn() function, and Message is the Erlang term. Spawning a new process is also very easy. Here's an example:




Pid = spawn(fun() -> io:format("hi!", []) end)


which calls the io:format function in a parallel process.



Erlang's pattern matching semantics make processing incoming messages very convenient. Here's an example:




receive
{hello, Text} ->
io:format("Got hello message: ~s", [Text]);
{goodbye, Text} ->
io:format("Got goodby message: ~s", [Text])
end


That's it. Simple, huh?



The simplicity of Erlang code is partly what makes Erlang such a great language: as Paul Graham said, succinctness is power.



Interestingly, Erlang's dynamic typing cuts down the effort required for sending and receiving messages because Erlang doesn't require you to define types for different messages before sending them. In Erlang, types -- or 'tags' -- are denoted by convention as the first element of a tuple, e.g. {person, Bob}. As you can see, making things and defining them are one and the same in Erlang.



At this point, fans of static languages would bemoan the lack of compiler checks in dynamic languages, including Erlang. Well, they should know that Erlang has a static analysis tool, Dialyzer, that catches many type-related errors. In addition, when you code in Erlang, you always have a shell running in a different window so you can test your code as you're writing it. It's very hard for type-related errors to sneak into your code when you program this way.



Update (8/17/06, 12:39AM): Also, if Erlang were statically typed, we wouldn't have Smerl! :)



One of the greatest aspects of Erlang's concurrency is the way it's implemented behind the scenes. Erlang code runs in a virtual machine called BEAM, which is responsible for spawning, scheduling, and cleaning up Erlang processes, as well as passing messages between them. Erlang processes are much more lightweight than OS threads, which means that you can potentially spawn millions of processes on a single box. Try that with a pthreads-based library and your machine would croak after the first 4000-10000 threads. Erlang's lightweight processes are the reason for this graph, which compares the number of simultaneous connections Yaws can handle compared to Apache:






It's very important for a language to get concurrency right from the beginning. I occasionally hear about attempts to add Erlang-style concurrency to other languages. However, even if those attempts are successful, it would take years for programmers to embrace them and for open source libraries and applications to use them. That's why Joe Armstrong is right when he says that Erlang programmers have a head start. I have quoted these words before, but I think they are worth quoting again in this context:




Erlang also maps nicely onto multi-core CPUs - why is this? - precisely because we use a non-shared lots of parallel processes model of computation. No shared memory, no threads, no locks = ease of running on a parallel CPU.



Believe me, making your favourite C++ application run really fast on a multi-core CPU is no easy job. By the time the Java/C++ gang have figured out how to throw away threads and use processes and how to structure their application into small lightweight processes they will be where we were 20 years ago.



Does this work? - yes - we are experimenting with Erlang programs on the sun Niagara - the results are disappointing: our message passing benchmark only goes 18 times faster on 32 CPU's - but 18 is not too bad - if any C++ fans want to try the Niagara all they have to do is make sure they have a multi-threaded version of their application, debug it -'cos it probably won't work and they can compare their results with us (and I'm not holding my breath).



Turning a sequential program in a parallel program for the Niagara is really easy. Just change map/2 to pmap/2 in a few well chosen places in your program and sit back and enjoy.



Efficency comes from a correct underlying architecture, in this case being able to actually use all the CPUs on a multi-core CPU. The ability to scale and application, to make it very efficient, to distribute it depends upon how well we can slit the application up into chuncks that can be evaluated in parallel. Erlang programmers have a head start here.




Well, I hope you find this interesting, and maybe even useful :) For additional reading, check out this well-written defmacro.org article about Erlang-style concurrency. If you haven't used Erlang, I recommend downloading it and following the Getting Started guide. This will give you the flavor of good concurrent programming with Erlang.



Update (8/17/06): For a great discussion about concurrent programming in Erlang vs. other languages, read this mailing list posting by Richard O'Keefe. It's pure gold.

Tuesday, August 15, 2006

Thoughts on Erlang Metaprogramming with Smerl

Writing Smerl was a great exercise, because it let me delve deep into the Erlang language. Now I realize how much power lies in Erlang's simple design: in Erlang, a module is just a list of functions. No types. No classes. No interfaces. No nonsense.



Just functions.



Why is this so great? Didn't we learn in college that "real" languages have classes, vtables, instance variables, abstract methods, inheritance information, static variables, static methods, etc, etc?



Nope, they do not.



Just like Lisp, Erlang is great because Erlang is simple.



In fact, Erlang and Lisp are quite similar. They are both functional, dynamically typed languages, and if you think of Erlang tuples as non-mutable lists, you'd realize that just like Lisp, Erlang is mostly about list manipulation.



Update (8/16/06): it's been brought to my attention that I made a mistake by saying that Lisp is a pure functional language because Lisp has mutable values. It's been a while since I've used Lisp so I deserve pardon :)



However, Erlang is a better language than Lisp IMO because Erlang is designed for concurrent and distributed programming -- key component of many non-trivial pieces of software -- whereas Lisp is not.



To highlighting Erlang's simplicity, with Smerl, which took me a few hours to write, changing a module in runtime boils down to 4 basic functions: smerl:for_module, smerl:add_func, smerl:remove_func and smerl:compile.



That's it.



Heck, Smerl can even modify itself, and it only takes 3 lines:




{ok, C1} = smerl:for_module(smerl),
{ok, C2} = smerl:add_func(C1, "foo() -> 1+1."),
smerl:compile(C2).


You could easily write a function that reduces the above example to 1 line:




add_func_to_module(Module, Func) ->
{ok, C1} = smerl:for_module(Module),
{ok, C2} = smerl:add_func(C1, Func),
smerl:compile(C2).


So the example would be written as:




add_func_to_module(smerl, "foo()-> 1+1.")


Pretty cool, huh?



Compare Smerl with this ominous list of libraries for Java bytecode manipulation, none of which comes close Smerl's power. (For a real trip, read this manual for BCEL -- Byte Code Engineering Library. You wouldn't know whether to laugh or cry.) Why? Because a) the Java compiler doesn't let you get away with using things that don't exist in compile time and b) Java is much more complex than Erlang, so using those libraries generally requires some serious mental gymnastics.



Ok, I know Java is an easy target to pick on, but I think that Smerl makes runtime metaprogramming in Erlang even easier than in Ruby.



So what's the conclusion? Smerl is just one more reason to love Erlang! :)

Monday, August 14, 2006

Smerl: Simple Metaprogramming for Erlang

Quick download link: smerl.erl



Update (8/14/06, 11:11PM ET): I added to smerl.erl the function has_func, which queries whether a module has a given function.



Update (8/14/06, 11:24PM ET): Take that, Ivan the Mean Java Dev! :)



Update (8/14/06, 11:56PM ET): A few minor bug fixes



Update (8/15/06, 14:55PM ET): Added smerl:for_file and wrote a follow-up posting.

Update (8/16/06, 23:30PM ET): Added smerl:get_func, which retrieves the abstract form for a function, and smerl:replace_func, which calls smerl:remove_func followed by smerl:add_func. Also wrote another follow-up article.

Update (8/17/06): Smerl now supports fun expressions! Read my follow-up article here.

Update (8/18/06): Added new functions, smerl:get_module, smerl:get_forms and smerl:get_exports. Also, the last update turns out to be less exciting than it sounds: fun expressions are only supported in the Erlang shell, at least for now.



Update (8/19/06): Big breakthrough: metacurrying! Read this follow-up article.



I posted a couple of weeks ago a short article discussing some thoughts I had about Metaprogramming with Erlang. It turns out that Erlang's dynamic typing, hot code swapping and built-in parsing and compilation tools make for some excellent runtime metaprogramming capabilities (I actually don't know of any other functional language that gives the programmer so much flexibility in the area of runtime code manipulation). The only problem is that using this capability is far from obvious and I doubt many programmers are aware of it. I was fortunate enough to discover it when I was digging through Ulf Wiger's rdbms_codegen.erl module in Jungerl, but otherwise it might have flown past my radar.



I wanted to make life easy for myself and my fellow Erlang (meta)programmers who want to take advantage of this capability, so I created created a small library called Smerl: Simple Metaprogramming for Erlang.



Following is the main documentation for Smerl, taken from smerl.erl (additional documentation is in the source).




Smerl is an Erlang library that simplifies the creation and manipulation of Erlang modules in runtime.


Smerl uses Erlang's capabilities for hot code swapping and abstract syntax tree parsing to do its magic. Smerl is inspired by the rdbms_codegen.erl module in the RDBMS application written by Ulf Wiger. RDBMS is part of Jungerl.



Here's a quick example illustrating how to use Smerl:




test_smerl() ->
C1 = smerl:new(foo),
{ok, C2} = smerl:add_func(C1, "bar() -> 1 + 1."),
smerl:compile(C2),
foo:bar(). % returns 2


New functions can be expressed either as strings of Erlang code or as abstract forms. For more information, read the Abstract Format section in the ERTS User's guide (link).



Using the abstract format, the 3rd line of the above example would be written as




{ok,C2} = smerl:add_func(C1, {function,1,bar,0,
[{clause,1,[],[],
[{op,1,'+',{integer,1,1},{integer,1,1}}]}]).


The abstact format may look more verbose in this example, but it's also more amenable to runtime manipulation.



To manipulate or query an existing module rather than a new module, the first line could be rewritten such as:




C1 = smerl:for_module(mnesia),



Detailed Description


With Smerl, you can both create new modules and manipulate existing modules in runtime. You can also query whether a module has a given function by calling smerl:has_func. To start creating a new module, call smerl:new(ModuleName). To start modifying an existing module, call smerl:for_module(ModuleName). (The module be accessible with code:which and either have been compiled debug_info or its source file must in the same directory as the .beam file or in a ../src directory relative to the .beam file's ./ebin directory.) Both these functions return an opaque context record for the module. To manipulate the module, use smerl:add_func and smerl:remove_func. Just remember not to add the same function name with the same arity twice as it will eventually result in a compilation error.



When you're ready to compile your module, call smerl:compile, passing in the opaque context record. If there are no errors, you can start using the new module.




If you have any problems or suggestions about Smerl, please let me know.



Smerl isn't just a programming exercise. It may not be obvious, but Smerl actually has very good uses. I will write more about Smerl soon.



New tip (8/18/06): Are you looking an easy way of knowing what the abstract form for a function is? Fire up the erlang shell and type




io:parse_erl_form(". ").


This will give you a one-line mini shell, in which you can type your function and get back its abstract form representation.

Sunday, August 13, 2006

The Reddit Effect

About a month ago, as I've been wresting with an unbelievably annoying installation procedure for Ruby on Rails on my MacBook, I wrote the posting Erlang + Yaws vs. Ruby on Rails, in which I highlighted some of the advantages Yaws, an web server for dynamic web applications written in Erlang, has you over Ruby on Rails. I really like Rails, but at the same time I'm sometimes puzzled at the disproportionate amount of buzz it gets, especially as compared to Erlang, which by all indications is the software industry's biggest undiscovered treasure.



I wanted to share my thoughts on Erlang and Rails with the community, and what is the best way to do that in this day and age? Blog about them, of course!



Well, there is actually one more thing to do after blogging: submitting the article to Digg, the biggest social news website. That's what I did, and sadly enough my article only got dugg 15 times -- not nearly enough to make the front page.



"Oh well," I thought, "I did my share of Erlang evangalism. The crowds apparently aren't interested. I guess Erlang will remain obscure to most web developers, at least for a while."



Jokingly, a few days later I wrote an article comparing Erlang and the One Red Paper Clip Guy, half-seriously thinking that maybe that would get people's attention. I just wrote it for fun and I didn't bother submitting it to Digg, though.



In a strange twist of fate, a couple of days ago, somebody has submitted my Erlang + Yaws vs. Ruby on Rails article to reddit.com, another big social news site, and lo and behold -- the article made the front page! From reddit.com, the posting then spilled over to del.icio.us and made the popular page. Not bad for an article on which I have already "given up." :)



In the two days since posting got "redditted," my blog has received almost 4000(!) visits. Here are the graphs from Google Analytics:








It looks as though against all odds, my ramblings have actually made some people interested in Erlang. I wonder if any of them will actually become real Erlanger hackers. I'm not expecting a mass conversion by any means, but one or if one or two good hackers adopted Erlang, I would feel like it was a job well done.



If you are one of these people, please let me know!

Saturday, August 12, 2006

Does Google Have No Interest in Functional Programmers?

Note: before I try to answer this question, I would like to clarify that by "functional programmers" I don't mean programmers who are functional human beings, but programmers who know functional programming. Whether Google is interested in hiring emotionally functional people is beyond the scope of this inquiry, but I would like venture a guess: of course not! Functional people are boring! Just kidding :)



To research this fascinating question, I went to the Google jobs website and ran the following queries:





Just to make sure I'm not completely delusional in assuming that Google actually creates software and therefore wants to hire *any* programmers, I ran the next few queries:






This is strange. I know that Google makes heavy use of Java, Python and C++, but I'm sure Google has many great hackers who love functional languages. Peter Norvig, the director of Research at Google, is a prominent figure in the functional programming world. In addition, Google's distributed computing infrastructure is largely built on MapReduce, a concept that originated in functional programming.



Speaking of MapReduce, let's take a short detour and think about MapReduce maps into Erlang -- no pun intended :)

In Erlang, the (linear) concept of MapReduce is embodied in lists:mapfold(). As Joe Armstrong has shown in his pmap implementation, it would be trivial in Erlang to write a parallelized version of mapfold(), which the Erlang runtime would execute on multiple CPUs and which could easily be distributed onto a cluster of Erlang nodes.



Following is the code, taken from his mailing list posting, for Joe's pmap implementation in Erlang. I doubt it would be possible to write an implementation that's nearly as concise in C++. This speaks well for Erlang, because as Paul Graham said, succinctness is power.




pmap(F, L) ->
S = self(),
Pids = map(fun(I) ->
spawn(fun() -> do_f(S, F, I) end)
end, L),
gather(Pids).

gather([H|T]) ->
receive
{H, Ret} -> [Ret|gather(T)]
end;
gather([]) ->
[].

do_f(Parent, F, I) ->
Parent ! {self(), (catch F(I))}.


I don't know of any language that would make MapReduce so straightforward to implement as Erlang. Also, in addition to Erlang's message passing primitives, Erlang has hot code swapping, supervision trees, lightweight processes, a distributed database engine called Mnesia and many other goodies that would facilitate a highly robust MapReduce implementation.



None of this is surprising, given that Erlang was designed to power massively scalable phone switches targeting %99.9999999 (nine nines!) availability, so concurrent, distributed programming is Erlang's bread and butter.



I may be crazy, but I just can't help but wonder whether Google would have been better off writing its MapReduce infrastructure in Erlang rather than C++. After all, Erlang has worked wonders for Ericsson in building its high-end distributed telephony products as Ulf Wiger's paper, Four-fold Increase in Productivity and Quality (pdf), points out. (Yes, I know that Ericsson isn't a search engine company, but let's not worry too much over such minutia :) )



Ok, I admit I'm not seriously proposing that Google should write its backend in Erlang. Doing so would totally defeat the purpose. If Google dropped C++ for Erlang, I would lose the competitive advantage I need for building my Erlang-Powered Google Killer! MOAHAHAHAHA! :)



Back to the jobs question: the only way I can explain my findings short of asking somebody at Google (and I don't know anybody who works there), is that Google simply doesn't build any applications in functional languages -- at least not the ones in my searches. So what's the conclusion? If you want to get a programming job at Google (actually, probably in 99.9% of software companies, especially in the US), you should forget those functional programming adventures you've been dreaming about and start cracking open those Java/Python/C#/C++ books.



Well... there is one more option. You can follow in Paul Graham's footsteps and use a functional language as the secret weapon of your very own startup. You'll have fun, and maybe -- just maybe -- you'll strike it rich. That doesn't sound too bad, does it? Just keep in mind that I take no responsiblity for any eventuality of financial ruin or loss of spouse or girlfriend in case the ideas I put in your head don't work out the way you hoped :)

Wednesday, August 09, 2006

Secret Weapons for Startups

In Paul Graham's famous essay, Beating the Averages, he explains how his decision to use Lisp as the backend language for Viaweb helped Viaweb gain a critical competitive advantage, making it the first successful commerial web application. (This happened ages ago: in 1996!) According to Paul, Lisp's conciseness and expressiveness produced a rapid development cycle that helped Viaweb's platform outpace its competition in stability and features. This rewarded Viaweb's founders very generously, as they eventually sold Viaweb to Yahoo for $49 mil(!).



Here's a passage from Paul's essay describing the competitive advantage Lisp gave Viaweb:




What were the results of this experiment [using Lisp]? Somewhat surprisingly, it worked. We eventually had many competitors, on the order of twenty to thirty of them, but none of their software could compete with ours. We had a wysiwyg online store builder that ran on the server and yet felt like a desktop application. Our competitors had cgi scripts. And we were always far ahead of them in features. Sometimes, in desperation, competitors would try to introduce features that we didn't have. But with Lisp our development cycle was so fast that we could sometimes duplicate a new feature within a day or two of a competitor announcing it in a press release. By the time journalists covering the press release got round to calling us, we would have the new feature too.



It must have seemed to our competitors that we had some kind of secret weapon -- that we were decoding their Enigma traffic or something. In fact we did have a secret weapon, but it was simpler than they realized. No one was leaking news of their features to us. We were just able to develop software faster than anyone thought possible.




One can learn a valuable lesson from Viaweb's story: the programming language(s) a startup chooses can have a decisive effect on its business success. Picking a language that fosters developer productivity is critical, because if your competitors use a more productive language, they would eventually have an advantage in the marketplace. In Viaweb, as well as Orbitz's case, Lisp has proven to be the winning language choice. Lisp's combination of power and obscurity has led Paul Graham to think of Lisp as a startup's secret weapon, so Viaweb has never advertised its use of Lisp, fearing Viaweb's competitors would adopt Lisp as well.



Viaweb has actually had another secret weapon that the essay doesn't discuss: great hackers. Viaweb's use of Lisp may have given it a productivity advantage, but even more important than the use of language is the skill of a company's hackers. I'm believe that in most cases, a team of good hackers can beat a team of average hackers even if the good team uses a less productive language.



The same holds even if the good team is smaller than the average team. In fact, the bigger the team, the more management and communications overhead tend to slow down development. Even worse, large teams sometimes become stratified bureaucracies in which hackers are regarded as mere implementors of a the suits' "vision." Depending on the industry, this can be a recipe for mediocrity if not outright failure. For an illustration, look at the rate at which Google, which has an agile, decentralized structure, churns out quality products and compare it to Microsoft's Vista nightmare.



I'd bet Paul and Robert Morris would have made Viaweb a success even if they had written it in a less productive language (maybe even C, which I sadly have to use at my day job) because they are top-notch hackers and because they ran a lean-and-mean, independent operation that gave free reign to their creativity and problem-solving finesse.



All else being equal, though, picking a great programming language can make a big difference for a startup. So, ignoring hackers' skills, what is a (web) startup's ultimate secret weapon? Well, I can't say I know for certain, but I do have some ideas :) Paul Graham thinks it's Lisp, and although I appreciate Lisp's clean syntax, dynamic typing, and great metaprogramming features, I can't say I like the way Lisp code looks (admittedly, the last time I touched Lisp was in college, so I may be oblivious to its wonders). I actually think that a startup's secret weapon arsenal boils down to the following:


  • If it's building an "ordinary" CRUD-based webapp and it has good or average hackers, it should go with Ruby on Rails or Python/Django. (haXe will be another appealing option in a few months.)

  • If it's building a CRUD-based app or a cutting-edge app that uses advanced Comet techniques like Meebo, and it has very good hackers who aren't scared of functional languages, it should go with Erlang.



(Update 8/17/06: Regarding haXe, I think haXe is an awesome language, especially for both Javascript and Flash development. The reason I said "in a few months" is because haXe/Neko, the server component of haXe, is not as feature rich as Rails -- yet. However, Nicolas Cannasse, haXe's creator, is moving it forward in a blazing speed. Keep that one on your radar, because with haXe you can actually write all aspects of a webapp in one great open source language. How's that for a secret weapon? :) )



Why would a startup need better hackers if it's using Erlang? After all, Erlang is actually a very simple language -- simpler than Ruby. There are two reasons:




  • Erlang is a dynamically-typed functional language, which apparently doesn't fit well into developer brains that are conditioned into thinking in Java. Sadly, this is the general case these days.

  • Erlang doesn't have as many web development libraries as Rails or Django, so the hackers would need the courage to roll up their sleeves and write their own framework-level code. (Yaws, "The" Erlang webserver, does have some good web development APIs, but not as many as Rails.)



In time, Erlang will have better web development tools. When it happens, I predict that Erlang will become the ultimate startup secret weapon. Why? Because Erlang is a better language than Ruby/Python, especially for building concurrent, distributed applications with scalable, highly-available backends, where Erlang is years ahead of any other language. Even if, hypothetically, the initial development would be somewhat slower with Erlang, Erlang is best positioned to prevent a startup's developers from turning into full-time sysadmins/optimizers once the app becomes popular. This is important because they can spend their time developing new features that give the startup a competitive advantage.



Erlang will get there sooner than you may realize. Stay tuned.

Thursday, August 03, 2006

Programming Language Trends

According to Oreilly Radar, a few notable trends are taking place in the programming language world. Java, Perl, VB and C/C++ are going down, C#, Javscript and Ruby are going up, and PHP and Python aren't changing much.



If you look at the list of languages more closely, you should notice that the strangest of trends is actually not depicted on the graph: functional languages don't exist. Apparently, between Lisp's invention in 1958 by John McCarthy and the present, a mass extinction has decimated every single functional language. Haskell, OCaml, Erlang and Lisp are mere memory fragments of lost civilizations that have been destroyed by big crashing objects shaped like the Flying Spaghetti Monster.



I like this theory, but I admit that there may be a couple of other possible explanations for this functional language anti-trend. The first is that functional languages are so advanced and cutting edge that their stealth technology helps them evade Oreilly's radar. As Paul Graham has said, Lisp can be used as a great secret weapon in a startup made of smart hackers. If Lisp were on Oreilly's graph, it wouldn't be such a secret weapon anymore. (I'm just using Lisp as an example. Any of the functional languages I mentioned would fit the bill.)



The second explanation is that the source of Oreilly's data is book sales, and since Oreilly doesn't publish books on functional languages, they're missing from the graph. This leads me to wonder, why doesn't Oreilly publish books about functional languages? As you have probably realized by now, the answer is obvious: hackers of functional languages can't read.


Paul Graham and co., please don't come hunt me down now -- I'm just kidding!



The real explanation is actually somewhere in between. Functional languages are very powerful, but their power scares away many people, especially those who have been inculcated into the Church of Java by Comp Sci 101 classes and an oppressive stack of Enterprise Java books. The Church of Java also controls the minds of the majority of programmers by evoking the coffee bean imagery to trigger feelings of dependency in their minds, which, conveniently for Sun, have a physical addiction to caffeine. However, Java mind-control isn't 100% effective, and some rebellious programmers have managed to break free from Java and join the heretical sects of Ruby or Python. Ruby and Python aren't as rigid as Java, which leads their freedom-seeking developers to believe that these newly-discovered gems are actually the holy grails of programming power. So, these developers keep buying similar books, but with different-sounding titles. Old habits are hard to break, after all.



I believe this is the real reason that, according to Oreilly, functional languages are trendless. However, don't write off the Flying Spaghetti Monster explanation just yet: if the FSM can challenge evolution, no school of rational thought is safe in its path!