My Erlang Wishlist

Every person can dream, and I have been dreaming quite a lot. In fact, it wouldn't be too far-fetched to say that I have taken this liberty and ran with it. Sometimes it's hard to fit all my dreams into my skull, but I think I've managed it pretty well so far.


I'm a programmer, so it shouldn't be a surprise that many of my dreams are about hopelessly geeky things. This blog has primarily served as an outlet for my Erlang musings thus far, so I figured this blog would be a good place to express my wishes for new capabilities and libraries I would like to have in my Erlang toolbox. I hope that some day the open source code fairy stumbles upon my blog and turns these wishes into reality.


Here are my Erlang wishes, in no particular order:

• Shorter repair times for broken Mnesia dets tables. This has been discussed in the erlang-questions list here, here and here . I have some web application ideas for which Erlang would be a great server side language, and I just hate the thought of using another DBMS just for the fear of long downtimes in the event of a crash when my data set gets big. If dets were more resilient to crashes, I would lose my motivation to use another DBMS.
  
  After I've used Mnesia, going back to MySQL/Postgres just feels lame. I want Mnesia to be good enough for all my needs.
  
  
• More scalable freelist handling in dets. Yup, dets has another problem: it uses a buddy system to maintain the freelist, and this freelist grows as the data becomes fragmented. The freelist is held in RAM, so this has a negative effect on memory consumption that gets worse over time. In addition, when you close a dets table, this potentially huge data structure needs to be serialized to disc, which can take a while. The only ways of shrinking the freelist is to close the table and reopen it in force repair mode, or to create a duplicate table that gets built from scratch without fragmentation. The first option can effect long downtimes, and the second option can be too expensive if the table is big. This is another force pulling me in the MySQL/Postgres direction, which, as you know, I'm trying to avoid.



• A robust join optimizer for QLC. QLC is the main query language for Mnesia. Unlike most DBMS, the QLC query engine doesn't have a join optimizer right now, so writing your query the wrong way could make it run off into the weeds. Postgres has what looks like a very sophisticated join optimizer, and I'd like QLC to have something similar. (Fortunately, this feature is planned for a coming release of OTP, but I wanted to mention it anyhow.)



• An advanced full-text indexing library. Java has Lucene, Ruby has Ferret, C++ has CLucene, Python has PyLucene, etc, etc. Erlang has a text search feature built into the RDMBS module in Jungerl, but AFAIK it's not as mature and feature-rich as the other libraries I mentioned. These days, almost every website needs a good text search feature, so having such a text indexing library would make my life much better.



• A graphing library like Gruff. I like graphs. Graphs make me happy. I want to make graphs with Erlang, but right now there's no easy way of doing it. I want an Erlang graphing library that's easy to use.



• A template language for Yaws. ehtml is pretty cool, but I think a template language produces more readable code. This has been discussed extensively on the mailing list, but I haven't heard of any progress in this direction lately.



• A RSS/Atom parsing library. Such a library would bring one of my project ideas one step closer to reality, and I really don't want to write one myself because of all the hairy incompatibilities between the outputs of different feed engines. Why should I suffer if somebody else may be willing to take on this pain?



• A better documentation website. It doesn't take a sharp artistic eye to notice that the current design is less than beautiful. I would welcome any aesthetic improvements, but even more importantly I would like the online documentation to integrate user comments. This will give the existing documentation the extra substance it needs on some places. The MySQL, PHP and Postgres documentation sites are good examples of this.



• A more active web development community. I don't hear of many large-scale websites using Erlang, and I know of only few independent developers who are building websites with Yaws. This graph validates my feelings. More people need to use Erlang for web development so I don't feel lonely and so I can use the solutions that other people contribute the to community. Sadly, I just don't have the time to implement everything myself, which, combined with an occasional bout of laziness, is why I'm writing this posting in the first place.



• A community website for Erlang developers,. This already exists thanks to the recently launched trapexit, but I wanted to list it anyway because I've been wishing for it for a while.



• An Erlang/OTP road map. Besides a few features, I have now idea what the OTP group is working on and what they are planning on working on in the future. It's possible they are already tackling some of the items I mentioned, but I'm in the dark about this, which is somewhat frustrating. I'd like to know where Erlang is going, not just where it is right now.



• An Erlang-powered machine that brings world peace, saves the environment, fights poverty and disease, provides univeral happiness and finds me a new apartment. I can dream, can't I? :)

Well, that's about it for now. There are probably a lot more things I could think about, but I don't want to overwhelm my poor readers, some of whom I must have bored to death by now. Now it's time to sit back and wait for the open source fairy to google for "erlang wishes" and hopefully land here on my blog.


It's bound to happen happen any day now -- I know it!