Sunday, May 10, 2009

Geeking out with Lisp Flavoured Erlang

One of the features I dislike the most about Erlang is records. They're ugly and they require too much typing. Erlang makes me write

Dog = #dog{name = "Lolo", parent = #dog{name = "Max"}},
Name =,
ParentName = (Dog#dog.parent),
Dog1 = Dog#dog{
name = "Lola",
parent = (Dog#dog.parent)#dog{name = "Rocky"}}

When I want to write

Dog = #Dog{name = "Lolo", parent = #dog{name = "Max"}},
Name =,
Size = Dog.size,
Dog1 = Dog{
name = "Lola",
parent = Dog.parent{name = "Rocky"}}

In the defense of records, they're just syntactic sugar over tuples and as such they enable fast access into a tuple's properties despite Erlang's dynamic nature. In compile time, they're converted into fast element() and setelement() calls that don't require looking up the property's index in the tuple. Still, I dislike them because 1) in many cases, I'd rather optimize for more concise code than faster execution and 2) if the Erlang compiler were smarter could allow us to write the more concise code above by inferring the types of variables in your program when possible. (I wrote a rant about this a long time ago, with a proposed solution to it in the form of a yet unfinished parse transform called Recless.)

Parameterized modules provide a somewhat more elegant and dynamic dealing with types, but they still require you to do too much work. You can define a parameterized module like this:

-module(dog, [Name, Size]).

This creates a single function called 'new/2' in the module 'dog'. You can call it as follows:

Dog = dog:new("Lola", "big").

It returns a tuple of the form

{dog, "Lola", "Big"}.

You can't set only a subset of the record's properties in the 'new' function. This doesn't work

Dog = dog:new("Lola").

To access the record's properties you need to define your own getter functions, e.g.

name() ->

which you can call it as follows:

Name = Dog:name().

(This involves a runtime lookup of the 'name' function in the 'dog' module, which is slower than a record accessor).

There's no way to set a record's properties after it's created short of altering the tuple directly with setelement(), which is lame and also quite brittle. To create a setter, you need do the following:

name(Val) ->
setelement(2, THIS, Val).

Then you can call

Dog1 = Dog:name("Lola").

to change the object's name.

When LFE came out I was hoping it would provide a better way of dealing with the problem. Unfortunately, records in LFE are quite similar to Erlang records, though they are a bit nicer. Instead of adding syntactic sugar, LFE creates a bunch of macros you can use to create a record and access its properties.

(record dog (name))
(let* ((dog (make-dog (name "Lola")))
(name (dog-name dog))
(dog1 (set-dog-name dog "Lolo")))
;; do something)

LFE still requires us to do too much typing when dealing with records to my taste, but LFE does give us a powerful tool to come up with our own solution to the problem: macros. We can use macros generate all those repetitive brittle parameterized module getters and setters that in vanilla Erlang we have to write by hand. This can help us avoid much the tedium involved in working with parameterized modules.

(ErlyWeb performs similar code generation on database modules, but it does direct manipulation of Erlang ASTs, which are gnarly.)

Let's start with the 'new' functions. We want to generate a bunch of 'new' functions allow us to set only a subset of the record's properties, implicitly setting the rest to 'undefined'.

(defun make_constructors (name props)
(let* (((props_acc1 constructors_acc1)
(: lists foldl
((prop (props_acc constructors_acc))
(let* ((params (: lists reverse props_acc))
`(defun new ,params
(new ,@params 'undefined))))
(cons prop props_acc)
(cons constructor constructors_acc)))))
(list () ())
`(defun new ,props (tuple (quote ,name) ,@props))))
(: lists reverse (cons main_constructor constructors_acc1))))

This function take the module name and a list of properties. It returns a list of sexps of the form

(defun new (prop1 prop2 ...prop_n-m)
(new prop1 prop2 ... prop_n-m 'undefined))

as well as one sexp of the form

(defun (prop1 prop2 ... prop_n)
(tuple module_name prop1 prop2... prop_n)

The first set of 'new' functions successively call the next 'new' function in the chain, passing into it their list of parameters together with 'undefined' as the last parameter. The last 'new' function takes all the parameters needed to instantiate an object and returns a tuple whose first element is the module, and the rest are the object's property values. (We need to store the module name in the first element so we can use the parameterized modules calling convention, as you'll see later.)

Now let's create a function that generates the getters and setters

(defun make_accessors (props)
(((accessors idx1)
(: lists foldl
((prop (acc idx))
(let* ((getter `(defun ,prop (obj) (element ,idx obj)))
(setter `(defun ,prop (val obj)
(setelement ,idx obj val))))
(list (: lists append
(list getter setter)
(+ idx 1)))))
(list () 2)

This function takes a list of properties and returns a list of sexps that implement the getters and setters for the module. Each getter takes the object and returns its (n + 1)th element, where n is the position of its property (the first element is reserved for the module name). Each setter takes the new value and the object and returns the tuple after setting its (n + 1)th element to the new value.

Now, let's tie it this up with the module declaration. We need to create a macro that generates the module declaration, constructors, getters, and setters, all in one swoop. But first, we need to expose make_constructors and make_accessors to the macro by nesting them inside a (eval-when-compile) sexp.

(defun make_constructors ...)
(defun make_accessors ...))

(defmacro defclass
(((name . props) . modifiers)
(let* ((constructors (make_constructors name props))
(accessors (make_accessors props)))
(defmodule ,name ,@modifiers)

(defclass returns a `(progn) sexp with a list of new macros that it defines. This is a trick Robert Virding taught me for creating a macro that in itself creates multiple macros.)

Now, let's see how this thing works. Create the file dog.lfe with the following code

;; import the macros here

(defclass (dog name size)
(export all))

Compile it from the shell:

(c 'dog)

Create a dog with no properties

> (: dog new)
#(dog undefined undefined)

Create a new dog with just a name

> (: dog new '"Lola")
#(dog "Lola" undefined)

Create a new dog with a name and a size

> (: dog new '"Lola" '"medium")
#(dog "Lola" "medium")

Get and set the dog's properties

> (let* ((dog (: dog new))
(dog1 (call dog 'name '"Lola"))
(dog2 (call dog1 'size '"medium")))
(list (call dog2 'name) (call dog2 'size)))
("Lola" "medium")

This code uses the same parameterized module function calling mechanism that allows us to pass a tuple instead of a function as the first parameter to the 'call' function. Erlang infers the name of the module that contains the function from the first element of the tuple.

As you can see, LFE is pretty powerful. I won't deny that sometimes I get the feeling of being lost in a sea of parenthesis, but over time the parentheses have grown on me. The benefits of macros speak for themselves. They give you great flexibility to change the language as you see fit.


Tom said...

I presume you mean

Dog = #dog{name = "Lolo", parent = #dog{name = "Max"}},


Dog = #Dog{name = "Lolo", parent = #dog{name = "Max"}},


Colin said...

Ever used the parse transform exprecs? An Ulf Wiger creation I think - it does much to remove the 'wreck' from 'record' imho. Very handy for writing simple specialized record access/manipulation functions too.

Abhijith said...

I had given LFE a cursory look but if the development moves forward I think it would be the ideal language for me - lisp syntax, macros and erlang features.
Is the development of LFE going anywhere ? It seems to have stalled.

Yariv said...

@Tom You're right, it was a bug.

@Abhijith I think Robert Virding is still working on it.

Laqrix said...

For comparison, consider a different Lisp macro (or in my case Scheme):

(define-erlang-record dog name parent)

define-erlang-record emits a new macro 'dog' which provides syntax that
starts with the type name.

( make [field value] ...)
( copy source [field value] ...)
( instance)

For creation, fields are named and required. Generally, I don't like
optional fields.

(define lola
(dog make [name "Lola"] [parent #f]))
(define rocky
(dog make [name "Rocky"] [parent #f]))

lola => #(dog "Lola" #f)
rocky => #(dog "Rocky" #f)

For functional updating provide copy. Any unspecified fields come from
the source record:

(define new-lola
(dog copy lola [parent rocky]))

new-lola => #(dog "Lola" #(dog "Rocky" #f))

Accessors can be used like this:

(dog name lola)

The macro can generate syntax errors for duplicate, missing, and
unknown fields. The macro can allow the fields to occur in any order.

If each accessor performs a type check, then with pattern matcher
support, you can do a single type check and extract multiple fields.

A possible matcher syntax might be:

(match x
[`(dog [name ,name] [parent ,top-dog])
;; name and top-dog are now bound in this scope

Food for thought.

Wei Zhu said...

Yariv, I have no idea what this article is about :-). I am just testing your Connect integration.

Robert Virding said...

Some responses:

- @Abhijith LFE has in no way died, just paused the last month. There is now a wiki (in github) to help people get started and a google group for discussions.

- LFE records look the way they do as they are defined to be compatible with vanilla erlang records. Tough I know, but not much to do about it. A goal of LFE is to be compatible with vanilla erlang and the OTP libraries.

- @Laqrix That is basically compatible with LFE records, though I based naming more on CL than scheme. This happened at the same time LFE went lisp-2 and adopted a more CL style. Defrecord also defines a macro to be used when matching, in the case of dog it would be match-dog and can be used like

(match-dog name "fido" size fido-size)

which match dogs called "fido" and return the dogs size in fido-size. All unmentioned fields will match anything of course.

- @Yariv You should be careful using the tuple structure like that, if parameterized module ever get adopted it might not work anymore. :-)

Yariv said...

@Robert Instead of using 'call' I'll just define a macro that performs the same operation. This way I won't be using any parameterized module features that come with Erlang.