Unofficial Erlang Planet

Zotonic destroys WordPress and rethinks the CMS with Erlang

2010-03-09T20:50:03+00:00

A chat with Marc Worrell, Lead Architect of Zotonic - a new Content Management System written entirely in Erlang.

CouchDB Case Study: Assay Depot

2010-03-06T23:40:35+00:00

...replication give this ability that would have been much more difficult to accomplish in MySQL. We could have done this in a relational manner but it would have been very challenging, very custom. CouchDB give a lot of the benefits for free.

CouchDB has changed the way I think about developing web applications

Talkr.IM XMPP/Jabber server gets Apple Push notifications

2010-03-04T09:43:17+00:00

The free public XMPP/Jabber server Talkr.IM gets Apple Push notifications for iPhone.

The ProcessOne's free, open, public XMPP/Jabber server Talkr.IM is gaining a new feature: Apple Push notifications. This well enable users to get notified of offline messages when they arrive, as well as simulate a continued XMPP session on iPhone devices that still do not implement multitasking.

For the latter feature, since the iPhone still does not accept to run applications in the background, if you want to use another application, you will have to shutdown your XMPP client and thus disconnect from Talkr.IM. With the Push feature enabled on your XMPP client (if this one supports Apple Push notifications), you can tell your contacts you are still online when you are using other applications. You will then automatically receive notifications of new messages. And when you re-open your XMPP client, you will receive these messages.

OneTeam for iPhone has the Apple Push feature enabled: you will be able to use it on you Talkr.IM account. Just use you Jabber ID, like username@talkr.im, and for a simple configuration, go to the "Settings" tab, then "Push settings".

The Talkr.IM XMPP server will stop at 10:00am CET. This means:

01:00am Los Angeles
04:00am New York
12:00 Moscow
18:00 Tokyo

UPDATE: the Apple Push module is installed on Talkr.IM, and working fine. You can use it on OneTeam for iPhone. You can run a APNS module on your own ejabberd server with IMpush.

ejabberd tip: simple health check

2010-03-03T16:27:29+00:00

An ejabberd health check mechanism might be useful in your deployments, but using the HTTP file server might be overkill...

Some people want to do regular health checks on ejabberd, which is always a good idea. Doing a health check over XMPP might not be the solution, here is at least two reasons:

with a pure ad-hoc solution like a cron script, you might need an XMPP library you might not want to install on your servers
monitoring systems might not talk XMPP at all...

So people first think of the internal ejabberd HTTP file server, which then they have to configure through a specific listener and the right module... possibly with logs, they would have to manage (rotate, parse, clean, etc.).

Sometimes people just don't think there is already an HTTP listener on their internal BOSH Connection Manager. You would argue that BOSH uses the HTTP POST method. But the ejabberd BOSH Connection Manager replies to simple GET:

wget http://myserver.net/http-bind

And ejabberd will answer:

ejabberd mod_http_bind v1.2

An implementation of XMPP over BOSH (XEP-0206)

This web page is only informative. To use HTTP-Bind you need a Jabber/XMPP client that supports it.

BBC and CouchDB

2010-03-01T22:11:06+00:00

Vaguely interesting KV/#couchdb stat from the BBC - 3.3 billion requests handled since last summer, running at about 150-170 million per day

http://twitter.com/endafarrell/status/9820160677

Memory matters - even in Erlang

2010-02-28T21:42:43+00:00

Some time ago we got an interesting bug report for RabbitMQ. Surprisingly, unlike other complex bugs, this one is easy to describe:

At some point basic.get suddenly starts being very slow - about 9 times slower!

Basic.get doesn’t do anything complex - it just pops a message from a queue. This behaviour was quite unexpected. Our initial tests confirmed that we have a problem when a queue contains thousands of elements:

queue_length: 90001  basic_get 3333 times took: 1421.250ms
queue_length: 83335  basic_get 3333 times took: 1576.664ms
queue_length: 60004  basic_get 3333 times took: 1403.086ms
queue_length: 53338  basic_get 3333 times took: 9659.434ms [ look at that! ]
queue_length: 50005  basic_get 3333 times took: 9885.598ms
queue_length: 46672  basic_get 3333 times took: 8562.136ms

Let me repeat that. Usually popping a message from a queue takes Xms. At some point, it slows down to 9*Xms.

It turned out that the problem is with the queue:len() function, which is executed during the basic.get. Actually, queue:len() calls only erlang:length() builtin. At some point it switches to the “slow” mode.

Erlang:length() is a builtin that iterates through a linked list and counts it’s length. It’s complexity is O(N), where N is the length of the list. This function is implemented in the VM so it’s expected to be very, very fast.

The problem is not with erlang:length() being slow. It’s about being unpredictably slow. Let’s take a look at Erlang interpreter source code (erl_bif_guard.c:erts_gc_length_1). Here’s the main loop for erlang:length():

i=0
while (is_list(list)) {
    i++;
    list = CDR(list_val(list));
}

It does nothing unusual - it just iterates through list elements. However, recompiling Erlang with some debugging information confirms that the problem is indeed here:

clock_gettime(CLOCK_REALTIME, &t0);
while (is_list(list)) {
    i++;
    list = CDR(list_val(list));
}
clock_gettime(CLOCK_REALTIME, &t1);
td_ms = TIMESPEC_NSEC_SUBTRACT(t1, t0) / 1000000.0;
if (i > 200000 || td_ms > 2.0) {
    fprintf(stderr, "gc_length_1(%p)=%i %.3fms\n\r", reg[live], i, td_ms);
}

gc_length_1(0x7f4dbfa7fc19)=499999 2.221ms
gc_length_1(0x7f4dbfa7fc19)=499999 2.197ms
gc_length_1(0x7f4dbfa7fc19)=499999 2.208ms
(hibernation)
gc_length_1(0x7f4db0572049)=499999 13.793ms
gc_length_1(0x7f4db0572049)=499999 12.806ms
gc_length_1(0x7f4db0572049)=499999 12.531ms

This confirms Matthias’ initial guess - the slowdown starts after Erlang process hibernation.

For those who aren’t Erlang experts: Hibernation is an operation that compacts an Erlang process. It does aggressive garbage collection and reduces the memory footprint of a process to absolute minimum.

The intended result of hibernation is recovering free memory from the process. However its side effect is a new memory layout of objects allocated on the heap.

Ah, how could I have forgotten! The memory is nowadays slow! What happens, is that before hibernation list elements are aligned differently, more dense. Whereas after hibernation they are sparse. It’s easy to test it - let’s count the average distance between pointers to list elements:

gc_length_1(0x7f5c626fbc19)=499999 2.229ms avg=16.000 dev=0.023
gc_length_1(0x7f5c626fbc19)=499999 3.349ms avg=16.000 dev=0.023
gc_length_1(0x7f5c626fbc19)=499999 3.345ms avg=16.000 dev=0.023
(hibernation)
gc_length_1(0x7f5c61f7d049)=499999 13.800ms avg=136.000 dev=0.266
gc_length_1(0x7f5c61f7d049)=499999 12.726ms avg=136.000 dev=0.266
gc_length_1(0x7f5c61f7d049)=499999 12.367ms avg=136.000 dev=0.266

Confirmed! Standard deviation is surprisingly small, so we can read the numbers as:

Before hibernation list elements are aligned exactly one after another, values are somewhere else.
After hibernation list elements are interleaved with values.

This behavior does make sense. In most cases when you traverse the list, you actually do something with the values. After hibernation, when you access list item, the value will be already loaded to the CPU cache.

Knowing the mechanism, it’s easy to write a test case that reproduces the problem.

The average distance between pointers in my case is constant - the standard deviation is negligible. This information has a practical implication - we can “predict” where the next pointer will be. Let’s use that information to “fix” the Erlang VM by prefetching memory!

while (is_list(list)) {
    i++;
    list2 = CDR(list_val(list));
    __builtin_prefetch((char*)list2 + 128*((long)list2-(long)list));
    list = list2;
}

Test script running on original Erlang VM:

length: 300001  avg:0.888792ms dev:0.061587ms
length: 300001  avg:0.881030ms dev:0.040961ms
length: 300001  avg:0.875158ms dev:0.019436ms
hibernate
length: 300001  avg:14.861762ms dev:0.150635ms
length: 300001  avg:14.833733ms dev:0.017405ms
length: 300001  avg:14.884861ms dev:0.220119ms

Patched Erlang VM:

length: 300001  avg:0.742822ms dev:0.029322ms
length: 300001  avg:0.739149ms dev:0.012897ms
length: 300001  avg:0.739465ms dev:0.014417ms
hibernate
length: 300001  avg:7.543693ms dev:0.284355ms
length: 300001  avg:7.342802ms dev:0.330158ms
length: 300001  avg:7.265960ms dev:0.053176ms

The test runs only a tiny bit faster for the “fast” case (dense conses) and twice as fast for the “slow” case (sparse conses).

Should this patch be merged into mainline Erlang? Not really. I have set the prefetch multiplier value to 128 and I don’t even know if it’s optimal. This was only an experiment. But it was fun to see how low-level system architecture can affect high-level applications.

Filtering lines efficiently

2010-02-24T09:00:02+00:00

Whenever you are dealing with log lines or that you're program is filtering data you always have to handle 'escaping' efficiently.

While developing a log module using a gen_event, I needed to escape simple quotes.
Sometime thoses quotes were already escaped...

I've found this regexp to handle gracefully the case:

re:replace( Bin, "(?<!\\\\)'", "\\\\'", [ global ] ).

Not so easy to read, and because of the various backslashes, this regexp needed some tests
before being fully usable.
Basically this regexp only filter simple quote when they're not already escaped...

Now that I've my filter for quotes and I need a filter for newline characters.
Everytime you find some newlines in your log files, you can be sure that many tools already in your network will not treat them efficiently, worse this may break everything after ...

So I came across this regexp:

re:replace( Bin, "[\\n\\r]+", " ", [ global ] ).

Finally having two filter functions for every line, I wanted to be able to add or remove easily any function.
You can reference functions with this notation:

fun filterquotes/1

Or a list of functions:

[ fun filterquotes/1, fun filternewlines/1 ]

With this notation and the famous 'lists:foldl', I can filter a line with many functions quite nicely:

% Data is the accumulator, but we don't change it :)
        lists:foldl( fun( Fun, Data ) ->
                         Fun(Data) 
        end, Bin, [ fun filterquotes/1, fun filternewline/1 ]).

Here's the code:

filter( Bin ) when is_atom(Bin) ->
        Bin;
filter( Bin ) when is_integer(Bin) ->
        Bin;
filter( Bin ) ->
        lists:foldl( fun( Fun, Data ) ->
                         Fun(Data) 
        end, Bin, [ fun filterquotes/1, fun filternewline/1 ]).

filterquotes(Bin) ->
        re:replace( Bin, "(?<!\\\\)'", "\\\\'", [ global ] ).

filternewline( Bin ) ->
        re:replace( Bin, "[\\n\\r]+", " ", [ global ] ).

Reading an openssl .priv.key file and extracting the key

2010-02-23T22:23:26+00:00

Extracting the private key from a .priv.key file is simple.
The private key is encrypted using a AES-128 with your passphrase.

The initial vector is also stored in the file, you can extract it directly from the first line:

get_salt( "Salted__", Salt:8/binary, Rest/binary>> ) ->
        {Salt, Rest}.

The last part is handled by some md5() of your passphrase and the initial vector:

Key = crypto:md5([ Password, Salt ]),
        IV = crypto:md5([ Key, Password, Salt ]),
        crypto:aes_cbc_128_decrypt( Key, IV, Rest).

Now the full module:

-module(priv_key).

-compile(export_all).

priv_key_file( File, Password ) ->
        {ok, Bin} = file:read_file(File),
        {Salt, Rest} = get_salt(Bin),
        Key = crypto:md5([ Password, Salt ]),
        IV = crypto:md5([ Key, Password, Salt ]),
        crypto:aes_cbc_128_decrypt( Key, IV, Rest).
        
get_salt( "Salted__", Salt:8/binary, Rest/binary>> ) ->
        {Salt, Rest}.

London Tech Meetups

2010-02-23T12:25:35+00:00

It can be quite hard to find groups in your area that share your interest, and even if you find out that they exist it can be pretty hard to work out when the next meetup is. Some of the meetup groups have frankly mental scheduling rules.

John Sutherland has solved this problem for Edinburgh tech community meetups by listing when various groups meet and who they'd be of interest to. After asking and receiving permission I've duplicated this idea for London tech community meetups.

If your meetup isn't listed and you'd like it to be, email me at craig@barkingiguana.com and let me know.

22 February 2010: Call for Summer 2010 Internships!

2010-02-22T12:16:21+00:00

Erlang Solutions regularly provides summer internships for qualified IT students. Many former interns have said that it was the best learning experience they have ever had. We are currently interviewing for this year's summer internships.

We are located in the heart of London. See contact us section. Applicants from all over the world are encouraged to seek the positions (we have 9 nationalities represented in our offices) but to apply, you have to be eligible to work in the EU.

The Internships run from June 1 until the end of August, and can continue through September, depending upon student availability and interest.

For more information and to apply, please visit our Jobs section here.

DSL : Grow your syntax on top of a clean semantic model

2010-02-22T10:59:16+00:00

A DSL primarily has two components - a semantic model that abstracts the underlying domain and a linguistic abstraction on top that speaks the dialect of the user. The semantic model is the model of the domain where you can apply all the principles of DDD that Eric Evans espouses. And the linguistic abstraction is a thin veneer on top of the underlying model. The more well abstracted your model is, easier will be the construction of the layer on top of it. Here's a general architecture of a DSL engineering stack :-

It's interesting to observe that the two components of the stack evolve somewhat orthogonally.

The Semantic Model evolves Bottom Up

The semantic model usually evolves in a bottom up fashion - larger abstractions are formed from smaller abstractions using principles of composition. It can be through composition of traits or objects or it can be through composition of functions as well. How beautiful your compositions can be depends a lot on the language you use. But it's important that the semantic model also speaks the language of the domain.

Here's an example code snippet from my upcoming book DSLs In Action that models the business rules for a trading DSL. When you do a trade on a stock exchange you get charged a list of tax and fee components depending on the market where you execute the trade. The following snippet models a business rule using Scala that finds out the list of applicable tax/fee heads for a trade ..

class TaxFeeRulesImpl extends TaxFeeRules {
  override def forTrade(trade: Trade): List[TaxFee] = {
    (forHKG orElse 
       forSGP orElse 
         forAll)(trade.market)
  }
   
  val forHKG: PartialFunction[Market, List[TaxFee]] = { 
    case HKG => 
      // in real life these can come from a database
      List(TradeTax, Commission, Surcharge)
  }
    
  val forSGP: PartialFunction[Market, List[TaxFee]] = {
    case SGP => 
      List(TradeTax, Commission, Surcharge, VAT)
  }
    
  val forAll: PartialFunction[Market, List[TaxFee]] = {
    case _ => List(TradeTax, Commission)
  }

  //..
}

The method forTrade clearly expresses the business rule, which reads almost as expressive as the English version ..

"Get the Hong Kong specific list for trades executed on the Hong Kong market OR Get the Singapore specific list for trades executed on the Singapore market OR Get the most generic list valid for all other markets"

Note how Scala PartialFunction s can be chained together to give the above model an expressive yet succinct syntax.

The Language Interface evolves Top Down

Here you start with the domain user. What dialect does he use on the trading desk ? And then you try to build an interpreter around that which uses the services that the semantic model publishes. I call this thin layer of abstraction a DSL Facade that sits between your DSL script and the underlying domain model and acts as the glue.

It also depends a lot on the host language as to how you would like to implement the facade. With a language like Lisp, macros can come in very handy in designing an interpreter layer for the facade. And with macros you do bottom up programming, bending the host language to speak your dialect.

When you are developing an external DSL, the EBNF rules that you specify act as the DSL Facade for growing your syntax. Within the rules you can use foreign code embedding to interact with your semantic model.

In summary, when you design a DSL, the semantic model is as important as the dialect that it speaks. Having a well designed semantic model is an exercise in designing well-engineered abstractions. And as I mention in my book, the four qualities of good abstractions are minimalism, distillation, extensibility and composability.

Code availability on git-hub

2010-02-21T08:54:32+00:00

Bored to switch code between my private subversion repository and git-hub for my open-source projects, I decided to use exclusively git-hub for them. My trac wiki pages are also moved to git-hub for improve the documentation of those projects. Selenium-RC Erlang binding Rake tasks to build erlang code Small mock library for erlang (Please don’t use mock to test [...]

About Me

2010-02-20T04:53:36+00:00

All about me. Me me me me me.

Vitals:
Birth date - October 24, 1973
Height - 6'1"
Eyes - brown
Hair - dark brown
Complexion - fair and freckly
Brain - wrinkly
Spouse - Laura Ann Katz (maiden name Toenjes)
Kids - Gwendolyn, Roseanna and Zack
Religion - unapologetically atheist

I live in Piedmont CA.

I am the creator of CouchDB.

I am the CEO of Couchio.

My good points:
I am very devoted, loving and supportive of my wife and daughters. I have a very positive attitude and am energetic. I am smart and a fast learner, especially when it comes to math, science and engineering stuff. I love solving really hard problems. I am a hard worker, and I am very passionate about my work. I am very honest. I like to make lots of jokes, sometimes they are even funny. I take good care of myself and exercise regularly. I am athletic. I read constantly (always non-fiction). I have good eye for aesthetics. I am good with my hands and like to make things. I like to bake. I generally try not to take life too seriously.

My bad points:
I am egotistical, judgmental and sometimes have a bad temper (but I'm working on those things). I am vain and self conscious. I don't take very good care of my teeth (but I do floss regularly: once every 4 months, whether I need it or not). I am messy and disorganized. I have a short attention span in meetings and lectures. I take my work too seriously. I am brutally honest (but I'm convinced it's not really a bad point). My self-created list of bad points is oddly short, hmmmm.

My weird points:
While I am pretty athletic (strong, fast, can jump, etc), I'm physically uncoordinated and hence suck at nearly every sport (except basketball, I'm decent at that). I am somewhat shy around new people, sometimes it comes off as being as being a snob. I spend waaay too much time in front of the computer, but that's the way I like it. I drink lots of diet soda, lots. I can't whistle. Sometimes my feet smell like hot buttered popcorn. I once was the proprietor of liquidpoop.com -- the internet's largest repository of diarrhea-related poetry. Although I'm vain, my wardrobe mostly consists of stained, freebie t-shirts and a few pairs of jeans.

Migrating Notes/Domino to CouchDB

2010-02-19T23:17:26+00:00

I've been talking to some IT shops who are migrating away from Lotus Notes and Domino. More than wanting to use something else, it seems the reasons they are migrating away is that people in management just don't like Lotus Notes.

The problem these IT shops face is they don't have many good options to migrate to. Sharepoint works for very generic collaborative activities, but not so much for custom business apps that are so pervasive in large Notes installs. That doesn't mean people don't try.

So these IT shops are interested in migrating to Apache CouchDB for obvious reasons. CouchDB is largely inspired by the Lotus Notes backend. CouchDB has a document database, peer based replication, views, full text indexing add-ons, security, and HTTP client access. It also has a very active and growing open source community.

So in theory, migrating to CouchDB from Notes/Domino should be easier than migrating to any other technology. But what we don't yet have is many tools, documentation and examples for migrating from Notes to CouchDB.

Anyone out there have experience with this type of thing for Notes? Anyone interested in helping us develop the tools and documentation for migrating to CouchDB? There is a lot of business opportunity here, and we are looking for partners and VARs to help customers with tools and migrations. If interested, have some ideas or feedback, feel free to email me at damien@couch.io.

Why I don't like ActiveRecord for Domain Model Persistence

2010-02-15T11:09:51+00:00

When it comes to a rich domain modeling, I am not a big fan of the ActiveRecord model. The biggest problem that it entails is invasiveness - the persistence model invades into my domain model. And this was also my first reaction to the Lift-CouchDB integration module which was released recently.

The moment I say ..

class Person extends CouchRecord[Person] {
  //..
}

my domain model becomes tied to the persistence concerns.

When I started scouchdb, my very first thought was to make it non-invasive. The Scala objects must be pure and must remain pure and completely oblivious of the underlying persistence model. In the age of polyglot persistence there is every possibility that you may need to persist a domain model across multiple storage engines. I may be using a JPA backed relational store as the enterprise online database, which gets synchronized with some offline processing that comes from a CouchDB backend. I need the domain model persistence in both my Oracle and my CouchDB engines. The ActiveRecord pattern is difficult to scale to such requirements.

Here's what I implemented in scouchdb ..

// Scala abstraction : pure
case class ItemPrice(store: String, item: String, price: Number)

// specification of the db server running
val couch = Couch("127.0.0.1")
val item_db = Db("item_db")

// create the database
couch(item_db create)

// create the Scala object : a pure domain object
val s = ItemPrice("Best Buy", "mac book pro", 3000)

// create a document for the database with an id
val doc = Doc(item_db, "best_buy")

// add
couch(doc add s)

// query by id to get the id and revision of the document
val id_rev = couch(item_db by_id "best_buy")

// query by id to get back the object
// returns a tuple3 of (id, rev, object)
val sh = couch(item_db by_id("best_buy", classOf[ItemPrice]))

// got back the original object
sh._3.item should equal(s.item)
sh._3.price should equal(s.price)

It's a full cycle session of interaction with the CouchDB persistence engine without any intrusion into the domain abstraction. I am free to use ItemPrice domain abstraction for a relational storage as well.

CouchDB offers a model of persistence where the objects that we store should be close to the granularity of domain abstractions. I should be able to store the entire Aggregate Root of my model components directly as JSON. ActiveRecord model offers a lower level of abstraction and makes you think more in terms of persistence of the individual entities. The thought process is so relational that you ultimately end up with a relational model both in terms of persistence and domain. With CouchDB you need to think in terms of documents and views and NOT in terms of relations and tables. I blogged on this same subject some time back.

The philosophy that I adopted in scouchdb was to decouple the domain entities from the persistence layer. You hand over a pure Scala object to the driver, it will extract a JSON model from it and write it to CouchDB. I use sjson for this serialization. sjson works totally based on reflection and can transparently serialize and deserialize Scala objects that you hand over to it. From this point of view, the three aspects of managing domain abstractions, JSON serialization and persistence into CouchDB are totally orthogonal. I think this is difficult to get with an ActiveRecord based model.

Wanted: Hosting/Infrastructure Engineer

2010-02-12T00:18:26+00:00

We are looking to hire someone with strong hosting or infrastructure experience to help us develop a CouchDB hosting platform.

Experience with CouchDB is nice but not necessary. We most want someone who is passionate about infrastructure with large scale hosting experience and with open source contributions.

We offer competitive pay, stock options, great health benefits and 6 weeks vacation, and an opportunity to change the world.

Email inquiries to damien@couch.io

First week in the new office

2010-02-09T00:40:32+00:00

Last week was our first week in our new office in Old Downtown Oakland. It's a really neat area with lots of restaurants and bars, and hardly any murders.

Oh yeah, we've changed our name to Couchio. Our new blog will be here http://blog.couch.io/ soon.

Our office:

Our office manager Claire:

Chris and Mikeal:

Claire and Jan:

Nitin:

Super Awesome Art by Julie Armbruster:

Me:

My Office:

So far we are really disorganized and discombobulated. But I'm are working on it! I even bought Management for Dummies. Things will be running smoothly in no time ;)

Also we are looking hard for someone to help us offer CouchDB support and hopefully build a whole support organization. Email me damien@couch.io if you are interested or know someone who is.

Scala Self-Type Annotations for Constrained Orthogonality

2010-02-08T11:58:15+00:00

I talked about orthogonality in design in one of my earlier posts. We had a class Address in Scala and we saw how we can combine it with other orthogonal concerns without polluting the core abstraction. We could do this because Scala offers a host of capabilities to compose smaller abstractions and build larger wholes out of them. A language is orthogonal when it allows such capabilities of composition without overlaps in functionalities between the composing featuresets.

The design of Scala offers many orthogonal features - I showed some of them in my earlier post. The power of mixins for adding orthogonal features ..

val a = new Address(..) with LabelMaker {
  override def toLabel = {
    //..
  }
}

and the power of Scala views with implicits ..

object Address {
  implicit def AddressToLabelMaker(addr: Address) = new LabelMaker {
    def toLabel =
    //..
  }
}

Here Address and LabelMaker are completely unrelated and offers truly orthogonal capabilities when mixed in. However there can be some cases where the mixins themselves are not completely orthogonal to the core abstractions, but really optional extensions to them. In fact the mixins implement some functionalities that may depend on the core abstraction as well. Let's see yet another feature of the Scala type system that makes this modeling wholesome.

Consider the following abstraction for a security trade that takes place in a stock exchange ..

// details ellided for clarity
case class Trade(refNo: String, account: String, 
                 instrument: String, quantity: Int,
                 unitPrice: Int) {
  // principal value of the trade
  def principal = quantity * unitPrice
}

For every trade executed on the exchange we need to have a set of tax and fees associated with it. The exact set of tax and fees depend on a number of factors like type of trade, instruments traded, the exchange where it takes place etc. Let's have a couple of tax/fee traits that model this behavior ..

trait Tax { 
  def calculateTax = //..
}
  
trait Commission { 
  def calculateCommission = //..
}

In the above definitions both methods calculateTax and calculateCommission depends on the trade being executed. One option is to keep them abstract in the above trait and provide their implementations after mixing in with Trade ..

val t = new Trade(..) with Tax with Commission {
  // implementations
  def calculateTax = principal * 0.2
  def calculateCommission = principal * 0.15
}

I did it at the instance level. You can very well use this idiom at the class level and define ..

class RichTrade extends Trade with Tax with Commission {
  //..
}

However the above composition does not clearly bring out the fact that the domain rules mandate that the abstractions Tax and Commission should be constrained to be used with the Trade abstraction only.

Scala offers one way of making this knowledge explicit at the type level .. using self-type annotations ..

trait Tax { this: Trade =>
  // refers to principal of trade
  def calculateTax = principal * 0.2
}
  
trait Commission { this: Trade =>
  // refers to principal of trade
  def calculateCommission = principal * 0.15
}

The traits are still decoupled. But using Scala's self type annotations you make it explicit that Tax and Commission are meant to be used *only* by mixing them with Trade.

val t = new Trade(..) with Tax with Commission
t.calculateTax
t.calculateCommission

Can I call this constraining the orthogonality of abstractions ? Tax and Commission provide orthogonal attributes to Trade optionally and publish their constraints explicitly in their definitions. It's not much of a difference from the earlier implementations. But I prefer to use this style to make abstractions closer to what the domain speaks.

02 February 2010: Erlang Solutions at QCon London 2010

2010-02-02T15:16:25+00:00

Erlang Solutions Ltd. sponsors QCon London 2010, hosts the track, gives the talks and organises Erlang User Group meeting there.

Qcon is an annual London enterprise software development conference designed for team leads, architects and project management. It gathers Java, .NET, Ruby, SOA, Agile, Erlang and architecture communities.

Erlang Solutions will be present at QCon on Friday, 12th March 2010, when Ulf Wiger (CTO of Erlang Solutions) will be hosting the Concurrency Challenge track. He will also give an introductory talk The Concurrency Challenge at 10:20 and a presentation on Death by accidental complexity at 4:30 pm.

Also, Francesco Cesarini (founder of Erlang Solutions) will be giving two tutorials: "Practical Erlang Programming" on Monday, 8th March and "Erlang/OTP System Principles" on Tuesday, 9th March. These are an all- day events, will start at 9:00 and finish at 16:00.

A day before, on 11th March 2010 from 18:30 until 20:30 Erlang Solutions organises special free London Erlang User Group Meeting at QCon. Everyone is welcome, even if you do not participate in QCon London 2010, just register here.There will be four talks:

Francesco Cesarini presents "Erlang community around the world"
Ulf Wiger talks about "Erlang in the Clouds"
Justin Sheehy will present "Introduction to RIAK" &
Joe Armstrong talks about "Erlang Libraries"

QCON 2010 will be held in The Queen Elizabeth II Conference Centre, in London from 8th until 12th March 2010.

See you there!

Tutorials: March 8-9, 2010

Conference: March 10-12, 2010

When registering for the QCon, use the Discount Code "erlangug" and save £50 off the price!

RabbitMQ-shovel: Message Relocation Equipment

2010-02-01T12:13:52+00:00

In several applications, it’s very useful to be able to take messages out of one RabbitMQ broker, and insert them into another. Many people on our mailing list have being asking for such a shovel, and we’ve recently been able to devote some time to writing one. This takes the form of a plugin for Rabbit, and whilst it hasn’t been through QA just yet, we’re announcing it so people who would like to play and even suggest further features for inclusion can do so sooner rather than later.

The shovel is written on top of the Erlang client. It supports both direct and network connections to nodes, SSL support, the ability to declare resources on nodes it connects to, basic round-~~robin~~rabbit balancing of both source and destination nodes, and allows you to configure many parameters controlling how messages are consumed from the source, and how they’re published to the destination. Multiple shovels can be specified, their statuses queried, and shovels can repeatedly reconnect to nodes in the event of failure.

The plugin is available from http://hg.rabbitmq.com/rabbitmq-shovel/, and is released under the MPL v1.1. There is a README included which contains full documentation. This is replicated below.

RabbitMQ-shovel

Introduction

This is a plug-in for RabbitMQ that shovels messages from a queue on one broker to an exchange on another broker. The two brokers may be the same. The plug-in allows several shovels to be specified at the same time. Each shovel may have a number of source and destination brokers specified, and one of each is chosen whenever the shovel attempts to make a connection: this permits simple round-rabbit load balancing.

Resources can be declared upon connection to both the source and destination brokers, and parameters can be specified for both the reception and publishing of messages.

Requirements

Currently, you must build the server from source, under branch bug16653. You must also have checked out the rabbitmq-public-umbrella hg repository, and have the rabbitmq-erlang-client built. From scratch, the following commands should build RabbitMQ with the shovel plug-in:

hg clone http://hg.rabbitmq.com/rabbitmq-public-umbrella
cd rabbitmq-public-umbrella
hg clone http://hg.rabbitmq.com/rabbitmq-codegen
hg clone http://hg.rabbitmq.com/rabbitmq-erlang-client
hg clone http://hg.rabbitmq.com/rabbitmq-server
hg clone http://hg.rabbitmq.com/rabbitmq-shovel
cd rabbitmq-server
hg up -C bug16653
make -j
mkdir -p plugins
cd plugins
ln -s ../../rabbitmq-erlang-client
ln -s ../../rabbitmq-shovel
cd ../../rabbitmq-erlang-client
make
cd ../rabbitmq-shovel
make
cd ../rabbitmq-server
./scripts/rabbitmq-activate-plugins
make cleandb run

Configuration

The RabbitMQ configuration file specifies the shovel configurations. This exists by default, in /etc/rabbitmq/rabbitmq.config under Linux systems, %RABBITMQ_BASE%\rabbitmq.config under Windows or somewhere else under OS X. This file configures both RabbitMQ-server and all the plugins installed in it. It is an Erlang-syntax file of the form:

[{section1, [section1-config]},
 {section2, [section2-config]},
 …
 {sectionN, [sectionN-config]}
].

thus a list of tuples, where the left element of each tuple names the applications being configured. Don’t forget the last element of the list doesn’t have a trailing comma, and don’t forget the full-stop is needed after closing the list. Hence if you configure RabbitMQ-server and the RabbitMQ-shovel, then the configuration file may have a structure like this:

[{rabbit,        [configuration-for-RabbitMQ-server]},
 {rabbit-shovel, [configuration-for-RabbitMQ-shovel]}
].

A full example of the shovel configuration is:

{rabbit_shovel,
  [{shovels,
    [{my_first_shovel,
      [{sources,      [{brokers,
                          ["amqp://fred:secret@host1.domain/my_vhost",
                           "amqp://john:secret@host2.domain/my_vhost"
                          ]},
                       {declarations,
                          ['queue.declare',
                           {'queue.bind',
                                  [{exchange, "my_exchange">>},
                                   {queue,    >}]}
                          ]}]},
       {destinations, [{broker, "amqp://"},
                       {declarations,
                          [{'exchange.declare',
                                  [{exchange, "my_exchange">>},
                                   {type, "direct">>},
                                   durable]}
                          ]}]},
       {queue, >},
       {qos, 10},
       {auto_ack, false},
       {tx_size, 0},
       {delivery_mode, keep},
       {publish_fields, [{exchange, "my_exchange">>},
                         {routing_key, "from_shovel">>}]},
       {reconnect, 5}
      ]}
     ]
   }]
}


Firstly, all shovels are named. Here we have one shovel, called
‘my_first_shovel’. We can have multiple shovels if you wish. Every
shovel must have all sub-fields specified: sources, destinations, qos,
auto_ack, delivery_mode, publish_fields, reconnect.

Sources and Destinations

Sources and destinations specify respectively where messages are
fetched from and delivered too. One of ‘broker’ and ‘brokers’ must be
specified, and ‘broker’ is simply shorthand for when only one broker
needs specifying. Using ‘brokers’ allows a list of brokers to be
specified: whenever the connection to a broker is lost, another one is
chosen at random from the list and a connection attempt is made to
that. The syntax for broker URIs is:

amqp://username:password@host:port/vhost

If username or password are omitted, the default values of guest and
guest are used. If the vhost is omitted, the default value of / is
used. If the host is omitted, then the plugin uses the “direct”
connection internally rather than a network connection: this means it
connects to the RabbitMQ-server node on which it is running without
going through the network stack. This is much more efficient. If port
is omitted then the default value is used (5672 or 5671 if SSL is
used).

SSL is implemented, for which additional parameters are needed:

amqps://username:password@host:port/vhost?cacertfile=/path/to/cacert.pem&certfile=/path/to/certfile.pem&keyfile=/path/to/keyfile.pem&verify=verifyOption&fail_if_no_peer_cert=failOption

(note, this is a single line)

All five parameters (3 paths: cacertfile, certfile and keyfile; 2
options: verify, fail_if_no_peer_cert) must be specified. See the SSL
guide at http://www.rabbitmq.com/ssl.html#configure-erlang for details
of SSL in RabbitMQ in general and specifically for the Erlang client
(on which the shovel is built).

Note that SSL cannot be used with the direct connection (i.e. a host
must be specified when using SSL), and that it is preferable to use
the non-SSL direct connection when connecting to the same node that’s
running the shovel.

Resource Declarations

Both sources and destinations can have an optional ‘declarations’
clause. The value of this is a list, consisting of AMQP Methods. If
default values are sufficient, then the method name alone can be
specified - e.g. ‘queue.declare’. If parameters need to be set then
the method should be given as a tuple, with the right hand side a
proplist specifying which fields need altering from their default
values. E.g:

    {'exchange.declare',[{exchange, "my_exchange">>},
                         {type, "direct">>},
                         durable]},


One very useful feature here is the Most-Recently-Declared-Queue
feature, in which RabbitMQ remembers the name of the most recently
declared queue. This means that you can declare a private queue, and
then bind it to exchanges without ever needing to know its name.

queue :: binary

This parameter specifies the name of the queue on the source brokers
to consume from. This queue must exist. Use the resource declarations
to create the queue (or ensure it exists) first. Note again that the
Most-Recently-Declared-Queue feature can be used here, thus an
anonymous queue can be used: use <<>> to indicate the
Most-Recently-Declared-Queue.

qos :: non-negative-integer

The shovel consumes from a queue. The QoS controls how many messages
are sent to the shovel in advance of the message the shovel is
currently processing.

auto_ack :: boolean

Setting this to ‘true’ turns on the no_ack flag when subscribing to
the source queue.

tx_size :: non-negative-integer

When set to 0, transactions are not used. Other values make publishes
transactional, with a commit every N messages. In lieu of the auto-ack
option, when transactions are not used, messages are acknowledged to
the source immediately after every publish. When transactions are
used, acks are only issued to the source on receipt of the commit-ok
message from the destination. This can thus be used to guarantee that
messages are only acknowledged (and thus forgotten about by the source
broker) when they are guaranteed to have been received by the
destination broker.

delivery_mode :: ‘keep’ | 0 | 2

This affects the delivery_mode field when publishing to the
destination. A value of ‘keep’ means that the same delivery_mode
should be used as when the message was originally published to the
source broker. 0 and 2 override the original setting.

publish_fields

This is a list of tuples which override fields in the publish method
when publishing to the destination. This can be used to direct
messages to a particular exchange on the destination, for example, or
change the routing key. By default, the routing key of the message as
it is received by the shovel is passed through, but this can be
overridden as necessary.

reconnect :: non-negative-integer

When an error occurs, the shovel will disconnect from both the source
and destination broker immediately. This will force uncommitted
transactions at the destination to be rolled back, and delivered but
unacknowledged messages from the source to be requeued. The shovel
will then try connecting again. If this is unsuccessful, then it’s not
a good idea for the shovel to very quickly and repeatedly try to
reconnect. The value specified here is the number of seconds to wait
between each connection attempt.

Note that if set to 0, the shovel will never try to reconnect: it’ll
stop after the first error.

Obtaining shovel statuses

From the broker Erlang prompt, call
rabbit_shovel_status:status(). This will return a list, with one row
for each configured shovel. Each row has three fields: the shovel
name, the shovel status, and the timestamp (a local calendar time of
{{YYYY,MM,DD},{HH,MM,SS}}). There are 3 possible statuses:


’starting’: The shovel is starting up, connecting and creating
resources.
‘running’: The shovel is up and running, shovelling messages.
{’terminated’, Reason}: Something’s gone wrong. The Reason should give
a further indication of where the fault lies.

Erlang Factory SF Bay Area 2010

2010-01-30T21:29:40+00:00

Interested in Erlang? You might consider attending Erlang Factory SF Bay Area 2010. Below is a message that Francesco Cesarini, conference organizer and co-author of the most excellent book Erlang Programming, sent to the erlang-questions list yesterday providing more information about the conference, especially pointing out that the Very Early Bird registration deadline is tomorrow (Sunday January 31). Hope to see you there!

Hi All,

a note to say that we are almost done with the programme for the 2010 SF Bay Area Erlang Factory. This year, we are lucky to have keynote speakers such as Joe Armstrong, Bjarne Dacker, Kenneth Lundin and Steve Vinoski. They will be giving four of the 35 scheduled talks on the 25th and 26th of March in the San Francisco Bay Area. The almost complete programme is available here:

http://erlang-factory.com/conference/SFBay2010/programme

The conference will be preceded by three days of University courses taught by experts such as Simon Thompson, John Hughes, Thomas Arts, Henry Nystrom and Kevin Smith. Come and learn Erlang, OTP, QuickCheck or Web Development with Erlang. More information on the courses are here:

http://erlang-factory.com/conference/SFBay2010/university

This *Sunday* (January 31st) is the deadline for the very early bird deadline. Register by Sunday night and save $400 on the on-site registration price.

The conference hotel and venue is the SF Airport Hilton, a short BART / Caltrain ride from SF and the Valley. We have secured a very competitive price of US$109 per room and night at the conference hotel, this being one of the reasons for us choosing it. The other is the lower price of the venue, allowing us to pass on the savings to the delegates through a higher very early bird discount. We are planning an ErlLounge open to everyone who can’t make the two days on the 25th, and hope we will be able to surpass last year’s success. If you have thoughts or questions, you are welcome to drop me a line.

Hope to see you all there!

Francesco
–
http://www.erlang-solutions.com

Erlang Factory SF Bay 2010 – Francesco Cesarini on the Conference and Old-School vs New-School Erlangers

2010-01-30T20:03:58+00:00

The Erlang Factory’s 2010 conference is March 25th and 26th in San Francisco, with the university three days before, starting the 22nd. The conference is at the Hilton San Francisco Airport, with three tracks, each on a different theme. The number of tracks gives attendees an unusually broad set of choices for talks. When attending [...]

Plugin exchange types for RabbitMQ

2010-01-22T14:38:54+00:00

An obvious extension point for an AMQP broker is the addition of new types of exchange. An exchange type essentially represents an algorithm for dispatching messages to queues, usually based on the message’s routing key, given how the queues are bound to the exchange — it’s a message routing algorithm.

At a minimum, supporting new exchange types requires only some scaffolding to plug in to (an exchange type registry) and a hook for routing messages. However, this wouldn’t support some more interesting use cases, and in particular it didn’t support our motivating use case. Exchange types that want to keep their own state need to be initialised, and be notified about other lifecycle events.

The ~~branch bug22169~~ default branch of RabbitMQ supports plugin exchange types, by providing a behaviour for exchange type modules to implement, and an exchange type registry to map a module to a type (i.e., what the client supplies in the type field of exchange.declare).

The behaviour requires exported hooks for validating exchange declarations, creating exchanges, recovering durable exchanges, publishing to an exchange (this is where the routing comes in), maintaining bindings, and deleting an exchange. RabbitMQ continues to maintain the database of exchanges and bindings, and calls the hooks after it’s done its own bookkeeping.

For simplicity, the hooks are not called atomically with the bookkeeping; so, it is possible for instance to publish to a new exchange for which the hook has not completed. However, provided there are no asynchronous operations in the hook implementation, the hook will have completed by the time the OK message is sent to the client. This is no more racey than AMQP itself, with consistency at the channel level where operations follow a single thread of control.

Here’s an example that simply io:formats things as they happen:

-module(rabbit_exchange_type_debug).
-include("rabbit.hrl").

-behaviour(rabbit_exchange_type).

-export([description/0, publish/2]).
-export([validate/1, create/1, recover/2, delete/2, add_binding/2, remove_bindings/2]).
-export([register_debug_types/0]).
-include(”rabbit_exchange_type_spec.hrl”).

-rabbit_boot_step({debug_exchange_types, 
                   [{description, "debugging exchange types"},
                    {mfa, {?MODULE, register_debug_types, []}},
                    {requires, rabbit_exchange_type_registry},
                    {enables, exchange_recovery}]}).

description() ->
    [{name, <<"debug">>},
     {description, <<"Debugging exchange">>}].

backing_module(#exchange{ type = Type }) ->
    %% Presume that Type is EITHER one of the standard types –
    %% i.e,. that we have been registered this module as direct,
    %% topic, fanout or match — or, for testing purposes, it’s
    %% registered (as in the boot steps above) as debug_direct, etc.
    Type1 = case atom_to_list(Type) of
                “x-debug-” ++ T -> T; 
                              T -> T
            end,
    list_to_existing_atom(”rabbit_exchange_type_” ++ Type1).

publish(Exchange, Delivery) ->
    io:format(”Publish ~p to ~p~n”, [Delivery, Exchange]),
    Module = backing_module(Exchange),
    Module:publish(Exchange, Delivery).

validate(X) ->
    io:format(”Validate ~p~n”, [X]),
    (backing_module(X)):validate(X).

create(X) ->
    io:format(”Create ~p~n”, [X]),
    (backing_module(X)):create(X).

recover(X, Bs) ->
    io:format(”Recover ~p with bindings ~p~n”, [X, Bs]),
    (backing_module(X)):recover(X, Bs).

delete(X, Bs) ->
    io:format(”Delete ~p with bindings ~p~n”, [X, Bs]),
    (backing_module(X)):delete(X, Bs).

add_binding(X, B) ->
    io:format(”Add binding ~p to ~p~n”, [B, X]),
    (backing_module(X)):add_binding(X, B).

remove_bindings(X, Bs) ->
    io:format(”Delete bindings ~p from ~p~n”, [Bs, X]),
    (backing_module(X)):remove_bindings(X, B).

register_debug_types() ->
    lists:foreach(
      fun (T) ->
              rabbit_exchange_type_registry:register(T, ?MODULE)
      end,
      [<<"x-debug-direct">>,
       <<"x-debug-topic">>,
       <<"x-debug-fanout">>,
       <<"x-debug-headers">>]).

Bit by important bit:

-behaviour(rabbit_exchange_type).
-export([description/0, publish/2]).
-export([validate/1, create/1, recover/2, delete/2, add_binding/2, remove_bindings/2]).

rabbit_exchange_type specifies these exported callbacks.

-include("rabbit_exchange_type_spec.hrl").

This include has the specs for each of the exported functions, if you’re using specs.

-rabbit_boot_step({debug_exchange_types, 
                   [{description, "debugging exchange types"},
                    {mfa, {?MODULE, register_debug_types, []}},
                    {requires, rabbit_exchange_type_registry},
                    {enables, exchange_recovery}]}).

This uses the new boot sequence mechanism to register the exchange type during boot. The “enables” and “requires” say that the function given as mfa above must be run after the exchange type registry is available, but before any exchanges are recovered.

publish(Exchange, Delivery) ->
    io:format("Publish ~p to ~p~n", [Delivery, Exchange]),
    Module = backing_module(Exchange),
    Module:publish(Exchange, Delivery).

This exchange type simply delegates to a “backing” exchange type.

register_debug_types() ->
    lists:foreach(
      fun (T) ->
              rabbit_exchange_type:register(T, ?MODULE)
      end,
      [<<"x-debug-direct">>,
       <<"x-debug-topic">>,
       <<"x-debug-fanout">>,
       <<"x-debug-headers">>]).

rabbit_exchange_type_registry maintains a registry of type to module; because of this indirection, we can register this module as many different types, then check the declared type of the exchange in our hook to see which type we’re expected to be. Note that the AMQP specification requires the “x-” prefix for non-standard exchange types.

~~This should reach default branch soon after RabbitMQ 1.7.1 is released.~~ This is in RabbitMQ’s default branch, and should be in the next release (1.7.2).

You can drop modules straight into src, but they are better packaged as plugins — follow the drill at the plugin development guide (and note that your plugin may only need to be a library application).

[EDITS: Updated sample code to keep up with name changes, and correct use of list_to_atom to list_to_existing_atom] [Further EDITS: slight API change, now in default branch]

A new way to think of Data Storage for your Enterprise Application

2010-01-21T19:24:15+00:00

A couple of posts earlier I had blogged about a real life case study of one of our projects where we are using a SQL store (Oracle) and a NoSQL store (MongoDB) in combination over a message based backbone. MongoDB was used to cater to a very specific subset of the application functionality, where we felt it made a better fit than a traditional RDBMS. This hybrid architecture of data organization is turning out to be an increasingly attractive option today with more and more specialized persistent storage structures being developed.

In many applications we need to process graph data structures. Neo4J can be a viable option for this. You can have your mainstream data storage still in an RDBMS and use Neo4J only for the subset of functionalities for which you need to use graph data structures. If you need to sync back to your main storage, use messaging as the transport to talk back to your relational database.

Multiple data storage use along with asynchronous messaging is one of the options that will looks very potent today. Drizzle has its entire replication based on a RabbitMQ based transport. And using AMQP messaging, Drizzle replicates data to a host of key/value stores like Voldemort, memcachedDB and Cassandra.

If we agree that messaging is going to be one of the most dominant paradigms in shaping application architectures, why not try to go one level up and look at some higher level abstractions for message based programming? Erlang programmers have been using the actor model for many years now and have demonstrated all the good qualities that the model imbibes. Inspired by Erlang, Scala also offers a similar model on the JVM. In an earlier post I had discussed how we can use the actor model in Scala to scale out messaging applications using a RabbitMQ storage.

Now with the developing ecosystem of polyglot storage, we can use the same model of actor based communication as the backbone for integrating multiple data storage options that you may plug in to your application. Have specific clients front end the storage that they need to work with and use messaging to sync that up with the main data storage backend to have a consistent system of record. Have a look at the following diagram that may not look that unreal today. You have a host of options that bring your data closer to the way you process them in your domain model, be it document oriented, graph based, key/value based or simple POJO based across a data grid like Terracotta.

When we have a bunch of architectural components loosely connected through messaging infrastructure, you can have a world of options managing interactions between them. In fact your options open up more when you get to interact with data shaped the way you would like to be. You now can think in terms of having a data model aligned with the model of your domain. You know once your rule base gets updated in Neo4J, it will somehow be synced up with the backend storage through some other service that will make it eventually consistent.

In a future post I will explore some of the options that a higher order middleware service like Akka can add to your stack. With Akka providing abstractions like transactors, pluggable persistence and out of the box integration modules for AMQP, there's a number of ways you can think of modularizing your application's domain model and storage. You can use peer to peer distributed actor based communication model that sets up synchronization options with your databases or you can use AMQP based transport to do the same much like what Drizzle does for replication. But that's some food for thought for yet another future post.

[ANN] ejabberd 2.1.2 bugfix release

2010-01-18T16:08:17+00:00

We are pleased to announce ejabberd 2.1.2, which contains several bugfixes over last month's maintenance release.

Brief summary of changes:

Fix SASL PLAIN authentication message for RFC4616 compliance
Fix support for old Erlang/OTP R10 and R11
If server start fails due to config file problem, display some lines and stop node
PubSub and PEP: several improvements and bugfixes
WebAdmin: fix offline message displaying
When server stops with new stop_kindly command: inform users, clients, MUC

Check the Release Notes for a more complete list of changes:
http://www.process-one.net/en/ejabberd/release_notes/release_note_ejabberd_2.1.2

If you upgrade from ejabberd 2.0.5 or older, read carefully the release notes of ejabberd 2.1.0 too, because there were several changes in the installation path and the configuration options.

The list of solved tickets since previous version is available on ProcessOne bug tracker:
http://redir.process-one.net/ejabberd-2.1.2

ejabberd 2.1.2 is available as source code package and binary installers for Linux 32 bits, 64 bits, Mac OS X Intel, and Windows:
http://www.process-one.net/en/ejabberd/downloads

My First Arduino Hack

2010-01-16T16:58:53+00:00

Yay! I got an Arduino as a present! So what do you do with it? I'm at DorkBotPDX and I'd just got as far as turning on an LED with a push button, and Ward suggested I try getting it to echo back to my a rhythm of clicks. I got that working, and then starting looking around in my starter kit. I had a speaker! Here's my circuit diagram and the code to go with it:

(The mystery component on the left is the buzzer - Fritzing didn't have a close match for that part)

The code:

#define BUZZER 3
#define BUTTON 7

int val = 0;
int old_val = 0;

long times[50];
long i = 0;

void setup() {
  pinMode(BUZZER, OUTPUT);
  pinMode(BUTTON, INPUT); 
}

void loop() {
  val = digitalRead(BUTTON);
  if ((val == HIGH) && (old_val == LOW)) {
    times[i] = millis();
    i++;
    analogWrite(BUZZER, 64);
    delay(10);
  }  
  if ((old_val == HIGH) && (val == LOW)) {
    times[i] = millis();
    i++;
    analogWrite(BUZZER, 0);
    delay(10);
  }  
  
  old_val = val;

  if (i > 0 && millis() - times[i-1] > 2000) {
    delay(1000);
    int state = 0;
    for(int y=0; y<i; y++) {
      state = 1 - state;
      analogWrite(BUZZER, 64*state);
      delay(times[y+1]-times[y]);
    }
    i = 0; // reset
    analogWrite(BUZZER, 0);     
  } 
}

Word to the wise: as with coding it pays to take small incremental steps!

The state of Haskell-Torrent

2010-01-16T10:31:46+00:00

Implementing processes

The Haskell Bittorrent project is evolving at a steady state these days. In the last couple of days, we have implemented most of the code relevant for carrying out choking of peers. Choking is the process by which you only communicate to a few peers at a time. Thus TCP/IP congestion can be avoided which drives up the download and upload rates. This post is not on this part however, which must wait a bit.

A fellow dane, Thomas Christensen, heeded my call and did some hlint runs over the code. Hopefully, we’ll see more work from him (github). Alex Mason has added even more parsing stuff through Cereal to the code, fixed a number of bugs in the parser and improved its general state. His venture is described on his blog (here).

Processes

I will be talking about processes in this post. When haskell-torrent started, our processes were simply “things spawned in the IO monad”. The approach works, but it quickly becomes inadequate for several reasons. In Haskell and FP in general, a lot of power stems from the idea that we can write a large set of small building blocks and then compose them together to form increasingly larger and larger blocks as we go. When composing, we use a fairly small number of helpers — readily present in the standard libraries.

When running in IO, we quickly end up with a lot of state. This can of course be passed around by “hand” and tail-calls. Unfortunately, this means we will end up using a lot of our precious coding time doing just exactly that. To optimize, we need a way to reflect away configuration and state when we don’t need it, and reify the information at certain points in the program where it is necessary to know the state.

The ubiquitious tool in Haskell for this are Monads. Not a single monad like IO, but a for-the-case relevant monad built from monad transformers. A couple of days ago, I installed XMonad by Spencer Janssen, Don Stewart, Jason Creighton and many more. This is a window manager — but surprisingly, it needs to solve some problems similar to the one in haskell-torrent. By inspiration from XMonad, we define

> newtype Process a b c = Process (ReaderT a (StateT b IO) c)
>    deriving (Functor, Monad, MonadIO, MonadState b, MonadReader a, Typeable)

That is, a Process is a type. It contains some configuration data a, an internal state b and is in the process of evaluating to a value of type c. We use a Reader transformer so we can ask for configuration data when we need it. We do not expect this configuration data to be altered when the process runs. A State Transformer takes care of the internal state of the process. Finally, we let the underlying monad be IO. This gives us access to CML and the outside world.

We let GHC derive a large set of things automatically (using several extensions in the process). This gives an easier way to manipulate the state when the process is running.

Running a Process is easy:

> runP :: a -> b -> Process a b c -> IO (c, b)
> runP c st (Process p) = runStateT (runReaderT p c) st

which is exactly like in XMonad. Spawning a new process is also fairly easy:

> spawnP :: a -> b -> Process a b () -> IO ThreadId
> spawnP c st p = spawn proc
>   where proc = do runP c st p
>                   return ()

In general different processes will have different configurations a. These will usually contain the channels on which the process can communicate. We then define a type class

> class Logging a where
>   getLogger :: a -> LogChannel
> 
> instance Logging LogChannel where
>   getLogger = id

of types that contain a logger channel. This means we can define a generic log function like this:

> log :: Logging a => String -> Process a b ()
> log msg = do
>     logC <- asks getLogger
>     liftIO $ logMsg logC msg

Type classes are a magnificent tool when we want to coerce a general function on top of different types. Many of our configurations in the client will instance the Logging class and then the log function knows how to access the logger in the Reader.

What does this buy us

The advantage of doing this change on the code is twofold: First, the amount of parameter passing is considerably reduced. Reflection into the monad solves this problem. We are now able to compose easier. Function composition is considerably harder when there are many parameters abound. With the change, preliminary restructuring of the Peer process shows a much simpler flow. Also, there are now ample refactoring opportunities available with the change.

Second, the monad means we can use locality much more to our advantage. A common idiom is to modify the state of the process or to retrieve the current state for query. This now happens locally at the point where it is needed. Before, we might have needed to pass a parameter through several functions and then use it.

What next?

There are a small number of things that needs to be addressed before we can claim that the client is a full bittorrent client:

The client needs to correctly handle the concept of interest. It must tell other clients if it is interested in the pieces they have at their disposal for transmission. I have some preliminary code for doing this.
The client needs to correctly tell the tracker how many bytes it uploaded and downloaded. This measure is needed on many private trackers as they require people to upload data back.
The client needs to be better at choosing the next eligible piece. Choosing one randomly is good enough.
The client needs to handle multi-file torrents. It is not as hard as it may sound — the only part of the system that needs to know about files is the code handling the file system. All other parts can just keep on transferring pieces.
For choking to work correctly, we must know how fast we are currently transferring to a peer. This is an interesting little problem if somebody feels their curiosity tickled :)
We currently take space proportional to torrent size due to our SHA1 calculation being slow and not use a file descriptor. Research into a faster SHA1 library would be really beneficial.
We need to accept incoming connections. The system only connects outward at the moment.

And of course, it needs some testing in a non-lab setting. Currently it can seed and leech, but my setup is very simple: Opentracker and rtorrent on another computer.

The main github repository for haskell-torrent is here.

Minor Erlang Interface Tricks

2010-01-14T12:41:28+00:00

erlrc requires that you drop a file whose name is your node name and whose contents is the node cookie into a particular directory so that the packaging system can find running Erlang VMs and ask them to do hot-code upgrades. It's not that hard, but I figured I would put some shell scripts into the erlrc google code project demonstrating how to do this and other minor tricks that make it a little nicer to talk to an Erlang VM from the (non-Erlang) command line.
All of these scripts are driven by a POSIX shell syntax configuration file describing an Erlang VM, which can be pretty short. Here's one I'm using right now for my personal stuff:

% cat /usr/share/myerlnode/myerlnode.rc
node_name_file='/etc/erlrc.d/nodes/erlang'

run_extra_args='+A 64 -noshell -noinput -s crypto -s mnesia -eval "case erlrc_boot:boot () of ok -> ok; _ -> init:stop () end" >erlang.out 2>erlang.err &'

node_name_file (the only required config setting) defines what the node name will be (via the basename). This config file drives four scripts:

erlstart-run-erlang: starts an Erlang VM. doesn't do that much, really: creates the node file for erlrc, and sets the heart command in case you want to use heart.
erlstart-remsh: starts a remote shell on an Erlang VM.
erlstart-eval: takes the argument, evals it on an Erlang VM, and prints the result to standard out. very useful shell script glue for maintenance scripts.
erlstart-etop: runs etop on an Erlang VM.

If you put your config file in the default location (/etc/erlstart.rc) or export an environment variable indicating the location (ERLSTART_CONFIG_FILE) then things are pretty zero-configuration. Doing this kind of thing at the (non-Erlang) shell gets very addictive:

% erlstart-eval 'application:which_applications()'
[{anwhereos,"Rest API for time series persistence and retrieval.","0.1.0"},
{drurlyjsclientsrv,"Serve the drurly jsclient from the drurly server.",
                "0.0.1"},
{drurly,"Social sharing server.","2.2.0"},
{mcedemo,"TinyMCE + Nitrogen demo.","1.3.0"},
{inets,"INETS  CXC 138 49","5.0.12"},
{mochiweb,"MochiWeb is an Erlang library for building lightweight HTTP servers.",
       "0.2009.05.26"},
{nitrogen,"Nitrogen web framework for Erlang.","0.2009.05.12.3"},
{nitromce,"A Nitrogen element which corresponds to a TinyMCE editor instance.",
       "4.0.1"},
{sgte,"String template language for Erlang.","0.7.1"},
{signzor,"Erlang library to generate signed printable encodings.","0.0.1"},
{tcerl,"Erlang driver for tokyocabinet.","1.3.1h"},
{webmachine,"An Erlang REST framework.","0.2009.09.24b"},
{erlrc,"Extensible application management.","0.2.3"},
{mnesia,"Mnesia storage API extensions.","4.4.7.6.1"},
{crypto,"CRYPTO version 1","1.5.3"},
{sasl,"SASL  CXC 138 11","2.1.5.4"},
{stdlib,"ERTS  CXC 138 10","1.15.5"},
{kernel,"ERTS  CXC 138 10","2.12.5"}]

erlstart-run-erlang is fairly low level, so for an init script I would do something like the following: first, a little stub installed into /etc/rc.d

#! /bin/sh

# chkconfig: 2345 20 80
# description: Control my personal Erlang node.

exec myerlnodectl "$@"

and then the actual guts installed as myerlnodectl

#! /bin/sh

eval_with_main_node () \
{
  erl -name myerlnodetmp$$ \
      -hidden \
      -setcookie "$cookie" \
      -noshell -noinput \
      -eval "MainNode = list_to_atom (\"$1\"), $2" \
      -s erlang halt
}

get_hostname () \
{
  erl -name myerlnodetmp$$ -setcookie $$ -noshell -noinput -eval '
    [ Host ] = tl (string:tokens (atom_to_list (node ()), "@")),
    io:format ("~s~n", [ Host ])
  ' -s init stop
}

id=`basename "$0"`

if test -d /root
then
  HOME=${HOME-/root}
else if test -d /var/root
  then
    HOME=${HOME-/var/root}
  fi
fi
export HOME

ERLSTART_CONFIG_FILE=${ERLSTART_CONFIG_FILE-/usr/share/myerlnode/myerlnode.rc}
export ERLSTART_CONFIG_FILE

. "$ERLSTART_CONFIG_FILE"

cookie=${cookie-turg}
user=${user-erlang}
hostname=${hostname-`get_hostname`}
node_name=`basename "$node_name_file"`
full_name="$node_name@$hostname"
shutdown_file=${shutdown_file-/var/run/myerlnode.shutting_down}

ERL_CRASH_DUMP=${ERL_CRASH_DUMP-/dev/null}
export ERL_CRASH_DUMP

case ${1-"status"} in
start)
  test "`id -u`" -eq 0 || exec sudo $0 "$@"

  printf "Starting Erlang... "

  if test -f "$node_name_file" &&                                     \
     test true = "`erlstart-eval 'true' 2>/dev/null`" 2>/dev/null
    then
      echo "already started."
      exit 0
    fi

  pid=`eval_with_main_node "$full_name" '
         io:format ("~p", [
           case rpc:call (MainNode, os, getpid, []) of
             { badrpc, _ } -> undefined;
             Pid -> list_to_integer (Pid)
           end ])'`

  test "$pid" -gt 0 2>/dev/null && {
    test -f "$shutdown_file" && {
      oldpid=`cat "$shutdown_file"`
      test "$pid" -eq "$oldpid" && {
        echo "shutdown in progress (pid = '$pid')." 1>&2
        exit 1
      }
    }
  }

  rm -f "$shutdown_file"

  # check for -s shell support
  su -l -s /bin/sh $user -c true >/dev/null 2>/dev/null

  if test $? = 0
    then
      dashs="-s /bin/sh"
    else
      dashs=""
    fi

  ${niceness+ nice -n $niceness}                                      \
  su -l $dashs "$user" -c                                             \
    "env cookie=\"$cookie\" erlstart-run-erlang \"$ERLSTART_CONFIG_FILE\""

  eval_with_main_node "$full_name"                                    \
      "Wait = fun (_, 0) ->
                    failed;
                  (Cont, Max) ->
                    case net_adm:ping (MainNode) of
                      pong ->
                        ok;
                      pang ->
                        timer:sleep (100),
                        Cont (Cont, Max - 1)
                    end
              end,
       DontTellMe = 100,
       ok = Wait (Wait, DontTellMe)" || {
    echo "" 1>&2
    echo "$id: could not connect to node after 10 seconds" 1>&2
    exit 1
  }

  eval_with_main_node "$full_name"                                    \
      "Wait = fun (_, 0) ->
                    failed;
                  (Cont, Max) ->
                    case rpc:call (MainNode, init, get_status, []) of
                      { started, _ } ->
                        ok;
                      { starting, _ } ->
                        timer:sleep (100),
                        Cont (Cont, Max - 1);
                      { Status, _ } ->
                        { failed, Status }
                    end
              end,
       DontTellMe = 100,
       ok = Wait (Wait, DontTellMe)" || {
    echo "" 1>&2
    echo "$id: node did not boot after 10 seconds" 1>&2
    exit 1
  }

echo "done."
;;

stop)
  test "`id -u`" -eq 0 || exec sudo $0 "$@"

  printf "Stopping Erlang... "

  if test ! -f "$node_name_file" ||                                   \
     test true != "`erlstart-eval 'true' 2>/dev/null`" 2>/dev/null
    then
      echo "not running."
      exit 0
    fi

  pid=`erlstart-eval 'os:getpid ()' 2>/dev/null`

  test "$pid" -gt 0 2>/dev/null && {
    printf '%s' $pid > "$shutdown_file"
    chown $user:$user "$shutdown_file"
  }

  erlstart-eval 'init:stop ()' >/dev/null 2>/dev/null

  eval_with_main_node "$full_name"                                    \
      "Wait = fun (_, 0) ->
                    failed;
                  (Cont, Max) ->
                    case net_adm:ping (MainNode) of
                      pong ->
                        timer:sleep (100),
                        Cont (Cont, Max - 1);
                      pang ->
                        ok
                    end
              end,
       DontTellMe = 100,
       ok = Wait (Wait, DontTellMe)" || {
    echo "" 1>&2
    echo "$id: node still responsive after 100 seconds" 1>&2
    exit 1
  }

  rm -f "$node_name_file"

  echo "done."
  ;;

status)
  if test -f "$node_name_file" &&                                     \
     test true = "`erlstart-eval 'true' 2>/dev/null`" 2>/dev/null
    then
      echo "Erlang is running"
    else
      echo "Erlang is not running"
    fi

  ;;

*)
  echo "$id: unknown command $1" 1>&2
  exit 1
esac

exit 0

Hopefully you found that inspirational.

WS-REST 2010 CfP

2010-01-14T05:45:25+00:00

I’m on the program committee for the WS-REST 2010 conference. Please consider submitting a paper, but note that the submission deadline is coming up quickly.

WS-REST 2010
http://ws-rest.org/
Paper Submission: February 8, 2010

Call for Papers

The First International Workshop on RESTful Design (WS-REST 2010) aims to provide a forum for discussion and dissemination of research on the emerging resource-oriented style of Web service design.

Background

Over the past few years, several discussions between advocates of the two major architectural styles for designing and implementing Web services (the RPC/ESB-oriented approach and the resource-oriented approach) have been mainly held outside of the research and academic community, within dedicated mailing lists, forums and practitioner communities. The RESTful approach to Web services has also received a significant amount of attention from industry as indicated by the numerous technical books being published on the topic.

This first edition of WS-REST, co-located with the WWW2010 conference, aims at providing an academic forum for discussing current emerging research topics centered around the application of REST, as well as advanced application scenarios for building large scale distributed systems.

In addition to presentations on novel applications of RESTful Web services technologies, the workshop program will also include discussions on the limits of the applicability of the REST architectural style, as well as recent advances in research that aim at tackling new problems that may require to extend the basic REST architectural style. The organizers are seeking novel and original, high quality paper submissions on research contributions focusing on the following topics:

Applications of the REST architectural style to novel domains
Design Patterns and Anti-Patterns for RESTful services
RESTful service composition
Inverted REST (REST for push events)
Integration of Pub/Sub with REST
Performance and QoS Evaluations of RESTful services
REST compliant transaction models
Mashups
Frameworks and toolkits for RESTful service implementations
Frameworks and toolkits for RESTful service consumption
Modeling RESTful services
Resource Design and Granularity
Evolution of RESTful services
Versioning and Extension of REST APIs
HTTP extensions and replacements
REST compliant protocols beyond HTTP
Multi-Protocol REST (REST architectures across protocols)

All workshop papers are peer-reviewed and accepted papers will be published as part of the ACM Digital Library. Two kinds of contributions are sought: short position papers (not to exceed 4 pages in ACM style format) describing particular challenges or experiences relevant to the scope of the workshop, and full research papers (not to exceed 8 pages in the ACM style format) describing novel solutions to relevant problems. Technology demonstrations are particularly welcome, and we encourage authors to focus on “lessons learned” rather than describing an implementation.

Papers must be submitted electronically in PDF format. Submit at the WS-REST 2010 EasyChair installation.

Important Dates

Submission deadline: February 8, 2010, 23.59 Hawaii time
Notification of acceptance: March 1, 2010
Camera-ready versions of accepted papers: March 14, 2010
WS-REST 2010 Workshop: April 26, 2010

Program Committee Chairs

Cesare Pautasso, Faculty of Informatics, USI Lugano, Switzerland
Erik Wilde, School of Information, UC Berkeley, USA
Alexandros Marinos, Faculty of Engineering & Physical Sciences, University of Surrey, UK

Program Committee

Rosa Alarcon, Pontificia Universidad Catolica de Chile
Subbu Allamaraju, Yahoo Inc., USA
Tim Bray, Sun Microsystems, USA
Bill Burke, Red Hat, USA
Benjamin Carlyle, Australia
Stuart Charlton, Elastra, USA
Joe Gregorio, Google, USA
Michael Hausenblas, DERI, Ireland
Rohit Khare, 4K Associates, USA
Frank Leymann, University of Stuttgart, Germany
Mark Nottingham, Yahoo Inc., Australia
Aristotle Pagaltzis, Germany
Ian Robinson, Thoughtworks, USA
Richard Taylor, UC Irvine, USA
Stefan Tilkov, innoQ, Germany
Steve Vinoski, Verivue, USA
Jim Webber, Thoughtworks, USA
Olaf Zimmermann, IBM Zurich Research Lab, Switzerland

Contact

WS-REST Web site: http://ws-rest.org/
WS-REST Email: chairs@ws-rest.org

14 December 2009: Erlang Training and Consulting's 10th Birthday – it's time to change our name!

2010-01-11T16:32:22+00:00

2009 has been a huge year for us. Not only have we had an incredible year assisting companies on six continents with their use of Erlang but we have also organised 4 conferences, 10 Erlang User Group Meetings and started a KTP project in e-learning with the University of Kent. To add to all that - it’s also our 10th Birthday!

It all began 10 years ago on 13th December 1999, when our founder Francesco Cesarini decided to set up the first non-Scandinavian company to offer Erlang/OTP training and consultancy services. At that time, Erlang wasn't known outside of the 'Ericsson world'. Over the years the business grew dramatically. We have expanded geographically, with offices now in London (UK), Uppsala (Sweden) and Krakow (Poland) and now we employ 40 employees worldwide. The services on offer have also expanded significantly. We are now a one-stop shop for all Erlang needs, including:

Erlang training at all levels
Certification in Erlang
Provision of experienced Erlang contractors
Erlang consulting services
In-house systems development
24/7 support
Assistance with building Erlang teams.

A year ago we started running the Erlang Factory conferences. These are aimed at all levels of the development community including Architects, Programmers, Project Managers, Software and Platform Innovators as well as existing Erlang users. They are also designed to present Erlang to prospective users, Erlang newbies and anyone else interested in making use if the many strengths of the language. The Erlang Factory takes place twice a year - in the San Francisco Bay Area in March/April and in London in June. In 2009 we also took over the organisation of the Erlang User Conference in Stockholm – the oldest Erlang conference in the world, which has been held annually since 1994. Finally this year the first Erlang day in Krakow (Poland) called Erlang Factory Lite was held to grow interest in Erlang of the substantial technology Community in that country.

Great service, a customer-focus and skilled and experienced staff have been the basis of our growth over the past ten years. This will not change. However in 2010 we will do it with a new name! In order to better describe the breadth of our service offering, on 1st January 2010 we will become Erlang Solutions Ltd.

09 December 2009: Marcus Taylor at Strengthen Your Business

2010-01-11T16:32:22+00:00

More than 50 representatives from the business world attended a special networking event at the University of Kent’s Medway campus. Called Strengthen Your Business, the free session on 25 November gave business leaders the chance to explore how their companies can team up with the University, and benefit from its commercial expertise and resources.

A variety of business people who had successfully worked with the University on student placements or Knowledge Transfer Partnerships spoke of their experiences, while others outlined the opportunities for collaboration in graduate recruitment and continuing professional development schemes. There was also the chance for individuals to network after the main talks.

Marcus Taylor, Chief Executive for London-based Erlang Training and Consulting, spoke of the strategic help his company had received with Knowledge Transfer Partnerships – a government-funded initiative that helps businesses develop their competitiveness, productivity and performance. ‘I’m not an experienced bid writer, but the University has a wealth of experience in this area and supported us wonderfully throughout the whole process – so much so that we’re thinking of applying for our fourth project,’ he said.

James Corbin, Placement Officer for the University of Kent, said it was a golden opportunity for businesses to find out about the resources on their own doorstep. 'The University is host to a wealth of knowledge that can often provide businesses with the solutions they seek,’ he said. ‘We’re convinced that firms who work with the University can gain a competitive edge.’

The event followed hot on the heels of the launch of the University’s ICE initiative, which took place at the Canterbury Innovation Centre on 17 November. ICE stands for innovation, creativity and enterprise, and is a project that aims to connect local, regional, national and international businesses to the University of Kent and to each other.

The Strengthen Your Business event at Medway was run in partnership with Thames Gateway Chamber of Commerce.

07 December 2009: Erlang Factory 2010 Dates Announced!!

2010-01-11T16:32:22+00:00

The dates of the Erlang Factory in the SF Bay Area and London have been announced.

SF Bay Area: 25-26 March 2010
London: 10-11 June 2010

Registration will be opening soon and we are currently accepting talk proposals. If you want to give a presentation at the Erlang Factory, you can find more information how to do that, here. Keep a look out and follow us on Twitter @erlangfactory

30 November 2009: Erlang Training and Consulting soon Erlang Solutions will be present at ACCU 2010

2010-01-11T16:32:22+00:00

Francesco Cesarini (founder of Erlang Training and Consulting, soon Erlang Solutions Ltd.) and Ulf Wiger (CTO of Erlang Training and Consulting, soon Erlang Solutions Ltd.) will be speaking at ACCU 2010. Francesco's talk on Styling your Architecture in an Evolving Concurrent World and Ulf's on Message-Passing Concurrency in Erlang, have both been accepted by the conference committee.

ACCU 2010 will take place from 14th to 17th April 2010 at the Barcelo Oxford Hotel, in Oxford, UK. The programme of the conference will feature a special track on software testing, and sessions on concurrent and distributed systems, C++, Java,agile development and Erlang!

Meet us at ACCU 2010!

23 November 2009: The Largest Erlang Open Source Project Directory!

2010-01-11T16:32:22+00:00

Trapexit has announced its own Erlang open source crawler. It will crawl the web, gathering searchable information on all open source Erlang projects from all the major repositories. Trapexit will allow discussing these projects in the forum, adding documentation in the Trapexit wiki or rating the various contributions. To view the project index repository, visit http://projects.trapexit.org.

19 November 2009: Learn Erlang from experts

2010-01-11T16:32:22+00:00

Do you want to learn about the basic, sequential and concurrent aspects of the Erlang programming language? And would you like to be taught by the expert, author of the 'Erlang Programming' book?

If the answer is yes, then we have something which may interest you:

On Wednesday, 3rd February 2010 in London Francesco Cesarini, the author of 'Erlang Programming' also the founder and CSO of Erlang Training and Consulting Ltd. will give a tutorial on Practical Erlang Programming.

You will learn the basics of how to read, write and structure Erlang programmes. The goal of tutorial is a hands-on introduction to the theory and concepts behind sequential and concurrent Erlang programming, explaining the Erlang syntax, semantics and concurrency model. We conclude with an overview of the error handling mechanisms used to build fault tolerant systems with five nines availability.

The target audience are software developers and engineers with an interest in server side applications and massively concurrent systems.

To learn more about the tutorial, please go here and if you want to book it click here.

The tutorial will take place in the Caesar Room, Imperial Hotel, Russell Square, London WC1B 5BB.

See you there!

19 November 2009: ETC at the 11th Symposium on Trends in Functional Programming

2010-01-11T16:32:22+00:00

Erlang Training and Consulting Ltd. sponsors the 11th Symposium on Trends in Functional Programming and sits on the programme committee.

The symposium is an international forum for researchers with interests in all aspects of functional programming. It embraces a broad view of current and future trends in functional programming and aspires to be a lively environment for presenting the latest research and applications.

The 11th Symposium on Trends in Functional Programming will be held at the University of Oklahoma, on campus in Norman, Oklahoma, May 17-19, 2010.

12 November 2009: Take our Erlang Certification Survey!

2010-01-11T16:32:22+00:00

Erlang Training and Consulting Ltd in conjunction with the University of Kent intends to launch a pilot version of an e-certification platform for Erlang in the near future.

To assist in this process, we have launched a survey on Erlang Certification. The objective of the study is to better understand the expectations of users, in order to ensure a well-focused product.

We would appreciate a few minutes of your time in completing this survey. To take the survey, please go to:

http://surveys.erlang-consulting.com/index.php?sid

The results of the survey will be shared with third parties, in publications and as part of research into Certification. When doing so, all results will be completely anonymised and any individual-identifying information removed.

Thank you for your time and cooperation!

28 October 2009: Erlang Web 1.4 has been released!

2010-01-11T16:32:22+00:00

We are happy to announce the next official release of the Erlang Web framework. The latest 1.4 version is full of new features and enhancements, like integration with EWGI, new template language support: ErlyDTL, Erlang Web Test Suite application bundled, multiple level request dictionary support.

Apart from the above, there are also some changes worth mentioning:
* new wpart introduced: wpart:comment
* embedding default values in wparts from templates (patch submitted by Elena Bobrova)
* top-down template inclusion feature
* experimental ejabberd-flavoured e_hook introduced
* upgraded to Yaws 1.85

Erlang Web 1.4 can be easily downloaded as a tarball from its official website and checked out from the bitbucket repository. The documentation can be found on Erlang Web's wiki page.

We do encourage everyone to try Erlang Web 1.4 out, send us comments, contribute and enjoy the coding!

13 October 2009: Ready for the Erlang courses in London?

2010-01-11T16:32:22+00:00

Have you always wanted to learn more about Erlang, but never knew how to get started?

We have a special offer, which may encourage you. If you sign up now for a five-day Erlang By Example or Erlang Open Telecom Platform courses in London you will get the Early-Bird Discount of £245 off the standard price! Don't miss out and register now.

To see the full list of our scheduled courses, please click here.

A Case for Orthogonality in Design

2010-01-11T12:46:07+00:00

In Learning, using and designing command paradigms, John M Carroll introduces the notion of lexical congruence. When you design a language, one of the things that you do is lexicalization of the domain. If we extrapolate the concept to software designs in general, we go through the same process with our domain model. We identify artifacts or lexemes and choose to name them appropriately so that the names are congruent with the semantics of the domain. This notion of lexical congruence is the essence of having a good mnemonics for your domain language. I found the reference to Carroll's work in Testing the principle of orthogonality in language design, which discusses the same issue of organizing your language around an optimal set of orthogonal semantic concepts. This blog post tries to relate the same concepts of orthogonality in designing domain models using the power that the newer languages of today offers.

The complexity of a modeling language depends on the number of lexemes, their congruence with the domain concepts being modeled and the number of ways you can combine them to form higher order lexemes. The more decoupled each of these lower level lexemes are, the easier they are to compose. When you have overlapping concepts being modeled as part of lexemes, the mixing is not easy. You need to squeeze in some special boundary conditions as part of composition logic. Making your concepts independent yet composable makes your design orthogonal.

The first time I came across the concept of orthogonality in design and consciously appreciated the power of unification that it brings on to your model, is through Andrei Alexandrescu's idea of policy based design that he evangelized in his book Modern C++ Design and in the making of the Loki library. You have orthogonal policies that are themselves reusable as independent abstractions. And at the same time you can use the language infrastructure to combine them when composing your higher order model. Consider this C++ example from Andrei's book ..

template
<
  class T,
  template <class> class CheckingPolicy,
  template <class> class ThreadingModel
>
class SmartPtr;

CheckingPolicy enforces constraints that need to be satisfied by the pointee object. The ThreadingModel abstraction defines the concurrency semantics. These two concerns are not related in any way between themselves. But you can use the power of C++ templates to plug in appropriate behaviors of these concerns when composing your own custom type of SmartPtr ..

template SmartPtr<Widget, NoChecking, SingleThreaded>
  WidgetPtr;

This is orthogonal design where you have a minimal set of lexemes to model otherwise unrelated concerns. And use the power of C++ templates to evolve a larger abstraction by composing them together. The policies themselves are independent and can be applied to construct arbitrary families of abstraction.

The crux of the idea is that you have m concepts that you can use with n types. There's no static relationship between the concepts and the types - that's what makes orthogonality an extensible concept. Consider Haskell typeclasses ..

class Eq a where 
  (==) :: a -> a -> Bool

The above typeclass defines the concept of equality. It's parameterized on the type and defines the constraint that the type as to define an equality operator in order to qualify itself as an instance of the Eq typeclass. The actual type is left open which gives the typeclass an unbounded extensibility.

For integers, we can do

instance Eq Integer where 
  x == y =  x `integerEq` y

For floats we can have ..

instance Eq Float where 
  x == y =  x `floatEq` y

We can define Eq even for any custom data type, even recursive types like Tree ..

instance (Eq a) => Eq (Tree a) where 
  Leaf a         == Leaf b          =  a == b
  (Branch l1 r1) == (Branch l2 r2)  =  (l1==l2) && (r1==r2)
  _              == _               =  False

Haskell typeclasses, like C++ templates help implement orthogonality in abstractions through a form of parametric polymorphism. Programming languages offer facilities to promote orthogonal modeling of abstractions. Of course the power varies depending on the power of abstraction that the language itself offers.

Let's consider a real world scenario. We have an abstraction named Address and modeled as a case class in Scala ..

case class Address(no: Int, street: String, 
                   city: String, state: String, zip: String)

There can be many contexts in which you would like to use the Address abstraction. Consider printing of labels for shipping that needs your address to be printed in some specific label format, as per the following trait ..

trait LabelMaker {
  def toLabel: String
}

Note that printing addresses in the form of labels is not one of the primary concerns of your Address abstraction. Hence it makes no sense to model it as one of the methods of the class. It's only required that in some situations we may need to use the Address to print itself in the form of a label as per the specification mandated by LabelMaker.

One other concern is sorting. You may need to have your addresses sorted based on zip code before submitting them to your Printer module for shpping. Sorting may be required in combination with label printing or as well as on its own - these two are orthogonal concerns that should never have any dependence amongst themselves within your abstraction.

Depending on your use case, you can decide to compose your Address abstraction as

case class Address(houseNo: Int, street: String, 
  city: String, state: String, zip: String)
  extends Ordered[Address] with LabelMaker {
  //..
}

which makes your Address abstraction statically coupled with the other two.

Or you may like to make the composition based on individual objects which would keep the base abstraction independent of any static coupling.

val a = new Address(..) with LabelMaker {
  override def toLabel = {
    //..
  }
}

As an alternative you can also choose to implement implicit conversions from Address using Scala views ..

object Address {
  implicit def AddressToLabelMaker(addr: Address) = new LabelMaker {
    def toLabel =
      "%d-%s, %s, %s-%s".format(
        addr.houseNo, addr.street, addr.city, addr.state, addr.zip)
  }
}

Whatever be the implementation, take note of the fact that we are not polluting the basic Address abstraction with the concerns that are orthogonal to it. Our model, which is the design language treats orthogonal concerns as separate lexemes and encourages ways to compose them non invasively by the user.

pm

2010-01-07T18:46:35+00:00

Apparently there is going to be a revolution in the software development world, backed by some very illustrious names in the industry. In the words of Ivar Jacobson, one of the people behind it:

We are some people who have observed software engineering theory and practice of the past decades and have realized that it is now time to revitalize this discipline. We have been quietly planning a “revolution”.

And here is the “Call for Action” statement for this revolution (called SEMAT — Software Engineering Method and Theory):

Software engineering is gravely hampered today by immature practices. Specific problems include:

The prevalence of fads more typical of fashion industry than of an engineering discipline.

The lack of a sound, widely accepted theoretical basis.

The huge number of methods and method variants, with differences little understood and artificially magnified.

The lack of credible experimental evaluation and validation.

The split between industry practice and academic research.

We support a process to refound software engineering based on a solid theory, proven principles and best practices that:

Include a kernel of widely-agreed elements, extensible for specific uses

Addresses both technology and people issues

Are supported by industry, academia, researchers and users

Support extension in the face of changing requirements and technology

The list of signatories to the “revolution” is quite impressive. See for your self on their web page. Sounds good? But not everyone agrees, as you can see on this blog. The author of the blog has an issue with the very framing of the problem:

But this is an entirely misguided movement. Let’s analyze the call for action, bit by bit. It begins:

Software engineering is gravely hampered today by immature practices. [...]

[...]

Never mind that people in the community (most prominently Tom DeMarco) have spoken against this paradigm, or that software development has little to do with most established engineering disciplines: the people behind SEMAT have chosen to ignore these criticisms and move on.

Fair enough. This critique reminds me of this really old paper, called “Masterpiece Engineering” that you might have heard of. Apparently, there were people doubting whether software development is really engineering even when the term was first coined circa 1968.

But what interests me is this statement by SEMAT:

The prevalence of fads more typical of fashion industry than of an engineering discipline.

So if Software Engineering is not engineering, could it be fashion? Let’s see. Here are my observations:

Formal study: (a) Electrical Engineer: formal degree required; (b) Software Engineer: formal degree not required; (c) Fashion Designer: formal degree not required
Work: Engineers: work hard; (b) Software developers: work harder; (c) Fashion Designers: while they are apprentices, work their ass off; as soon as they “become” a designer, stop calling it work and start calling it art; the amount of work/art that needs to be done everyday is inversely proportional to how good you are considered
Respect: (a) Electrical Engineer: gets some respect from the people who work for him, for his knowledge; gets respect from people he works for (the “business folks”) only if he can keep the cost low; (b) Software Engineer: gets no respect from anyone, including the janitor services, unless he accidentally makes some money from a dot.com IPO; (c) Fashion Designer: gets respected both by people who work for him, and people who he works for, directly proportional to how much he/she is exploiting them
Fame: (a) Electrical Engineer: not famous, with rare exceptions of people who have managed to make something that has the astonishing capability of being really appealing to customers and their company’s marketing department at the same time; (b) Software Engineer: not famous as long as they are doing the right thing; you need to write a worm that brings down the ‘net to be famous and maybe not even then (c) Fashion Designer: often famous, especially if the product simultaneously manages to be feckless (you don’t know whether its a skirt or a bandanna) and useless (does not cover the body)
Discipline: (a) Electrical Engineer: practices great discipline in work; (b) Software Engineer: is expected to work with discipline during office hours and continue to work with passion until midnight; (c) Fashion Designer: discipline is death!
Creativity: (a) Electrical Engineer: as his experience grows, tries his hand at more creative things; (b) Software Engineer: to start with, because he does not know much yet, everyone around is too busy to teach, there is no training budget, and anyway the product has to ship by month end with no time to learn anything, has to be creative by using the only tool he know to do everything; only after gaining much experience can he stop being creative and actually do thing properly; (c) Fashion Designer: your creativity is defined to be inversely proportional to how many people can afford your clothes

So, from the above, it is clear, fad or no fad, a Software Engineer’s life is nothing like a Fashion Designer’s. It is also not like an Electrical Engineer’s.

P.S.: while I have a Bachelors and Master degree in Electrical Engineering, and I have been developing software professionally for some time now, my profound knowledge of the fashion industry is gained entirely from the movie The Devil Wears Prada.

Tagged: controversy

Thoughts on process hierarchies in Haskell.

2010-01-05T12:50:22+00:00

On Process Hierachies

I have been in thinking mode the last couple of days. It all started when I decided to look at the problem of wrong in haskell-torrent. What will happen when something fails? Joe Armstrong has part of the answer by arguing that we must proactively expect a process to fail and then let some other process clean it up (paraphrased a lot by me, these are not his exact words).

Take a look at this image,

It is a hypergraph where each vertex corresponds to a process and each edge corresponds to an channel. The graph is undirected because in principle each process can both send and receive on the channel in question. In practice however, one can limit the communication as some processes will only transmit on a channel, and others will only receive (and some will do both).

To simplify the graph, I omit local RPC channels. It is customary to implement RPC by creating a local channel and transmitting that channel over one of the hypergraph edges. Response can then work directly on the local channel, simplifying the problem of giving identity to processes in many cases.

Termination

A somewhat scary part of such a network is what happens when we want to terminate it or if something goes wrong in one of the processes. In particular, we have the problem that while Haskell is a pure, nice, protected Nirvana — the real world inside IO is a dank, dark place of twisted little passages (all alike). Thus it might very well be that we get to experience an exception first-hand in one of the above processes.

I have toyed with the idea of copying the excellent CHP library by Neil Brown. Neil defines a concept of poison for channels. A channel that has been poisoned will always transmit poison. Poison thus propagates through the hypergraph and slowly but surely kills off each process. Neil has a description of the process on his blog. I like the idea because it uses the existing infrastructure of the hypergraph. It would be fairly easy to add support for writing poison handlers at each node.

Process Hierarchies

I have pondered on a different scheme for the last couple of days however. We can impose a location graph on the hypergraph above:

This graph, a tree, describes the location or hierarchy of the processes in question. The processes named S0, S1, … and so forth are supervisors. Their responsibility is only one: Keep their children running according to a policy. Erlang programmers fluent in the OTP high level libraries present in the Erlang distribution will recognize the supervisor term.

A normal (worker) process has two rules:

If the process terminates, it must inform its supervisor.
If the process gets an asynchronous KillThread exception, it must die.

A supervisor process needs to obey the following scheme:

If a process under its supervision dies it must take affair according to its policy. This may mean that it kills off all other children and reports back to its supervisor for instance. Or it may mean that it simply restarts the process that got killed. What happens is dependent on the policy we configured the supervisor with.
If a supervisor is asked to shutdown by a KillThread exception, it must first kill all of its children.

Note that with these rules, termination is almost free: Terminate the tree. Because everything is linked to the tree, termination will happen. We will terminate by asynchronous exceptions aggressively.

An interesting difference from Erlang is that of communication, which is a certain sense is Dual in Haskell+CML: In Erlang, the process id is what you need to hold in order to transmit an (async.) message to the process. In our CML-based system the channel is the name you must hold in order to communicate. In other words, this lets us, in some circumstances, replace the process with a new one but keeping the channel. Note, though, that the interface is not completely clean: if the Haskell runtime figures out a thread is indefinitely blocked, it will in general throw an exception — but I have not yet read the CML code in detail so I do not know if this will happen. It depends on what MVar references a channel hold.

Another view

The hypergraph and location graph constitutes what is sometimes called a Bigraph. This is not to be confused with the term Bipartite graph which is a completely different term but sometimes also called a bigraph unfortunately. I somewhat resist the idea to use the same term for different concepts, but work done in this twi-graph concept presented in this post on bigraphs suggests that the term is quite used in this respect.

The research seems different from what I need however. ITU has done extensive research in bigraphical programming languages, letting them describe various known calculi (Lambda, Pi, etc.). I just want to use the descriptive power of the bigraphs to discriminate location from communication.

So to conclude: Does that give anybody out there a brilliant idea or does my design have an inherent flaw I should be aware of? If not, I'll probably try to sketch some code in Haskell for the Supervisor tree and processes.

ProcessOne, a year in review

2010-01-04T14:24:13+00:00

At ProcessOne, we wanted to wish you all a very happy 2010 year. To give you a good sense of all what has been accomplished, we have written this small review of the 2009 year at ProcessOne. We hope you will enjoy the reading.

Software

First, we have released ejabberd 2.0.3 to 2.0.5, and we finally made it with ejabberd 2.1 (with its bugfix 2.1.1). It has received much scalability improvements, as well as a big PubSub boost. Even more changes have made it an even more rock-solid and featureful real-time server. A STUN server has been added in ejabberd, for multimedia purposes. The version 3.0 is in the works, based on the opensource exmpp library, freshly released this year also.

On the desktop, OneChannel, the PubSub client working on AIR, has been released, optionally using the xmpp-sandbox.org preconfigured real-time feeds.

Our gateway packaging is stable, scalable and works pretty well, covering Microsoft Live Messenger, Yahoo! IM, AIM and ICQ, XMPP, and also non-IM systems, like Twitter and Wave gateway.

On the mobile front, OneTeam for iPhone, the mobile XMPP client, has been released in versions 2.1, 3.0, 3.1, and 3.2, and we believe it is the best XMPP client out there, with room for improvements and innovations in the coming year.

OneWeb, the extension for Firefox, has been released as an alpha product, with a Fennec port (Firefox mobile). It basically adds data sync and remote control between browsers, demoing the power of XMPP in the browser.

Our supervision console TeamLeader has had many improvements in 2009 and is now clean and efficient, being deployed on large customer projets.

Our XMPP desktop client OneTeam for Firefox has become a strong client and should become a major component in our offering in 2010.

We also are offering a module for the APNS, or Apple Push Notifications Service, for the iPhones.

Announced

We have announced and demoed the OneTeam Media Server, our Flash server implementation, acting as a media relay and much more, for multimedia applications.

The ProcessOne Wave Server is in the work, gaining features and improving fast. A demo on Tkabber has been done, showing very basic client/server interactions, like create a wavelet, join and edit it, the basics of collaborative real-time editing of a text document.

Our OCS Bridge enables a real federation of ejabberd servers with the Microsoft's IM+VoIP corporate servers, while this one only painfully offers gateways to XMPP since this year.

Services

We have given birth to a number of services:

We have opened the ProcessOne Labs, showing all our non-production ready, but promising products.
Hosted.IM offers for XMPP hosting services for small businesses.
IMstore enables customers to buy products and services, the fast and easy way, on the web via a simple form.
The APNS, or Apple Push Notifications Services, runs on our infrastructure for OneTeam for iPhone clients using our OneTeam.IM and Talkr.IM services.
Tweet.IM is a XMPP to Twitter gateway, offering nice features like delivery modes.
xmpp-sandbox.org is the XMPP service for developers.
Talkr.IM is our XMPP service for the masses.

Events, Talks & Articles

Here are the places where we have been, and what has been said about us:

In December, we have organized the Sea Beyond event, for real-time communications.
The concept of "Web Augmented Reality" came to birth.
A Strategic Guide to Instant Messaging and Security has been published.
Facebook has been caught working on XMPP with ejabberd.
id Software's QuakeLive online game is using ejabberd for its ingame chat infrastructure.
Mickaël Rémond went to Mobile World Congress, Erlang Factory in San Jose and London, XMPP Summit in Brussel and Palo Alto, Erlang User Conference in Stockholm, Le Web in Paris.
We have been covered by Datamonitor and Giga OM.
The series "Scalable XMPP bots" has been published, promoting the use of exmpp (I, II, III).

Roadmap 2010

In 2010, our focus will be mainly on the following points:

Hosted.IM will be offering new simple and extendable services.
OneTeam for Firefox will be released, with WAR (Web Augmented Reality), and much more innovative features.
ejabberd will become more industrial, with 3.0 getting to production ready state, better TeamLeader support, good test suite coverage and more modules on the IMstore.
Talkr.IM will offer the seamless migration from/to another XMPP service.

Conclusion

It has been a busy year, with many achievements, much of which we cannot talk about, or we have just forgotten. But the coming year 2010 will be even more exciting.

Pragmatics of Impurity

2010-01-04T11:51:59+00:00

James Hague, a long time Erlanger, drives home a point or two regarding purity of paradigms in a couple of his latest blog posts. Here's his take on being effective with pure functional languages ..

"My real position is this: 100% pure functional programing doesn't work. Even 98% pure functional programming doesn't work. But if the slider between functional purity and 1980s BASIC-style imperative messiness is kicked down a few notches--say to 85%--then it really does work. You get all the advantages of functional programming, but without the extreme mental effort and unmaintainability that increases as you get closer and closer to perfectly pure."

Purity is not necessarily pragmatic. In my last blog post I also tangentially touched upon the notion of purity while discussing how a *hybrid* model of SQL-NoSQL database stack can be effective for large application deployments. Be it with programming languages or with databases or any other paradigms of computation, we need to have the right balance of purity and pragmatism.

Clojure introduced transients. Rich Hickey says in the rationale .. "If a pure function mutates some local data in order to produce an immutable return value, is that ok?". Transients in Clojure allow localized mutation in initializing or transforming a large persistent data structure. This mutation will only be seen by the code that does the transformation - the client gets back a version for immutable use that can be shared. In no way does this invalidate the benefits that immutability brings in reasoning of Clojure programs. It's good to see Rich Hickey being flexible and pragmatic at the expense of injecting that little impurity into his creation.

Just like the little compromise (and big pragmatism) with the purity of persistent data structures, Clojure also made a similar compromise with laziness by introducing chunked sequences that optimize the overhead associated with lazy sequences. These are design decisions that have been taken consciously by the creator of the language that values pragmatism over purity.

Enough has already been said about the virtues of purity in functional languages. Believe me, 99% of the programming world does not even care for purity. They do what works best for them and hybrid languages are mostly the ones that find the sweetest spots. Clojure is as impure as Scala is, considering the fact that both allow side-effecting with mutable references and uncontrolled IO. Even Erlang has uncontrolled IO and a mutable process dictionary, though its use is often frowned upon within the community. The important point is that all of them have proved to be useful to programmers at large.

Why do creators infuse impurity into their languages ? Why aren't every language created as pure as Haskell is ? Well, it's mostly related to a larger thought that the language often targets to. Lisp started as an incarnation of the lambda calculus under the tutelage of John McCarthy and became the first significant language promoting the purely applicative model of programming without side-effects. Later on it added the impurities of mutation constructs based on the von Neumann architecture of the machines where Lisp was implemented. The obvious reason was to get an improved performance over purely functional constructs. Scala and Clojure both decided to go for the JVM as the primary runtime platform - hence both languages are susceptible to the pitfalls of impurity that JVM offers. Both of them decided to inherit all the impurities that Java has.

Consider the module system of Scala. You can compose modules using traits with deferred concrete definitions of types and objects. You can even compose mutually recursive modules using lazy vals, somewhat similar to what Newspeak and some dialects of ML offer. But because you have decided to bite the Java pill, you can also wreak havoc through shared mutable state at the top level object that you compose. In his post titled A Ban on Imports Gilad Bracha discusses all evil effects that an accessible global namespace can bring to the modularity aspects of your code. Newspeak is being designed as pure in this respect, with all dependencies being abstract and need to be plugged together explicitly as part of configuring the module. Scala is impure in this respect, allows imports to bring in the world on to your module definitions, but at the same time opens up all possibilities of sharing the huge ecosystem that the Java community has built over the years. You can rightfully choose to be pure in Scala, but that's not enforced by the language.

When we talk about impurity in languages, it's mostly related to how it handles side-effects and mutable state. And Haskell has a completely different take on this aspect than what we discussed with Lisp, Scala or Clojure. You have to use monads in Haskell towards any side-effecting operation. And people with a taste for finer things in life are absolutely fine with that. You cannot just stick in a printf to your program for debugging. You need to return the whole stuff within an IO monad and then do a print. The Haskell philosophy looks at a program as a model of mathematical functions where side-effects are also implemented in a functional way. This makes reasoning and optimization by the compiler much easier - you can make your pure Haskell code run as fast as C code. But you need to think differently. Pragmatic ? What do you think ?

Gilad Bracha is planning to implement pure subsets of Newspeak. It will be really exciting to get to see languages which are pure, functional (note: not purely functional) and object-oriented at the same time. He observes in his post that (t)he world is slowly digesting the idea that object-oriented and functional programming are not contradictory concepts. They are orthogonal, and can be arranged to be rather complementary. This is an interesting trend where we can see families of languages built around the same philosophy but differing in aspects of purity. You need to be pragmatic to choose and even mix them depending on your requirements.

Erlang Quick Tip: The user_default module

2010-01-03T15:09:38+00:00

If you are anything like me you spend a lot of your time in the erlang shell debuging and testing your code. And some commands are used more than others but they may be a bit on the long side and you may make regular errors while typing it. One of these wonderful functions is make:all([load]). that will compile using your Emakefile and then magically reload the newly compiled modules into the running shell. What I reglarly do is to type load instead of [load].

Enter the user_default module. Any function defined in the user defined user_default module or in the Erlang defined shell_default module will be allowed to run without typing in the module name. One of these functions that you may use often is the c(module\_name). function. First start by creating a directory for your utility modules. I like having it in the ~/.ebin/ diretory. Now add that direcory to the code path by opening or creating the file ~/.erlang and inserting this line:

code:add_pathz("/Users/username/.ebin").

Create your user_default.erl file and but in the following:

-module (user_default).
-export ([sync/0]).

%% Compiles all files in Emakefile and load into current shell.
sync() ->
  make:all([load]).

Then compile it with erlc user_default.erl and start an Erlang shell. Now inside a project you can run sync(). and it will recompile and load you project.

For those interested I will put up my ~/.ebin code up at github.com/JonGretar/erlang_user_utilities. Please fork and make your own changes and share with the world.

Fun with the CouchDB _changes feed and RabbitMQ.

2010-01-01T23:14:01+00:00

I was recently introduced to yajl-ruby, ruby bindings to the C based yajl json parsing/encoding libraries. After discovering that it can parse HTTP streams it seemed like it would be a perfect fit for use with CouchDB. A while back I wrote some code to push update notifications to RabbitMQ and a commenter mentioned using the _changes feed instead. Combining the _changes feed and yajl-ruby’s HttpStream seemed like a good way to do it.

The _changes feed is a running list of all the documents that have changed in a database listed in order by sequence number. This is similar to update notifications but gives more information such as the document IDs and is HTTP based (with multiple feed styles) rather than stdout. Additionally you can create design document filters which can be specified as a query parameter to give you only the parts of the feed you want. All in all _changes is a pretty powerful feature.

Now for the fun stuff, the code. There are a few dependencies I used to do this, specifically focused on making it fast. As such I used EventMachine based libraries for AMQP and HTTP requests. The first bit of code takes the _changes feed for the “test” database, parses the feed, uses the document ID to request that document and publish it to the queue. One key item to note is that this code requires the latest yajl-ruby from github to run properly. Additionally, this works nicely with feed=continuous so it grabs the documents as they are changed without a need for polling.

Note that there is a variable for since, this allows you to start from a specific sequence number so you can skip over old changes.

The next bit of code works from the other side of the queue. It subscribes to the queue, parses the JSON, performs some operations on it and puts the results back into another CouchDB database called “results”.

What could it be used for? My first thought is some sort of parallel computation, boot up a few dozen EC2 nodes and start dumping data into CouchDB. Have all those nodes pop messages off the queue, process them and dump the results back into Couch. Legitimately one could chain these together to process the results again. The queue ends up being a simple job management system with the EC2 nodes popping new messages as they finish processing them. With a little bit of work, features and the right use case I think could be a pretty powerful system.

Check out the code, my other projects and follow me on twitter @williamsjoe.

[edit: made a slight improvement to changes_sub.rb on 20100107]

Network server programming with SML/NJ and CML

2010-01-01T15:30:32+00:00

My experience with SML/NJ has been almost uniformly positive, over the years. We used it extensively in a previous project to write a compiler (targeting the .NET CLR) for a pi-calculus-based language, and it was fantastic. One drawback with it, though, is the lack of documentation. Finding out how to (a) compile for and (b) use CML takes real stamina. I’ve only just now, after several hours poring over webpages, mailing lists, and library source code, gotten to the point where I have a running socket server.

Download source code, building, and running

The following example is comprised of a .cm file for building the program, and the .sml file itself. The complete sources:

Running the following command compiles the project:

ml-build test.cm Testprog.main

The ml-build output is a heap file, with a file extension dependent on your architecture and operating system. For me, right now, it produces test.x86-darwin. To run the program:

sml @SMLload=test.x86-darwin

substituting the name of your ml-build-produced heap file as necessary.

On Ubuntu, you will need to have run apt-get install smlnj libcml-smlnj libcmlutil-smlnj to ensure both SML/NJ and CML are present on your system.

The build control file

The test.cm file contains

Group is
    $cml/basis.cm
    $cml/cml.cm
    $cml-lib/smlnj-lib.cm
    test.sml

which instructs the build system to use the CML variants of the basis and the standard SML/NJ library, as well as the core CML library itself and the source code of our program. For more information about the SML CM build control system, see here.

The example source code

Turning to test.sml now, we first declare the ML structure (module) we’ll be constructing. The structure name is also part of one of the command-line arguments to ml-build above, telling it which function to use as the main function for the program.

structure Testprog = struct

Next, we bring the contents of the TextIO module into scope. This is necessary in order to use the print function with CML; if we use the standard version of print, the output is unreliable. The special CML variant is needed. We also declare a local alias SU for the global SockUtil structure.

open TextIO
structure SU = SockUtil

ML programs end up being written upside down, in a sense, because function definitions need to precede their use (unless mutually-recursive definitions are used). For this reason, the next chunk is connMain, the function called in a new lightweight thread when an inbound TCP connection has been accepted. Here, it simply prints out a countdown from 10 over the course of the next five seconds or so, before closing the socket. Multiple connections end up running connMain in independent threads of control, leading automatically to the natural and obvious interleaving of outputs on concurrent connections.

fun connMain s =
    let fun count 0 = SU.sendStr (s, "Bye!\r\n")
          | count n = (SU.sendStr (s, "Hello " ^ (Int.toString n) ^ "\r\n");
                       CML.sync (CML.timeOutEvt (Time.fromReal 0.5));
                       count (n - 1))
    in
        count 10;
        print "Closing the connection.\n";
        Socket.close s
    end

The function that depends on connMain is the accept loop, which repeatedly accepts a connection and spawns a connection thread for it.

fun acceptLoop server_sock =
    let val (s, _) = Socket.accept server_sock
    in
        print "Accepted a connection.\n";
        CML.spawn (fn () => connMain(s));
        acceptLoop server_sock
    end

The next function is the primordial CML thread, responsible for creating the TCP server socket and entering the accept loop. We set SO_REUSEADDR on the socket, listen on port 8989 with a connection backlog of 5, and enter the accept loop.

fun cml_main (program_name, arglist) =
    let val s = INetSock.TCP.socket()
    in
        Socket.Ctl.setREUSEADDR (s, true);
        Socket.bind(s, INetSock.any 8989);
        Socket.listen(s, 5);
        print "Entering accept loop...\n";
        acceptLoop s
    end

Finally, the function we told ml-build to use as the main entry point of the program. The only thing we do here is disable SIGPIPE (otherwise we get rudely killed if a remote client’s socket closes!) and start CML’s scheduler running with a primordial thread function. When the scheduler decides that everything is over and the program is complete, it returns control to us. (The lone end closes the struct definition way back at the top of the file.)

fun main (program_name, arglist) =
    (UnixSignals.setHandler (UnixSignals.sigPIPE, UnixSignals.IGNORE);
     RunCML.doit (fn () => cml_main(program_name, arglist), NONE);
     OS.Process.success)

end

Mustache – GitHub’s framework agnostic templating ported to Erlang

2009-12-30T14:00:29+00:00

Mustache is an Erlang port of the Ruby framework of the same name, originally written by Chris Wanstrath, one of the founders of GitHub. A powerful feature of this type of framework is the ability to write a ‘template’ that you could use in a Rails application and then reuse in an Erlang application with [...]

Yay!

2009-12-29T08:20:55+00:00

def groovyPomUrl =
    "http://svn.codehaus.org/groovy/trunk/groovy/groovy-core/pom.xml"
def contributors = new XmlSlurper().parse(groovyPomUrl).contributors
def me = "Merlyn Albery-Speyer"
assert me in contributors.contributor.collect {
    contributor -> contributor.name.text() }

Features:
[GROOVY-3897] [1.7 release notes]

Improvements:
[GROOVY-3958]
[GROOVY-3910]

Bug fixes:
[GROOVY-3954]
[GROOVY-3820]

Failing test submissions:
[GROOVY-3867]
[GROOVY-3870]
[GROOVY-3708]

The State of Erlang Web Middleware – EWGI and SimpleBridge

2009-12-28T15:34:31+00:00

In the interest of publishing more often and covering smaller topics, lets review web-server-agnostic middleware for Erlang.

The current standard for Erlang web application middleware is EWGI, inspired by Python’s PEP 333 and providing similar functionality as Ruby’s Rack. All major Erlang web servers support EWGI, and most web frameworks.

Created by Rusty Klophaus, SimpleBridge “takes the pain out of coding to multiple Erlang web servers by creating a standardized interface.” It is used by the soon to be released new version of The Nitrogen Web Framework. According to Rusty, SimpleBridge has some improvements over EWGI, such as a smaller code base; more easily extendable (Takes about 150 lines to add support for a new HTTP server, vs. ~350 for EWGI; support for multipart file uploads, with size limits and handle-able errors; static file support; more specific interface functions for getting and setting cookies.

However, based on my experience with Rack, Multipart file uploads, static file support, and anything but basic cookie manipulation are all best handled by a web server such as nginx or a reverse proxy like HAProxy anyway. Perhaps it is better that EWGI tries to do less in this case…

Let’s look at a simple EWGI example with MochiWeb (shamelessly pulled from the docs)…

First we startup mochiweb calling the loop/1 function. Mochiweb will call our loop/1 function whenever a request is received.

In loop/1 we setup a call to our simple implementation of the EWGI interface.

Now the EWGI implementation is called with a tuple that aways starts with ewgi_context, and contains a Request and a Response. This code returns a 5-tuple response (further details in the documentation).

EWGI solves the problem of creating a standard interface to middleware components. It has become the standard, and the power of such standards lies in their implementations’ ubiquity. Making your own hit web framework at home in your spare time has never been easier.

Integrating Heman into CalendERL About Nothing

2009-12-28T08:00:00+00:00

A while ago I started a project called Heman. The goal was to create a way to use Erlang to measure and assess application health. Application health is a score used to determine how your application is doing. This could be as general as it's immediate resource consumption or it's general activities but can also be embed to give you detailed information on it's internals and inner workings.

This evening I started integrating heman into calenderl about nothing. It is a good way to put it to practical use and also flesh out some use cases and conditions. You can see the updated cerlan code base as well as the stats system on calenderlaboutnothing.com.

On Peer Processes

2009-12-27T16:58:08+00:00

On Peer Processes

The diagram above is the control-flow in the Peer Process as it stands right now. This process is responsible for communicating with another Peer in haskell-bittorrent. This post is not about the details of the inner workings. Rather it serves to document the legend of the diagram.

All the ellipses that are blue signifies other processes. The inbound process waits on the socket for incoming messages and sends them into the Peer process. The outbound process is a queue which dequeues messages as fast as possible to the peer in the other end. Peer Manager is the process which manages a set of peers -- we have one Peer process for each peer. Finally, Piece Manager is responsible for managing the state of the torrent: what subsets of pieces do we have and what is missing before we have the full file?

A little bit of terminology: a piece is the smallest array of bytes we can SHA1 check for correctness. A block is a subset of a piece which is transferred by the wire protocol. In practice these are always 16K although the protocol provisions for alternative sizes. The reason for this split is that the 16K blocks means we can interleave other messages in between. Further, we can get a hunch on how fast we transfer to a peer by looking at the amount of bytes we can dequeue and smaller blocks means a better granularity.

The diagram depicts the control flow of the client. Black arrows designate how control transfers from one function to the next (usually by tail-calling this is functional programming after all). The blue dotted arrows specifies synchronization points with external processes. If a blue dotted arrow enters a box it usually means that control can only transfer to this point if the corresponding event tied to the arrow fires. The Piece Mgr and "Grab up to ..." is the exception from the rule however: when arrows are back and forth, it is a rpc call.

If a blue dotted arrow leaves towards an external process it means we will wait for a synchronization towards that process before continuing out of the box.

The current state

Modulo some work in the debugging and QA department, the client should be able to leech in a laboratory setting. We cheat by having no Peer Manager yet and we can essentially not really support many Peers properly yet. When the client portion stabilizes it should be fairly easy to add proper Peer managent. I hope to support leeching in the coming days.

Wanna help?

If you want to help out, there is plenty to do. I am still, deliberately, leaving many things open so it should be fairly easy to pick up the code from github and then do some of the simpler stuff. I've not run the excellent hlint by Neil Mitchell - so there should be an opportunity there. There is also an amortized queue implementation which could do with some documentation. Finally, the TODO list is littered with what needs to be done.

On github: haskell-torrent.

A case for hybrid SQL - NoSQL stack

2009-12-24T11:31:22+00:00

Alex Popescu talks about Drizzle replication in his MyNoSql column. He makes a very interesting observation in his post regarding Drizzle's replication capabilities into a host of NoSQL storage backends ..

"Leaving aside the technical details — which are definitely interesting .., the solution using the Erlang AMQP .. implementation RabbitMQ .. — I think this replication layer could represent a good basis for SQL-NoSQL hybrid solutions".

We are going to see more and more of such hybrid solutions in the days to come. Drizzle does it at the product level. Using RabbitMQ as the transport, Drizzle can replicate data as serialized Java objects to Voldemort, as JSON marshalled objects to Memcached or as a hashmap to column family based Cassandra.

Custom implementation projects have also started using the hybrid stack of persistent stores. When you are dealing with real high volumes and patterns of access where you cannot use joins, anyway you need to denormalize. ORMs cease to be a part of your solution toolset. Data access patterns vary widely across the profile of clients using your system. If you are running an ecommerce suite and your product is launched you may have an explosive use of the shopping cart module. It makes every sense to move your shopping cart from the single relational data store where it was lying around and have it served through a more appropriate data store that lives up to the scalability requirements. It's not that you need to throw away your relational database that has served you so long. Like Alex mentioned, you can always go along with a hybrid model.

In one of our recent projects, we were using Oracle as the main relational database for a securities trading back office solution implementation. The database load was computed based on all calculations that were done initially. In a very late stage of the project a new requirement came up that needed heavy processing and storage of semi-structured data and meta-data from an external feed. Both the data and the meta-data were extensible which meant that it was difficult to model them with a fixed schema.

We could not afford frequent schema changes since it would entail long downtime of the production database. But there also was the requirement that after processing of these semi-structured data lots of them will have to be made available in the production database. We could have modeled it following the key/value paradigm in Oracle itself, which we were using anyway as the primary database. But that's again going down the age old saying of the hammer and nail story.

We decided to supplement the stack with another data store that fits the bill for this specific use case. We used MongoDB, that gave us phenomenal performance for our requirements. We were getting the feed from external data sources and loaded our MongoDB database with all the semi-structured data and meta-data. All necessary processing was done in MongoDB on those data and relevant information from MongoDB were pushed to JMS based queues for consumption by appropriate services that copied data asynchrnously to our Oracle servers.

What did we achieve with the above architecture ?

Kept Oracle free to do what it does the best.

Took away unnecessary load from production database servers.

Introduced a document database for serving a requirement tailor made for its use - semi structured data, mainly reads, no constraints, no overhead of ORM ceremony. MongoDB supports a very clean programming model, a very decent query interface, simple to use and easy to convince your client.

Used message based mapping to sync up data ASYNCHRONOUSLY between the nosql MongoDB and sql based Oracle. Each of the data stores were doing what they do the best, keeping us away from the blames of the hammer-nail paradigm.

With more and more of the nosql stores coming up, message based replication is going to play a very important role. Even within the nosql datastore, we are seeing choices of sql based storage backends being offered. Voldemort offers MySql as one of the storage backends - so the hybrid model starts right up there. It's always advisable to use multiple storage that fits your use case than trying to force-fit everything into a single paradigm.

erlrc and rpm

2009-12-23T15:12:51+00:00

Two of the Dukes now work at OpenX, which is starting to dip its toe into the Erlang waters. They use CentOS so they agreed to fund porting framewerk and erlrc to rpm; previously I'd only used them with deb.

A bit of background: erlrc is a set of Erlang modules and shell scripts that are designed to be integrated into packaging hooks so that installation, upgrade, or removal of a package causes corresponding hot-code activity to happen inside registered Erlang VMs on the box. Since erlrc was designed to easily integrate with multiple package managers, getting it to work with rpm was mostly about me understanding rpm's package hooks model. The result is an experience like this,


% sudo yum -q -y install lyet
erlrc-start: Starting 'lyet': (erlang) started
% sudo yum -q -y remove lyet
erlrc-stop: Stopping 'lyet': (erlang) unloaded

i.e., installing an rpm causes a running Erlang VM on the box to hot-code load the new modules and start the associated application, and removing the rpm causes the associated application to be stopped and the corresponding modules to be hot-code unloaded.

If you use fw-template-erlang than the appropriate packaging hooks are added for you automatically, both for deb and now rpm. However even manual creation of rpm spec files is pretty easy:

erlrc-stop is called in %preun if the installation count indicates removal
erlrc-upgrade is called in %preun if the installation count indicates upgrade
erlrc-start is called in %posttrans

Also, the erlrc shell scripts want to know the previously installed version, so I call rpm -q in a %pretrans hook and save the result. Longer term, erlrc should probably ask the Erlang VM it is talking to what version is running to eliminate the need for this argument (I was a bit surprised that rpm doesn't provide this argument to the package hook like debian does; it seems very useful for creating version-specific upgrade fixes).

[ANN] exmpp 0.9.2 new release

2009-12-23T14:59:25+00:00

We are pleased to announce a new release of exmpp.

exmpp is a XMPP library written in Erlang and released under the Erlang Public License. It helps the development of XMPP entities by providing functions to build and interpret XML stanzas. Once compiled, check the generated HTML documentation.

List of main changes since the previous release six months ago:

Adding support for stream error stanzas in exmpp_session when logged in
Autodetect compilation parameters for Mac OS X Snow Leopard
Component support for exmpp, and some fixes
Fix segmentation fault with exmpp and expat
New functions exmpp_client_pubsub:discover_nodes/1 and 2
Stringprep Bidi checking doesn't reject strings with RandALCat and LCat characters
queryns was not defined for IQ packets in received_packet record

The API of exmpp 0.9.2 is considered to be stable, so there aren't any
major changes expected in the API before reaching 1.0.0.

exmpp home page:
http://exmpp.org/

Download exmpp 0.9.2 source code package from:
http://download.process-one.net/exmpp/

You can also check the ProcessOne Labs page:
http://www.process-one.net/en/labs/

[ANN] ejabberd 2.1.1 bugfix release

2009-12-23T14:54:42+00:00

We are pleased to announce the bugfix release ejabberd 2.1.1, which was published on the 17th of December. ejabberd 2.1.1 contains several important bugfixes over last month's major release. None of the fixes is critical, but you are encouraged to update if you are running ejabberd 2.1.0.

Short list of bugfixes:

Fix connection if starttls_required and zlib are set
S2S: fix allow_host/2 on subdomains. added hook s2s_allow_host
MUC: Add support for serving a Unique Room Name
MUC: Route vCard request to the occupant bare JID
MUC: Support converting one-to-one chat to MUC
PubSub: Receive same last published PEP items at reconnect if several resources online
PubSub: Typo in mod_pubsub_odbc breaks Service Discovery and more
Web: Fix memory and port leak when TLS is enabled in HTTP
WebAdmin: report correct last activity with odbc backends
Change captcha.sh to not depend on bash
Generate main XML file also when exporting only a vhost
Fix last newline in ejabberdctl result
Guide: fix -setcookie, mod_pubsub_odbc host, content_types

Check the Release Notes for a more complete list of changes:
http://www.process-one.net/en/ejabberd/release_notes/release_note_ejabberd_2.1.1

If you upgrade from ejabberd 2.0.5 or older, read carefully the release notes, because there were several changes in the installation path and the configuration options.

The list of solved tickets since 2.1.0 is available on ProcessOne bug tracker:
http://redir.process-one.net/ejabberd-2.1.1

ejabberd 2.1.1 is available as source code package and binary installers for Linux 32 bits, 64 bits, Mac OS X Intel, and Windows:
http://www.process-one.net/en/ejabberd/downloads

ProcessOne launches Talkr.IM: a new open, free public XMPP service

2009-12-23T11:19:58+00:00

Talkr.IM is the new XMPP service provided by ProcessOne. It is an open and free service for all audiences worldwide.

What is talkr.IM?

In a few words, Talkr.IM offers scalable and robust presence and chat services for all standard XMPP clients of your choice. Just create an account now, by filling the registration form, and you will be able to login and start chats. It is fully interoperable with XMPP services such as Google Talk, Nokia Ovi, LiveJournal, Yandex Online, jabber.org, jabber.ru, jabbim.cz, jabberes.org, and jabber.fr for example.

Why talkr.IM?

Although this is a plain good XMPP service like others are providing, we want to provide a service that is professionally managed on a daily basis, with a strong service level, responsiveness and uptime.

We also emphasize on privacy and freedom. Privacy is there to ensure that your data are safe with us and will not be abused or transfered. We would like the service to be as much as a safe as possible. Freedom put the emphasize on using as much as free software as possible and ensure data portability. The service runs on free software and the data are yours. We plan to provide a migration feature. You will be able to import your data from another server, but we are also working on the reverse, so that you are able to export them and take them to another server, maybe your own server at some point. That's your data, that's your life and its important for you to keep the control on it.

Instant messaging conveys the value of the pionneer of Internet. One of the first simple messaging client was talk on various Unix flavour in the 1970's. It was free software and was a new way to communicate. Jabber and XMPP after that has been the heir of this tradition of freedom. Talkr.IM wants, like many other XMPP public servers to be a testimonial to this spirit: be free and open minded.

Why another XMPP hosting service ?

There are XMPP services backed by big companies like Google Talk, Nokia Ovi, LiveJournal or Yandex Online. There are also the valuable and much respected XMPP services administered by groups of volunteers and enthusiats, like jabber.org, jabber.ru, jabbim.cz, jabberes.org, and jabber.fr.

In none of these services, we had the opportunity to fully help, as much as we could, and really be implicated into the service administration and optimization.

With Talkr.IM, it is intended to provide a strong service to all users, which we can fully support. It is not intended to compete with any of the above mentionned XMPP services, as it is fully compliant and interoperated, in the pure tradition of the fair-play of the open Internet.

We believe we can achieve these goals since we are the makers of ejabberd. This work will of course benefit the product, in terms even more robustness, features, scalability and security.

Services

That said, besides the simple presence and IM, we also offer more services:

conference will let you create and join multi-party chats with friends, family and colleagues (Multi user chat)
irc is the gateway to the IRC networks that will enable you to join channels you wish from your XMPP client
pubsub and PEP enables you to publish information for yourself and receive information from your contacts such as your mood, activity, geolocation, or the titles of the music you are listening to
users is the directory that enables you to search for friends

The Talkr.IM service will also enhance the user experience in the future, by adding more services.

Feedback

Please feel free to express your needs for the Talkr.IM service and its evolution, as we are listening to and value your feedback.

Please join Talkr.IM, and contribute to the forums.

Merry Christmas: Toke — Tokyo Cabinet driver for Erlang

2009-12-21T17:01:01+00:00

Tokyo Cabinet is a rather excellent key-value store, with the ability to write to disk in a sane way (i.e. not just repeatedly dumping the same data over and over again), operate in bounded memory, and go really fast. I like it a lot, and there’s a likelihood that there’ll be a RabbitMQ plugin fairly soon that’ll use Tokyo Cabinet to improve the new persister yet further. Toke is an Erlang linked-in driver that allows you to use Tokyo Cabinet from Erlang.

There is already a Tokyo Cabinet driver for Erlang, tcerl, however, I couldn’t make it work: even after fixing the C so that it compiles (I hit this bug), I still couldn’t make it work. Inspecting the code, I get the feeling it’s bit-rotted - the Tokyo Cabinet API has moved on, and tcerl hasn’t kept up.

The other issue with tcerl is that it’s not a linked-in driver. Erlang allows two different types of drivers: the first are external C programs — these have a main() and run in their own process. Communication is done by stdin/stdout. These are a bit safer because if they crash they don’t take out the Erlang VM, but they’re never going to be blazingly fast. Toke, on the other hand, is a fully linked-in driver. It dynamically links with the Erlang VM, exists in the same address space and goes as fast as it possibly can (using the Erlang driver callbacks which avoid all copying of data passed from the Erlang). My tests show it’s about three times slower driving Tokyo Cabinet from Erlang via Toke, than driving it natively through C (which is quite good: some googling suggests both the Ruby and Python bindings to Tokyo Cabinet are rather slower). Toke is also about twice as slow as the Erlang ets module, which is in-memory only.

Toke only implements the Tokyo Cabinet hash table (tchdb*) functions, and doesn’t even support all of those: I only wrapped exactly what I needed. You’ll want to read the documentation for Tokyo Cabinet for these. The functions implemented are as follows (refer to the Tokyo Cabinet documentation to explain these further):

toke_drv:new/1 — Set up the driver with a new TCHDB object.
toke_drv:delete/1 — Destroy the driver’s TCHDB object.
toke_drv:tune/5 — Tune the driver’s TCHDB Object.
toke_drv:set_cache/2 — Set the number of records to cache.
toke_drv:set_xm_size/2 — Set the extra amount of memory mapped in.
toke_drv:set_df_unit/2 — Set the steps between auto defrag.
toke_drv:open/3 — Open a db.
toke_drv:close/1 — Close an open db.
toke_drv:insert/3 — Insert. If the key already exists, value is updated.
toke_drv:insert_new/3 — Insert new. If the key already exists, the old value is silently kept.
toke_drv:insert_concat/3 — Concatenate the supplied value with an existing value for this key.
toke_drv:insert_async/3 — Asynchronously insert. If the key already exists, value is updated.
toke_drv:delete/2 — Delete a key from the db.
toke_drv:get/2 — Fetch a key from the db. Returns ‘not_found’ on occasion.
toke_drv:fold/3 — Fold over every value in the db. This internally uses the iteration functions. It’s just wrapped up as a fold to make it appear more functional.
toke_drv:stop/1 — Stop the driver and close the port.

You should be able to use Mercurial to clone it:

# hg clone http://hg.opensource.lshift.net/toke/

Make sure you have Tokyo Cabinet installed (ideally from source. If you’re using a package, make sure you have the development headers available. If you do compile from source, make sure you ldconfig to make your system pick up the new library once it’s installed.). Then it should just be a case of make. There’s also a make run target that starts up an Erlang shell with the paths set up correctly for testing Toke:

toke# make run
erl -pa ebin +K true +A30
Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [hipe] [kernel-poll:true]

Eshell V5.7.4  (abort with ^G)
1> toke_test:test().
passed
2>

Toke is licensed under the MPL. As ever, feedback is very welcome, as are patches!

Concurrency, Bittorrent clients, and Haskell

2009-12-19T16:05:15+00:00

I have been writing code lately on a BitTorrent client in Haskell. Here is the grand old history of that effort. Many years ago, I attempted to start a client, Conjure, written in Haskell. Sadly, it really did not make it to the point where it was really usable, for me at least. Then a couple of years passed, I became involved with programming in Erlang. Erlang has a really nice Actor-based concurrency model. Since BitTorrent is highly concurrent and event-driven it was easy to start another implementation effort in Erlang. Thus etorrent was born. Sadly, I lost interest in etorrent - partially because I am partial to statically typed languages of which Erlang is not one. Thus, we come to this december, where I decided to figure out the state of concurrency in Haskell. To really give GHC a whirl, we need to implement something which is fairly complex. Some people implement ray tracers when toying with programming languages, some solve projecteuler problems. I ... write BitTorrent clients.

BitTorrent is a good maturity check. First, you need decent Disk I/O, decent network I/O and a fairly complete HTTP client. Second, the protocols are easy, but still requires some nontrivial parsing to occur. Finally, the system is fairly concurrent, so it fits nicely into concurrency models if the language has one.

Initial Thougths:

Initally, there is the choice of a concurrency framework. In Haskell, there are several such. The built-in Control.Concurrent is fairly low level and fairly simple. At the CS department at Copenhagen University, DIKU, there is currently a course running in CSP and some students decided to use Control.Concurrent.CHP by Neil Brown. I think it will suit them well for the course, but it irritated me that it mapped so weakly to the etorrent model. So Control.Concurrent.CML became the basic library for concurrency. This concurrency model is not entirely unfamiliar to me: it is written by John Reppy for ML.

The CML model is somewhat reminiscent of the Pi-calculus. In particular, we can transmit channels on channels. This gives a neat method for doing RPC-style operations. Send off a message together with a responder channel and then wait for a response on that channel. This means we are easily able to emulate most of what the Actor model has if needed.

Haskell-torrent design:

Haskell-torrent uses a basic design, where a fairly large number of processes communicate with each other to solve the problem of uploading and downloading files. We have a limitation of one single torrent right now, but that should be fixable in time. The process diagram at the header describes the basic layout. Each box is a single process, apart from the "Peer" box, which is multiple processes. The diamond-shapes describes external communication over the network.

Console: A process responsible for User communication. The idea is to have a simple console-interface on which to log messages (it is heavily used for debugging). The Console can in time also be used for parsing user input and affecting the system. It only responds to a "quit" command at the moment. Logging is done with a LogChannel on which the Console process syncs and prints out information.

Main: Initial program entry point. Spawns everything else and is the last thing that closes again.

Status: Keeps track of various torrent-specific statuses. How much is left to download? How many seeders are there? How fast is the torrent currently? The intention is to keep this kind of information around in that process.

Tracker: Communicates with the tracker. Is a specialized HTTP client. It talks to the Timer so it won't continually request the tracker for information. In due time, it should also be able to do UDP tracking. It is using bencoded documents as per the tracker spec. These bcoded documents are handled as in Conjure with the monadic parser generator Parsec.

PeerMgr: Handles a set of peers. Gets lists of other peers via the Tracker process and spawns a pool of them. In due time, this process will handle choking/unchoking of peers so we don't clog the TCP/IP stack by trying to talk to everybody at once.

Listener: Listens on a TCP port and spawns off Peers when they try to connect. Not implemented at all yet, but is fairly easy to get to work.

OMBox: A very simple implementation on top of CML of what is present in Control.Concurrent.SampleVar. This will let the PeerMgr be able to place things for the Peer to read later and it avoids a deadlock between those two which otherwise would occur. OMBox is currently not cleaned up properly. We need some mechanism for poisoning the communication channel so it closes down again gracefully.

Peer: The system contains zero or more peers. Each peer is really four processes. One controls the state of the peer and reacts on the events from the peer. Two are responsible for sending messages to the peer: one as a queue, and one blocks on the socket. Finally, a process receives messages on the socket and then sends them to the controller, blocking in the process.

PieceMgr: Keeps track of the current piece-state. It knows what pieces we have, what we are missing and what to request next. Peers will be using it for asking what to download next.

FS: Abstracts away the file system for the rest of the client. Is implemented as a communication process which peers ask for blocks to send.

Current State:

Currently, the client is pretty flaky. It is able to seed in a controlled lab-setting, but needs much work to be able to leech data from other clients. The TODO-list is fairly long and some of the things on it are pretty hard. But there are also many things which are fairly simple to fix. Feel free to grab anything and help out if it sounds interesting. I am deliberately keeping things simple and leaving doors open. It is my hope that even aspiring Haskell programmers might stand a chance at solving some of the things on the TODO list. I'll happily take any help I can get as in the long run, this is fairly hard to carry out as a single person. The code is on github in my haskell-torrent repository if this should indulge you.

Sea Beyond event summary

2009-12-18T15:58:19+00:00

This thursday 17th of December, ProcessOne has held an event in Paris, France, around real-time communications. Here are the bits.

As announced and programmed, we held our event "Sea Beyond", on this thursday 17th of december. It took place at Le Before, an art gallery in the center of Paris. Usually, we have snow on the french capital only once a year... And the weather chose that special day, to cover the streets with the white jacket.

We had almost 40 attendants during the whole day, either for the XMPP Sandbox part for developpers, or the Lighthouse part for presentations.

We had a wide range of people, from pure players to operators, including manufacturers and freelancers, and even the press. Let's quote a few, like Nokia, Erlang Consulting, Yoono, af83, Ohm Force, StudiVZ, Meetic, Tandberg, Mozilla Foundation Europe, Orange, INRIA, Yoono, ...

During this real-time communications event, there has been a real sandbox as well as a real (tiny) lighthouse. There was also real pizzas for lunch, as well as real Champagne at the end of the day.

The first part of the day should have been a hacking session, at least this was what we had planned. But people took a lot of interest into workshops about PubSub and the Jingle Nodes. People then talked to each other about real-time technologies, and the future of internet and telecom. We have demoed our products, like the OneTeam client that we hope to release soon, also our supervision console TeamLeader. Mathieu Barcikowski also demonstrated Yoono to some people.

Then we reorganized the room for the presentations. We started with the lightning talks, with Philippe Sultan from INRIA, Asterisk developer and book author, who has presented the Asterisk module for Jingle. Nicolas Vérité from ProcessOne Project Manager, then presented the Talkr.IM XMPP service soon to be launched. Jérôme Sautret, ProcessOne CTO, did a summary of Sea Beyond XMPP sandbox. And Mathieu Barcikowski from Yoono, finished the session by demonstrating to the whole audience this time, the Yoono Firefox extension.

The presentations started with Thiago Camargo from Nimbuzz, who introduced his concept of Jingle Nodes, a network of XMPP peers for relay and relay discovery. This presentation has impressed quite a lot of people: the technology, as well as the ease of the Thiago on these subjects. Christophe Romain from ProcessOne the detailed the PubSub use at BBC, for their LiveText over IP in their radios channels in the browser. The session has been closed by Mickaël Rémond from ProcessOne, who talked and demoed our alpha products implementing the Wave real-time communication system from Google.

Then we had a cocktail to conclude the day, with once again ver interesting and enriching talks.

We believe people enjoyed the all day. We hope they enjoyed it as much as we did!

Thank you to everyone who joined and helped made this event a big success!