Unofficial Erlang Planet

There is a common saying that Erlangs I/O layer is slow. Bullshit. The etorrent program can easily sustain 70 megabit with less than 5% of CPU usage on my old machines. The reason is that etorrent is shuffling 16k binary blocks around mostly. But that means you have a pointer to each 16k block and you are only a writev(2) call away from the disk.

This means I/O is not itself a problem. The problem that people often hit is Erlangs string handling. In Erlang a string can either by represented as an immutable array of 8-bit characters, called a binary; or it can be represented as a list of integers (ie, a linked list of integers). The latter is mutable in the sense that you can generate new integers and update the linked list from the front only.

Whenever you mutate the list of integers, garbage is generated. Allocation will happen and this directly translates to more garbage collection pressure later on. Had you had destructible arrays in Erlang it would have been very easy to circumvent the worst performance problems, but you will have to do with a limited zoo of types. In better languages, like Ocaml, Standard ML or Haskell, your type zoo is much richer here. You can choose between several immutable and mutable variants so the leverage is much greater.

First trick: Erlang has the concept of an IO list. These are lists of binary/string chunks. They are fast because they lend themselves to a writev(2) in the bottom of the system. Thus you don't have to concatenate strings in the Erlang language but you can hand a quasi-concatenated list to the VM and let C do the heavy-lifting for you.

Second trick: Mutating data as strings is slow. But if you have a abstract tree of tuples then it can be manipulated fast. If you think about the command [Bin | IOList] where we add a Binary to the front of an IOList, you'll know it happens in O(1) (constant) time. If we build a tuple {atom, D1, D2, ...}, we can build it in constant time. So the trick is to maintain your data in a non-string form for as long as possible and then only change it to a string at the very last point in time.

Third trick: Correct data structure. Ropes. Finger Trees. M-way trees. Don't brute force yourself over a string. Process the structure by pattern matching in a clever way so you minimize the amount of change needed. A good tree will have around logarithmic change whereas a list will have linear change. This leads to the next point: allocation.

Another crap problem in Erlang is that you have few (traditional) ways to minimize allocation. Any kind of operation will allocate data if you are not wary of it. Hence you should try to build programs that do not gloss over data and allocate all the time. Mapping of lists via list comprehensions is a good example. It is easy to understand but it will build a new list and garbage collection will cost you. If you can do multiple things in one list comprehension, you win traversals and allocations.

Whenever I hear about "Slow string processing" in Erlang I immediately think: "Another Ruby programmer trying to cram his language into Erlang". If you hit slowness the trick in Erlang is to redefine the problem! Don't try to string process yourself into oblivion. If the sole thing your problem concerns is string-processing, go use perl. Please. Or be clever and use Ocaml. Pre-process the data in other languages and make them into erlang terms which can be read fast by erlang. Erlangs power is the concurrency at which it is really good. There were no reason for giving it fast string processing.

Also, don't get bitten by the mnesia bug, please. It is almost always the case that mnesia and ETS are the wrong way to go. I got bitten by it multiple times in etorrent and now I am contemplating ripping all of it out again. Plain old data structures baby! There are some nasty caveats to mnesia and ETS and they are not related to the size limit of the database.

For mnesia, be aware that a failed transaction will be re-run! Now take 100 transactions tripping over each other. They will never finish. Etorrent did that when peers should update their state. Instant snail-speed application. Mnesia is crap at doing joins. Just forget it.

ETS is also problematic: Any store or get from ETS means a data copy. Don't do as etorrent and add large rows (128k+) to ETS. Your app will come to a grinding halt.

Don't message big data structures around. Any message incurs a copy (in the default VM). If you need to work with a big data structure, shove it into a process and then send the Pid of that process around or register him globally. You can think of a Pid as "Passing a pointer around" in this case.

And finally: Measure measure measure! Be scientific about your performance problems. You need to know exactly what causes it before jumping to conclusions. And don't worry too much about being parallel at the beginning. You can always split things up in finer granularity later.

It’s been a few weeks since we released the initial chapters of O’Reilly’s CouchDB book. The next release ought to be happening “soon” – There’s more work to be done on it, but the major skeleton of the chapters has been written, and the figures drawn.

I’m posting the outline of the next section, so that people’s expectations about the book are inline with what we provide. The next released section will be a few chapters walking through the creation of a simple web app (yes, a weblog). The point is not to teach Ajax, but to teach the CouchDB HTTP API in a hands on way.

Yes, all the code runs, and the in-progress version of the example blog is even available on github. One of my main tasks between now and the release is to make the installation proceedure much easier. So you are welcome to play now, but if it seems messy or confusing, please try again when the chapters are released.

If anyone has input on pacing and ordering of the example application I’d love to hear it. I think this section should be relatively fast-paced, and focussed on how easy it is to build applications using CouchDB, with the occasional bits of API specification or deeper material presented as asides and sidebars.

2. Learning CouchDB

Touring the Futon Administration Interface

A survey of CouchDB’s http api, through the eyes of Futon. Touches on most of the core concepts. This material is might be better suited for Section 1, as it does stand alone as an introduction to the http API.

Introducing the Example Blog

Why we chose JavaScript for the book examples. Also: even if you are storing scientific data, what you’ll learn by following along is applicable (views, document semantics, design docs).

Setting Up Your Development Environment

Getting the example app to your local drive, and being able to push it to CouchDB. Also cover design-doc / application correspondence while explaining the couchapp script.

Storing Blog Posts in CouchDB

The JSON doc format, validation functions, plus the HTML and Javascript it takes to support the blog post form. (This will be a longer one, but the code for it shouldn’t be changing.)

Rendering Documents with CouchDB Forms

What forms are. Why they have the single-doc constraint. Mention that you can write raw JS, but then show CouchApp’s require / include system as a best practice, using templates and a basic templating engine.

Browsing Recent Posts with Map Views

Introduce a simple view by published at date. Show basics of pagination. Explain the view-query API by looking at the JavaScript client API used to query the view.

Mention that there will be forms for views in the future, so that this page can be served as HTML as well. (Forms for views should be in Couch 1.0 – and thus in the book, so polishing this section now is lower priority. Pagination will also be easier with view forms)

Accepting Reader Comments

The JSON format, validations, and the HTML form for updating a comment. This is a short one, mostly repeats what happened in “Storing Blog Posts in CouchDB” but its the second time, so we can touch on different aspects and explain other API details. (One neat difference is that we’re adding Javascript behavior to a page that was rendered as a CouchDB form, rather than a static document.)

Listing Related Comments Using View Collation and Compound Keys

(I didn’t say “joins”) ;)

Currently this uses an Ajax query that lists comments by [post_id, date]. When view forms are available, we’ll change it to Christopher Lenz’s query, which includes the post itself, so the whole thing can be rendered in a single request. This chapter works on the same files as the previous chapter, they could be “joined” but I think this is material we should take the space to cover as clearly as possible.

---

If you’ve gotten this far, then you must be seriously interested in CouchDB. If you are, then you should familiarize yourself with the test suite. It’s written in JavaScript, so anyone can understand it, and it’s the main gateway for new features. Plus, code speaks louder than words.

Abstractions at a higher level through type constructor polymorphism. Good type systems are expressive enough to conceal the implementation complexity, and expose *only* what it matters to the developer. People often cringe about the complexity of Scala's type system and how it serves as a barrier to the entry point in mainstream programming. As Michael Feathers recently noted on Tweeter, the unfortunate fact is that people often jump at the esoteric parts of a language before looking at the simpler subset, which he will be using 90% of the time. And, I think Scala has that sweet spot, where you need not mess around too much with variances, implicits and existentials and yet come up with a nice, concise and functional codebase.

In this post, I discuss the already introduced intimidating phrase "Type Constructor Polymorphism" through a series of code snippets ranging from toys to some real-world stuff. The idea is, once again, not to evangelize type theory intricacies, but share some of the experiences of how this feature in Scala's type system can help write idiomatic code, while staying away from the complexities of its underlying implementation.

Jump on ..

We have a list of Option[String] that we need to abstract over and compute some value. Say, for the sake of keeping the example simple, we will calculate the sum of lengths of all the strings present ..

val employees: List[Option[String]] =
  List(Some("dave"), None, Some("john"), Some("sam"))

val n: List[Int] =
  employees.map { x =>
    x match {
      case Some(name) => name.length
      case None => 0
    }
  }.elements.reduceLeft[Int](_ + _)
println(n)


Let us take another problem that needs to abstract over a different List structure, a List of List of Strings, and compute the same result as before, i.e. the sum of lengths of all the strings encountered in the collection ..

val brs: List[List[String]] =
  List(List("dave", "john", "sam"), List("peter", "robin", "david"), List("pras", "srim"))

val m: List[Int] =
  brs.flatMap {x => x.map {_.length}}
     .elements.reduceLeft[Int](_ + _)
println(m)


Do you see any commonality in the solution structure in the above snippets ? After all, the problem space has a common generic structure ..

1. we have a List with some String values abstracted in different forms


2. need to iterate over the list


3. do some stuff with elements in the list and compute an Int value

Unfortunately the actual solution structures look quite different and have to deal a lot digging into the abstractions of the underlying representations within the collection itself. And this is because, we cannot abstract over the type constructor (the List in this case) that takes another type constructor as an argument (Option[String] in the first case and List[String] in the second case).

Enter type constructor polymorphism.

Sounds intimidating ? Maybe, but ask the Haskellers .. they have been using typeclasses ever since so successfully in comprehensions, parser combinators and embedded DSLs and programming at a different level of abstraction.

Scala supports type constructor polymorphism since 2.5, and the details have been discussed in a recent paper by Adriaan Moors et al in OOPSLA 2008.

Here is a snippet of the Scala code that works seamlessly for both of the above cases ..

val l: List[Int] = employees.flatMapTo[Int, List]{_.map{_.length}}
val sum: Int = l.elements.reduceLeft[Int](_ + _)
println(sum)


The above code abstracts over List through higher order parametric polymorphism, i.e. independent of whether the List parameter is an Option[] or another List[]. Incidentally both of them (List and Option) are monads and flatMapTo abstracts a monadic computation, hiding all details of type constructor polymorphism from the developer.

Now here is some real life example (elided for simplicity) ..

Here are the simple domain models for Customer, Instrument and Trade, used for modeling a use case where a Customer can order for the Trade of an Instrument in an exchange.

case class Customer(id: Int, name: String)
case object nomura extends Customer(1, "NOMURA")
case object meryll extends Customer(2, "MERYLL")
case object lehman extends Customer(3, "LEHMAN")

case class Instrument(id: Int)
case object ibm extends Instrument(1)
case object sun extends Instrument(2)
case object google extends Instrument(3)

case class Trade(ref: Int, ins: Instrument, qty: Int, price: Int)


And we fetch the following list through a query from the database. It is a List of tuples where each tuple consists of a Customer and a trade that has been executed based on the Order he placed at the exchange. And here is the snippet of the code that computes the sum total of the values of all trades executed in the day for all customers.

val trades: List[(Customer, Option[Trade])] =
  List((nomura, Some(Trade(100, ibm, 20, 12))),
       (meryll, None), (lehman, Some(Trade(200, google, 10, 10))))

val ts: List[Option[Trade]] = trades.map(_._2)
val t: List[Int] = ts.flatMapTo[Int, List]{_.map{x => x.qty * x.price}}
val value = t.elements.reduceLeft[Int](_ + _)
println(value)


Really not much different from the above simple cases where we were dealing with toy examples - isn't it ? The structure of the solution is the same irrespective of the complexity of data being stored within the collections. The iteration is being done at a much higher level of abstraction, independent of the types stored within the container. And as I mentioned above, flatMapTo is the secret sauce in the above solution structure that abstracts the building of the new container hiding the inner details. To get more into the innards of flatMapTo and similar higher order abstractions, including the new form of Iterable[+T], have a look at the OOPSLA paper referenced above.

Postscript: In all the snippets above, I have been explicit about all type signatures, just for the sake of easier understanding of my passionate reader. In reality, almost all of these will be inferred by Scala.

Planet Erlang is fastly becoming a large index for Erlang projects resources, including ejabberd.

Since its relaunch, Planet Erlang is now indexing content from tens of Erlang related blogs. Thank to its search engine, you can now find many blog posts and articles on lots of different Erlang topic. ejabberd is one of the most covered topic there.

You should have a look and try searching for ejabberd in the search engine (search field on the right).

The list of resources is growing every day as we are adding even more older resources.

Your comments and feature requests are welcome !

A few months back Klacke gave me committer rights for Yaws. I’ve made a few fixes here and there, including adding support for passing “*” as the request URI for OPTIONS, enabling OPTIONS requests to be dispatched to appmods and yapps, and completing a previously-submitted patch for configuring the listen backlog. Klacke has just started putting a test framework into the codebase and build system so that contributors can include tests with any patches or new code they submit, and I’ve contributed to that as well.

The biggest feature I’ve added to date, though, is a new linked-in driver that allows Yaws to use the sendfile system call on Linux, OS X, and FreeBSD. I never wrote a linked-in driver before, so I was happy and fortunate to have an Erlang expert like Klacke providing hints and reviewing my code.

I did some preliminary testing that showed that sendfile definitely improves CPU usage across the board but depending on file size, sometimes does so at the cost of increasing request times. I used my otherwise idle 2-core 2.4GHz Ubuntu 8.04.1 Dell box with 2 GB of RAM, and ran Apache Bench (ab) from another Linux host to simulate 50 concurrent clients downloading a 64k data file a total of 100000 times. I saw that user/system CPU on the web host tended to run around 33%/28% without sendfile, while with sendfile it dropped to 22%/17%. The trade-off was request time, though, where each request for the 64k file averaged 0.928ms with sendfile but 0.567ms without. With larger files, however, sendfile is slightly faster and still has better CPU usage. For example, with a 256k file, sendfile averaged 2.251ms per request with user/system CPU at 8%/16% whereas it was 2.255ms and 16%/27% CPU without sendfile, which makes me wonder if the figures for the 64k file are outliers for some reason. I performed these measurements fairly quickly, so while I believe they’re reasonably accurate, don’t take them as formal results.

On my MacBook Pro laptop running OS X 10.5.6, CPU usage didn’t seem to differ much whether I used sendfile or not, but requests across the board tended to be slightly faster with sendfile.

I ran FreeBSD 7.0.1 in a Parallels VM on my laptop, and there I saw significantly better system CPU usage with sendfile than without, sometimes as much as a 30% improvement. Requests were also noticeably faster with sendfile than without, sometimes by as much as 17%, and again depending on file size, with higher improvements for larger files. User CPU was not a factor. All in all, though, I don’t know how much the fact that I ran all this within a VM affected these numbers.

Given that Yaws is often used for delivering mainly dynamic content, sendfile won’t affect those cases. Still, I think it’s nice to have it available for the times when you do have to deliver file-based content, especially if the files are of the larger variety. Anyway, I committed this support to the Yaws svn repository back around December 21 or so. If you’d like to do your own testing, please feel free — I’d be interested in learning your results. Also, if you have ideas for further tests I might try, please leave a comment to let me know.

I'm releasing OpenPoker, a scalable poker server written in Erlang, under a Creative Commons Non-Commercial License.

Please make a $100 donation at GitHub and subscribe to The Erlang Journal if you want to discuss the OpenPoker architecture!

The donations button is in the upper right corner of this page, or you can just PayPal me at joelr1@gmail.com.

We are building a platform for message switching, in Erlang. Everything looks OK on stability and features. It actually has run more than half year with zero down. We tested its performance on our 2-core CPU machine before, and got about 140 transactions/second, it's good enough.

Then, we got a 8-core CPU machine several weeks ago, and we did same performance testing on it, to see the scalability. Since Erlang is almost perfect on scalability, you can image the result, yes, about 700 transactions/second now, scaled almost linear. Until it crashed with "out of memory" when million hits processed.

It left a very big "erl_crash.dump" file there, I had to dig the issue. My first guess was, were some remote requests (access db, access remote web service etc) timeout but the process itself was not timeout yet, and cause more and more processes kept in VM?

A quick grep "=proc:" erl_crash.dump showed that the total number of processes was about 980, which was reasonable for our case.

So, which process ate so many memory? A quick grep "Stack+head" erl_crash.dump showed that there was indeed a process with 285082125 size of Stack+head there.

Following this clue, I caught this process:

=proc:0.4.0>
State: Garbing
Name: error_logger
Spawned as: proc_lib:init_p/5
Last scheduled in for: io_lib_format:pad_char/2
Spawned by: 0.1.0>
Started: Sun Apr  1 01:21:50 2012
Message queue length: 2086029
Number of heap fragments: 1234053
Heap fragment data: 281266956
Link list: [0.27.0>, 0.0.0>, {from,0.42.0>,#Ref0.0.0.88>}]
Reductions: 72745575
Stack+heap: 285082125
OldHeap: 47828850
Heap unused: 121777661
OldHeap unused: 47828850
Program counter: 0x0764c66c (io_lib_format:pad_char/2 + 4)
CP: 0x0764c1b4 (io_lib_format:collect_cseq/2 + 124)

This process was error_logger, which is from OTP/Erlang standard lib: error_logger, writing received messages to log file or tty. The typical usage is:

error_logger:info_msg("~p:~p " ++ Format, [?MODULE, ?LINE] ++ Data))

Which will format Data to a String according to the Format string, and write it to tty or log file.

The above case showed the message queue length of process "error_logger" had reached 1234053, and the Stack+heap was 285082125, about 272M size.

So the cause may be, that the message queue could not be processed in time, the messages were crowded in error_logger's process and finally caused "out of memory". The bottle-neck was that when error_logger tried to format the message to String, Erlang VM was weak on processing them, which seemed to need a lot of CPU cycles.

In my previous blog, I talked about Erlang is bad on massive text processing. Erlang processes String/Text via List, which is obvious bottle-neck in Erlang now, with Erlang is getting much and much popular and more and more Erlang applications are written.

But, why this did not happen on our 2-core CPU machine? It's an interesting scalability related problem:

"error_logger" module will registered one and only one process to receive and handle all log messages. But Erlang VM's scheduler can not distribute ONE process to use multiple CPUs' computing ability. In our 2-core machine, the whole ability is about 140 transactions/second, the one process of "error_logger" just happened to have the power to handle corresponding log messages in time. Under 8-core CPUs machine, our platform scales to handle 700 transactions/second, but there is still only one process of "error_logger", which can not use 8-core CPUs' ability at all, and finally fail on it.

Erlang treats every process fairly (although you can change the priority manually), we can do a simple/quick evaluation:

1. 2-Core machine, keeping hits at 140 trans/second:
The number of simultaneous processes will be about 200, each process shares the CPU cycles: 1/200 * 2 Core = 1%

2. 8-Core machine, keeping hits at 700 trans/second:
The number of simultaneous processes will be about 980, each process shares the CPU cycles: 1/980 * 8 Core = 0.82%

So, the CPU cycles shared by error_logger process actually not increases.

BTW, I think error_logger should cut its message queue when can not process them in time (disk IO may also be slower than receiving messages).

Erlang is like an exotic beautiful woman with no dressing sense. I came across Erlang about a year ago. It is a language of a kind - precise, crafted and powerful. Its been there for about two decades now, and people have been using it for serious real-time applications. But the hype Erlang is getting in [...]

An MD5 hash for any given message is 16 bytes (128 bits) in size and is represented by a unique 32 digit Hexadecimal number. Erlang has a built-in-function to calculate the MD5, which returns the hash in the form of a binary data-structure. $ erlang:md5("hello"). <93,65,64,42,188,75,42,118,185,113,157,145,16,23,197,146> But what we usually need is a string representation of the Hexadecimal value. [...]

Although there are a few other editors for Erlang, I prefer to use Emacs for Erlang and its the only major reason I use Emacs for. Erlang now has an eclipse plugin too, called Erlide. Erlang comes with the emacs mode as part of its standard distribution, so you only need to customize your emacs settings [...]

On a fresh install of Leopard, the following is what I did in order to set up Erlang working on my Mackbook Pro: 1. Download Xcode by going to Apple ADC. Xcode is the Apple’s developer tool set, and its available for free. At the time of my install, the latest is Xcode 3.0 and its [...]

Update 1: Retest data (using different machine and Erlang kernel polling) added near bottom of post.

Update 2: More details and testing on the weird MochiWeb kernel polling results, bottom of post.

Almost a year ago I did some Apache and Nginx performance testing. Apparently I have the bug again and have done some performance testing on Nginx, Yaws and MochiWeb. The latter two being Erlang based. Again deathmatch may be an overstatement but this is my attempt at gleaning some interesting performance data from some high performance web servers. Also, I attempted to improve the graphs this time around since they were a bit hard to read the last time.

The Setup:

I was not able to use the same server and setup as the last time, so comparing between this and my last deathmatch probably isn’t very accurate. For this test I used a Intel Dual Core 2.2GHz, 4GB RAM machine running Ubuntu 8.10 (64bit) and for the test server. Erlang (R12B-3), Yaws (1.77) and Nginx (0.6.32) are installed from the standard repository and mochiweb from subversion (rev 88). All are using the default configurations outside of adjusting listening port numbers. The test is again against a basic robots.txt file. The tests were done using a consumer grade 100mb switch and all tests originated from an old laptop I had laying around. I think that about covers the test bed, if you have any questions let me know.

For the tests I used autobench (httperf under the hood) with the following command, each test ran ten minutes apart. The order of the tests were done in was MochiWeb then Yaws and lastly Nginx.

autobench –single_host –host1 HOST –port1 PORT –uri1 /robots.txt –low_rate 10 –high_rate 200 –rate_step 10 –num_call 10 –num_conn 5000 –timeout 5 –file SERVER-results-`date +%F-%H:%M:%S`.tsv

The Results:

There are a few results from httperf/autobench that I would like to show, errors, network I/O, reply rate (and it’s standard deviation) and response time. (click on the graphs for a larger view)

MochiWeb and Yaws both seem to be the most consistent here. Nginx had a couple of funky spikes, I do not know if this was an issue with Nginx or with my tests and/or test bed. Take from it what you will.

Nginx seems to use a bit more network I/O consistently through the lower ranges of this test and then again as some spikes. MochiWeb and Yaws seem to have some inconsistencies as well.

The reply rate and network I/O graphs certainly seem to be tied, which would make sense. Edit: Average reply rate is average replies per second.

In the higher reaches of the tests Yaws seems to be most consistent.

MochiWeb seems to have consistently the highest response times with Nginx has the lowest. This also follows the data from the first deathmatch. Nginx had consistently low response times against Apache. Edit: Response time is how quickly replies are sent in milliseconds.

Next up are the system graphs, I have CPU usage (both cores combined), context switches, interrupts and load. To help read these please note recall that each test ran ten minutes apart and the order of the tests was MochiWeb then Yaws and lastly Nginx. The data was gathered using sar at five minute intervals and graphed using ksar.

It seems Nginx is the clear winner here. Kernel polling may be the answer here, a retest may be in order to see if it makes a difference.

MochiWeb and Nginx seem pretty even on context switches with Yaws a little higher. I suppose turning on kernel polling might make this a bit more even, since Erlang and Nginx both use epoll. This may also account for the CPU usage difference above.

Interrupts are fairly even across all of them.

Again Nginx takes it, again likely due to kernel polling being disabled. That’s my best guess anywho.

The data I used to create the graphs and etc is available here.

Let me know if you are interested in me retesting anything, I may try to enable kernel polling and try again if I get a chance.

Note that these are *my* experiences with each webserver, your testing and experiences may be different. As with most things there are pro’s, con’s, trade offs and pitfalls. The only way to find out what will work best for your environment is to test, test and test.

Update:

I performed the upper half of the tests again to see if there were any changes to sporadic jumps in the graphs http performance graphs. My initial test using the old laptop I saw the same results. I then ran the tests from a VM (running Ubuntu 8.10 in a KVM VM) on my dual core machine and found that the results were much more even. Unfortunately it’s the same machine that the webservers are running on but the results look much better. The first set is using the same setup as before but just adjusted to have the top half test. The second is the same test but with kernel polling turned on in Erlang.

All of them are very even and close, no real winners here.

Looks like Nginx is the clear winner with Yaws next, followed by MochiWeb.

Pretty much the same as last time (likely a little higher across the board due to running the tests in a VM on the same machine). Note that Nginx is a system process, so for Yaws and MochiWeb follow the blue line and Nginx follow the green.

About the same as before, other than being higher due to running a VM.

Pretty much the same as before again, Nginx seems the lowest.

Now for the tests with kernel polling enabled in Erlang (erl +K true).

With kernel polling on it looks like Yaws actually performs better in the reply rate test with MochiWeb performing worse and Nginx in the middle

In the response time test a huge change is noted, MochiWeb goes from roughly a ~14 ms response time at 2000 requests to ~65 ms. Also noted Yaws performs much better matching or beating Nginx.

With kernel polling in the Erlang webservers Nginx still seems to come out on top for CPU usage.

Following the performance trend we saw above Yaws sees a drop in context switches and MochiWeb increases.

Load-wise things stay roughly the same with Nginx being the lowest.

While it certainly seems that my old laptop that I did the original tests on is too slow or has a network issue, hopefully with these new tests we have some more clarity. It seems that Yaws improves with kernel polling enabled and competes well with Nginx. MochiWeb on the other hand apparently has issues with kernel polling and actually degrades performance. If anyone has more info on the internals of MochiWeb and possible causes I would be certainly interested.

If anyone would like the data from the second round of tests it is available here.

Update 2:

I did some more testing to see what the issue might be with MochiWeb, response times and kernel polling. I did a few tests with different versions of Erlang, with and without kernel polling and testing from within and outside a KVM VM. From what I can tell the issue seems to be isolated to testing from within a VM with MochiWeb and kernel polling. Seems to be sorta strange but all my testing and retesting shows the same issue. Just to be clear on my setup, I am running httperf from with in a VM to MochiWeb running outside the VM. Here is the latest round of testing to show this point.

Even though the numbers are higher from within the VM without kernel polling, it certainly seems to be an issue with the combination of MochiWeb, KVM and kernel polling. Since I did not see the same spike from within a VM in the earlier tests with Yaws and kernel polling I assume it is not an issue with Erlang or it’s kernel polling mechanism conflicting with KVM. I am not entirely sure what to make of this other than MochiWeb, kernel polling and KVM don’t play well together and that kernel polling actually helps MochiWeb significantly when KVM is not involved. If anyone has any ideas on why that may be I am all ears.

Sriram Krishnan was recently lamenting the lack of SHA-256 support in Erlang’s crypto module, so just for fun I wrote a simple sha256 module based on the pseudocode in Wikipedia. The tests use the test data from this C implementation.

[Update: I've posted a new module that implements more SHA digest variants and so renders this module obsolete.]

Yesterday I posted a SHA-256 Erlang module, but I figured since other SHA algorithms are similar, I might as well finish the job. I grabbed the Secure Hash Standard and went to work.

The resulting new module is named sha2 and it implements SHA-224, SHA-256, SHA-384, and SHA-512.

• Code
• Light documentation
• Tests are included in the module but you get the test vectors here
• It’s open source, BSD license

I hope you find it useful.

Happy New Year! It’s been a great 2008 for Erlang, which we’ll cover in our 2008 Retrospective in another post later this week. For today, it’s a new, simple web framework for Erlang called BeepBeep. Written by Dave Bryson, it uses MochiWeb, which most Erlang Web developers are familiar with, and ErlyDTL, which is an Erlang port of the Python Template Language. Travis Slicegood has a demo application written in BeepBeep grabbing photos from Flickr, and Dave Bryson has a sample Blog Application as well.

The new set of Erlang web frameworks is more proof of the number of ambitious, sharp developers moving to the language is growing.

2008 was a great year of activity and excitement. I've got the sneaking suspicion that 2009 is going to be no less busy.

• Finished and published my first book!
• Spoke at CUFP and met a lot of really smart people.
• Gave several tech-talks and presentations.
• Learned how to develop apps for the iPhone.
• Migrated I Play WoW from a Perl app to an Erlang app.
• Learned about CouchDB.
• Learned Git.
• Accepted a lead position at EA.

This is what I'd like to accomplish in 2009.

• Finish my second book.
• Do more tech-talks and presentations.
• Have a 100 day streak of open source commits on GitHub.
• Write more technical articles and entries.
• Learn Python.
• Hit level 80 with Korale and get into tier 9.
• Grow I Play WoW to 75,000 monthly active users.

It's 2009 now, in Beijing.

1==0.999999999......

I met Erlang 2 years ago, which finally brings me to Scala. I learnt a lot from Erlang, and I entered the Scala world with Erlang atmosphere surrounding me. The FP, the Pattern Match, the Actor/Process, I found these familiar friends in Scala everywhere.

Scala has extra bonus, to me, static types and OO/FP. The domains I face are usually with a lot of business logic, or, the worlds I try to describe are not only messages, they are, models I don't think are suitable to describe in Function only.

The world itself is OO/FP mixed, like Martin's quote: Two sides of coin. It's something like the Particle/Wave in Quantum. The world is an infinite whole, but the reason of Human Being is always finite, we are using out finite reason to measure the infinite world, it's an unsolvable contradiction: Infinity vs Finite. We have to read our world in OO and, in FP, in snapshot and in continuation.

There won't be "Super Hero" in computer languages, the world is getting self-organization and harmony, so do the languages. Each language is living in an eco-system, born, growing via interacting with environment, disappear ...

The Economy

It was bad in 2008. I tried to do some computing on stock market based on my neural network. What I can say is it will be swing in the next half-year, no big drop, no big rise. The Shanghai Stock Index will swing between 1200 and 3000. At least, no big worse any more.

My Self

I need to make some big decisions in this a year.

• Writing Your Own GPS Applications - Programmer's Heaven
• CodeProject: GPS- Deriving British Ordnance Survey Grid Reference from NMEA data. Free source code and programming help
• NMEA library
• NMEA:: Wiring / Arduino library for easy decoding of GPS data
• NMEA Parser Design
• http://vancouver-webpages.com/peter/nmeafaq.txt
• SourceForge.net: pyserial » pySerial
• reg:exp: GPS + Google Maps Mash-up in 42 lines of code
• Glenn Baddeley - GPS - NMEA sentence information
• http://gpsd.berlios.de/NMEA.txt
• NMEA data
• Product CAN-Ethernet Gateway - SYS TEC electronic GmbH
• Clutter Toolkit
• How to wrap C functions to OCaml
• Oxford Oberon-2 Compiler

Keeping Erlang in mind, I've coded two months in Scala, I'm thinking something called "Scala vs Erlang", I wrote some benchmark code to prove me (the code and result may be available someday), and I'd like to do some gradually summary on it in practical aspect. These opinions may be or not be correct currently due to lacking of deep experience and understanding, but, anyway, I need to record them now and correct myself with more experiences and understanding got on both Scala and Erlang.

Part I. Syntax

List comprehension

Erlang:

Lst = [1,2,3,4],
[X + 1 || X - Lst],
lists:map(fun(X) -> X + 1 end, Lst)

Scala:

val lst = List(1,2,3,4) 
for (x - lst) yield x + 1
lst.map{x => x + 1}
lst.map{_ + 1} // or place holder

Pattern match

Erlang:

case X of
   {A, B} when is_integer(A), A > 1 -> ok;
   _ -> error
end,

{ok, [{A, B} = H|T]} = my_function(X)

Scala:

x match {
   case (a:Int, b:_) if a > 1 => OK // can match type
   case _ => ERROR
}

val ("ok", (h@(a, b)) :: t) = my_function(x)

List, Tuple, Array, Map, Binary, Bit

Erlang:

Lst = [1, 2, 3] %% List
[0 | Lst]  %% List concat
{1, 2, 3}  %% Tuple
1, 2, “abc”>>  %% Binary
%% no Array, Map syntax

Scala:

val lst = List(1, 2, 3)  // List
0 :: lst  // List concat
(1, 2, 3) // Tuple
Array(1, 2, 3) // Array
Map(“a” -> 1, “b” -> 2) // Map
// no Binary, Bit syntax

Process, Actor

Erlang:

the_actor(X) -> 
   receive 
      ok -> io:format(“~p~n”, [X]);
      I -> the_actor(X + I) %% needs to explicitly continue loop
   end.
P = spawn(mymodule, the_actor, [0])
P ! 1
P ! ok

Scala I:

class TheActor(x:Int) extends Actor { 
   def act = loop {
      react {
         case “ok” => println(x); exit // needs to explicitly exit loop
         case i:Int => x += i
      }
   }
}
val a = new TheActor(0)
a ! 1
a ! “ok”

Scala II:

val a = actor { 
   def loop(x:Int) = {
      react {
         case "ok" => println(x)
         case i:Int => loop(x + i)
      }
   }
   loop(0)
}
a ! 1
a ! "ok"

Part II. Processes vs Actors

Something I

Erlang:

• Lightweight processes
• You can always (almost) create a new process for each new comer
• Scheduler treats all processes fairly
• Share nothing between processes
• Lightweight context switch between processes
• IO has been carefully delegated to independent processes

Scala:

• Active actor is delegated to JVM thread, actor /= thread
• You can create a new actor for each new comer
• But the amount of real workers (threads) is dynamically adjusted according to the processing time
• The later comers may be in wait list for further processing until a spare thread is available
• Share nothing or share something upon you decision
• Heavy context switch between working threads
• IO block is still pain unless good NIO framework (Grizzly?)

Something II

Erlang:

• Try to service everyone simultaneously
• But may loss service quality when the work is heavy, may time out (out of service)
• Ideal when processing cost is comparable to context switching cost
• Ideal for small message processing in soft real-time
• Bad for massive data processing, and cpu-heavy work

Scala:

• Try to service limited number of customers best first
• If can not service all, the later comers will be put in waiting list and may time out (out of service)
• It's difficult for soft real-time on all coming concurrent customers
• Ideal when processing cost is far more than context switching cost (context switch time is in ns on modern JVM)
• When will there be perfect NIO + Actor library?

PlanetErlang, the Erlang news and blog post aggregation site has been rewritten.

The new site should be easier to read. It includes a search engine, that allows users to search inside the history of Erlang posts among all the known blogs. The sitebar also contains latest Twitter post on Erlang.

Planet Erlang is available on the following address: http://www.planeterlang.org.

Enjoy !

This isn't a beginner's walk through. It assumes you know how to use and read Erlang as well as install Erlang and Perl libraries

I love TAP. TAP is a protocol used to create and interact with testing frameworks. The general idea is that your unit tests interact with a TAP client which outputs the TAP protocol that is then consumed by a TAP server. What it boils down to is a really simple text based structure used to convey unit test results, with a few added bonuses here and there.

With etap and Perl's TAP::Harness we can create really slick test result output and move closer to having a continuous integration build environment. Before you get ahead of yourself, you'll need to make sure that you've got etap downloaded and in your Erlang lib path. You'll also have to install TAP::Harness and it's dependancies.

To demonstrate how this is done, we'll use the erlang_protobuffs library. This library is composed of two modules that provide functionality through direct interaction with their exported functions. They don't extend any OTP behaviors and have very few side effects. The project's sources are contained in the src directory and it's tests in the t directory. There is also a support directory that provides the Makefile include for building the actual modules.

The test target in the root Makefile calls the prove command with the verbose flag and a glob of the *.t files in the t diretory.

test: all
    prove -v t/*.t

The actual tests for the project are escript files with the .t extension. Using the escript hash bang lets us treat each file as it's own complete unit test. They can be run in any order and have their own configuration and directives. Below is the protobuffs_t_001.t file.

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -pa ./ebin -sasl errlog_type error -boot start_sasl -noshell

main(_) ->
    etap:plan(8),
    etap_can:loaded_ok(protobuffs, "module 'protobuffs' loaded"),
    etap_can:can_ok(protobuffs, encode),
    etap_can:can_ok(protobuffs, encode, 3),
    etap_can:can_ok(protobuffs, decode),
    etap_can:can_ok(protobuffs, decode, 2),
    etap_can:loaded_ok(protobuffs_compile, "module 'protobuffs_compile' loaded"),
    etap_can:can_ok(protobuffs_compile, scan_file),
    etap_can:can_ok(protobuffs_compile, scan_file, 1),
    etap:end_tests().

Lines 1, 2 and 3 are used by the shell and escript command to set any Erlang vm options necessary. In this case we want to include the ebin directory, start sasl with minimal logging and disable the shell.

The main/0 function is called escript with whatever relevant arguments that may apply, but we don't care about them so we ignore them for now. Here we perform several tests to assert that the project's modules load and export the proper functions.

The etap:plan/1 and etap:end_tests/0 functions wrap the etap*:* function calls and is used to define the test plan. At this point etap is not designed to work without finite plans.

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -pa ./ebin -sasl errlog_type error -boot start_sasl -noshell

-record(person, {name, address, phone_number, age, location}).

main(_) ->
    etap:plan(1),
    etap:is(protobuffs_compile:scan_file("t/simple.proto"), ok, "simple.proto compiled"),
    compile:file("simple_pb.erl"),
    Data = [{1, "Nick">>, string}, {2, "Mountain View">>, string}, {3, "+1 (000) 555-1234">>, string}, {4, 25, int32}],
    BinData = erlang:iolist_to_binary([protobuffs:encode(Pos, Value, Type) || {Pos, Value, Type} - Data]),
    #person{ name = "Nick">>, address = "Mountain View">>, phone_number = "+1 (000) 555-1234">>, age = 25} = simple_pb:decode_person(BinData),
    BinData = simple_pb:encode_person(#person{ name = "Nick">>, address = "Mountain View">>, phone_number = "+1 (000) 555-1234">>, age = 25}),
    etap:end_tests().

In the protobuffs_t_005.t file you see that you can do things like define records. In this test unit we call several of the library's exported functions and test some of the more complex functionality. The protobuffs_compile:scan_file/1 function actually writes several files which are then compiled by compile:file/1 and used by subsequent test unit function calls.

The output of the make test command, as processed by the TAP::Harness perl module, is very clean.

mbp:erlang_protobuffs ngerakines$ make test
mkdir -p ebin/
(cd src;make)
erlc -W -I ../include  +debug_info -o ../ebin protobuffs.erl
erlc -W -I ../include  +debug_info -o ../ebin protobuffs_compile.erl
prove t/*.t
t/protobuffs_t_001....ok   
t/protobuffs_t_002....ok   
t/protobuffs_t_003....ok   
t/protobuffs_t_004....ok   
t/protobuffs_t_005....ok   
t/protobuffs_t_006....ok   
All tests successful.
Files=6, Tests=17,  1 wallclock secs ( 0.04 usr  0.02 sys +  0.97 cusr  0.27 csys =  1.30 CPU)
Result: PASS

Depending on the error and severity, failed tests will either output in a similarly clean fashion or stop the entire suite from processing.

mbp:erlang_protobuffs ngerakines$ make test
mkdir -p ebin/
(cd src;make)
erlc -W -I ../include  +debug_info -o ../ebin protobuffs.erl
erlc -W -I ../include  +debug_info -o ../ebin protobuffs_compile.erl
prove t/*.t
t/protobuffs_t_001....ok   
t/protobuffs_t_002....ok   
t/protobuffs_t_003.... Failed 1/3 subtests 
t/protobuffs_t_004....ok   
t/protobuffs_t_005....ok   
t/protobuffs_t_006....ok   

Test Summary Report
-------------------
t/protobuffs_t_003 (Wstat: 0 Tests: 3 Failed: 1)
  Failed test:  2
Files=6, Tests=17,  2 wallclock secs ( 0.04 usr  0.02 sys +  0.96 cusr  0.26 csys =  1.28 CPU)
Result: FAIL
make: *** [test] Error 1

The prove command has a number of options and arguments that augment test execution. Please refer to prove --help or the documentation available on CPAN for more information.

There are a few things to take note of. All of the paths to included and referenced files, such as the simple.proto file, are relative to where the prove command is called. In this case it's in the root directory as part of the test target. Also, for some more complex and lengthy tests, please read up on the escript documentation. The compile mode can be used to pre-compile the file before being called but has a few caveats that you should be aware of.

I attended CN Erlounge III last weekend, it was a 2-day conference. I did a presentation about Scala vs Erlang.

I met Jackyz who is one of the translators of Chinese version "Programming Erlang". And Aimin who is writing a Delphi module to support Erlang c-node and c-driver in Delphi.

There is a commercial network monitoring product using Erlang from a major telecom company in China. And our Mobile-Banking platform (in Erlang) is scheduled to launch at middle of January too.

I talked with Yeka and Diuera from Broadview, a leading publisher in IT in China, they are really interested in importing "Programming in Scala" to mainland China.

And many thanks to Shiwei Xu, who is heavy working on Erlang community in China, and took the place to organize this conference.

I gave some encouragements to younger developers on learning Erlang and reading "Programming Erlang", since I'm the oldest one in attendees :-). Erlang is one of the best pragmatic and clear languages to learn concurrent/parallel and functional programming, and the book, is a very thoughtful and philosophic one on these perceptions.

And I'd like to see "Programming in Scala" also appeals in China soon, Scala is another pragmatic language on solving real world problems and, the book, is also thoughtful and philosophic one on our real world on Types, OO and FP.

Of course, choosing Scala or Erlang for your real world project should depend on the requirements.

I may be back to Vancouver next month for a while. Oh, it will be the beginning of new year.

CN Erlounge III photos by krzycube

Not that I spend much time compiling erlang code in Windows (unless I’m using cygwin or coLinux), but the issue does pop up from time to time. Here are a few minor tricks that can help a bit.

Starting Erlang in the current directory
For a unix user, it’s of course odd that this should be a problem.

John Hughes used a nice trick that’s so simple that I slap myself for not having thought of it: Right-click on a .beam file, select “Open With…” and locate werl.exe. Now you can open an erlang shell in the current working directory by double-clicking a .beam file.

Modifying the Windows context menu
So what if you don’t have any .beam files, and you’re trying to get to an erlang shell
in order to create some?

I did some googling and found this tutorial on youtube on how to add custom entries to the context menu.

It seemed simple enough, but Vista still served me a few hours of utter confusion since it took what I inserted, and then copied it to another place in the registry (without letting me know)… but only the first time. My changes and additions were simply ignored. The solution? Search the registry for the key you inserted, and you’ll figure out where Vista wants it to be, then make your changes there.

What I’ve experimented with so far is to add under Computer\HKEY_CLASSES_ROOT\ErlangSource\shell\ the following entries:


Compile with erlc
- Command = "C:\Program Files\erl5.6.3\bin\erlc.exe" "%1"
Make all
- Command = "C:\Program Files\erl5.6.3\bin\werl.exe" "-make"
Erlang shell
- Command = "C:\Program Files\erl5.6.3\bin\werl.exe" "%1"


A problem with running erlc this way is that the window is destroyed immediately upon completion. I’ve poked around a bit for good workarounds. One option is of course to write an erlang function that compiles the file, then either sleeps a short while, or waits for input.

The death of useful insight from a new idea encountered or approach studied begins a time of denial for me and then a time of mourning. Erlang stopped providing me with useful insight months ago, yet I doggedly pursued it hoping I would experience more of the mind-opening joy I had when I first encountered it. This has not happened. Worse than that I feel a seething resentment for Erlang now since it continually pops up in my head as a possible solution to things I am ill-equipped to solve using it. None of this is a fault of Erlang, but rather a reality of my inexperience with Functional Programming and the lack of modules in Erlang for structuring and developing RESTful Web Services the way I would like.

Certainly Yaws and mochiweb let you build RESTful Web Services, but I find myself fumbling with them rather than solving my real domain problems. I am not sure if this is a side-effect of transparent data structures in Erlang vs objects in Java or what. I do know that I find it easier to grow a solution gradually with Restlet (a Java REST framework). Nothing inherently prevents Erlang from providing developers with a good framework for building RESTful Web Services. I believe Erlang is inherently better suited to the purpose at a runtime level than Java, but Yaws and mochiweb do not fit my way of thinking.

It would be very interesting to see OTP-style behaviours for Resources and Routers (and the various other actors of the Restlet framework) which could be implemented simply in Erlang. Much of the design is already there in Restlet. I simply do not have the time or expertise to develop it in Erlang. I imagine that much of the high-level architecture of Restlet could be reproduced in Erlang with ease by an experienced Erlang developer.

So I am now mourning the loss of useful insight from Erlang and the likely impossibility of ever using it productively beyond desktop automation. Alas poor Erlang, I knew ye well!

Briefly this evening, I thought about spending some time playing around with Processing.

I'd take a few days, learn some new stuff, create animations that do nothing in particular but look cool, set it to some music. I have lots of ideas I want to explore, things that I think will look interesting and cool and maybe interactive. A toy someone can play with.

I'd get better, develop a following, buzz, excitement, yada yada. Eventually I'm like a Processing guru, making wicked cool animations at a whim. Sounds neat.

But when would I start this? Hmmmm. How about, do it fucking now? Ok, sounds good.

Go to the website, start looking at some code. It all looks interesting and approachable. Cool.

Hmmm, wait. Should I really be fooling around with this? I really need to do the replication testing for the new security features of CouchDB. For some reason I don't want to. But If I can fool around with Processing, then I clearly have the mental focus necessary to do the tests. The replication tests need to be done, then I can work on the Lucene integration and get closer to 1.0. So then I decide to fire up TextMate and have at it.

But before I do, I have this idea for a blog post. The CouchDB testing work will have to wait...

I have been putting off web development for as long as I could, until I couldn't any longer. I tried Ruby/Rails, and Python/Django and then I tried Allegro Common Lisp and Webactions.

I ditched Common Lisp Pages (CLP) in favor of generating the HTML programmatically and skinning it with CSS but I'm a very happy camper. I'm using AllegroCache as my database backend.

The project is the web portion of a translator from EasyLanguage, a trading language used by Omega Research TradeStation to NinjaTrader C#. There's a sample translation you can take a look at, generated code first and original code at the bottom.

This is my way of saying thanks to Franz for a great set of products!

To preempt some of your questions...

Q: Do you shill?

A: Nah! My licensing needs are covered but only because there's a Cocoa bridge for ACL.

Q: Shouldn't you be using Hunchentoot?

A: I know about it and Edi is a master of his work, I just don't like the extra libraries it pulls in and have everything I need in AllegroServe.

Q: Man, you suck! Why not use Erlang and a great framework like Nitrogen?

A: Erlang is the new Java. Lisp is the new Erlang!

Q: I'm not convinced! Aren't you the Erlang expert?

A: Suppose you had the source code to a top-notch commercial OODBMS and had the opportunity to try to add scalability and replication. Would you pass that chance? I wouldn't and I don't. You will pry my macros from my cold dead hands!

As I'm programming I haven't seen an instance where an array is better for storing information than another form thereof. I had indeed figured the added "features" in programming languages had improved upon this and by that replaced them. I see now that they aren't replaced but rather given new life, so to speak.

http://stackoverflow.com/questions/392397/arrays-whats-the-point

I saw this on Reddit, and the reaction wasn't kind. Most of the discussion centered around how it was such a dumb question, and what is it that has failed in our educational or professional communities that such basic knowledge isn't known. My initial reaction, unfortunately, wasn't much different.

But then as I thought a little more about it, I realized it really isn't such a dumb question. I found I really don't use arrays in Erlang, they aren't necessary. In C and C++, I used them all the time, you pretty much have to. But in Erlang, lists rule the day. Erlang has facilities for arrays, but the closest thing I use is small fixed-sized tuples. It's weird, but until just now I never really noticed how little I use arrays any more. The only place I can think of ever using them in Erlang is in the CouchDB btree code for doing binary key searches.

The truth is, unless you are doing low level programming, arrays (or even vectors) are rarely necessary or even optimal for most coding problems. The most notable feature of arrays are constant time access to any value given it's ordinal key. The problem is few programming problems call for a structure that gives them constant time access to ordinal mapped values. Most uses of arrays are just as a collection of values, and can be better satisfied with a list.

I guess the changes that I made back in November have had a positive impact.

In late October / early November I ported over the very last perl piece of the I Play WoW Facebook Application into the Erlang application set. I'm thrilled with the results that I'm seeing. The idea of using Erlang to power a web service has been reaffirmed as a great idea time after time now. The application is supporting well over 100,000 total users and just under 40,000 monthly active users. The system knows about 175,000 unique characters.

I've been telling Janl for a while now that I'd post some stats and information about the app and it's usage patterns. Here you go.

Before I go any further, I want to state the primary and driving reason for doing all of the changes that I've made. The goal was to lower the cost per user. This cost means time developing features, fixing bugs, interacting with user data (storing, fetching, aggregating, crawling, etc) and supporting users.

The application is powered by Erlang, CouchDB, MySQL, Xapian and Amazon S3. Erlang powers the web interface, the interaction with the Facebook API as well as the other services. CouchDB acts as a primary database with writes and updates still being sent to MySQL. For what it's worth, the application entirely relies on CouchDB for document reads and aggregates, but realistically CouchDB is still being developed and there are changes from time to time that warrant not depending on it alone.

Xapian is used internally for a search feature that I'm testing. I decided to go with xapian instead of Lucene because it has a smaller footprint and I don't need anything big and bulky at this point. Amazon S3 is used for image and file storage. Using Amazon S3 was much cheaper (time and money) than rolling my own file storage system.

I Play WoW started as a Perl application using the Catalyst framework using MySQL and Memcached. Since then it's evolved quite a bit. The Erlang component of the application is really a cluster of applications.

• ipwapp handles CouchDB, MySQL and Facebook Platform interfacing. This is where the bulk of application's core functionality and features exist.
• ipwweb provides the web interface that the Facebook Platform makes requests to for canvas page content.
• ipwcache provides a write-through cache and storage system for crawled data. This includes World of Warcraft characters, guilds, etc.
• ipwsearch provides the xapian search interface and rebuilds the index from time to time.

Although these applications aren't open source, there are a lot of pieces that are. The erlang_couchdb, erlang_facebook, erlang_wowarmory and etap projects available on GitHub (open source with the MIT license) are all actively used and developed by I Play WoW. This project also uses the extremely powerful MochiWeb application.

There are currently three Erlang nodes running this application. Each node runs one of the services and the other nodes are set as fallback nodes using distributed Erlang configuration. The erlang_wowarmory module is running on all nodes to distributed World of Warcraft Armory crawling evenly. The database, search

The CouchDB data store currently has 537,248 documents. Most of those documents are either character or guild documents. There is one primary document per character, guild and realm as well as a number of supportive documents for storing all sorts of other things. The MySQL database has around 383,000 rows. There are 5 design documents being used and about 25 views. This number has grown and shrunk over the past few months as I've learned to use documents more efficiently with the data structures the contain. It really has been a trial-and-error process with it's ups and downs, but ultimately it's worked out very well for me and I'm very satisfied with the results.

The one thing that I do slightly differently with CouchDB is actively track many document versions. For World of Warcraft characters I don't just keep one document per character, I keep one document per version of a character. This means that document ids are completely arbitrary and aren't used to identify characters unlike guilds and realms in the system. This is done to let me track different configurations of a given character over time and activity. This done through creating a hash of several mutable and immutable fields that represent a character and when that hash changes, track the information separately in a new or different document. This makes me rely very heavily on views to retrieve character documents from the system but it hasn't translated into "more work" yet.

The results of rewriting it have been great. I use fewer servers that cost less to support a larger number of users. At first I had three machines with either one or two gigs of ram as well as one database machine. Now I've got 4 machines with either 256 or 512 megs of ram, one of them still dedicated to MySQL, CouchDB and Xapian. The three nodes on the application grid average about 10% of memory and about 1% CPU. Some simple load tests show that it can sustain an increase of 9x without having to expand, and even at that point expanding is just adding another node to the grid. I'm effectively serving 3x the traffic at a third of the cost. Making changes to the application has been simplified with less downtime and the time to debug and fix errors has decreased.

Message me through Facebook, email or twitter if you've got questions or want to know more about a specific subject.

We have been very happy to see more and more interest for the Erlang material we were providing (our video has just reached 1000 plays).

And with the growing number of posts and tutorials, we decided it was time to give them a real dedicated Erlang page

In the future, more content and more links will be added to this page. All your feedback is also very welcome to help us create a comprehensive and complete guide.

It is really nice to see so much enthusiasm and a vibrant community starting around Erlang for web apps. Two new Erlang blogs we came across and that are definitively worth a look:

• Erlang Inside
• Poking Around Erlang

Good luck to Chad and Lloyd.

While it seems that most (?) Erlangers use emacs as their editor of choice, I enjoy using gedit. Here’s what I do to get things just right for writing some code (many of the changes I found here):

• Plugins: Auto Completion, Bracket Completion, Document Statistics, Embedded Termal, File Browser Pane, Session Saver, Snippets, Spell Check (if you are rolling ubuntu, many of these can be installed via apt-get install gedit-plugins)
• Theme: Darkmate
• Font: droid sans mono

One additional change I make is to the snippet plugin. I have basically copied the code skeletons from erlang mode in emacs and put them into the snippets plugin. You can download my export file here. This file can be imported into the snippets plugin and will provide similar functionality to the skeletons in emacs.

Here’s what it looks like when I’m done, with an example of a snippet on the upper right.

I was happy to read these two quotes from Steve Sanderson’s new blog post: Oh CouchDB, Why Do I Love Thee So…? (Emphasis mine, throughout)

For a piece of ‘middleware’, CouchDB has great elegance and great congruence with the bevy of potential uses, i.e. it appears to afford us the possibility to think about our problem/solution domains and our softwares internal models in in very similar fashions – with some very interesting beneficial side-effects (e.g. eventual consistency, availability)

This is what excited me about CouchDB when I was first hearing about it. It’s. all. just. the. web. Only with a sexy scalable BTree storage model and Map Reduce.

The second quote gets to the heart of the most revolutionary (well it’s an old idea, but it’s gonna work this time) aspect of peer-based distributed systems:

A CouchDB instance is sufficiently capable to become an application platform and that via the synchronization model, multiple instances of an application could run in a distributed & isolated manner and then synchronize and migrate as needed to different environments, i.e. run locally on laptops and sync and migrate up to centralized servers and then back down as needed. Instead of a rigid & pre-defined deployment structure, it’s more of an organic ’shape’ adjusting as needed. If this is a new degree of freedom, the potential is huge.

This was the part that surprised me, as I started to get more familiar with CouchDB. I’m still not sure what CouchApps are gonna look like, but exploring the potential is a great way to spend a winter break. Look for a walk-through on some RESTful blog software with our next book update.

I am continuing my rendezvous with Joy, and this post is yet another rant about my perception of why combinators are real first class citizens in the paradigm of concatenative languages. Many of today's languages offer combinator libraries, but, with the possible exception of Haskell, none match the elegance that Joy offers.

Combinators help programming at a higher level of abstraction. In any functional language, using maps and folds definitely make your program more expressive than explicit recursion or iteration. Consider this snippet that uses map as a transformer over a list to convert every element of the list to upper case ..

upperCase xs = map toUpper xs


and here is the same using explicit recursion ..

upperCase (x:xs) = toUpper x : upperCase xs
upperCase []     = []


Apart from the former being more expressive, using the map combinator also helps you separate the strategy from the implementation of your program. map is a higher level abstraction and the imp