Unofficial Erlang Planet

Avoiding Erlang deadlocks

2009-01-06T18:33:01+00:00

A friend of mine - a long time and very good Java developer - mentioned in his blog that Erlang can produce deadlocks too. In his example a process A send a message to process B and waits for a reply. During the processing of the message in process B this one sends a message to process A and waits for a reply too. But this message won't be processed due to the waiting of process A. And indeed, even if this example is trivial, he's right. The situation may for example be more complex, with a circle of several dozen processes all sending synchronous requests until the lock occurs.

But the model behind Erlang isn't the traditional request/response way like it is done between instances in OOP. It's more like a communication between colleagues in a bureau all busy doing their stuff. Sometimes one has a question. He may ask his co-worker and wait for an answer, and waits, and waits, and waits. But at some point it will be enough and he will react, through asking somebody else, through asking again, or through throwing a book at his colleague. *smile* That's like using a timeout in a receive statement, or the default behaviour of gen_server:call(). No deadlock, no endless waiting like a dumb sheep - hmm, even those won't wait very long.

Another way would just continue doing the own work until the answer comes. This is highly asynchronous and needs to split the logic into small pieces and to remember the question in the state of the process. This way processes could juggle multiple tasks. Using OTP with supervisor trees and gen_fsm for state machines helps a lot here. Additionally if needed all those concurrent working processes may share a process with a pure getting and setting functionallity for common status data or they may use one or more ets tables.

So, once again, yes it is possible to produce deadlocks in Erlang. But the way to produce it is an improper way of developing in Erlang. Using receive with timeout or using the OTP default timout behaviour - no calls with the argument infinity - is the first step, create solutions using the concurrency right the second one.

Common Erlang misconceptions

2009-01-06T08:34:05+00:00

There is a common saying that Erlangs I/O layer is slow. Bullshit. The etorrent program can easily sustain 70 megabit with less than 5% of CPU usage on my old machines. The reason is that etorrent is shuffling 16k binary blocks around mostly. But that means you have a pointer to each 16k block and you are only a writev(2) call away from the disk.

This means I/O is not itself a problem. The problem that people often hit is Erlangs string handling. In Erlang a string can either by represented as an immutable array of 8-bit characters, called a binary; or it can be represented as a list of integers (ie, a linked list of integers). The latter is mutable in the sense that you can generate new integers and update the linked list from the front only.

Whenever you mutate the list of integers, garbage is generated. Allocation will happen and this directly translates to more garbage collection pressure later on. Had you had destructible arrays in Erlang it would have been very easy to circumvent the worst performance problems, but you will have to do with a limited zoo of types. In better languages, like Ocaml, Standard ML or Haskell, your type zoo is much richer here. You can choose between several immutable and mutable variants so the leverage is much greater.

First trick: Erlang has the concept of an IO list. These are lists of binary/string chunks. They are fast because they lend themselves to a writev(2) in the bottom of the system. Thus you don't have to concatenate strings in the Erlang language but you can hand a quasi-concatenated list to the VM and let C do the heavy-lifting for you.

Second trick: Mutating data as strings is slow. But if you have a abstract tree of tuples then it can be manipulated fast. If you think about the command [Bin | IOList] where we add a Binary to the front of an IOList, you'll know it happens in O(1) (constant) time. If we build a tuple {atom, D1, D2, ...}, we can build it in constant time. So the trick is to maintain your data in a non-string form for as long as possible and then only change it to a string at the very last point in time.

Third trick: Correct data structure. Ropes. Finger Trees. M-way trees. Don't brute force yourself over a string. Process the structure by pattern matching in a clever way so you minimize the amount of change needed. A good tree will have around logarithmic change whereas a list will have linear change. This leads to the next point: allocation.

Another crap problem in Erlang is that you have few (traditional) ways to minimize allocation. Any kind of operation will allocate data if you are not wary of it. Hence you should try to build programs that do not gloss over data and allocate all the time. Mapping of lists via list comprehensions is a good example. It is easy to understand but it will build a new list and garbage collection will cost you. If you can do multiple things in one list comprehension, you win traversals and allocations.

Whenever I hear about "Slow string processing" in Erlang I immediately think: "Another Ruby programmer trying to cram his language into Erlang". If you hit slowness the trick in Erlang is to redefine the problem! Don't try to string process yourself into oblivion. If the sole thing your problem concerns is string-processing, go use perl. Please. Or be clever and use Ocaml. Pre-process the data in other languages and make them into erlang terms which can be read fast by erlang. Erlangs power is the concurrency at which it is really good. There were no reason for giving it fast string processing.

Also, don't get bitten by the mnesia bug, please. It is almost always the case that mnesia and ETS are the wrong way to go. I got bitten by it multiple times in etorrent and now I am contemplating ripping all of it out again. Plain old data structures baby! There are some nasty caveats to mnesia and ETS and they are not related to the size limit of the database.

For mnesia, be aware that a failed transaction will be re-run! Now take 100 transactions tripping over each other. They will never finish. Etorrent did that when peers should update their state. Instant snail-speed application. Mnesia is crap at doing joins. Just forget it.

ETS is also problematic: Any store or get from ETS means a data copy. Don't do as etorrent and add large rows (128k+) to ETS. Your app will come to a grinding halt.

Don't message big data structures around. Any message incurs a copy (in the default VM). If you need to work with a big data structure, shove it into a process and then send the Pid of that process around or register him globally. You can think of a Pid as "Passing a pointer around" in this case.

And finally: Measure measure measure! Be scientific about your performance problems. You need to know exactly what causes it before jumping to conclusions. And don't worry too much about being parallel at the beginning. You can always split things up in finer granularity later.

Couch Book Progress

2009-01-05T20:03:29+00:00

It’s been a few weeks since we released the initial chapters of O’Reilly’s CouchDB book. The next release ought to be happening “soon” – There’s more work to be done on it, but the major skeleton of the chapters has been written, and the figures drawn.

I’m posting the outline of the next section, so that people’s expectations about the book are inline with what we provide. The next released section will be a few chapters walking through the creation of a simple web app (yes, a weblog). The point is not to teach Ajax, but to teach the CouchDB HTTP API in a hands on way.

Yes, all the code runs, and the in-progress version of the example blog is even available on github. One of my main tasks between now and the release is to make the installation proceedure much easier. So you are welcome to play now, but if it seems messy or confusing, please try again when the chapters are released.

If anyone has input on pacing and ordering of the example application I’d love to hear it. I think this section should be relatively fast-paced, and focussed on how easy it is to build applications using CouchDB, with the occasional bits of API specification or deeper material presented as asides and sidebars.

2. Learning CouchDB

Touring the Futon Administration Interface

A survey of CouchDB’s http api, through the eyes of Futon. Touches on most of the core concepts. This material is might be better suited for Section 1, as it does stand alone as an introduction to the http API.

Introducing the Example Blog

Why we chose JavaScript for the book examples. Also: even if you are storing scientific data, what you’ll learn by following along is applicable (views, document semantics, design docs).

Setting Up Your Development Environment

Getting the example app to your local drive, and being able to push it to CouchDB. Also cover design-doc / application correspondence while explaining the couchapp script.

Storing Blog Posts in CouchDB

The JSON doc format, validation functions, plus the HTML and Javascript it takes to support the blog post form. (This will be a longer one, but the code for it shouldn’t be changing.)

Rendering Documents with CouchDB Forms

What forms are. Why they have the single-doc constraint. Mention that you can write raw JS, but then show CouchApp’s require / include system as a best practice, using templates and a basic templating engine.

Browsing Recent Posts with Map Views

Introduce a simple view by published at date. Show basics of pagination. Explain the view-query API by looking at the JavaScript client API used to query the view.

Mention that there will be forms for views in the future, so that this page can be served as HTML as well. (Forms for views should be in Couch 1.0 – and thus in the book, so polishing this section now is lower priority. Pagination will also be easier with view forms)

Accepting Reader Comments

The JSON format, validations, and the HTML form for updating a comment. This is a short one, mostly repeats what happened in “Storing Blog Posts in CouchDB” but its the second time, so we can touch on different aspects and explain other API details. (One neat difference is that we’re adding Javascript behavior to a page that was rendered as a CouchDB form, rather than a static document.)

Listing Related Comments Using View Collation and Compound Keys

(I didn’t say “joins”) ;)

Currently this uses an Ajax query that lists comments by [post_id, date]. When view forms are available, we’ll change it to Christopher Lenz’s query, which includes the post itself, so the whole thing can be rendered in a single request. This chapter works on the same files as the previous chapter, they could be “joined” but I think this is material we should take the space to cover as clearly as possible.

If you’ve gotten this far, then you must be seriously interested in CouchDB. If you are, then you should familiarize yourself with the test suite. It’s written in JavaScript, so anyone can understand it, and it’s the main gateway for new features. Plus, code speaks louder than words.

Higher Order abstractions in Scala with Type Constructor Polymorphism

2009-01-05T12:25:08+00:00

Abstractions at a higher level through type constructor polymorphism. Good type systems are expressive enough to conceal the implementation complexity, and expose *only* what it matters to the developer. People often cringe about the complexity of Scala's type system and how it serves as a barrier to the entry point in mainstream programming. As Michael Feathers recently noted on Tweeter, the unfortunate fact is that people often jump at the esoteric parts of a language before looking at the simpler subset, which he will be using 90% of the time. And, I think Scala has that sweet spot, where you need not mess around too much with variances, implicits and existentials and yet come up with a nice, concise and functional codebase.

In this post, I discuss the already introduced intimidating phrase "Type Constructor Polymorphism" through a series of code snippets ranging from toys to some real-world stuff. The idea is, once again, not to evangelize type theory intricacies, but share some of the experiences of how this feature in Scala's type system can help write idiomatic code, while staying away from the complexities of its underlying implementation.

Jump on ..

We have a list of Option[String] that we need to abstract over and compute some value. Say, for the sake of keeping the example simple, we will calculate the sum of lengths of all the strings present ..

val employees: List[Option[String]] =
  List(Some("dave"), None, Some("john"), Some("sam"))

val n: List[Int] =
  employees.map { x =>
    x match {
      case Some(name) => name.length
      case None => 0
    }
  }.elements.reduceLeft[Int](_ + _)
println(n)

Let us take another problem that needs to abstract over a different List structure, a List of List of Strings, and compute the same result as before, i.e. the sum of lengths of all the strings encountered in the collection ..

val brs: List[List[String]] =
  List(List("dave", "john", "sam"), List("peter", "robin", "david"), List("pras", "srim"))

val m: List[Int] =
  brs.flatMap {x => x.map {_.length}}
     .elements.reduceLeft[Int](_ + _)
println(m)

Do you see any commonality in the solution structure in the above snippets ? After all, the problem space has a common generic structure ..

we have a List with some String values abstracted in different forms

need to iterate over the list

do some stuff with elements in the list and compute an Int value

Unfortunately the actual solution structures look quite different and have to deal a lot digging into the abstractions of the underlying representations within the collection itself. And this is because, we cannot abstract over the type constructor (the List in this case) that takes another type constructor as an argument (Option[String] in the first case and List[String] in the second case).

Enter type constructor polymorphism.

Sounds intimidating ? Maybe, but ask the Haskellers .. they have been using typeclasses ever since so successfully in comprehensions, parser combinators and embedded DSLs and programming at a different level of abstraction.

Scala supports type constructor polymorphism since 2.5, and the details have been discussed in a recent paper by Adriaan Moors et al in OOPSLA 2008.

Here is a snippet of the Scala code that works seamlessly for both of the above cases ..

val l: List[Int] = employees.flatMapTo[Int, List]{_.map{_.length}}
val sum: Int = l.elements.reduceLeft[Int](_ + _)
println(sum)

The above code abstracts over List through higher order parametric polymorphism, i.e. independent of whether the List parameter is an Option[] or another List[]. Incidentally both of them (List and Option) are monads and flatMapTo abstracts a monadic computation, hiding all details of type constructor polymorphism from the developer.

Now here is some real life example (elided for simplicity) ..

Here are the simple domain models for Customer, Instrument and Trade, used for modeling a use case where a Customer can order for the Trade of an Instrument in an exchange.

case class Customer(id: Int, name: String)
case object nomura extends Customer(1, "NOMURA")
case object meryll extends Customer(2, "MERYLL")
case object lehman extends Customer(3, "LEHMAN")

case class Instrument(id: Int)
case object ibm extends Instrument(1)
case object sun extends Instrument(2)
case object google extends Instrument(3)

case class Trade(ref: Int, ins: Instrument, qty: Int, price: Int)

And we fetch the following list through a query from the database. It is a List of tuples where each tuple consists of a Customer and a trade that has been executed based on the Order he placed at the exchange. And here is the snippet of the code that computes the sum total of the values of all trades executed in the day for all customers.

val trades: List[(Customer, Option[Trade])] =
  List((nomura, Some(Trade(100, ibm, 20, 12))),
       (meryll, None), (lehman, Some(Trade(200, google, 10, 10))))

val ts: List[Option[Trade]] = trades.map(_._2)
val t: List[Int] = ts.flatMapTo[Int, List]{_.map{x => x.qty * x.price}}
val value = t.elements.reduceLeft[Int](_ + _)
println(value)

Really not much different from the above simple cases where we were dealing with toy examples - isn't it ? The structure of the solution is the same irrespective of the complexity of data being stored within the collections. The iteration is being done at a much higher level of abstraction, independent of the types stored within the container. And as I mentioned above, flatMapTo is the secret sauce in the above solution structure that abstracts the building of the new container hiding the inner details. To get more into the innards of flatMapTo and similar higher order abstractions, including the new form of Iterable[+T], have a look at the OOPSLA paper referenced above.

Postscript: In all the snippets above, I have been explicit about all type signatures, just for the sake of easier understanding of my passionate reader. In reality, almost all of these will be inferred by Scala.

Large set of ejabberd resources

2009-01-05T10:01:05+00:00

Planet Erlang is fastly becoming a large index for Erlang projects resources, including ejabberd.

Since its relaunch, Planet Erlang is now indexing content from tens of Erlang related blogs. Thank to its search engine, you can now find many blog posts and articles on lots of different Erlang topic. ejabberd is one of the most covered topic there.

You should have a look and try searching for ejabberd in the search engine (search field on the right).

The list of resources is growing every day as we are adding even more older resources.

Your comments and feature requests are welcome !

Sendfile for Yaws

2009-01-05T07:22:54+00:00

A few months back Klacke gave me committer rights for Yaws. I’ve made a few fixes here and there, including adding support for passing “*” as the request URI for OPTIONS, enabling OPTIONS requests to be dispatched to appmods and yapps, and completing a previously-submitted patch for configuring the listen backlog. Klacke has just started putting a test framework into the codebase and build system so that contributors can include tests with any patches or new code they submit, and I’ve contributed to that as well.

The biggest feature I’ve added to date, though, is a new linked-in driver that allows Yaws to use the sendfile system call on Linux, OS X, and FreeBSD. I never wrote a linked-in driver before, so I was happy and fortunate to have an Erlang expert like Klacke providing hints and reviewing my code.

I did some preliminary testing that showed that sendfile definitely improves CPU usage across the board but depending on file size, sometimes does so at the cost of increasing request times. I used my otherwise idle 2-core 2.4GHz Ubuntu 8.04.1 Dell box with 2 GB of RAM, and ran Apache Bench (ab) from another Linux host to simulate 50 concurrent clients downloading a 64k data file a total of 100000 times. I saw that user/system CPU on the web host tended to run around 33%/28% without sendfile, while with sendfile it dropped to 22%/17%. The trade-off was request time, though, where each request for the 64k file averaged 0.928ms with sendfile but 0.567ms without. With larger files, however, sendfile is slightly faster and still has better CPU usage. For example, with a 256k file, sendfile averaged 2.251ms per request with user/system CPU at 8%/16% whereas it was 2.255ms and 16%/27% CPU without sendfile, which makes me wonder if the figures for the 64k file are outliers for some reason. I performed these measurements fairly quickly, so while I believe they’re reasonably accurate, don’t take them as formal results.

On my MacBook Pro laptop running OS X 10.5.6, CPU usage didn’t seem to differ much whether I used sendfile or not, but requests across the board tended to be slightly faster with sendfile.

I ran FreeBSD 7.0.1 in a Parallels VM on my laptop, and there I saw significantly better system CPU usage with sendfile than without, sometimes as much as a 30% improvement. Requests were also noticeably faster with sendfile than without, sometimes by as much as 17%, and again depending on file size, with higher improvements for larger files. User CPU was not a factor. All in all, though, I don’t know how much the fact that I ran all this within a VM affected these numbers.

Given that Yaws is often used for delivering mainly dynamic content, sendfile won’t affect those cases. Still, I think it’s nice to have it available for the times when you do have to deliver file-based content, especially if the files are of the larger variety. Anyway, I committed this support to the Yaws svn repository back around December 21 or so. If you’d like to do your own testing, please feel free — I’d be interested in learning your results. Also, if you have ideas for further tests I might try, please leave a comment to let me know.

I suck

2009-01-05T05:12:34+00:00

I seem to be particularly good at picking very narrow, most likely unprofitable niches. A translator from a particular trading language to another product with a much smaller user base is one example, writing Mac OSX device drivers is another one.

If I were a trader, I could short myself and do the opposite of any idea I came up with. How do I apply this strategy to business, though?

On a different tangent, this blog gained about 100 more subscribers to this blog all of a sudden, whereas Think Erlang has less than 100 subscribers a few weeks after launch.

Should I assume that you are not here to read about my Erlang progress but rather about my multi-language antics? Let me know!

I'm about to dive into Linux device drivers, Forth, low-power embedded devices, FPGAs, marine electronics and navigation software. I hope you'll still be around in a year! :D

Update 1:

Whipsawing back and forth too much between ideas is a side effect of being burned by OpenPoker, which I spent way too much time on. It's a disaster from a commercial point of view and so whenever I sense that an idea may turn out to be unprofitable, I drop it like a hot potato.

Granted, I may not be giving ideas enough time to develop and should probably have a method of delivering bare-bones working implementations, if only to see them fail quickly.

As for sticking with one language, it just doesn't work, period. There's a language for every task.

Mac OSX device drivers (IOKit) are programmed in C++, Linux device drivers in C. FPGAs are programmed in VHDL and Forth is an excellent tool to program cheap low-power controllers.

Update 2:

The baby less than two months old so I'm a little foggy. Still, I think I have reached enlightenment on at least one issue.

I'll short myself on device drivers...

Hot: Linix. Not: Mac and IOKit.

linux device driver, mac iokit Job Trends

linux device driver jobs - mac iokit jobs

OpenPoker is open again

2009-01-05T04:54:24+00:00

I'm releasing OpenPoker, a scalable poker server written in Erlang, under a Creative Commons Non-Commercial License.

Please make a $100 donation at GitHub and subscribe to The Erlang Journal if you want to discuss the OpenPoker architecture!

The donations button is in the upper right corner of this page, or you can just PayPal me at joelr1@gmail.com.

A Case Study of Scalability Related "Out of memory" Crash in Erlang

2009-01-04T02:59:59+00:00

We are building a platform for message switching, in Erlang. Everything looks OK on stability and features. It actually has run more than half year with zero down. We tested its performance on our 2-core CPU machine before, and got about 140 transactions/second, it's good enough.

Then, we got a 8-core CPU machine several weeks ago, and we did same performance testing on it, to see the scalability. Since Erlang is almost perfect on scalability, you can image the result, yes, about 700 transactions/second now, scaled almost linear. Until it crashed with "out of memory" when million hits processed.

It left a very big "erl_crash.dump" file there, I had to dig the issue. My first guess was, were some remote requests (access db, access remote web service etc) timeout but the process itself was not timeout yet, and cause more and more processes kept in VM?

A quick grep "=proc:" erl_crash.dump showed that the total number of processes was about 980, which was reasonable for our case.

So, which process ate so many memory? A quick grep "Stack+head" erl_crash.dump showed that there was indeed a process with 285082125 size of Stack+head there.

Following this clue, I caught this process:

=proc:0.4.0>
State: Garbing
Name: error_logger
Spawned as: proc_lib:init_p/5
Last scheduled in for: io_lib_format:pad_char/2
Spawned by: 0.1.0>
Started: Sun Apr  1 01:21:50 2012
Message queue length: 2086029
Number of heap fragments: 1234053
Heap fragment data: 281266956
Link list: [0.27.0>, 0.0.0>, {from,0.42.0>,#Ref0.0.0.88>}]
Reductions: 72745575
Stack+heap: 285082125
OldHeap: 47828850
Heap unused: 121777661
OldHeap unused: 47828850
Program counter: 0x0764c66c (io_lib_format:pad_char/2 + 4)
CP: 0x0764c1b4 (io_lib_format:collect_cseq/2 + 124)

This process was error_logger, which is from OTP/Erlang standard lib: error_logger, writing received messages to log file or tty. The typical usage is:

error_logger:info_msg("~p:~p " ++ Format, [?MODULE, ?LINE] ++ Data))

Which will format Data to a String according to the Format string, and write it to tty or log file.

The above case showed the message queue length of process "error_logger" had reached 1234053, and the Stack+heap was 285082125, about 272M size.

So the cause may be, that the message queue could not be processed in time, the messages were crowded in error_logger's process and finally caused "out of memory". The bottle-neck was that when error_logger tried to format the message to String, Erlang VM was weak on processing them, which seemed to need a lot of CPU cycles.

In my previous blog, I talked about Erlang is bad on massive text processing. Erlang processes String/Text via List, which is obvious bottle-neck in Erlang now, with Erlang is getting much and much popular and more and more Erlang applications are written.

But, why this did not happen on our 2-core CPU machine? It's an interesting scalability related problem:

"error_logger" module will registered one and only one process to receive and handle all log messages. But Erlang VM's scheduler can not distribute ONE process to use multiple CPUs' computing ability. In our 2-core machine, the whole ability is about 140 transactions/second, the one process of "error_logger" just happened to have the power to handle corresponding log messages in time. Under 8-core CPUs machine, our platform scales to handle 700 transactions/second, but there is still only one process of "error_logger", which can not use 8-core CPUs' ability at all, and finally fail on it.

Erlang treats every process fairly (although you can change the priority manually), we can do a simple/quick evaluation:

1. 2-Core machine, keeping hits at 140 trans/second:
The number of simultaneous processes will be about 200, each process shares the CPU cycles: 1/200 * 2 Core = 1%

2. 8-Core machine, keeping hits at 700 trans/second:
The number of simultaneous processes will be about 980, each process shares the CPU cycles: 1/980 * 8 Core = 0.82%

So, the CPU cycles shared by error_logger process actually not increases.

BTW, I think error_logger should cut its message queue when can not process them in time (disk IO may also be slower than receiving messages).

Erlang - overhyped or underestimated?

2009-01-04T01:30:54+00:00

Erlang is like an exotic beautiful woman with no dressing sense. I came across Erlang about a year ago. It is a language of a kind - precise, crafted and powerful. Its been there for about two decades now, and people have been using it for serious real-time applications. But the hype Erlang is getting in [...]

MD5 in Erlang

2009-01-04T01:30:54+00:00

An MD5 hash for any given message is 16 bytes (128 bits) in size and is represented by a unique 32 digit Hexadecimal number. Erlang has a built-in-function to calculate the MD5, which returns the hash in the form of a binary data-structure. $ erlang:md5("hello"). <93,65,64,42,188,75,42,118,185,113,157,145,16,23,197,146> But what we usually need is a string representation of the Hexadecimal value. [...]

Erlang mode on Emacs

2009-01-04T01:30:54+00:00

Although there are a few other editors for Erlang, I prefer to use Emacs for Erlang and its the only major reason I use Emacs for. Erlang now has an eclipse plugin too, called Erlide. Erlang comes with the emacs mode as part of its standard distribution, so you only need to customize your emacs settings [...]

Erlang on Mac OSX

2009-01-04T01:30:54+00:00

On a fresh install of Leopard, the following is what I did in order to set up Erlang working on my Mackbook Pro: 1. Download Xcode by going to Apple ADC. Xcode is the Apple’s developer tool set, and its available for free. At the time of my install, the latest is Xcode 3.0 and its [...]

Nginx vs Yaws vs MochiWeb : Web Server Performance Deathmatch, Part 2 [Update x 2]

2009-01-03T23:19:33+00:00

Update 1: Retest data (using different machine and Erlang kernel polling) added near bottom of post.

Update 2: More details and testing on the weird MochiWeb kernel polling results, bottom of post.

Almost a year ago I did some Apache and Nginx performance testing. Apparently I have the bug again and have done some performance testing on Nginx, Yaws and MochiWeb. The latter two being Erlang based. Again deathmatch may be an overstatement but this is my attempt at gleaning some interesting performance data from some high performance web servers. Also, I attempted to improve the graphs this time around since they were a bit hard to read the last time.

The Setup:

I was not able to use the same server and setup as the last time, so comparing between this and my last deathmatch probably isn’t very accurate. For this test I used a Intel Dual Core 2.2GHz, 4GB RAM machine running Ubuntu 8.10 (64bit) and for the test server. Erlang (R12B-3), Yaws (1.77) and Nginx (0.6.32) are installed from the standard repository and mochiweb from subversion (rev 88). All are using the default configurations outside of adjusting listening port numbers. The test is again against a basic robots.txt file. The tests were done using a consumer grade 100mb switch and all tests originated from an old laptop I had laying around. I think that about covers the test bed, if you have any questions let me know.

For the tests I used autobench (httperf under the hood) with the following command, each test ran ten minutes apart. The order of the tests were done in was MochiWeb then Yaws and lastly Nginx.

autobench –single_host –host1 HOST –port1 PORT –uri1 /robots.txt –low_rate 10 –high_rate 200 –rate_step 10 –num_call 10 –num_conn 5000 –timeout 5 –file SERVER-results-`date +%F-%H:%M:%S`.tsv

The Results:

There are a few results from httperf/autobench that I would like to show, errors, network I/O, reply rate (and it’s standard deviation) and response time. (click on the graphs for a larger view)

MochiWeb and Yaws both seem to be the most consistent here. Nginx had a couple of funky spikes, I do not know if this was an issue with Nginx or with my tests and/or test bed. Take from it what you will.

Nginx seems to use a bit more network I/O consistently through the lower ranges of this test and then again as some spikes. MochiWeb and Yaws seem to have some inconsistencies as well.

The reply rate and network I/O graphs certainly seem to be tied, which would make sense. Edit: Average reply rate is average replies per second.

In the higher reaches of the tests Yaws seems to be most consistent.

MochiWeb seems to have consistently the highest response times with Nginx has the lowest. This also follows the data from the first deathmatch. Nginx had consistently low response times against Apache. Edit: Response time is how quickly replies are sent in milliseconds.

Next up are the system graphs, I have CPU usage (both cores combined), context switches, interrupts and load. To help read these please note recall that each test ran ten minutes apart and the order of the tests was MochiWeb then Yaws and lastly Nginx. The data was gathered using sar at five minute intervals and graphed using ksar.

It seems Nginx is the clear winner here. Kernel polling may be the answer here, a retest may be in order to see if it makes a difference.

MochiWeb and Nginx seem pretty even on context switches with Yaws a little higher. I suppose turning on kernel polling might make this a bit more even, since Erlang and Nginx both use epoll. This may also account for the CPU usage difference above.

Interrupts are fairly even across all of them.

Again Nginx takes it, again likely due to kernel polling being disabled. That’s my best guess anywho.

The data I used to create the graphs and etc is available here.

Let me know if you are interested in me retesting anything, I may try to enable kernel polling and try again if I get a chance.

Note that these are *my* experiences with each webserver, your testing and experiences may be different. As with most things there are pro’s, con’s, trade offs and pitfalls. The only way to find out what will work best for your environment is to test, test and test.

Update:

I performed the upper half of the tests again to see if there were any changes to sporadic jumps in the graphs http performance graphs. My initial test using the old laptop I saw the same results. I then ran the tests from a VM (running Ubuntu 8.10 in a KVM VM) on my dual core machine and found that the results were much more even. Unfortunately it’s the same machine that the webservers are running on but the results look much better. The first set is using the same setup as before but just adjusted to have the top half test. The second is the same test but with kernel polling turned on in Erlang.

All of them are very even and close, no real winners here.

Looks like Nginx is the clear winner with Yaws next, followed by MochiWeb.

Pretty much the same as last time (likely a little higher across the board due to running the tests in a VM on the same machine). Note that Nginx is a system process, so for Yaws and MochiWeb follow the blue line and Nginx follow the green.

About the same as before, other than being higher due to running a VM.

Pretty much the same as before again, Nginx seems the lowest.

Now for the tests with kernel polling enabled in Erlang (erl +K true).

With kernel polling on it looks like Yaws actually performs better in the reply rate test with MochiWeb performing worse and Nginx in the middle

In the response time test a huge change is noted, MochiWeb goes from roughly a ~14 ms response time at 2000 requests to ~65 ms. Also noted Yaws performs much better matching or beating Nginx.

With kernel polling in the Erlang webservers Nginx still seems to come out on top for CPU usage.

Following the performance trend we saw above Yaws sees a drop in context switches and MochiWeb increases.

Load-wise things stay roughly the same with Nginx being the lowest.

While it certainly seems that my old laptop that I did the original tests on is too slow or has a network issue, hopefully with these new tests we have some more clarity. It seems that Yaws improves with kernel polling enabled and competes well with Nginx. MochiWeb on the other hand apparently has issues with kernel polling and actually degrades performance. If anyone has more info on the internals of MochiWeb and possible causes I would be certainly interested.

If anyone would like the data from the second round of tests it is available here.

Update 2:

I did some more testing to see what the issue might be with MochiWeb, response times and kernel polling. I did a few tests with different versions of Erlang, with and without kernel polling and testing from within and outside a KVM VM. From what I can tell the issue seems to be isolated to testing from within a VM with MochiWeb and kernel polling. Seems to be sorta strange but all my testing and retesting shows the same issue. Just to be clear on my setup, I am running httperf from with in a VM to MochiWeb running outside the VM. Here is the latest round of testing to show this point.

Even though the numbers are higher from within the VM without kernel polling, it certainly seems to be an issue with the combination of MochiWeb, KVM and kernel polling. Since I did not see the same spike from within a VM in the earlier tests with Yaws and kernel polling I assume it is not an issue with Erlang or it’s kernel polling mechanism conflicting with KVM. I am not entirely sure what to make of this other than MochiWeb, kernel polling and KVM don’t play well together and that kernel polling actually helps MochiWeb significantly when KVM is not involved. If anyone has any ideas on why that may be I am all ears.

SHA-256 in Erlang

2009-01-03T17:14:59+00:00

Sriram Krishnan was recently lamenting the lack of SHA-256 support in Erlang’s crypto module, so just for fun I wrote a simple sha256 module based on the pseudocode in Wikipedia. The tests use the test data from this C implementation.

[Update: I've posted a new module that implements more SHA digest variants and so renders this module obsolete.]

More SHA in Erlang

2009-01-03T17:11:54+00:00

Yesterday I posted a SHA-256 Erlang module, but I figured since other SHA algorithms are similar, I might as well finish the job. I grabbed the Secure Hash Standard and went to work.

The resulting new module is named sha2 and it implements SHA-224, SHA-256, SHA-384, and SHA-512.

Code
Light documentation
Tests are included in the module but you get the test vectors here
It’s open source, BSD license

I hope you find it useful.

BeepBeep Is Another New Erlang Framework

2009-01-03T12:14:39+00:00

Happy New Year! It’s been a great 2008 for Erlang, which we’ll cover in our 2008 Retrospective in another post later this week. For today, it’s a new, simple web framework for Erlang called BeepBeep. Written by Dave Bryson, it uses MochiWeb, which most Erlang Web developers are familiar with, and ErlyDTL, which is an Erlang port of the Python Template Language. Travis Slicegood has a demo application written in BeepBeep grabbing photos from Flickr, and Dave Bryson has a sample Blog Application as well.

The new set of Erlang web frameworks is more proof of the number of ambitious, sharp developers moving to the language is growing.

2008 in Review & 2009 Goals

2009-01-02T00:00:00+00:00

2008 was a great year of activity and excitement. I've got the sneaking suspicion that 2009 is going to be no less busy.

Finished and published my first book!
Spoke at CUFP and met a lot of really smart people.
Gave several tech-talks and presentations.
Learned how to develop apps for the iPhone.
Migrated I Play WoW from a Perl app to an Erlang app.
Learned about CouchDB.
Learned Git.
Accepted a lead position at EA.

This is what I'd like to accomplish in 2009.

Finish my second book.
Do more tech-talks and presentations.
Have a 100 day streak of open source commits on GitHub.
Write more technical articles and entries.
Learn Python.
Hit level 80 with Korale and get into tier 9.
Grow I Play WoW to 75,000 monthly active users.

The Year That Will Be

2009-01-01T19:39:00+00:00

It's 2009 now, in Beijing.

1==0.999999999......

I met Erlang 2 years ago, which finally brings me to Scala. I learnt a lot from Erlang, and I entered the Scala world with Erlang atmosphere surrounding me. The FP, the Pattern Match, the Actor/Process, I found these familiar friends in Scala everywhere.

Scala has extra bonus, to me, static types and OO/FP. The domains I face are usually with a lot of business logic, or, the worlds I try to describe are not only messages, they are, models I don't think are suitable to describe in Function only.

The world itself is OO/FP mixed, like Martin's quote: Two sides of coin. It's something like the Particle/Wave in Quantum. The world is an infinite whole, but the reason of Human Being is always finite, we are using out finite reason to measure the infinite world, it's an unsolvable contradiction: Infinity vs Finite. We have to read our world in OO and, in FP, in snapshot and in continuation.

There won't be "Super Hero" in computer languages, the world is getting self-organization and harmony, so do the languages. Each language is living in an eco-system, born, growing via interacting with environment, disappear ...

The Economy

It was bad in 2008. I tried to do some computing on stock market based on my neural network. What I can say is it will be swing in the next half-year, no big drop, no big rise. The Shanghai Stock Index will swing between 1200 and 3000. At least, no big worse any more.

My Self

I need to make some big decisions in this a year.

Links for 2008-12-31 [del.icio.us]

2009-01-01T06:00:00+00:00

Thinking in Scala vs Erlang

2008-12-30T18:15:17+00:00

Keeping Erlang in mind, I've coded two months in Scala, I'm thinking something called "Scala vs Erlang", I wrote some benchmark code to prove me (the code and result may be available someday), and I'd like to do some gradually summary on it in practical aspect. These opinions may be or not be correct currently due to lacking of deep experience and understanding, but, anyway, I need to record them now and correct myself with more experiences and understanding got on both Scala and Erlang.

Part I. Syntax

List comprehension

Erlang:

Lst = [1,2,3,4],
[X + 1 || X - Lst],
lists:map(fun(X) -> X + 1 end, Lst)

Scala:

val lst = List(1,2,3,4) 
for (x - lst) yield x + 1
lst.map{x => x + 1}
lst.map{_ + 1} // or place holder

Pattern match

Erlang:

case X of
   {A, B} when is_integer(A), A > 1 -> ok;
   _ -> error
end,

{ok, [{A, B} = H|T]} = my_function(X)

Scala:

x match {
   case (a:Int, b:_) if a > 1 => OK // can match type
   case _ => ERROR
}

val ("ok", (h@(a, b)) :: t) = my_function(x)

List, Tuple, Array, Map, Binary, Bit

Erlang:

Lst = [1, 2, 3] %% List
[0 | Lst]  %% List concat
{1, 2, 3}  %% Tuple
1, 2, “abc”>>  %% Binary
%% no Array, Map syntax

Scala:

val lst = List(1, 2, 3)  // List
0 :: lst  // List concat
(1, 2, 3) // Tuple
Array(1, 2, 3) // Array
Map(“a” -> 1, “b” -> 2) // Map
// no Binary, Bit syntax

Process, Actor

Erlang:

the_actor(X) -> 
   receive 
      ok -> io:format(“~p~n”, [X]);
      I -> the_actor(X + I) %% needs to explicitly continue loop
   end.
P = spawn(mymodule, the_actor, [0])
P ! 1
P ! ok

Scala I:

class TheActor(x:Int) extends Actor { 
   def act = loop {
      react {
         case “ok” => println(x); exit // needs to explicitly exit loop
         case i:Int => x += i
      }
   }
}
val a = new TheActor(0)
a ! 1
a ! “ok”

Scala II:

val a = actor { 
   def loop(x:Int) = {
      react {
         case "ok" => println(x)
         case i:Int => loop(x + i)
      }
   }
   loop(0)
}
a ! 1
a ! "ok"

Part II. Processes vs Actors

Something I

Erlang:

Lightweight processes
You can always (almost) create a new process for each new comer
Scheduler treats all processes fairly
Share nothing between processes
Lightweight context switch between processes
IO has been carefully delegated to independent processes

Scala:

Active actor is delegated to JVM thread, actor /= thread
You can create a new actor for each new comer
But the amount of real workers (threads) is dynamically adjusted according to the processing time
The later comers may be in wait list for further processing until a spare thread is available
Share nothing or share something upon you decision
Heavy context switch between working threads
IO block is still pain unless good NIO framework (Grizzly?)

Something II

Erlang:

Try to service everyone simultaneously
But may loss service quality when the work is heavy, may time out (out of service)
Ideal when processing cost is comparable to context switching cost
Ideal for small message processing in soft real-time
Bad for massive data processing, and cpu-heavy work

Scala:

Try to service limited number of customers best first
If can not service all, the later comers will be put in waiting list and may time out (out of service)
It's difficult for soft real-time on all coming concurrent customers
Ideal when processing cost is far more than context switching cost (context switch time is in ns on modern JVM)
When will there be perfect NIO + Actor library?

Planet Erlang updated

2008-12-30T09:23:27+00:00

PlanetErlang, the Erlang news and blog post aggregation site has been rewritten.

The new site should be easier to read. It includes a search engine, that allows users to search inside the history of Erlang posts among all the known blogs. The sitebar also contains latest Twitter post on Erlang.

Planet Erlang is available on the following address: http://www.planeterlang.org.

Enjoy !

Using Test::Harness with etap

2008-12-30T00:00:00+00:00

This isn't a beginner's walk through. It assumes you know how to use and read Erlang as well as install Erlang and Perl libraries

I love TAP. TAP is a protocol used to create and interact with testing frameworks. The general idea is that your unit tests interact with a TAP client which outputs the TAP protocol that is then consumed by a TAP server. What it boils down to is a really simple text based structure used to convey unit test results, with a few added bonuses here and there.

With etap and Perl's TAP::Harness we can create really slick test result output and move closer to having a continuous integration build environment. Before you get ahead of yourself, you'll need to make sure that you've got etap downloaded and in your Erlang lib path. You'll also have to install TAP::Harness and it's dependancies.

To demonstrate how this is done, we'll use the erlang_protobuffs library. This library is composed of two modules that provide functionality through direct interaction with their exported functions. They don't extend any OTP behaviors and have very few side effects. The project's sources are contained in the src directory and it's tests in the t directory. There is also a support directory that provides the Makefile include for building the actual modules.

The test target in the root Makefile calls the prove command with the verbose flag and a glob of the *.t files in the t diretory.

test: all
    prove -v t/*.t

The actual tests for the project are escript files with the .t extension. Using the escript hash bang lets us treat each file as it's own complete unit test. They can be run in any order and have their own configuration and directives. Below is the protobuffs_t_001.t file.

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -pa ./ebin -sasl errlog_type error -boot start_sasl -noshell

main(_) ->
    etap:plan(8),
    etap_can:loaded_ok(protobuffs, "module 'protobuffs' loaded"),
    etap_can:can_ok(protobuffs, encode),
    etap_can:can_ok(protobuffs, encode, 3),
    etap_can:can_ok(protobuffs, decode),
    etap_can:can_ok(protobuffs, decode, 2),
    etap_can:loaded_ok(protobuffs_compile, "module 'protobuffs_compile' loaded"),
    etap_can:can_ok(protobuffs_compile, scan_file),
    etap_can:can_ok(protobuffs_compile, scan_file, 1),
    etap:end_tests().

Lines 1, 2 and 3 are used by the shell and escript command to set any Erlang vm options necessary. In this case we want to include the ebin directory, start sasl with minimal logging and disable the shell.

The main/0 function is called escript with whatever relevant arguments that may apply, but we don't care about them so we ignore them for now. Here we perform several tests to assert that the project's modules load and export the proper functions.

The etap:plan/1 and etap:end_tests/0 functions wrap the etap*:* function calls and is used to define the test plan. At this point etap is not designed to work without finite plans.

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -pa ./ebin -sasl errlog_type error -boot start_sasl -noshell

-record(person, {name, address, phone_number, age, location}).

main(_) ->
    etap:plan(1),
    etap:is(protobuffs_compile:scan_file("t/simple.proto"), ok, "simple.proto compiled"),
    compile:file("simple_pb.erl"),
    Data = [{1, "Nick">>, string}, {2, "Mountain View">>, string}, {3, "+1 (000) 555-1234">>, string}, {4, 25, int32}],
    BinData = erlang:iolist_to_binary([protobuffs:encode(Pos, Value, Type) || {Pos, Value, Type} - Data]),
    #person{ name = "Nick">>, address = "Mountain View">>, phone_number = "+1 (000) 555-1234">>, age = 25} = simple_pb:decode_person(BinData),
    BinData = simple_pb:encode_person(#person{ name = "Nick">>, address = "Mountain View">>, phone_number = "+1 (000) 555-1234">>, age = 25}),
    etap:end_tests().

In the protobuffs_t_005.t file you see that you can do things like define records. In this test unit we call several of the library's exported functions and test some of the more complex functionality. The protobuffs_compile:scan_file/1 function actually writes several files which are then compiled by compile:file/1 and used by subsequent test unit function calls.

The output of the make test command, as processed by the TAP::Harness perl module, is very clean.

mbp:erlang_protobuffs ngerakines$ make test
mkdir -p ebin/
(cd src;make)
erlc -W -I ../include  +debug_info -o ../ebin protobuffs.erl
erlc -W -I ../include  +debug_info -o ../ebin protobuffs_compile.erl
prove t/*.t
t/protobuffs_t_001....ok   
t/protobuffs_t_002....ok   
t/protobuffs_t_003....ok   
t/protobuffs_t_004....ok   
t/protobuffs_t_005....ok   
t/protobuffs_t_006....ok   
All tests successful.
Files=6, Tests=17,  1 wallclock secs ( 0.04 usr  0.02 sys +  0.97 cusr  0.27 csys =  1.30 CPU)
Result: PASS

Depending on the error and severity, failed tests will either output in a similarly clean fashion or stop the entire suite from processing.

mbp:erlang_protobuffs ngerakines$ make test
mkdir -p ebin/
(cd src;make)
erlc -W -I ../include  +debug_info -o ../ebin protobuffs.erl
erlc -W -I ../include  +debug_info -o ../ebin protobuffs_compile.erl
prove t/*.t
t/protobuffs_t_001....ok   
t/protobuffs_t_002....ok   
t/protobuffs_t_003.... Failed 1/3 subtests 
t/protobuffs_t_004....ok   
t/protobuffs_t_005....ok   
t/protobuffs_t_006....ok   

Test Summary Report
-------------------
t/protobuffs_t_003 (Wstat: 0 Tests: 3 Failed: 1)
  Failed test:  2
Files=6, Tests=17,  2 wallclock secs ( 0.04 usr  0.02 sys +  0.96 cusr  0.26 csys =  1.28 CPU)
Result: FAIL
make: *** [test] Error 1

The prove command has a number of options and arguments that augment test execution. Please refer to prove --help or the documentation available on CPAN for more information.

There are a few things to take note of. All of the paths to included and referenced files, such as the simple.proto file, are relative to where the prove command is called. In this case it's in the root directory as part of the test target. Also, for some more complex and lengthy tests, please read up on the escript documentation. The compile mode can be used to pre-compile the file before being called but has a few caveats that you should be aware of.

CN Erlounge III

2008-12-28T18:29:16+00:00

I attended CN Erlounge III last weekend, it was a 2-day conference. I did a presentation about Scala vs Erlang.

I met Jackyz who is one of the translators of Chinese version "Programming Erlang". And Aimin who is writing a Delphi module to support Erlang c-node and c-driver in Delphi.

There is a commercial network monitoring product using Erlang from a major telecom company in China. And our Mobile-Banking platform (in Erlang) is scheduled to launch at middle of January too.

I talked with Yeka and Diuera from Broadview, a leading publisher in IT in China, they are really interested in importing "Programming in Scala" to mainland China.

And many thanks to Shiwei Xu, who is heavy working on Erlang community in China, and took the place to organize this conference.

I gave some encouragements to younger developers on learning Erlang and reading "Programming Erlang", since I'm the oldest one in attendees :-). Erlang is one of the best pragmatic and clear languages to learn concurrent/parallel and functional programming, and the book, is a very thoughtful and philosophic one on these perceptions.

And I'd like to see "Programming in Scala" also appeals in China soon, Scala is another pragmatic language on solving real world problems and, the book, is also thoughtful and philosophic one on our real world on Types, OO and FP.

Of course, choosing Scala or Erlang for your real world project should depend on the requirements.

I may be back to Vancouver next month for a while. Oh, it will be the beginning of new year.

CN Erlounge III photos by krzycube

Windows tricks for erlangers

2008-12-27T09:00:05+00:00

Not that I spend much time compiling erlang code in Windows (unless I’m using cygwin or coLinux), but the issue does pop up from time to time. Here are a few minor tricks that can help a bit.

Starting Erlang in the current directory
For a unix user, it’s of course odd that this should be a problem.

John Hughes used a nice trick that’s so simple that I slap myself for not having thought of it: Right-click on a .beam file, select “Open With…” and locate werl.exe. Now you can open an erlang shell in the current working directory by double-clicking a .beam file.

Modifying the Windows context menu
So what if you don’t have any .beam files, and you’re trying to get to an erlang shell
in order to create some?

I did some googling and found this tutorial on youtube on how to add custom entries to the context menu.

It seemed simple enough, but Vista still served me a few hours of utter confusion since it took what I inserted, and then copied it to another place in the registry (without letting me know)… but only the first time. My changes and additions were simply ignored. The solution? Search the registry for the key you inserted, and you’ll figure out where Vista wants it to be, then make your changes there.

What I’ve experimented with so far is to add under Computer\HKEY_CLASSES_ROOT\ErlangSource\shell\ the following entries:

Compile with erlc - Command = "C:\Program Files\erl5.6.3\bin\erlc.exe" "%1" Make all - Command = "C:\Program Files\erl5.6.3\bin\werl.exe" "-make" Erlang shell - Command = "C:\Program Files\erl5.6.3\bin\werl.exe" "%1"

A problem with running erlc this way is that the window is destroyed immediately upon completion. I’ve poked around a bit for good workarounds. One option is of course to write an erlang function that compiles the file, then either sleeps a short while, or waits for input.