Unofficial Erlang Planet

Getting Your Open Source Project to 1.0

2010-07-29T18:55:56+00:00

The project I founded, Apache CouchDB, recently hit 1.0. I'm very proud :)

It's been a long time, but we finally produced a release that's complete, performs well and is rock solid.

Already CouchDB is on over 10 million machines. It's used by big respected websites (like the BBC) and groundbreaking organizations (Mozilla and Canonical). We run on most *nix, OS X, Windows, and even Android phones. Have dozens of frameworks and client libraries available. There are 2 books available for sale right now. There is a venture capital backed startup, Cloudant, that offers CouchDB hosting and scales to huge datasets. And I'm CEO of another venture backed ($2 million invested) 12 person start-up, Couchio.

So how did I get here? It took a lot of time and effort (almost 5 years!), and the help of a lot of people. Here are some tips of what it took to get CouchDB to 1.0.

Why?

Successful open source projects need a reason for being. You need to decide why you are creating a project and what problems it solves. Whether it's one or many reasons, you need to figure out what they are and explain them.

Perhaps you are making something new, that hasn't existed. Why hasn't it existed it before? No had the idea? No one had the will to carry it through? Or maybe you are making something that's already in existence, like an HTTP server. What are your reasons? Simpler, faster, more features, different license?

If you are just doing it as a learning exercise, that's fine. But don't expect to attract a community until you can explain why it's useful beyond you own personal goals.

With CouchDB, my reasons were:
1. A schemaless document database with views, bi-directional replication and conflict detection to enable disconnected operation would be really useful.
2. I wanted to understand more about creating distributed systems and database internals.

No one cares about reason #2 except for me. But the first reason is compelling.

Make sure you can tell people why your project exists and what it's good for. And put the reasons on your project site where people can find them.

Code Comes First

Don't start a project unless you have a deep commitment to being a strong coder.

Now I'm not saying you must be a strong coder to participate in a project. Not at all. I'm saying that you must be strong coder to lead one. Maybe you'll get lucky and somehow attract a some really good coders to your project. But most really good coders go to projects with already solid codebases, or start their own.

Also, you don't have to be a strong coder when you start out, but you should know the basics and have a strong desire to learn and get better. Don't expect to attract anyone to your project until you have a substantial amount of working code that isn't a big ball of spaghetti.

With CouchDB, I always emphasized the quality of the high-level design and code implementation. We cannot under any circumstances lose or corrupt your committed data, or get things into an inconsistent state. Reliability and durability are absolutely imperative. Any design or implementation that doesn't meet these goals doesn't make it into the project.

Some projects might not have an emphasis on the reliability, but on absolute performance. That's a fine choice to make, but make sure your users know what they are trading off. And then actually deliver on the performance.

As the project moves along, you will need to ensure the code quality (reliability, performance, resource, usage, etc) is improving over time. If you aren't a good coder, you won't be able to do this.

Know What You Aren't

Almost as important as knowing what your project is trying to accomplish, is know what it isn't trying to accomplish.

When your project starts to get traction, but before it's done, you'll get a lot of people who want the project to work more like things they've used in the past. New users might think your goals and abilities are cool, but they'd trade it all for just a little more. They'll want everything your project does, plus a pony.

The problem is feature and scope creep. Even if you are successfully keeping the project on track, the community may get slowed down dealing with people trying to make it something it's not. Stating clearly what your project isn't trying to do or be helps make it much easier to explain what you can't implement or change.

Now, you can't define everything your project isn't. (It's not a video game. It's not accounting software. It's not a banana. It's not a rainbow. etc.). But you can find the things it's related to, overlaps with, or might be confused with, and explicitly say it's not those things.

With CouchDB, because we are a database, people often asked us to add features that were in traditional RDBMS's, but didn't fit well with the CouchDB data model. Not being intimately familiar with CouchDB's model and how it all fits together, they don't realize that what they're asking for simply doesn't work. But because we explicitly stated on the project site we aren't a relational database and aren't trying to replace relational databases, it made it much easier to explain why those features weren't a good fit for what CouchDB is trying to accomplish.

So if you don't clearly define what your project isn't, often people will try to make it into those things. This can damage the community, as moving forward is slower and people feel like they aren't being listened to. Be explicit what you aren't, and it makes it much easier to focus on what you actually are.

Don't Do Everything (Well)

So you are superstar coder, your code is clear and concise and high quality, you write clear complete documentation, your create all the tests, and you fix every bug. You are awesome!

Thing is, you might be awesome, but until you actually get a community behind the project, the project will be limited in an absolute sense by what a single person can produce. And if you are doing everything, that's not a whole lot. Trying to do everything well means you'll probably never actually release anything.

Unfortunately, at first, you _will_ need to do everything. But just don't do everything really well. Instead, you'll have do some things crappily, and then move on. In addition to writing all the code, you'll need to: Create a project site. Explain your project. Write documentation. Do the releases. Start a bug tracker. Create a mailing list and answer questions. And you'll have to do most of these things poorly if you want to keep moving the ball forward.

You'll have to do some things poorly. But you'll need to pick a few things that you do really well and execute on those things. (The code should be one of the things you do well).

And everything you do, you'll need to make it easy for others to participate. To add patches, to update and create documentation, make bug reports and send patches. And make it clear that help is desired.

Don't get hung up on trying to make everything perfect. That just paralyzes you. But by picking a few things to do well, you will attract people to help you with the things you aren't doing well.

Community Wants to Help

Open Source is awesome in the way it attracts people who just want to help make something cool. Many people want to contribute their time, but only if they think their help will amount to something in the long run. They don't want to spend time and effort on something that doesn't yet show potential or might be abandoned if the creators lose interest.

If you have a solid codebase, then it becomes much easier to attract people to your community. If people can recognize there is at least something high quality about your project, but it's lacking in some areas, people will want to help you in those areas. But you have to have the high quality pieces in place. People don't want to be the one excellent contributor to dreck. They'd rather not have their efforts associated at all.

They do want to be a part of something great. They want to add their work and make it even better. They want to contribute to projects where the total excellence of the project reflects well on them and their efforts. They want to make the world a better place, and don't want their efforts wasted.

And people who like making the world a better place are exactly the kind of people you want to attract. You want people to have pride in their contributions, and to feel like they are really positively affecting the things they care about. Those people have lots of projects to choose where they can add their time and talents. If they feel their efforts on your project are wasted, they are gone. Make sure the people who show a strong desire to contribute aren't ignored, and feel like their efforts will eventually amount to something.

Being a part of Apache has helped CouchDB tremendously. Partially it's because Apache has helped our visibility and credibility. But it's also because we've adopted the "Apache Way", which is more focused on the community aspects of a project than on any specific contributor. Without our amazingly active community, CouchDB would be far behind where it is now.

Community Is Often Incompetent

Unfortunately, many people who will want to help you will produce contributions of poor quality. You will have deal with this "help", and do so diplomatically. The best way to point out the shortcomings with their contributions is to identify what needs improvement without denigrating their overall effort. This can be hard, and many don't want to hear why their efforts aren't up to the project's standards.

Sometimes you have to hurt peoples feelings. But it's better to be honest then to have the quality of your project brought down. If they can't handle the feedback, so be it. The good news is the people who do listen to constructive criticism and actually improve the quality of their contributions are incredibly valuable. Look for these people and nurture their involvement.

With CouchDB, we try to listen to all members of our community, but we only grant commit access to the ones who have shown high quality contributions. Our committers are our first line of defense against poor code and design.

Paul Graham Was Right

It seems to make sense to choose a mainstream language for your project. The more mainstream it is, the larger the potential community you can attract. While that's true to an extent, the quality of the community is more important than its absolute size. Much more important.

Using a mainstream language means you are also competing for contributor's time from other projects in the same language. So the pool is large, but in the end, you still have to attract quality developers from other things competing for their attention. And the competition might actually be stronger in that larger pool.

The more mainstream a language, the more likely it is that a random developer knows it because it's what they use at work. They aren't necessarily interested in being more productive, being more reliable, or whatever. They are interested in getting paid, and they choose their language not for elegance, power or performance, but for the number of job openings available.

If you pick a non-mainstream, more esoteric language, you tend to get a higher quality of developer. You tend to find people who absolutely love programming and building, and choose their languages not based on the scale of pay, but because they make the developers and projects more powerful. So while the total pool of contributors is smaller, they tend to give a higher quality of contribution. You get a much better signal to noise ratio.

As Paul Graham explained in Beating the Averages, the exotic languages tend to attract devs who love to learn and expand their toolbox. You'll attract more of the types of devs who don't mind creating new code to fill in the gaps, or diving into source to find a bug. They aren't afraid of what they don't know, they actually get excited by the chance to learn and do something new.

But if you pick enterprisey language X, you might find you are spending more of your time fixing problems and dealing with developers who just don't "get it". If you aren't careful, this can drown your project and bring the total code quality down to the point where you can't find good devs to help you anymore. With the less popular, esoteric languages, that tends to be less of a problem and you get a higher quality of contribution in general.

Use Your Brain

I can keep listing all the stuff we did, but you aren't creating the same project under the same circumstances. Pretty much everything I've said here, we've not followed at some point during the project. Often it was to the detriment of the project, but sometimes it just didn't make sense to blindly follow a rule or guideline.

You have a brain, and using it is the most important thing to remember at anytime. Projects can't follow cookie cutter rules. Even the "Apache Way", as I've discovered, means different things to different people, often at different times.

So take my advice here with a grain of salt, and use your brain to figure out what's actually important to you, your project and it's community. Good luck!

CouchCamp is Coming Soon!

2010-07-26T17:10:07+00:00

CouchCamp, September 8-10

This is the place to be to learn and hack on Apache CouchDB. In honor of the recent 1.0 release, for a limited time it's only $500, with accommodations.

In addition to unconference style discussions, we've got some great speakers: Selena Deckelman, Stuart Langridge, Ted Leung, Josh Berkus, Dion Almaer and me :)

One thing I'm really excited to talk about is our work porting CouchDB to mobile platforms. Android, iOS, RIM, etc. We've got some very cool stuff coming :)

Erlang and REST – an interview with Steve Vinoski

2010-07-22T12:23:21+00:00

Interview recorded at Erlang Factory, where Steve Vinoski discusses the key concepts of RPC frameworks such as CORBA, and how that compares to REST, and how Erlang fits into both worlds

Zotonic rethinks the CMS with Erlang

2010-07-21T13:36:41+00:00

A chat with Marc Worrell, Lead Architect of Zotonic - a new Content Management System written entirely in Erlang.

Adding Health Checks to Deckard from Chef.

2010-07-19T20:52:09+00:00

Recently, we (at Cloudant) open sourced Deckard, a HTTP content check monitoring system based on CouchDB. One of the best bits about using Couch is that it gives you a ReST API and with Deckard it can be used to add new health checks. Doing a simple PUT adds new URLs to monitor. At Cloudant we love Chef and use it for everything. Chef has things called resources and providers. Resources are abstractions that describe the state you want a machine to be in. Providers perform the actions described by a resource. A good example is using the package resource on Centos uses yum while on Ubuntu it uses apt-get. The resource abstracts that away, letting the provider (and node) deal with the specifics on how to install the package. This makes your recipes nice and DRY, use the same code to install packages on all sorts of platforms. There are resources and providers for anything from installing packages to even one I wrote for executing Erlang code via erl_call. One resource that works well with Deckard is the HTTP request resource, using it makes it very easy to add health checks from your cookbooks. We use something like the following code to add checks to new nodes at Cloudant:

This code will add the document describing the check to the monitor_content_check database and then create a file so we can use “not_if” and Chef won’t attempt to add the check twice. Pretty cool stuff and even more reason that everything should have an API. Even cooler than this example would be to use Chef Search to do the same thing but I’ll save that for another blog post.

New Column, New Interview

2010-07-17T18:01:47+00:00

A couple newsworthy items:

The July/August 2010 issue (PDF) of The Functional Web column is available, this time with guest columnist Aaron Bedra writing about using Clojure and the Compojure web framework on the Google App Engine platform. Pretty interesting read, and plenty of code examples.
Sadek Drobi interviewed me about Erlang and REST at Erlang Factory back in March, and the interview is now up at InfoQ.

PostgreSQL Erlang client library epgsql supports asynchronous messages from LISTEN/NOTIFY events

2010-07-08T19:27:05+00:00

If you use PostgreSQL with Erlang, you're probably already familiar with the epgsql client library. Something that slipped my attention was the addition of...

ProcessOne releases OneCached, a Memcached in erlang

2010-07-05T20:41:54+00:00

ProcessOne has just released OneCached, a Memcached server and client implementation written in Erlang.

OneCached is a new Memcached server and client implementation written from scratch in Erlang by ProcessOne.

From the Memcached website:

What is Memcached?

Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

Memcached is simple yet powerful. Its simple design promotes quick deployment, ease of development, and solves many problems facing large data caches. Its API is available for most popular languages.

OneCached supports the set, add, replace, get, incr, decr, delete, flush_all and quit commands. It doesn't handle expiration time.

You call pull the source code from the public repository at: https://git.process-one.net/onecached

git clone git://git.process-one.net/onecached/mainline.git

To compile, just run make, and to start, just type:

bin/onecachedctl start

OneCached is released under the Erlang Public License (EPL), version 1.1. It is available from ProcessOne Labs.

New Erlang job board – totally-erlang.com

2010-07-05T11:27:25+00:00

The Zotonic guys put together a new Erlang job board at http://totally-erlang.com/

erldocs.com updated with R14A support, mochiweb, and available for your own projects

2010-06-29T12:53:43+00:00

Dale Harvey just recently updated erldocs.com with R14A support. If you are doing any erlang development, or even looking to just learn a bit about erlang, I highly recommend using erldocs over the official documentation.

A Phase Shift for the ORM

2010-06-28T12:55:25+00:00

I came to know about Squealer from one of Dean's tweets over the last weekend. Over there at the git repo README, there's a statement which makes a very succinct point on the role that relational mappers will be playing in the days to come. It says "... ORMs had it the wrong way around: that the application should be persisting its data in a manner natural to it, and that external systems (like reporting and decision support systems - or even numbskull integration at the persistence layer) should bear the cost of mapping."

I have expressed similar observations in the past, when I talked about the rationale of modeling data close to the way applications will be using them. I talk about this same architecture in an upcoming IEEE Software Multi-Paradigm Programming Special Issue for Sep/Oct 2010.

In most of the applications that churn out domain models in an object oriented language and persist data in a relational store, we use the ORM layer as follows :

It sits between the domain model and the relational database, provides an isolation layer between the two at the expense of an intrusive framework invasion within an otherwise non-complicated application architecture.

The scenario changes if we allow the application to manipulate and persist data in the same form that it uses for modeling its domain. My email application needs an address book as a composite object instead of being torn apart into multiple relational tables in the name of normalization. It will be better if I can program the domain model to access and persist all its entities in a document store that gives it the same flexibility. So the application layer does not have to deal with the translation between the two data models that adds a significant layer of complexity today. The normal online application flow doesn't need an ORM.

How does the translation layer get shifted ?

Consider an example application that uses Terrastore as the persistent storage for implementing online domain functionalities. Terrastore is a document database having some similarities with each of CouchDB, MongoDB and Riak in the sense that data is stored in the form of JSON documents. However unlike others it has an emphasis on the "C" component of the CAP theorem and provides advanced scalability and elasticity features without sacrificing consistency.

Terrastore is built on top of Terracotta, which is itself an ACID based object database that offers storage of data larger than your RAM size clustered across multiple machines. Terrastore uses the storage and clustering capabilities of Terracotta and adds more advanced features like partitioning, data manipulation and querying through client APIs.

As long as you are using Terrastore as your persistent database for the domain model, your data layer is at the same level of abstraction as your domain layer. You manipulate objects in memory and store them in Terrastore in the same format. Terrastore, like all other NoSQL stores is schemaless and offers a collection based interface storing JSON documents. No impedance mismatch to handle so far between the application and the data model.

Have a look at the following figure where there's no additional framework sitting between the application model and the data storage (Terrastore).

However there are many reasons why you would like to have a relational database as an underlying store. Requirements like ad hoc reporting, building decision support systems or data warehouses are some of the areas which are best supported with relational engines or any of the extensions that relational vendors offer. Such applications are not real time and can very well be served out of a snapshot of data that has a temporal lag from the online version.

Every NoSQL store and many SQL ones as well offer commit handlers for publishing async jobs. In Terrastore you can write custom event handlers that you can use to publish information from the document store to an underlying relational store. It's as simple as implementing the terrastore.event.EventListener interface. This is well illustrated in the above figure. Translation to the relational model takes place here which is one level down the stack in your application architecture. The Terrastore event handlers are queued up in a synchronous FIFO manner while they execute asynchronously which is exactly what you need to scale out your application.

I took up Terrastore just as an example - you can do most of the stuff (some of them differently) with other NoSQL stores as well front ending as the frontal persistent store of your application. In real life usage choose the store that best maps the need of your domain model. It can be a document store, it can be a generic key value store or a graph store.

The example with which I started the post, Squealer, provides a simple, declarative Ruby DSL for mapping values from document trees into relations. It was built to serve exactly the above use case with MongoDB as the frontal store of your application and MySQL providing the system of record as an underlying relational store. In this model also we see a shift of the translation layer from the main online application functionalities to a more downstream component of the application architecture stack. As the document says "It can be used both in bulk operations on many documents (e.g. a periodic batch job) or executed for one document asynchronously as part of an after_save method (e.g. via a Resque job). It is possible that more work may done on this event-driven approach, perhaps in the form of a squealerd, to reduce latency". Nice!

All the above examples go on to show us that the translation layer which has so long been existing between the core application domain logic and the persistent data model has undergone a phase shift. With many of today's NoSQL data stores allowing you to model your data closer to how the domain needs it, you can push away the translation further downstream. But again, you can only do this for some applications that fit this paradigm. With applications that need instant access to the relational store, this will not be a good fit. Use it whereever you deem it's applicable - besides the simplicity that you will get in your application level programming model, you can also scale your application more easily when you have a scalable frontal database along with non-blocking asynchronous writes shoveling data into the relational model.

Spanish Language Erlang List

2010-06-25T01:53:35+00:00

If you’re looking for Spanish language group to discuss Erlang, Mariano Guerra recently started “ErlAr“, a mailing list for all things Erlang, but in Castellano. Mariano is the author of Efene, a Javascript-syntax language that compiles to Erlang bytecode. I’m planning an interview with Mariano on Efene soon. There is a small but growing group [...]

Case study: Arc International

2010-06-23T10:30:21+00:00

Arc International has migrated away from a proprietary solution to the open source, and open standards ejabberd and Psi, realising cost-savings and productivity increase.

Arc International is well known for distributing famous tableware brands such as Luminarc®, Arcoroc, Chef&Sommelier, Pyrex and Cristal d’Arques Paris. With 12,000 employees around the world, from Europe to Asia, and the Middle East to North America, they needed to lower the costs of their previous solutions.

They have chosen ejabberd, as an open-source, scalable XMPP platform. In conjunction, with the Psi cllent Arc International has achive their goals of cost-savings and high-quality service.

The cost-savings have benefitted from a lower TCP on IT, as well as reduced telephone bills, while providing a same feature-set to the employees worldwide.

Read our case study on the website, or as a PDF (582 KB).

Migrating From MySQL to CouchDB

2010-06-22T18:08:51+00:00

Here is a detailed white paper from John P. Woods of Interactive Mediums, going into depth about their migration of their mobile marketing archive from MySQL to CouchDB.

http://www.couch.io/migrating-to-couchdb

CouchCamp, aka BurningCouch!

2010-06-22T17:12:57+00:00

Registration for CouchCamp is now open. CouchCamp is our conference about CouchDB, September 8th - 10th 2010 at Walker Creek Ranch, north of the Bay area.

This is the event to learn all about CouchDB and it's innards, the community, and how much beer I can drink before hurling. It's going to be fun!

How Software Is Built: Apache CouchDB

2010-06-18T18:31:23+00:00

Here is a detailed interview I recently did for HowSoftwareIsBuilt.com about CouchDB, document databases, cloud and replication:

http://howsoftwareisbuilt.com/2010/06/18/interview-with-damien-katz-apache-couchdb/

Miscellaneous CouchDB News

2010-06-15T16:58:21+00:00

Awesome New Hires: We've recently hired Volker Mische of GeoCouch, and CouchDB contributor Filipe Manana. These guys are damn good!

CouchDB Case Study: Anologue

"It was difficult to continue programming at times, as I could hardly see through the tears of joy."
- Jon Adams, Chief Architect of Anologue

CouchCamp: September 8th - 10th 2010 Registration will open soon.

Just Opensourced: Gaff and Deckard

2010-06-04T21:18:50+00:00

This post was stolen from my original post on the Cloudant blog.

Today we released two open source projects that have been in use internally at Cloudant for some time now, Gaff and Deckard.

All of our infrastructure is in the cloud and as such we need a way for disperate systems to all request resources, this is where Gaff comes in. Gaff is a pubsub daemon for asynchronously talking to cloud APIs using AMQP. Currently it supports a subset of the Dynect (DNS), Slicehost and EC2 APIs and uses geemus‘ awesome fog Ruby library. The basic workflow for Gaff is to send JSON-RPC formated messages to an AMQP exchange with a routing key corresponding to the API you are talking to, you could be sending these messages from a web application or another service. Each message gets routed to an API specific queue and is picked up by Gaff and turned into the appropriate API call, starting, stopping, modifying your servers on EC2 or elsewhere.

We have a lot of CouchDB instances to keep tabs on to do this we wrote Deckard. Deckard is a HTTP check monitoring system based on CouchDB. Yo dawg! What better than to monitor CouchDB with CouchDB (and some Ruby)? Deckard supports basic HTTP content checks, email alerts, SMS alerts (via email) for on-call rotations, basic maintenance scheduling, replication latency alerts (between two Couches) and even has EC2 Elastic IP support for failover between two EC2 instances. Best of all since it’s based on Couch you get an API for free, just PUT a doc in the HTTP checks database and you get a new HTTP check the next time Deckard runs.

Checkout these and my other projects on GitHub and follow Cloudant and myself on Twitter.

ejabberd 2.1.4 and exmpp 0.9.4 bugfix releases

2010-06-04T11:22:42+00:00

We are pleased to announce the bugfix releases ejabberd 2.1.4 and exmpp 0.9.4.

Regarding ejabberd 2.1.4:

Brief summary of changes:

Bounce messages when closing c2s session
Bugfixes when handling Service Discovery to contacts
Don't send error stanza as reply to error stanza
Don't store blocked messages in offline queue
Extauth: Optionally cache extauth users in mnesia
Full support for XEP-0115 Entity Capabilities v1.5
HTTP-Bind: Remove 100ms delay, export functions for prebinding methods
LDAP: Inband password change, Extensible match support, and ldap_tls_verify
Localization: Updated most translations
MUC: Refactor code to reduce calls to get_affiliation and get_role
ODBC: Add created_at column also to PostgreSQL schema
PAM: New option pam_userinfotype to provide username or JID
Pubsub: Send node notification. Enforce presence_based_delivery, show_values
Vcard: Automatic vcard avatar addition in presence
WebAdmin: New Access rule webadmin_view for read-only

Check the Release Notes for a more complete list of changes:

http://www.process-one.net/en/ejabberd/release_notes/release_note_ejabberd_2.1.4

If you upgrade from ejabberd 2.0.5 or older, read carefully the release notes of ejabberd 2.1.0 too, because there were several changes in the installation path and the configuration options.

The list of solved tickets since previous version is available on ProcessOne bug tracker:
http://redir.process-one.net/ejabberd-2.1.4

Download

ejabberd 2.1.4 is available as source code package and binary installers for Linux 32 bits, 64 bits, Mac OS X Intel, and Windows:
http://www.process-one.net/en/ejabberd/downloads

Regarding exmpp 0.9.4:

Brief summary of changes:

Add unexpected-request with cancel needed by pubsub
Caps: Add exmpp_caps to the compilation scripts
Caps: Move Caps erlang records to a header file
Handle the case that the payload is incorrectly defined in error stanzas
Preliminary documentation of functions from ejabberd 2 available in exmpp
Remove unexpected-request with cancel as this is pubsub specific
Support for SRV DNS query

exmpp home page:

http://support.process-one.net/doc/display/EXMPP/
or easier to remember: http://exmpp.org/

Download

Download exmpp 0.9.4 source code package from:

http://download.process-one.net/exmpp/

You can also check the ProcessOne Labs page:
http://www.process-one.net/en/labs/

CouchDB on your TV

2010-06-04T00:53:54+00:00

After initially getting rejected for sound issues, Google Ads has finally approved our awesome Couchio commercial.

GoogleAds Commercial Version

So we've aired this commercial 11 times so far, 5 times during Robot Chicken, twice duringAqua Teen Hunger Force, once during Tim and Eric's Awesome Show, Great Job! and 3 times on VH1.

It has gone to a total of 321,253 impressions, which means 321,253 people have seen Chris running like a lunatic around the streets of Oakland, singing the CouchDB praises.

This has cost us a total of $709.57!

Just saw a commercial for CouchDB. Seriously. I'm at a loss for words. #couchdb #couchio
@jrgarcia

Someday CouchDB will actually be inside your TV, you can watch CouchDB while watching your CouchDB TV from your couch.Yo dawg.

Grokking Functional Data Structures

2010-05-31T12:20:46+00:00

With so many languages espousing functional programming paradigms, it's no wonder that functional implementations of common data structures are also becoming more and more popular. As a programmer it's important that we learn to adopt them into our repertoire of tools and techniques.

Imperative data structures operate through in-place mutation of data. In our everyday usage of data structures in Java we mutate lists, arrays and any other type to realize the algorithm that we implement. Most of the books on data structures and algorithms explain all the complexity analyses of them using imperative data structures as case studies. When we create an object in Java (an instance of an abstract data type) and invoke some methods on it (some of which can potentially mutate its state) we get some result as an observable outcome of the sequence of operations. This sequence of operations on the value of the abstract data type constitutes a logical future of that type.

With functional data structures, immutability is the buzz of the day, though it's not mandatory that all of them have to be immutable. Many functional languages allow mutable data structures, which, more often than not have also proved to be more pragmatic than the pure school of thought. I wrote a blog post on this same topic some time back.

The other point of view that purists say is that the main point is not that whether you need a mutable array or a hash table. The more important part is that you need to access or update a data structure within a certain bound of time and space complexity. Okasaki's book Purely Functional Data Structures contains a rich collection of functional implementations of many of the popular imperative data structures. And quite contrary to what we tend to believe, given proper language support, these implementations are often as efficient as their imperative counterparts.

When we talk about functional programming, we talk about immutability of data structures and purity of functions. In the absence of mutable state, these are the two cornerstones that make reasoning of concurrent programs much easier in the functional paradigm. Since data structures are immutable, there's no in-place update of values, which means that every operation that changes the value of a data structure will create a new instance of it. So at the end of n operations on an instance of an ADT we will have all the previous versions of the data structure available for access. This is called persistence of data structures, unlike the imperative ones that we discussed earlier, where one update destroys its previous state (we call them ephemeral data structures). Unlike ephemeral data structures which has a single logical future (as we saw earlier), persistent data structures can potentially have multiple logical futures.

Have a look at the following figure, where the list l can have 2 logical futures based on the two possible mutating operations that we make on it. None of them mutate the original list and at the end we still have l as '(1 2 3). Only the two logical futures point to two different instances of the ADT.

Since Clojure sequences are persistent, we have all instances of the ADT accessible even after the entire sequence of operations. Had persistence meant brutal copying of instances, obviously we would not have the same performance guarantees as the imperative ones. Persistent data structures are implemented through careful sharing of structures across instances that make them affordable in real life applications. Have a look at the following tree structure that employs copying of paths from the root down to the newly inserted leaf while sharing the rest with the original ADT.

As with the earlier scenario, both the versions of the tree are available even after adding the new node 8 to the tree - xs points to the original tree, while ys points to the new one.

Despite the fact that functional data structures are persistent, it's possible that their implementations make use of mutable storage. It's mostly done for efficiency of implementation, as we have seen with transients in Clojure. However, for all practical purposes, the data structure is published to its clients as an immutable one.

It's also possible that we can implement a persistent data structure using an imperative language like Java. In fact, FunctionalJava does exactly the same and offers a rich suite of persistent data structures developed in Java. But of course there's a trade-off. First, the usage looks intimidating with Java being quite verbose and without much of type-inferencing capabilities. And, as Okasaki points out in his book, you need to have an implementation of call-by-need in order to have improved amortized complexity of functional data structures. That's the subject of another post, some other day.

What would be a life like without mutable arrays and hash-tables, the bread and butter data structures on which imperative programming thrives upon?

Well, in functional languages lists, trees and tuples take the place of arrays and hash-tables. The basic unit of a functional data structure is to find a general representation of a sequence. And then apply recursive transformations on it to implement specific operations. In this post I give you a very brief look at a couple of such building blocks that are used extensively to implement persistent data structures in many languages.

Consider the use case for a data structure where you would like to reach a specific item (often called a focus point) within the structure and do manipulations around it. With mutable structures you can reach a specific node within a list, an array or a tree and make changes to it. Or the very sole reason for which doubly linked lists exist is to allow you to reach a node within the list and move in either direction. Or you may want to have a new node and splice it within an existing doubly linked list using a minimum number of pointer manipulations.

In the functional world we have zippers which can be adapted to this very use case. Zipper, invented by Gerard Huet, is a generic data structure which allows you to identify a focal point within an ADT and make all manipulations around it as cheap as in-place updates. And it does all this in a completely immutable and persistent way. Every focal point within the tree is identified by a pair of Tree and Context, each of which is again a recursive data structure. The root of the Tree is the focal point around which we would like to do mutable operations. The functional pearl paper by Huet has all the details of this functional data structure. Clojure has a nice implementation of zipper as part of its core distribution. Zipper is a general purpose data structure and has lots of exciting use cases. I hope to cover some of them in my upcoming posts.

Another interesting data structure used functionally is the finger tree, invented by Hinze and Paterson. A finger tree is a 2-3 tree representation of a sequence that gives you cheap access to 'fingers', where you would like to do your manipulations on. And then you can define transformation functions using reductions and monoids that implement specific operations. For example, you can annotate every node of a finger tree using a characteristic function (modeled as a monoid), which you can use to look up specific elements having that property. If you want to implement fast positional operations (like accessing the nth element of the sequence), annotate the finger tree with the 'size' function, so that every node has the size of the subtree stored as its measure. Given size annotations we can now find the nth element in O(log n) time. Similarly annotating the tree with a 'priority' function turns a finger tree into a priority queue. Similarly you can implement deques, interval trees and a host of other data structure variants using the single underlying representation.

This post is just an introduction to functional data structures and I hope to cover some of the specific ones in future. The key difference in implementation with mutable data structures is that the functional ones are based on recursive transformations, easier to model using the power of pure functions. They are persistent and as we discussed above can have multiple logical futures. This makes their runtime analyses a big challenge. In order to obtain amortized complexity that match their mutable counterparts you need to have support for laziness and call-by-need from the underlying implementation language. Computations once forced need to be memoized so that the next logical future can use it without any additional expense on its realized cost of execution.

High Performance CouchDB

2010-05-25T18:47:06+00:00

At Couchio we are working with a company that makes high performance database appliances. These are complete system stacks where all the components, software and interconnects are carefully chosen, modified and optimized. They can deliver performance and scalability that's often an order of magnitude better than what can be accomplished by assembling your own systems with commodity components, and reducing overall hardware and admin costs.

So if you are a user of Apache CouchDB and are interested in high performance CouchDB servers, we'd love to talk with you to help us shape our plans and offerings, and potentially join an early adopters program. Email me at damien@couch.io.

Adding support for more than 1024 files to etorrent.

2010-05-23T07:15:23+00:00

Some torrent files contain a humongous amount of files. Thousands. This is one of the problems you have to cope with as a client-writer and I plan to take care of both etorrent and combinatorrent. However, the solution I've adopted for etorrent is sinisterly beautiful, so I decided to write it down.

The Problem

Open files are limited on UNIX systems. This is to protect different applications against each other and to circumvent an eventual resource exhaustion on the system. A typical limit is 1024 files, or fewer in some cases. In etorrent, a file is governed by a process which plays the role of that file. Whenever you want to do a file operation on that file, you get hold of the process Pid and send it a message. Simple.

Resources are limited by a timeout in these file processes. When a file has not been in use in 60 seconds, the process governing it terminates and frees up the resources on that file. It works reasonably well. The problem, however, is that some torrents have more files than the file descriptor limit. When we check the file upon starting up, we unfortunately open more than 1024 and then hit the limit.

The Solution

The solution is deceptively simple. We add a janitor process to the game. Whenever a new file is opened the janitor gets informed and the file process enters itself into an ETS table. Whenever we do an operation on the file a timestamp is bumped in the ETS table. This goes on and on; if a process dies, a monitor in the janitor cleans out the entry from the ETS table.

Now, whenever a new file is opened we check the size of the table against a high watermark, 128 by default. If more processes are opened, we extract the full table and order it by last bump. Thus, the first elements in the resulting list are the processes which have been used the farthest back in time. We then ask enough of these to terminate to bring ourselves back under a low watermark - ensuring we won't be hitting the resource collection all the time we add a new file system process to the game.

Updating the ETS table is expected to be rather cheap. The table is public, so each file governing process maintains its own entry. I don't think they will spend much time waiting for each other on the table. And if they do, there is always {write_concurrency, true} we can set on the table.

WaveOne, a wave server and client by ProcessOne

2010-05-21T13:31:42+00:00

WaveOne, the new wave server developped by ProcessOne, has been demoed at Google I/O.

ProcessOne has been invited to join the Wave area in Google Developer Sandbox at Google I/O 2010 in San Francisco.

Mickaël Rémond, the founder of ProcessOne, has been invited by Google in the context of the Google I/O event, to write a blog post to introduce WaveOne, our own wave server implementation.

WaveOne is basically an ejabberd module, so it is written in Erlang. It features an Operationnal Transform engine, as well as a Wave Store. It means that with the help of a client, you can - on your own WaveOne server installation - create, edit, store and share wavelets.

The WaveOne extension to our XMPP client OneTeam enables you to manage your wavelets (and blips), like create, edit, as well as add participants to wavelets, still on your own WaveOne server installation.

With the WaveOne client and server, you can collaborate in real-time internally, as well as externally, thanks to the federation capability. WaveOne supports federation with Google Wave in both directions. You can host the wavelets on your local server and have remote participants or you can do the reverse and participate to wavelets hosted on the Google server. This enables you to add participants, or be added as participant, to wavelets hosted on any wave server. Everything works in real time with character by character support.

Here is a screenshot that shows a wavelet (containing two blips) edited both on Google Wave and ProcessOne's WaveOne:

The result is much like having your own e-mail or XMPP client and server under your own control. This is built on the power of XMPP-based federation.

A handy feature of our WaveOne implementation is the capability to share a textarea of a webpage, for collaborative real-time text editing. This has been demoed at Google I/O, and has received much interest. Here are some sample screenshots.

The user right-clicks in a textarea, and shares it:

He adds participants to the wavelet:

All participants can collaboratively edit the text in real-time:

Migrate Notes data to CouchDB: Smart Notes 2 Couch

2010-05-21T00:54:35+00:00

can migrate your Lotus Domino data to CouchDB data. It's free and it's open source.

http://www.smartlotus.com/smart-notes-2-couch.html

I'm in your phone

2010-05-21T00:53:14+00:00

CouchDB on Motorola Droid

And on Nokia n900

CouchDB Rap

2010-05-21T00:44:40+00:00

I Use Couch DB.

Premiered today on The H. Nice.

17 May 2010: Erlang Solutions awarded a Knowledge Transfer Partnership with the Kent Business School

2010-05-17T10:15:58+00:00

Erlang Solutions is proud to announce that a third KTP project has been accepted by the Knowledge Partnerships Programme (KTP). This is in addition to the Reusable Components project that recently completed and the E-Learning/E-Certification Project that began in October 2009. This project will be a strategic marketing project together with the University of Kent, who have been our Academic partners in the other two projects and with whom Erlang Solutions has a strong partnership.

The main purpose of the project will be to establish and nurture a marketing orientation within the company and improve current marketing processes from a tactical and strategic perspective.

10 May 2010: Erlang Solutions Announces Opening of New Office in Stockholm, Sweden

2010-05-16T19:16:06+00:00

Erlang Solutions Ltd has opened a new office in central Stockholm. The new office and and staff with extensive knowledge of Erlang will allow the company to better serve its large client-base in Sweden and worldwide. The expansion also provides an expanded training portfolio and facilities. It is envisaged that Erlang community will meet here at regularly organised Erlang User Group Meetings. In addition Erlang Solutions will be able to accommodate more students for internships, training and certification.

Erlang Solutions’ new office is located at Saltmätargatan 5, 113 59 Stockholm, Sweden and can be reached by phone at +46 - (0) 8 - 120 424 00

CouchDB Case Studies

2010-05-06T21:10:31+00:00

We've just put up our new CouchDB case studies page. So far we have the BBC, Assay Depot and Interactive Mediums, and more in the works. And if you are interested in being profiled for a case study, see this post.

Google Denies Our Commercial

2010-05-05T22:26:54+00:00

Recently Chris and Claire produced a commercial for Couchio and CouchDB.

We tried to run it on real TV via Google AdWords, but they denied it:

Hello Claire,

Thank you for your email. I have reviewed your account and found that the ad was rejected for the following reasons:

* Your ad clearly and accurately describes your business, product, or service.

* Rich Media and TV ads: your ad clearly identifies the sponsor of the ad.

* Online Audio and TV ads: Your ad does not include a 'shout-out' that is not commercial in nature.

Please note that the audio quality is not clear enough to explain the business information. This is the reason why your ad was disapproved. I suggest you improve the audio quality and resubmit your ad.

Please follow these technical Specifications and Instructions in future:

http://adwords.google.com/support/aw/bin/answer.py?hl=en&answer=78641

Once you edit your ad to comply with our policies, your ad is automatically resubmitted back to us for review. We appreciate your effort in editing your ad.

Sincerely,

Shilpi
Google AdWords

I think it's that they just can't handle our awesomeness, but they claim it's something about the audio. I'm thinking we'll resubmit with the lyrics scrolling at the same time, so there is no confusion what's being sung.

Announcing New Erlang Native GeoCouch

2010-05-05T22:25:58+00:00

GeoCouch: The Future is Now

An idea has become reality. Exactly two years after the blog post with the initial vision, a new version of GeoCouch is finished. It's a huge step forward. The first time the dependencies were narrowed down to CouchDB itself. No Python, no SpatiaLite any longer, it's pure Erlang. GeoCouch is tightly integrated with CouchDB, so you'll get all the nice features you love about CouchDB.

Awesome work Volker!

Availability, the Cloud and Everything

2010-05-01T17:37:42+00:00

Finally posted my presentation at Erlang Factory, WTIA Cloud SIG and Seattle Scalability Meetup here on the blog.

Availability, the Cloud and Everything

View more presentations from logicalstack.

Create a Custom CouchDB indexer

2010-04-30T02:43:04+00:00

Recently I worked with Volker Mische to help him create a new R-Tree geo-spacial indexer for CouchDB, the details of which he'll be announcing soon.

And not only that, he took the time to write up a tutorial how to create you own customer indexer. If you wanted to write a new full text indexer in Erlang for CouchDB, this tutorial is the place to start. Awesome work!

http://vmx.cx/couchdb/tutorial/indexer.html

ProcessOne at Google I/O Developer Sandbox 2010

2010-04-29T15:03:47+00:00

We have been selected to demonstrate our WaveOne server and preview client in Google I/O developer sandbox.

ProcessOne will be presenting many of its new products as a preview technology implementation, among which:

WaveOne server and extension module for ejabberd XMPP server.
Various example client implementations using the Wave protocol from XMPP: OneTeam for Firefox client extension, and Tkabber support.
Jingle and Jingle Nodes XMPP calling with our OneTeam client.
The roadmap for the future of our XMPP and Wave collaborative platform.

We are also preparing various surprises for the people who can attend Google I/O (Did I say Android?).

This is going to be an extraordinary event for us. See you there!

[More...]

Haskell vs. Erlang for bittorent clients

2010-04-26T17:51:01+00:00

Haskell vs. Erlang

Since I wrote a bittorrent client in both Erlang and Haskell, etorrent and combinatorrent respectively, I decided to put up some bait. This might erupt in a language war and “My language is better than yours”, but I feel I am obligated to write something subjective. Here is to woes of programming in Haskell and Erlang.

Neither Haskell, nor Erlang was a first language for me. I have programmed serious programs in C, Standard ML, Ocaml, Python, Java and Perl; tasted the cake of Go, Javascript, Scheme and Ruby; and has written substantial stuff in Coq and Twelf. I love static type systems, a bias that will rear its ugly head and breathe fire.

I have written Haskell code seriously since 2005 and Erlang code seriously since 2007. I have programmed functionally since 1997 or so. My toilet reading currently is “Categories for the working mathematician” by Mac Lane. Ten years ago it was “ML for the working programmer” by Paulson.

Enough about me.

Caveats:

With any language war material follows a disclaimer and a healthy dose of caveats. This is subjective. You have to live with it being subjective. My writing can’t be objective and colorful at the same time. And I like colors in my life. Also, it is no fun reading a table listing the comparison. Rather, I will try to make it into a good foil duel with attacks, parries, guards, pierces, bananas, and barbed wire.

I built etorrent in Erlang first and combinatorrent in Haskell second. Hence, the 2nd time around, with the sole goal of redoing the functionality of etorrent was much easier and could proceed much faster. The Erlang code is slightly fattier at 4.2K lines versus 3.6K lines of Haskell (SLOCs). The performance of the two clients is roughly equal, but more time was spent at optimizing the Haskell code.

My hypothesis is this: The Erlang VM is much more optimized at the IO layer than the current IO layer I use in GHC (specifically, the way incoming data is handled allocates more memory. This might change in the future do to a new IO layer). GHC kills the Erlang VM for everything else though, perhaps including message passing.

Also, the quality of the Erlang code could be better, relatively compared to the Haskell code.

Enough!

Enough with the caveats!

Haskell cons:

What weighs against using Haskell for the project? First is laziness. Sometimes you want your code to be strict and sometimes lazy. In combinatorrent, we do some statistics which we don’t really need to calculate unless we want to present them. Stuff like bytes uploaded and downloaded for instance. Since you do not necessarily ask for these statistics, the compiler is free to build up thunks of the calculation and you have a neat little space leak. This is a recurring problem until you learn how to harness the strictness annotations of Haskell. Then the problem disappears.

IO in Haskell is somewhat weak if you naively assume a String is fast. But there is help from Bytestrings, attoparsec and low-level Socket networking. Combinatorrent could use more help with getting the speed up here. I have substituted the IO layers lowest level some 2–3 times in combinatorrent. Contrast this with Erlang, where the original protocol parser and IO is the one still standing. If you want fast network IO in Haskell, you should be using bytestrings and network-bytestring. It is not worth the simplicity going over String in my experience.

The GHC compiler has, comparatively, more performance regressions compared to the Erlang VM. It should come as no surprise: GHC is acting as both a research vehicle and a compiler implementation. I want to stress however, that this has not worried me a lot. When asking the GHC developers for help, the response has been fast and helpful, and in every case it was easy to fix or work around. Also, change is a necessary thing if you want to improve.

Haskell pros:

Haskell has one very cool thing: Static typing (remember the bias!). The type system of Haskell is the most advanced type system for a general purpose language in existence. The only systems which can beat it are theorem provers like Coq, and they are not general purpose programming languages (Morriset and the YNot team might disagree though!). Static typing has some really cool merits. Bugs are caught fast and early; types ensure few corner cases in the programs (why check for null when it can’t be represented). The types is my program skeleton and the program inhabiting the type is the flesh. Getting the skeleton right yields small and succinct programs. The abstraction possibilities from this is unparalleled in any language I have seen (and I’ve seen a few).

The GHC compiler provides programs which have excellent execution speed. You don’t need to worry a lot about speed when the compiler simply fixes most of the problems for you. This in turn means that you can write abstract code without worrying too much about the result. This yields vastly more general and simpler programs.

One very big difference in the implementations is that of STM channels versus Erlangs message passing. In Erlang, each process has a mailbox of unbounded size. You send messages to the mailbox, identified by the process ID of the mailbox owner. In Haskell, we use STM Channels for most communication. Thus, you send messages not to the PID of a process, but to a specific channel. This effectively changes some rules in channel network configuration. In Erlang you must either globally register a process or propagate PIDs. In Haskell, channels are created and then propagated to communicating parties. I find the Haskell approach considerably easier - but also note that in a statically typed language, channels is the way to go. The sum type for a PID mailbox would be cumbersome in comparison.

Haskell has excellent library and data structure support. For instance you have access to priority search queues via Hackage. PSQueues are useful for implementing the piece histogram in a bittorrent client: knowing how rare a given piece is so you can seek to fetch the rarest first.

Haskell can create (im-)mutable (un-)boxed arrays. These are useful in a bittorrent client in several places. Immutable arrays for storing knowledge about pieces is an example. Or bit-arrays for storing knowledge about the pieces a given peer has. Erlang has no easy access to these and no guarantee of the data representation.

Bryan O’Sullivans attoparsec library allows for incremental parsing. When you get a new chunk of data from the network, you feed it to attoparsec. It will either give you a parsed message and the remaining bytes, or it will hand you back a continuation. This continuation, if invoked with more food, will continue the parsing. For network sockets the incrementality is pure win.

The GHC compiler has some awesome profiling tools, including a powerful heap profiler. Using this, the run-time and memory usage of combinatorrent was brought down.

Finally, testing in Haskell is easy. QuickCheck and Test.Framework provides a —tests target built into the combinatorrent binary itself. Self tests are easy.

Haskell mistakes:

I made some mistakes when writing the Haskell client. For one I relied on the CML library until I realized STM would do an equal or better job. The amount of Haskell developers with STM experience compared to the CML head-count made the decision to change easy.

Furthermore, I should have focused on laziness earlier in the process. The first combinatorrent releases leak memory because of lazy thunk buildup. The latter versions, after I understood it intuitively, does not leak.

Erlang cons:

In Erlang, dynamic typing is the norm. Rather than enforce typing, you can get warnings by a type analyzer tool, the dialyzer, if need be. Running this on the code is a good idea to weed out some problems quickly. When building etorrent I had much use of the dialyzer and used a at that time experimental extension: spec() specifications. Yet, I think that 19/20 errors in my erlang programs were errors which a type system would have caught easily. This means you spend more time actually running the program and observing its behavior. Also note that dynamic typing hurts less in Erlang compared to other languages. A process is comprehensible in its own right and that reduces the interface to the process communication - a much simpler task.

Etorrent has less stability than combinatorrent and has erred more. Yet, this is no problem for a bittorrent client since the supervisor-tree in Erlang/OTP will automatically restart broken parts of the system. For a bittorrent client we can live with a death once a week or once a day without any troubles.

You have no mutability in Erlang and you have far less options for data representation. This in turn make certain algorithms rather hard to express or you have to opt for variant with a larger space usage. There were no Cabal-equivalent at the time I wrote the code and thus fewer libraries to choose from.

For the built-in libraries, the HTTP library was more strict with respect to correctness. In turn, many trackers would not communicate with it and I had to provide a wrapper around the library. Today, this might have changed though. Haskells HTTP library worked out of the box with no changes.

Erlangs syntax, compared to Haskell, is ugly, clunky and cumbersome. Make no mistake though: Tanks are ugly, clunky and cumbersome. It does not make tanks less menacing.

Erlang pros:

One application SASL. SASL is a system logger which will record in a ring-buffer any kind of process death and process restart. I used this a lot when developing. I would load a couple of torrents in the client and go to bed. Next morning I would check the SASL log for any error that might have occurred and fix those bugs. This way of developing is good for a bittorrent client: utmost stability is not needed. We just to get the number of errors below a certain threshold. Rather than waste time fixing a bug which only occurs once every year, we can concentrate on the things that matter.

The IO layer in Erlangs VM is FAST! It is written in C, and it is optimized heavily because this is what Erlang does best. For file IO it uses asynchronous threads to circumvent having to wait on the kernel. For the network, it plugs into epoll() getting good performance in turn.

The Beam VM of Erlang is a beast of stability. Basically, it doesn’t quit unless you nuke it from orbit. One of the smaller things I learned some weeks ago was the rudimentary flow control trick. Erlang schedules by counting reductions in an Erlang process and then switching process context when it has no more reductions in its time share. Sending a message never fails but it costs reductions proportional to the queue size of the receiving process. Hence, many senders have a harder time overloading a single receiver. The trick is simple, easily implementable and provides some simple flow control. While not fail-safe, it ups the ante for when communication overload happens.

Erlang has OTP, the Open Telecom Platform, which is a callback-framework for processes. You implement a set of callbacks and hand over control to the OTP-portion of your process. OTP then handles a lot of the ugly, gritty details leaving your part simple. OTP also provides the supervision of processes, restarting them if they err. Supervisor-processes form trees so they are in turn supervised. It isn’t turtles all the way down in an Erlang VM…

Erlang executes fast enough for most things. Haskell gives you faster execution, but Erlang was more than adequate for a bittorrent client in the speed department. As an example of how this plays together with the IO layer, an early version of etorrent could sustain 700 megabit network load on a local network of 1 gigabit when seeding. The current version of etorrent can do the same as a seeder I suspect. Also, message passing in Erlang is blazing fast. It feels like a function call - a key to good Erlang I think.

The Erlang shell can easily be used as a poor mans user interface. Etorrent simply responds to some functions in the shell, showing status of the running system. I suspect GHCi can do the same, but I never got around to doing it and it doesn’t seem as easy to pull off.

I love the Erlang way of programming. You assume your code does the right thing and let it crash otherwise. If it crashes too often you handle that case. Code is not lingered with error handling for things that never happen and should it happen occasionally, the supervisor tree saves the day.

Erlang mistakes:

Unfortunately, I made a number of mistakes in Etorrent. Most of these has to do with being the first version. Fred P. Brooks hinted that you want to throw away things when building the first version. And I did. I used ETS tables in places where they are not good. ETS is a table in which you can store any erlang term and later retrieve it. They give you a way to circumvent the representation limitation in Erlang. But they are no silver bullet: When you pull out a term, you copy it to the process pulling it. When your terms are 4–8 megabyte in size, that hurts a lot.

I relied far too much on mnesia, the database in Erlang. Mnesia is basically using software-transactional-memory so locking is optimistic. When you have something like 80 writers wanting access to the same row in a table, then the system starves. Also, there is no need for a bittorrent application to require a mnesia store. A simple serialization of key data to a file based on a timer is more than adequate.

I made several mistakes in the process model. I thought that choking was local to a torrent while in reality it is a global thing for all torrents currently being downloaded. These reorganizations require quite some refactoring - and missing in the static typing department these are somewhat more expensive compared to Haskell refactorings.

I thought autotools were a good idea. It is not. Autotools is the Maven of C programming.

Finally, I shedded unit-tests. In a dynamically typed environment you need lots and lots of these. But I decided against them early on. In hindsight this was probably a mistake. While unit-testing Erlang code is hard, it is by no means impossible.

Future:

The future brings exciting things with it. I will continue Combinatorrent development. I am almost finished with the Fast-extension (BEP 0006) for combinatorrent and have some more optimization branches ready as well. I still follow Erlang in general because it is an interesting language with a lot of cool uses. I do check that etorrent compiles on new releases of Erlang. If anyone shows interest in any of the client implementations, feel free to contact me. I will happily answer questions.

There is no clear winner in the duel. I prefer Haskell, but I am biased and believe in static typing. Yet I like programming in Erlang - both languages are good from different perspectives.

26 April 2010: Hans Nilsson joins Erlang Solutions as Senior Consultant

2010-04-26T10:30:42+00:00

Erlang Solutions Ltd proudly welcomes one of the world's most experienced and capable Erlang experts into our team.

Hans Nilsson joined the Ericsson Computer Science Lab in 1988, the year after Joe Armstrong. For the first two years, he worked on the development of the highly regarded SICSTUS Prolog. In 1995 together with Claes Wikström, he developed the Mnesia DBMS and Mnemosyne query language, and then went on to study the budding Voice over IP field. He implemented his first Erlang-based SIP stack in 1998, while the standard (RFC2543) was not yet in stable draft. Hans presented his SIP-ISDN Gateway at the Erlang User Conference 1997.

After a couple of years developing a residential Ethernet switch at Ericsson, Hans returned to Voice over IP. He established himself as one of Ericsson's leading SIP experts, and was the company's representative in the GSM Association SIP Technical Expert Group. Meanwhile, he was the technical lead on a large product development project based on his SIP implementation. Before joining Erlang Solutions, he worked in a tiger team doing advanced prototyping with Erlang in close cooperation with the Intel Performance Lab.

Hans is also a key member of the ProTest EU Project, where his work on property-baseddata mining of packet trace logs and QuickCheck verification of SIP parsers have been highly influential. As Erlang Solutions is an active member of that project, his involvement will continue.

With the addition of Hans Nilsson, Erlang Solutions is able to offer world-class expertise in the Voice over IP field.

19 April 2010: Erlang Solutions at TFP 2010 and AMOOCON2010

2010-04-19T13:01:45+00:00

We are proud to announce that the talk abstracts that Erlang Solutions' employees submitted have been accepted to TFP2010 and AMOOCON2010.

Michal Ptaszek, Consultant at Erlang Solutions Ltd. will be giving a talk on "ComputErl - Erlang-based Framework for Many Task Computing" at the Symposium on Trends in Functional Programming at the University of Oklahoma this May and Tino Breddin, Systems Engineer at Erlang Solutions Ltd, will be giving a talk on "Concurrent Programming with Erlang" at the AMOOCON2010 in June 2010.

If you are planing to attend any of these conferences, meet us there!

Why is this man so damn happy?

2010-04-17T19:34:46+00:00

CouchDB!

NetBeans(6.8) plugin for Scala 2.8.0 RC1

2010-04-15T05:02:06+00:00

I'm please to announce the NetBeans (6.8) plugin for Scala 2.8.0 RC1 is available now. It's a maintain release to catch up with Scala 2.8.0 RC1.

The packed plugins is downloadable at: http://sourceforge.net/projects/erlybird/files/nb-scala/6.8v1.1.0rc2/

It requires Scala 2.8.0 RC1, which is available at: http://www.scala-lang.org/downloads. For maven project, it's under http://www.scala-tools.org/repo-releases, with version number "2.8.0.RC1"

Please refer to http://wiki.netbeans.org/Scala68v1 for the installation/upgrading information.

SpringSource / VMWare Acquire Rabbit Technologies

2010-04-14T13:05:28+00:00

SpringSource, a division of VMware, Inc. today announced the acquisition by VMware of Rabbit Technologies, Ltd, a company set up by LShift and partners Monadic and CohesiveFT.

Read the full story

RabbitMQ acquired by SpringSource/VMWare

2010-04-13T13:45:55+00:00

Erlang-based RabbitMQ was acquired today by SpringSource...

12 April 2010: Erlang Solutions and Quviq Cement Partnership to Deliver Test Driven Development to the Market

2010-04-12T11:15:51+00:00

Erlang Solutions and Quviq AB, makers of the widely-adopted Quickcheck software tools, announce the formation of a partnership to deliver Quickcheck-based software tools and support services to the enterprise market. Erlang Solutions will be able to offer the full range of Quickcheck products and services to their enterprise clients. Quickcheck, is uniquely designed to test products and services that require high quality and reliability as well as an automated process.

The partnership formalises Erlang Solutions ability to provide the market with training, consultancy and support for the Quickcheck product portfolio that today covers Erlang and C based solutions.

Slides: Real time Web Application with XMPP and Wave

2010-04-07T16:30:33+00:00

This is the slides of the talk I have given at Erlang Factory 2010 event in San Francisco.

Here are the slides of my presentation on XMPP and our implementation of Wave protocol:

Real time Web Application with XMPP and Wave

View more presentations from Mickaël Rémond.

Next blog post will present a screencast of the federation demonstration between our OneWave server and Google Sandbox.

Dusty Old Projects: Erlang/Dojo Settlers of Catan

2010-04-07T16:36:57+00:00

A couple of years ago I got it in my head that I should write a Settlers of Catan clone with an Erlang server and a Dojo Toolkit client. I got fairly far with it, but the rubber never hit the road. Instead of letting the experience, the bruises and the documents generated from that go to waste I have made all available on Github at http://github.com/AlainODea/erlang_dojo_settlers.

I think this is a worthwhile project to look at to see the first intuitions of an experienced Java developer attempting to build a system in Erlang. Particularly the heavy use of processes as objects of a sort. It also makes some interesting use of JavaScript and The Dojo Toolkit as well.

Enjoy :)

Enabling SyntaxHighlighter in Blogger Step-by-Step

2010-04-07T07:06:06+00:00

Learn about SyntaxHighlighter
Go to Layout from you Blogger Dashboard
Select Edit HTML

Immediately after <head>, add the following code:

<script src='http://alexgorbatchev.com/pub/sh/2.1.364/scripts/shCore.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/2.1.364/scripts/shBrushErlang.js' type='text/javascript'></script>
<script src='http://alexgorbatchev.com/pub/sh/2.1.364/scripts/shBrushXml.js' type='text/javascript'></script>
<link href='http://alexgorbatchev.com/pub/sh/2.1.364/styles/styles/shCore.css' rel='stylesheet' type='text/css' />
<link href='http://alexgorbatchev.com/pub/sh/2.1.364/styles/shThemeDefault.css' rel='stylesheet' type='text/css' />

Select Save Template
Select Page Elements
Select Add a Gadget
Select HTML/JavaScript
In the Title field, enter SyntaxHighlighter

In the Content box enter the following code:

<script class='javascript'>//<![CDATA[
SyntaxHighlighter.config.clipboardSwf = 'http://alexgorbatchev.com/pub/sh/2.1.364/scripts/clipboard.swf';
SyntaxHighlighter.all();
//]]></script>

Select Save
Select Edit HTML
Make note of the widget id of the b:widget whose title is SyntaxHighlighter (for me it was HTML1)
Immediately before ]]></b:skin>, add the following code (with your widget id in place of HTML1):
```
#HTML1 {
display:none;
}
```
Select Save Template (this step hides the unnecessary visible gadget from your readers)
Enjoy syntax highlighting :)

Erlang.org traffic no longer exponential

2010-04-05T11:00:07+00:00

Bjarne Däcker, who has been tracking download and web site statistics on erlang.org for years, just produced a new graphic.

Visits to http://www.erlang.org up to March 2010

For years, the erlang.org graph has been nicely exponential. This latest peak seems to break the pattern. It will be interesting to see what follows.

(Update: Looking closer at the graph, it appears to stretch as far as to March 2010. I changed the caption, but left the URL unchanged.)

(Update 2: Closer inspection suggests that this is indeed measurement error. Google analytics, which also takes mirrors into account, reports a steady, but decidedly less dramatic, rise. The number of visitors to erlang.org is now at ca 80,000/month, up from ca 57,000/month in September 2008.)

New Strategic Guide released: from Real-time to Interpersonal Web

2010-04-01T13:22:03+00:00

ProcessOne has just released a new strategic guide explaining the underlying trends of the web evolution and how this will lead to the interpersonal web, the core business of ProcessOne.

Instant messaging has grown so rapidly in popularity and usage that it is shaping the next era in the evolution of the Internet. This Strategic Guide explains how demand for more real-time communication and personalized content is leading to the emergence of the Interpersonal Web.

The reference document is published on IMtrends: Strategic Guide: The Interpersonal Web.

Further articles with explain how to build this Interpersonal web thanks to ProcessOne technologies.

Your comments are welcome.

Is this stupid? Drop Last Element of Erlang List

2010-03-30T16:38:18+00:00

I wanted to drop the last element of a list. I didn't want a queue, I simply wanted to remove the last character of a string. Specifically I wanted to remove the \n from ref: refs/heads/master\n that comes back when reading the .git/HEAD file.

Code listing for trim.erl module:

-module(trim).

-export([string_strip_right/1, reverse_tl_reverse/1, bench/0]).

bench() -> [nbench(N) || N - [1,1000,1000000]].

nbench(N) -> {N, bench(["a" || _ - lists:seq(1,N)])}.

bench(String) ->
    {{string_strip_right,
    lists:sum([
        element(1, timer:tc(trim, string_strip_right, [String]))
        || _ - lists:seq(1,1000)])},
    {reverse_tl_reverse,
    lists:sum([
        element(1, timer:tc(trim, reverse_tl_reverse, [String]))
        || _ - lists:seq(1,1000)])}}.

string_strip_right(String) -> string:strip(String, right, $\n).

reverse_tl_reverse(String) ->
    lists:reverse(tl(lists:reverse(String))).

Benchmark transcript:

Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> trim:bench().
[{1,{{string_strip_right,11261},{reverse_tl_reverse,1000}}},
  {1000,
    {{string_strip_right,55131},{reverse_tl_reverse,17915}}},
  {1000000,
    {{string_strip_right,79881856},                                  
     {reverse_tl_reverse,119920353}}}]
2>

“Erlang the Movie” Poster

2010-03-29T16:47:44+00:00

They handed out this poster at Erlang Factory. Brilliant!

Domain Services and Bounded Context using Akka - Part 2

2010-03-29T13:43:32+00:00

In Part 1 of this series you saw how we can model a domain repository as an actor in Akka. It gives you declarative transaction semantics through Akka's STM and pluggable persistence engine support over a variety of data stores. As a result the domain model becomes cleaner. The repository that you design can take advantage of Akka's fault tolerance capabilities through supervisors that offer configurable lifecycle strategies.

One other important artifact of a domain model are the domain services. A domain service is not necessarily focused on any particular entity and is mostly centered around the verbs of the system. It models some actions or use cases involving multiple entities and is usually implemented as a stateless abstraction.

Using Akka you model a service as yet another actor. Domain services are coarse level abstractions and are the ones to receive requests from the clients. It can invoke other services or use any other entitites to do the job that it's supposed to do. No wonder a busy service gets requests from lots of consumers. Not only does it need to be stable, but it needs to ensure that all of it's other services with which it collaborates also stay alive while serving requests.

One of the services that it interacts with is the Domain Repository, which I discussed in the last post.

When you design a domain service using Akka actors, you can ensure that the service can make its collaborating services fault-tolerant through declarative or minimal programming effort. Akka runtime offers all the machinery to make implementation of fault tolerant services quite easy.

Consider the following domain service for management of Accounts, continuing our earlier example from the last post ..

trait AccountServer extends Actor {
  // handle crash
  faultHandler = Some(OneForOneStrategy(5, 5000))
  trapExit = List(classOf[Exception])
  
  // abstract val : the Repository Service
  val storage: AccountRepository

  // message handler
  def receive = {
    case Open(no, name) => 
      storage ! New(Account(no, name, Calendar.getInstance.getTime, None, 100))
    case msg @ Balance(_) => storage forward msg
    case msg @ Post(_, _) => storage forward msg
    case msg @ OpenM(as) => storage forward msg
  }
  
  // shutdown hook
  override def shutdown = { 
    unlink(storage)
    storage.stop
  }
}

The message handler is a standard one that forwards client requests to the repository. Note tha use of the abstract val storage: AccountRepository that helps you defer committing to the concrete implementation class till instantiation of the service object.

AccountServer plays the role of a supervisor for the repository actor. The first 2 lines of code defines the strategy of supervision. OneForOneStrategy says that only the component that has crashed will be restarted. You can make it AllForOne also when the supervising actor will restart all of the actors that it's supervising if one of them crashes. trapExit defines the list of exceptions in the linked actor for which the supervising actor will take an action.

AccountServer is the aupervising actor for the repository. When it is shutdown it has to be unlinked fro the linked actors. This we do in the shutdown hook of the AccountServer.

But how do we link the Repository actor to our domain service actor ? Note that AccountServer is not a concrete object yet. We need to instantiate a concrete implementation of AccountRepository and assign it to storage in AccountServer. And link the two during this instantiation.

We define another trait that resolves the abstract val that we defined in AccountServer and provides a concrete instance of AccountRepository. spawnLink not only starts an instance of Redis based repository implementation, it also links the repository actor with the AccountServer ..

trait RedisAccountRepositoryFactory { this: Actor =>
  val storage: AccountRepository = spawnLink(classOf[RedisAccountRepository]) 
}

Now we have all the components needed to instantiate a fault tolerant domain service object.

object AccountService extends 
  AccountServer with 
  RedisAccountRepositoryFactory {

  // start the service
  override def start: Actor = {
    super.start
    RemoteNode.start("localhost", 9999)
    RemoteNode.register("account:service", this)
    this
  }
}

Have a look at the start method that starts the service on a remote node. We start a remote node and then register the current service under the id "account:service". Any client that needs to use the service can get hold of the service actor by specifying this id .. as in the following snippet ..

class AccountClient(val client: String) { 
  import Actor.Sender.Self
  val service = RemoteClient.actorFor("account:service", "localhost", 9999)

  def open(no: String, name: String) = service ! Open(no, name)
  def balance(no: String): Option[Int] = 
    (service !! Balance(no)).getOrElse(
      throw new Exception("cannot get balance from server"))
  def post(no: String, amount: Int) = service ! Post(no, amount)
  def openMulti(as: List[(String, String)]) = service !!! OpenMulti(as)
}

Services defined using Akka can be made to run on remote nodes without much of an engineering hack. Akka runtime offers APIs for doing that. However, Akka never tries to hide from you the paradigms of distribution. You need to be aware of your distribution requirements and process and supervising hierarchies. Akka facilitates you to define them at the application level, doing the heavy lifting within its underlying implementation. This principle is inspired from Erlang's philosophy and is in sharp contrast to the RPC way of defining APIs. Along with the benefits of message based computation that decouples the sender and the receiver, Akka also enables you to handle states using its built-in STM and pluggable storage engine.

When you model a complex domain, you need to deal with multiple contexts, which Eric calls Bounded Contexts. Within a specific context you have a cohesive model with a set of domain behavior and abstractions. The interpretations of the same abstractions may change when you move to a different context within the same application. As a domain modeler you need to define your context boundaries very carefully using context maps and implement appropriate translation maps between multiple contexts.

Messaging is a great way to implement translation between contexts. You process messages within a context using domain services as above and at the end forward the same message or a translated one to the other contexts of the application. It can be in the form of push or you can also implement a publish/subscribe model between the related contexts. In the latter case, you use Akka-Camel integration that allows actors to send or receive messages through Camel end-points. In either case Akka provides you a world of options to implement loosely coupled domain contexts that form the components of your domain model.

On the limits of concurrency: Worker Pools in Erlang

2010-03-29T09:35:18+00:00

A worker pool is a very common pattern, and they exist in the standard libraries for many languages. The idea is simple: submit some sort of closure to a service which commits to running the closure in the future in some thread. Normally the work is shared out among many different threads and in the absence of anything fancier, one assumes a first-come-first-served queue of closures.

Erlang, with its light-weight process model is not a language which you would expect would require such an approach: processes are dirt cheap, and the scheduler maps processes onto threads when they are ready to be run — in many ways, the ErlangVM is a glorified implementation of a worker pool, only one that does pre-emption and other fancy features, in a very similar way to an OS kernel. However, we recently found in RabbitMQ a need for a worker pool.

At various points in RabbitMQ, we use mnesia transactions to ensure that state that is held by mnesia (for example, the existence or otherwise of a queue) is updated safely. A client recently informed us that he was creating 10,000 connections, each connection was creating a queue, with exclusive set (which means the queue must be deleted automatically when the connection disappears), and then dropping all the connections at the same time. RabbitMQ would become very unresponsive, for a very very long time. A small amount of head scratching and some quick testing led to the conclusion that this was creating 10,000 transactions, all on the same table, all of which were continuously colliding with each other and thus having to be restarted. Here, the realisation is that there is frequently no good reason to allow more mnesia transactions to go on at the same time than you have cores available — the probability of a collision will rise exponentially with the number of concurrently in flight transactions.

The solution here is to submit all of these transactions to a worker pool which has the same number of workers as there are CPU cores in the system. As a result, the probability of a collision is greatly reduced, and things progress much much faster. In many ways, this is a good example of where hinting to the underlying VM as to how it should schedule different jobs would result in massively improved performance.

One interesting gotcha with this solution is the issue of nested transactions. Mnesia handles these very well indeed, but if we wrap every transaction in a submission to the worker pool, we stop mnesia from knowing that these transactions are nested (which means we lose the unrolling of the inner ones if the outer one subsequently fails), and furthermore, we risk deadlock if every worker in the pool is waiting on another nested transaction to complete. To solve this problem, on submission of a closure, we detect whether or not we are already in a worker pool process, and if we are, we just run the closure in the current process without submission at all. This means that nested transactions are correctly run in the same process as their parent, and we don’t reach deadlock.

The code is already QA’d and merged into our default branch and is in the three modules worker_pool_sup (which is the module you just need to start up, it’ll then start everything else), worker_pool_worker (the actual workers who do the, erm, work), and worker_pool (which is the module to which you submit your jobs). Note that the submission is synchronous, thus you will be returned the result of your closures. Furthermore, your closures are responsible for catching any errors that might occur and returning them to you as values. Finally, note that you’ll also need our gen_server2 module as the worker_pool and worker_pool_worker modules both use this.

Erlang Factory: You Should’ve Been There

2010-03-29T03:44:23+00:00

The Erlang Factory SF Bay Area 2010 conference just wrapped up last Friday, and it was fantastic. The talks were of high quality and the conference organization was excellent. The slides for all the talks should be on the Erlang Factory site soon.

If you missed this one, I strongly encourage you to attend the London Erlang Factory coming up in June if you can.

Concise HOWTO's: Message from Erlang to Java

2010-03-26T17:46:02+00:00

First create an EchoServer in Java that receives messages in the format {Sender,Data} and echoes the Data portion back to the Sender. You will need OtpErlang.jar in your build classpath and runtime classpath. I found it in /opt/local/lib/erlang/lib/jinterface-1.5.1/priv/ on Mac OS X. I had to install Erlang from source on Ubuntu since the APT package is missing Jinterface.

Listing for EchoServer.java:

 
import com.ericsson.otp.erlang.*;

public class EchoServer {
    public static void main(String[] args) throws Exception {
        OtpNode node = new OtpNode("java");
        OtpMbox mbox = node.createMbox("echo");
        OtpErlangAtom SHUTDOWN = new OtpErlangAtom("shutdown");
        while (true) {
            OtpErlangObject message = mbox.receive();
            System.out.format("%s received: %s%n", mbox.self(), message);
            if (SHUTDOWN.equals(message)) {
                System.out.format("%s shutting down...%n", mbox.self());
                break;
            } else if (message instanceof OtpErlangTuple) {
                OtpErlangTuple messageTuple = (OtpErlangTuple) message;
                if (messageTuple.arity() == 2 && messageTuple.elementAt(0) instanceof OtpErlangPid) {
                    OtpErlangPid sender = (OtpErlangPid) messageTuple.elementAt(0);
                    OtpErlangObject sendersMessage = messageTuple.elementAt(1);
                    mbox.send(sender, sendersMessage);
                }
            }
        }
    }
}

Once you have compiled and started the EchoServer you can communicate with it from the Erlang shell. First you need to start a new named node with erl -sname erl_node.

Listing of the Erlang shell interaction:

Erlang R13B01 (erts-5.7.2) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.2  (abort with ^G)
(erl_node@alains_desktop)1> net_adm:ping(java@alains_desktop).                   
pong
(erl_node@alains_desktop)2> {echo,java@alains_desktop} ! {self(),"Hello, Java!"}.
{0.39.0>,"Hello, Java!"}
(erl_node@alains_desktop)3> % use f(M) to "forget" M (make it unbound)
(erl_node@alains_desktop)3> f(M), receive M -> {ok,M} after 100 -> {error,timeout} end.
{ok,"Hello, Java!"}
(erl_node@alains_desktop)4> {echo,java@alains_desktop} ! {self(),{tuples,can,be,sent,too}}.
{0.39.0>,{tuples,can,be,sent,too}}
(erl_node@alains_desktop)5> f(M), receive M -> {ok,M} after 100 -> {error,timeout} end.
{ok,{tuples,can,be,sent,too}}
(erl_node@alains_desktop)6> {echo,java@alains_desktop} ! shutdown.
(erl_node@alains_desktop)6>
(erl_node@alains_desktop)7> {echo,java@alains_desktop} ! shutdown.                         
shutdown
(erl_node@alains_desktop)8> {echo,java@alains_desktop} ! {self(),"Are you there?"}.        
{0.39.0>,"Are you there?"}
(erl_node@alains_desktop)9> f(),receive M -> {ok,M} after 100 -> {error,timeout} end.
{error,timeout}
(erl_node@alains_desktop)10>

Listing of EchoServer Java process output:

#Pid received: {#Pid,"Hello, Java!"}
#Pid received: {#Pid,{tuples,can,be,sent,too}}
#Pid received: shutdown
#Pid shutting down...

Concise HOWTOS: Install Erlang/OTP R13B01 from Source on Ubuntu 9.04

2010-03-26T17:42:44+00:00

Generally it is better to use packages, so why have this HOWTO? Ubuntu 9.04's Erlang package is missing Jinterface, but it works when you build Erlang from source. Debian or Ubuntu will address this omission at some point. Until then we can build and install the latest Erlang release fairly easily.

The only real frustration in building from source is making sure all of the necessary build dependencies are installed first. Well, it also takes a significant chunk of time to build...

sudo apt-get install build-essential libncurses-dev \
                     unixodbc-dev libssl-dev \
                     libwxgtk2.8-dev default-jdk
curl http://erlang.org/download/otp_src_R13B01.tar.gz
tar xzvf otp_src_R13B01.tar.gz
cd otp_src_R13B01/
./configure
make
sudo make install

The make step takes a long time. It made me really appreciate the pre-built packages I can get so easily and quickly with APT.

Concise HOWTO's: Message from Java to Erlang

2010-03-26T17:41:51+00:00

First create an Erlang module called server that receives messages in the format {Sender,Data} and echoes the Data portion back to the Sender. Then create a Java class called EchoClient that prompts for a node to connect to and prompts in a loop for messages to send. You will need OtpErlang.jar in your build classpath and runtime classpath. I found it in /opt/local/lib/erlang/lib/jinterface-1.5.1/priv/ on Mac OS X. I had to install Erlang from source on Ubuntu since the APT package is missing Jinterface.

Listing of `server.erl`:

-module(server).
-compile(export_all).

start() -> register(server,spawn(fun loop/0)).
loop() ->
    receive
        {Sender,Data} ->
            Sender ! Data,
            loop();
        shutdown -> ok
    end.

Listing of `EchoClient.java`

import java.io.BufferedReader;
import java.io.InputStreamReader;

import com.ericsson.otp.erlang.*;

public class EchoClient {
    public static void main(String[] args) throws Exception {
        OtpNode node = new OtpNode("java");
        OtpMbox mbox = node.createMbox("admin_gui");
        BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
        String defaultServerNodeName = "erl_node@" + node.host();
        System.out.format("Server Node to contact [%s]> ", defaultServerNodeName);
        String serverNodeName = in.readLine();
        if (serverNodeName == null || "".equals(serverNodeName)) {
            serverNodeName = defaultServerNodeName;
        }
        OtpErlangTuple serverPidTuple = new OtpErlangTuple(new OtpErlangObject[] {
                new OtpErlangAtom("server"), new OtpErlangAtom(serverNodeName)});
        while (true) {
            if (!node.ping(serverNodeName, 1000)) {
                System.out.println("Erlang node is not available: " + serverNodeName);
                System.exit(1);
            }
            System.out.print("Message (Hit Enter to send)> ");
            String message = in.readLine();
            if (message != null) {
                mbox.send("server", serverNodeName, new OtpErlangTuple(
                        new OtpErlangObject[] {
                                mbox.self(), new OtpErlangList(message)}));
                OtpErlangObject serverReply = mbox.receive(1000);
                if (serverReply == null) {
                    System.out.println("WARN: Timeout when receiving reply");
                } else {
                    System.out.format("%s replied : %s%n", serverPidTuple, serverReply);
                }
            }
        }
    }
}

Open an Erlang shell in the directory with server.erl by running erl -sname erl_node. From that shell compile and run the Erlang server module.

Listing of the Erlang shell interaction

$> erl -sname erl_node
Erlang R13B01 (erts-5.7.2) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.2  (abort with ^G)
(erl_node@alains_desktop)1> server:start().
true
(erl_node@alains_desktop)2> server ! {self(), hello}.
{0.84.0>,hello}
(erl_node@alains_desktop)3> f(M), receive M -> {ok, M} after 100 -> {error,timeout} end.
{ok,hello}
(erl_node@alains_desktop)4>

Run the EchoClient Java program and either accept the default node or specify the name of another Erlang node you want to work with. Type messages in and see them echoed back to the EchoClient console.

Listing of the `EchoClient` Java program console interaction:

Server Node to contact [erl_node@alains_desktop]> [ENTER]
Message (Hit Enter to send)> Hello, Erlang![ENTER]
{server,erl_node@alains_desktop} replied : "Hello, Erlang!"
Message (Hit Enter to send)>

OneTeam Media Server by ProcessOne

2010-03-26T10:48:36+00:00

ProcessOne has just released OneTeam Media Server, a Flash server written in erlang.

OneTeam Media Server, or OMS for short, is a new Flash server implementation, written in erlang, by ProcessOne. This will enable your users to broadcast voice and/or video streams to multiple subscribers.

OneTeam Media Server, although being at an early stage, behaves already quite well, since it is more scalable than red5. It might not be a fair comparison though, since it OMS is not as featurefull as red5.

Key features:

Support for streaming live/recorded content
Support for recording live content
Support for ActionScript3.0 Shared Objects
Clustering support
Ability to write application modules in Erlang/OTP

It still does not include RTMPT for HTTP tunneling. The license chosen for OneTeam Media Server is the Erlang Public License version 1.1. OMS needs Erlang/OTP R11B-0 or higher.

The source code is available here: https://git.process-one.net/oms

To pull the source code form git, please type:

git clone git://git.process-one.net/oms/mainline.git

The usual procedure is required (./configure, make, sudo make install). To start OMS, just type:

omsctl start

Please check the documentation in:

doc/doc.html

Replication

2010-03-23T21:57:07+00:00

Some cool Apache CouchDB replication related links:

The fine art of holding a file descriptor

2010-03-23T17:26:56+00:00

People tend to like certain software packages to be scalable. This can have a number of different meanings but mostly it means that as you throw more work at the program, it may require some more resources, in terms of memory or CPU, but it nevertheless just keeps on working. Strangely enough, it’s fairly difficult to achieve this with finite resources. With things like memory, the classical hierarchy applies: as you use up more and more faster memory, you start to spill to slower memory — i.e. spilling to disk. The assumption tends to be that one always has enough disk space.

Other resources are even more limited, and are harder to manage. One of these is file descriptors. This is especially difficult to manage in a VM such as Erlang’s where lots of other systems are potentially using file descriptors and you have no control over them. In the released versions of Rabbit, Rabbit’s persister used only one or two file descriptors, and the queues themselves used none. This would be an obvious scalability issue in that it would restrict to a single process access to messages stored on disk, were it not for the fact that released versions of Rabbit hold all messages in memory all the time, thus negating that problem (at the expense of a larger scalability issue). However the upside is that as more queues appear, you don’t need more file descriptors, so almost all file descriptors could be set aside for network sockets. If you need to allow more network connections to Rabbit than your OS provides, then just raise the ulimit (and often the default Erlang process limit too) and away you go.

With the new persister, each queue requires at least two file descriptors each, and can use a number of file descriptors bounded only by the number of messages stored on disk. Whilst this has removed all sorts of bottlenecks, it has also made the probability of running out of file descriptors rather higher. Introducing a central file descriptor allocator would reintroduce a bottleneck that we have sought to avoid, and so we have developed an alternative scheme for managing file descriptors. This scheme can go wrong: it is really only probabilistic that it works. But, given the way in which Rabbit works, it seems to be working very successfully.

Firstly, there is a central process. However, this is never asked for a file descriptor, instead all processes who open files asynchronously tell the central process that they’ve opened a file. They also tell the central process when they close a file descriptor. With both of these messages, processes include another piece of information which is the time at which the least recently used file descriptor that the process has open was used. Thus the processes maintain a mapping from file descriptor to the timestamp at which they were last used. The smallest value in this mapping is the value which is included with these messages to the central process. Whenever a process uses a file descriptor, it takes a new timestamp and updates this mapping. No communication is made to the central process on use of a file descriptor.

The central process detects the ulimit imposed by the OS on the number of file descriptors that can be opened. It imposes an artificial limit, 100 less than the real limit. This gives both buffer space, and allows the rest of the Erlang VM some file descriptors beyond our own control. When we reach this artificial lower limit, the central process does the following calculation: for every process that has some open files, it finds the difference between the current timestamp and the most recently reported least recently used file descriptor timestamp. It sums and averages these ages to give the average time since the least recently used file descriptors were used. It then asynchronously sends messages to all the processes with open files, telling them to close any file descriptor that has not been used for more than this average time.

The first time the processes receive this message, they may very well find that they don’t have any file descriptors that have not been used for this long. This is because the central process is only informed or processes’ least recently used file descriptor timestamps when the process opens or closes a file descriptor. As such, if the process then uses the file descriptor then the central process will immediately have out of date information. Thus all processes, when they receive a request to close files descriptors older than the calculated average, they always inform the central process of the timestamp of their current least recently used file descriptor. Thus at this point, the central process is brought up to date, and if it finds that it’s still at or over the limit of open file descriptors, it recalculates the average age (which will now be less than before) and asks all the processes again to close file descriptors older than the new, smaller average age.

What this means is that processes are never blocked from opening files, even when Rabbit’s over the limit of file descriptors. However, immediately after opening a file, when it goes to receive its next message, a process may find a request from the central process, asking it to close the file it’s just opened. Thus the limit is enforced softly, in a way which does as little as possible to impact performance. This is the reason why we have the lower artificial limit: to try and guard against the possibility of lots of processes opening files at the same time, pushing us over the limit before they or any other process receive the close request from the central process. However it can still go wrong: if a process is hell bent on opening as many files as possible then it can do so, and still hit the hard OS limit, crashing the VM. Cooperation from the processes is obviously vital: for example if a process can never open more than one file descriptor before checking its mailbox again, then you’re very likely to be safe in this scheme.

All of this is implemented in a module called file_handle_cache.erl which is available in the new persister branch of RabbitMQ. This module also wraps many of the functions of Erlang’s file module, providing many more optimisations (at the cost of, for example, only ever being able to append to files). These optimisations aim to reduce to an absolute minimum the number of OS calls. So, much better control of write buffers is provided, and seeks which would position the file handle at the same location as it currently is are optimised out. Further calls are provided, e.g. to throw away the write buffer contents without writing them out to disk.

When the request to close a file comes in, the file_handle_cache module works out which file descriptors to close. If it finds a file handle to close, it flushes any outstanding writes, closes the file, but keeps track of the last state that it was in. Thus the next time the process decides to use that file descriptor, the module can silently reopen the file and seek to the last location. As a result, when writing to this module, you never need to find out whether or not the central process has asked the module to close files and what, if anything, the result of that request was. The result is a system which dynamically closes old and unused file descriptors but without imposing arduous constraints on the client: the module manages all the state of the file descriptors.

Finally, there are some file descriptors which we have decided, after careful consideration, not to arbitrarily close. These are network sockets to AMQP clients. For these, it is indeed right to have a central process controlling whether further sockets can be created. This is simply implemented as a pair of synchronous calls (acquire and release) to the central process which lowers and raises, respectively, the artificial limit on the number of allowed file descriptors.

StarCraft2 AI Coding

2010-03-22T11:28:40+00:00

So I was curious as to how come the StarCrack AI folks are finding it so easy to develop new and improved AI for StarCraft 2. It turns out that there's an SC2Data file that's an mpq archive. So I downloaded their AI distribution and extracted their code. The first thing I noticed is that there's more than AI in here, there's actual game balancing information. E.g.:

<CWeaponLegacy id="Marauder">
  <EditorCategories value="Race:Terran"/>
  <DisplayEffect value="MarauderU"/>
  <DisplayAttackCount value="1"/>
  <TargetFilters value="Ground,Visible;Missile,Stasis,Dead,Hidden,Invulnerable"/>
  <Range value="6"/>
  <Period value="1.5"/>
  <DamagePoint value="0"/>
  <Backswing value="0"/>
  <Effect value="MarauderLM"/>
  <Icon value="Assets\Textures\btn-upgrade-terran-infantryweaponslevel1.dds"/>
</CWeaponLegacy>

That makes me wonder if it's possible to edit these files in order to artificially make an opposing race harder or easier to play against.

Looking at the contents of TriggerLibs, this all looks very much like C code. Sweet. My next thought is that this can probably be translated into one big Groovy script without too much hassle, and then these StarCrack folks could write test harnesses! It must be a real pain to have to crank up StarCraft itself to see if you coded the right thing. I don't see all the TriggerDebugOutput logging as being a good substitute.

Edit: I gave this a quick stab. The first real problem with this idea is that there are multiple declarations of the method InitCounters.

Unofficial Erlang Planet

Getting Your Open Source Project to 1.0

CouchCamp is Coming Soon!

Erlang and REST – an interview with Steve Vinoski

Zotonic rethinks the CMS with Erlang

Adding Health Checks to Deckard from Chef.

New Column, New Interview

PostgreSQL Erlang client library epgsql supports asynchronous messages from LISTEN/NOTIFY events

ProcessOne releases OneCached, a Memcached in erlang

New Erlang job board – totally-erlang.com

erldocs.com updated with R14A support, mochiweb, and available for your own projects

A Phase Shift for the ORM

Spanish Language Erlang List

Case study: Arc International

Migrating From MySQL to CouchDB

CouchCamp, aka BurningCouch!

How Software Is Built: Apache CouchDB

Miscellaneous CouchDB News

Just Opensourced: Gaff and Deckard

ejabberd 2.1.4 and exmpp 0.9.4 bugfix releases

Regarding ejabberd 2.1.4:

Brief summary of changes:

Check the Release Notes for a more complete list of changes:

Download

Regarding exmpp 0.9.4:

Brief summary of changes:

exmpp home page:

Download

CouchDB on your TV

Grokking Functional Data Structures

High Performance CouchDB

Adding support for more than 1024 files to etorrent.

WaveOne, a wave server and client by ProcessOne

Migrate Notes data to CouchDB: Smart Notes 2 Couch

I'm in your phone

CouchDB Rap

17 May 2010: Erlang Solutions awarded a Knowledge Transfer Partnership with the Kent Business School

10 May 2010: Erlang Solutions Announces Opening of New Office in Stockholm, Sweden

CouchDB Case Studies

Google Denies Our Commercial

Announcing New Erlang Native GeoCouch

Availability, the Cloud and Everything

Create a Custom CouchDB indexer

ProcessOne at Google I/O Developer Sandbox 2010

Haskell vs. Erlang for bittorent clients

Haskell vs. Erlang

Caveats:

Haskell cons:

Haskell pros:

Haskell mistakes:

Erlang cons:

Erlang pros:

Erlang mistakes:

Future:

26 April 2010: Hans Nilsson joins Erlang Solutions as Senior Consultant

19 April 2010: Erlang Solutions at TFP 2010 and AMOOCON2010

Why is this man so damn happy?

NetBeans(6.8) plugin for Scala 2.8.0 RC1

SpringSource / VMWare Acquire Rabbit Technologies

RabbitMQ acquired by SpringSource/VMWare

12 April 2010: Erlang Solutions and Quviq Cement Partnership to Deliver Test Driven Development to the Market

Slides: Real time Web Application with XMPP and Wave

Dusty Old Projects: Erlang/Dojo Settlers of Catan

Enabling SyntaxHighlighter in Blogger Step-by-Step

Erlang.org traffic no longer exponential

New Strategic Guide released: from Real-time to Interpersonal Web

Is this stupid? Drop Last Element of Erlang List

“Erlang the Movie” Poster

Domain Services and Bounded Context using Akka - Part 2

On the limits of concurrency: Worker Pools in Erlang

Erlang Factory: You Should’ve Been There

Concise HOWTO's: Message from Erlang to Java

Listing for EchoServer.java:

Listing of the Erlang shell interaction:

Listing of EchoServer Java process output:

Concise HOWTOS: Install Erlang/OTP R13B01 from Source on Ubuntu 9.04

Concise HOWTO's: Message from Java to Erlang

Listing of server.erl:

Listing of EchoClient.java

Listing of the Erlang shell interaction

Listing of `server.erl`:

Listing of `EchoClient.java`

Listing of the `EchoClient` Java program console interaction: