Archive for the ‘Lunchbox’ Category

Equalizer 1.4 released

7. September 2012

The last two weeks have been quiet, since I was on biking through Switzerland. Meanwhile, poor Daniel back at work churned through most of the Collage changes outlined in the last post. You can see the changes in the Collage endian branch on github, which will be merged back into master in the next couple of weeks.

Now back to the news: After finally figuring out how to build Equalizer and dependencies using MacPorts portfiles on Mac OS X, I released the long-standing 1.4 version of Equalizer, GPU-SD, Lunchbox and vmmlib. Below is the release announcement – enjoy!

Neuchatel, Switzerland – September 7, 2012 – Eyescale is pleased to announce the release of Equalizer 1.4.

Equalizer is the standard framework to create and deploy parallel, scalable 3D applications. This modular release includes Collage 0.6, a cross-platform C++ library for building heterogenous, distributed applications, GPU-SD 1.4, a C++ library and daemon for the discovery and announcement of graphics processing units using zeroconf networking and Lunchbox 1.4, a C++ library for multi-threaded programming. All software packages are available for free for commercial and non-commercial use under the LGPL open source license.

Equalizer 1.4 is a feature release extending the 1.0 API, introducing major new features, most notably asynchronous readbacks, region of interest and thread affinity for increased performance during scalable rendering. It culminates over seven years of development and decades of experience into a feature-rich, high-performance and mature parallel rendering framework and related high-performance C++ libraries.

Equalizer enables software developers to easily build interactive and scalable visualization applications, which optimally combine multiple graphics cards, processors and computers to scale the rendering performance, visual quality and display size.

Equalizer Applications

Eyescale provides software consulting and development services for parallel 3D visualization software and GPU computing applications, based on the Eyescale software products or other open and closed source solutions.

Please check the release notes on the Equalizer website for a comprehensive list of new features, enhancements, optimizations and bug fixes. A paperback book of the Programming and User Guide is available.

We would like to thank all individuals and parties who have contributed to the development of Equalizer 1.4.

Left image courtesy of Cajal Blue Brain/ / Blue Brain Project. Second from left copyright Realtime Technology AG, 2008. Right image courtesy University of Siegen, 2008.


Programming and User Guide for Equalizer 1.4

29. June 2012

Equalizer Programming and User Guide

Equalizer Programming and User Guide

I’ve just uploaded the review version of the Programming and User Guide for the upcoming 1.4 release of Equalizer, and to a certain extent, Collage.

This one packs 111 pages of content (118 total) and 63 figures. In a couple of weeks I’ll create the final hardcopy version on Amazon/CreateSpace.

What’s New?

This edition has, among the customary full review pass, a lot of new content. Starting with a full new chapter on Sequel and the associated polygonal rendering example, continuing with new section for application-specific scaling factors in immersive environments, region of interest for compositing and load-balancing and zeroconf discovery in Collage, finally finishing with a substantial rewrite of the section on distributed objects in Collage.


The book is structured in two parts: The User Guide, laying the foundation on parallel rendering and scalability algorithms, and then explaining the configuration of visualization systems for Equalizer applications. The appendix contains a full reference on the file format.

The second part, the Programming Guide, gradually introduces programming parallel rendering applications. Starting with the basics in eqHello, the complexity is gradually increased with a chapter on Sequel and Equalizer using the respective example application. After this, an advanced features section focuses on introducing and demonstrating on specific features in isolation. It finishes of with a chapter on the Collage network library.


The Programming and User Guide is the ‘OpenGL red book’ of Equalizer. It consolidates all the documentation available in various places (Equalizer website, mailing list, github feature issues, my head) into a single document. Apart from gathering this information, through the format emerges a bigger picture, putting design decisions in context.

Introducing lunchbox::Servus

22. June 2012

Yes, we’re back to Lunchbox. During the 1.4 beta release (see previous post) we decided to merge a separate library in Lunchbox, since keeping it separate would have been overkill. In the end, it’s just a single class: Servus.

For the 999‰ of you who can’t get the reference: It’s a play on words to Bonjour. This class implements a simple C++ interface to announce and discover key-value pairs over zeroconf networking (aka. Bonjour).

The usage is really simple: You instantiate the class with the service name you’re interested in, e.g., “_collage._tcp”. Then you can register key-value pairs and start announcing them, and you can discover existing key-value pairs announced by other processes, typically within the subnet. It’s also legal to update key-value pairs on an announced service. Sounds simple enough, but if you’ve ever used the callback-driven dns_sd C API, you’ll know it’s much easier to use.

We use this class for two things right now: Collage node discovery and in GPU-SD.

In Collage, each co::LocalNode announces his node identifier and connection descriptions. This information is used as a fallback path when performing a LocalNode::connect( NodeID ) to identify and to connect to a previously unknown node. The Servus handle is exposed to applications through co::Zeroconf, which is a wrapper to ensure thread-safety. Applications can then use additional key-value pairs specific to their implementation.

In GPU-SD, Servus is used to announce and discover graphics cards in a visualization cluster, which in turn is used by Equalizer for auto-configuration.

1.4 beta release of the Eyescale open source packages

20. June 2012

We are pleased to announce the 1.4 beta release of the Eyescale open source packages. This release is a preview for testing the upcoming 1.4 stable release. It is the first modular release, and contains the following libraries and new features:

  • Equalizer: parallel rendering framework
    • Various scalable rendering performance features: asynchronous readbacks, region of interest and thread affinity.
  • Collage: C++ library for building heterogenous, distributed applications
    • Zeroconf support and node discovery
    • Blocking object commits
    • Increased InfiniBand RDMA performance
  • GPU-SD: discovery and announcement of GPUs using zeroconf
    • VirtualGL detection
    • Hostname command line parameter for gpu_sd daemon
  • Lunchbox: C++ library for multi-threaded programming
    • Servus, C++ interface to announce, discover and iterate over key-value pairs stored in a zeroconf service description
    • LFVector, a thread-safe, lock-free vector
  • Buildyard: A CMake-based superbuilder to download, configure and build the packages and dependencies for this release
    • Generates Unix Makefiles and solution files for Visual Studio 2008/10
    • Simple CMake project configuration scripts
    • Support for local overrides and user forks
    • Extensible with custom in-house or open source projects
  • A website for API documentation of all
    the aforementioned packages

Please test this release extensively and report any bugs on the respective project page at The release notes are part of the API documentation at

We would like to thank all contributors who made this release possible.

Buildyard + doxygen + github =

15. June 2012

Sometimes pieces just fall into place and you know you’re on the right track. While preparing the 1.4 beta release of our Eyescale software stack (Equalizer and friends), I needed to put the API documentation of all these projects on a web server. In the past, with Equalizer as a single project, I just had doxygen dump it somewhere and then copied the stuff at release time to equalizergraphics. With five or more projects, that’s not really an option anymore.

Meet Our new home for all doxygen-generated API documentation. For us, the process is almost fully automated:

  • github provides a git repository for all the web pages, and automatically serves them at
  • Buildyard has a configuration which clones this git repository before all other projects.
  • A per-project CMake rule installs the project, runs doxygen on the installed headers, and copies the result to the git repository into project-version
  • CMake magic in the git repository rebuilds the index page and adds all new files to the repository
  • A manual git commit; git push uploads the new pages to the git repository, and github automatically updates the website.

This took about one day to setup, and now it takes almost no time to update API documentation. It has the added benefit that you can easily get all the reference documentation by simply cloning the git repository.

Lunchbox Folly

8. June 2012

Facebook recently released folly, which caught my eye due to its similarity to Lunchbox. It’s definitely a library to watch.

I don’t think we’ll be using it right now, since it’s missing some of the stuff we’re using, e.g, the LFVector, and since the implementation so far seems to be mainly tested on Linux + gcc 4.6. On the other hand, it has some interesting components which we might need in the future, e.g, atomic hash containers. It also contains optimized version of some standard components such as vector and string, which haven’t really shown up as hotspot during profiling our code.

Interesting enough, folly also has some components which are almost identical to the Lunchbox counterparts, such as r/w spin locks and lock-free queues. It’s always good to see when ideas converge to a common design.

Buildyard – C++ source project management

1. June 2012

The Problem

The Equalizer source code always was modularized, but did build everything into a single shared library for convenience. Lately new projects, e.g., dash/codash, required functionality from what was called eq::base. This lead to Lunchbox, which you should know by now.

Managing the development of multiple, source dependent projects is a pain. You’ve got to download them, configure, build and install them in the right order. There are solutions for this in the Java world, but the ones I found for C/C++ were either in limbo (ryppl) or are more for source-based distribution, but not that easy for developers (0install).

Our Solution

After some tinkering we came up with Buildyard, a CMake-based meta build system. It builds on top of the ExternalProject CMake module, which is good but requires quite some customization when used. We’ve simplified this further, and adding a new CMake-based project is a couple of lines in a configuration file, e.g.:

set(EQUALIZER_VERSION 1.3.1)                                                     
set(EQUALIZER_DEPENDS gpu-sd Boost hwloc vmmlib Lunchbox GLStats)                
set(EQUALIZER_ROOT_VAR EQ_ROOT)                                                  
set(EQUALIZER_REPO_TAG master)                                                   

Creating this simple configuration files allows you to build a project such as Equalizer, and all it’s dependencies, much easier. You simply clone Buildyard and use make Equalizer in the cloned directory. This will first configure Buildyard to resolve the dependencies, download the ones not installed yet, and then configure/build/install them in the correct order into a local installation directory. This even works with parallel builds.

The development process is largely unchanged as well. Simple go to src/project, code and compile away. Buildyard will inject a Makefile into each project source which sets up the proper targets for building. When compiling in a source directory, other project dependencies are not considered by default to get speedy compiles.

The whole thing is easily extendable. Internally we’ve got a config.bbp repository for the closed-source project configurations, which build on to of the open source ones pre-configured with Buildyard. Developers clone that into the Buildyard directory, where it will be picked up on the next CMake run.

This post just scratches the surface, have a look at the Readme for more details and post questions below. While it’s not as sophisticated as ryppl, it works and we use it daily already in quite a number of projects. It also facilitates creating new projects a lot, such as dash, codash, Lunchbox and GLStats.

Introducing lunchbox::RequestHandler

25. May 2012

The last post –for now– in the lunchbox series is about the RequestHandler. This class is the most specific in Lunchbox, and makes most sense in the context of Collage. The primary use case is to register a pending operation and wait on its completion by another thread. The pattern is similar to futures, except that a future can be easily identified by a single integer.

In Collage, it is heavily used for synchronous, remote operations. The caller registers a request, sends a packet containing the request ID to a remote node which eventually replies using the request ID. The local reply handler then serves the local request, unblocking the caller waiting on the request. For convenience, data can be attached to the request, and the thread serving the request can provide a (return) value passed on to the waitRequest().

This concludes the lunchbox introduction. I’ve intentionally skipped over the following classes since I believe their concepts are more commonplace: Buffer, Clock, DSO, Lock, SpinLock, TimedLock, ScopedMutex, Log, MemoryMap, PerThread, Pool, RefPtr, RNG, Thread and UUID. If you want to hear some background on one of them, please post below.

Next week I’ll cover another topic – BuildYard, Collage, dash are on the list.

Introducing lunchbox::LFVector< T >

18. May 2012

Today’s post is about the LFVector. If you’ve paying attention to the previous posts in this series, you’ll know that this is a thread-safe, lock-free, STL-like vector implementation. It is based on the algorithm described by Damian Dechev et al in Lock-free dynamically resizable arrays.

Since the vector provides resizing operations, it is not completely thread-safe for all combinations of concurrent operations. We’ve also decided to make operations modifying the vector thread-safe by using a spin lock, which costs two compare-and-swap operations in the uncontented case. All read operations are fully lock-free and wait-free.

The LFVector algorithm uses a clever reallocation strategy. Compared to the STL vector, the data is not stored in a single array, since this would invalidate existing storage during resize operations. Instead, it uses a number of slots pointing to arrays of size 2^slot_index. This allows to allocate new storage during concurrent read accesses to existing elements. It also keeps the read access very fast, using just a few operations to access an element:

const int32_t slot = getIndexOfLastBit( ++i );
const size_t index = i ^ ( size_t( 1 )<<slot );
return slots_[ slot ][ index ];

A minor static overhead of this approach is that the slots have to be pre-allocated, which costs 504 bytes in the worst-cast scenario (63 slots of 8 bytes). The number of slots is a configurable template parameter, preset to 32 which seems reasonable for most use cases.

This slot-based storage requires a different approach to implementing iterators, which are based on maintaining the current index and using the operator [] instead of the pointer magic used by the STL vector iterators. In practice, they have the same complexity as the STL iterators and use the same interface.

Insert and erase operations first acquire the spin lock, and then potentially allocate or free the storage corresponding to a slot. Insert operations are completely thread-save with read operations, since existing end() iterators will keep pointing to the old end of the vector. Furthermore, the size is updated after the element is inserted, so size() followed by a read is also thread-safe with a concurrent insert. For erase operations, a concurrent read on the remove element produces undefined results. This affect in particular the end() and back() methods.

The LFVector is one of the magic pieces in DASH, which I guess warrants a whole set of posts in the future. DASH is very exciting work-in-progress for generic, efficient multi-threaded data access. Drop me a note if you want to beta test, provide feedback and contribute to this already.

As an added bonus, LFVector has support for boost::Serialization.