Archive for the ‘Darwin’ Category

Equalizer 1.4 released

7. September 2012


The last two weeks have been quiet, since I was on biking through Switzerland. Meanwhile, poor Daniel back at work churned through most of the Collage changes outlined in the last post. You can see the changes in the Collage endian branch on github, which will be merged back into master in the next couple of weeks.

Now back to the news: After finally figuring out how to build Equalizer and dependencies using MacPorts portfiles on Mac OS X, I released the long-standing 1.4 version of Equalizer, GPU-SD, Lunchbox and vmmlib. Below is the release announcement – enjoy!

Neuchatel, Switzerland – September 7, 2012 – Eyescale is pleased to announce the release of Equalizer 1.4.

Equalizer is the standard framework to create and deploy parallel, scalable 3D applications. This modular release includes Collage 0.6, a cross-platform C++ library for building heterogenous, distributed applications, GPU-SD 1.4, a C++ library and daemon for the discovery and announcement of graphics processing units using zeroconf networking and Lunchbox 1.4, a C++ library for multi-threaded programming. All software packages are available for free for commercial and non-commercial use under the LGPL open source license.

Equalizer 1.4 is a feature release extending the 1.0 API, introducing major new features, most notably asynchronous readbacks, region of interest and thread affinity for increased performance during scalable rendering. It culminates over seven years of development and decades of experience into a feature-rich, high-performance and mature parallel rendering framework and related high-performance C++ libraries.

Equalizer enables software developers to easily build interactive and scalable visualization applications, which optimally combine multiple graphics cards, processors and computers to scale the rendering performance, visual quality and display size.

Equalizer Applications

Eyescale provides software consulting and development services for parallel 3D visualization software and GPU computing applications, based on the Eyescale software products or other open and closed source solutions.

Please check the release notes on the Equalizer website for a comprehensive list of new features, enhancements, optimizations and bug fixes. A paperback book of the Programming and User Guide is available.

We would like to thank all individuals and parties who have contributed to the development of Equalizer 1.4.

Left image courtesy of Cajal Blue Brain/ / Blue Brain Project. Second from left copyright Realtime Technology AG, 2008. Right image courtesy University of Siegen, 2008.

Introducing Collage: DataI/OStream

10. August 2012

The co::DataOStream and co::DataIStream form the core of the co::Object data distribution. They will gain even more importance in the next couple of weeks, when they will replace the current packet-based messaging (see 145). They will become the core of any communication between Collage nodes.

The data iostreams provide a std::iostream-like interface to send data over the network. They hide all the network connection details, allow overlapping of data serialization and sending through bucketization, do configurable compression and allow application-provided serializers for custom data types. We’re currently working on extending them to also do automatic endian conversion.

First of all they allow object serialization without a need of the application to know if the data has to be saved for later used (buffered objects), who will receive the data, whether or not to use multicast or how to compress it. The serialization in the application code is as simple as possible, here for example the eqPly::FrameData:

void FrameData::serialize( co::DataOStream& os, const uint64_t dirtyBits )       
{                                                                                
    co::Serializable::serialize( os, dirtyBits );                                
    if( dirtyBits & DIRTY_CAMERA )                                               
        os << _position << _rotation << _modelRotation;                          
    if( dirtyBits & DIRTY_FLAGS )                                                
        os << _modelID << _renderMode << _colorMode << _quality << _ortho        
           << _statistics << _help << _wireframe << _pilotMode << _idle          
           << _compression;                                                      
    if( dirtyBits & DIRTY_VIEW )                                                 
        os << _currentViewID;                                                    
    if( dirtyBits & DIRTY_MESSAGE )                                              
        os << _message;                                                          
}                                                                                

The deserialize method looks exactly the same, except using a DataIStream and the >> operator instead of <<. Applications can write free-standing serialization functions, similar to free-standing std::iostream operators.

Behind the scene Collage sets up the ostream with all connections to the nodes which have slave instance of the FrameData, preferring multicast connections. The output is bucketized, that is, whenever the accumulated data reaches a certain threshold (default ~64k), the current block is sent. This allows the OS to send data while the application prepares the next buffer.

Each DataOStream has a configurable compressor. The default algorithm uses heuristics to choose the best tradeoff between speed and compression ratio. Each outgoing packet is compressed before transmission. For buffered objects, the compressed data is retained to optimize memory usage.

Currently we are working on automatic endian conversion to form Collage networks between little and big endian hosts (see 146). Since most of the data types are known at deserialization time, a templated swap function will provide the endianness conversion. For void data we simply have to assume that it is already endian safe, or that the application doesn’t need endian safeness. In Collage and Equalizer we will make sure everything is endian-safe, e.g., by using portable boost serialization archives for the co::DataOStreamArchive.

Introducing Collage: github

3. August 2012

Collage

Collage Dependencies

Good news, everybody: Since this week, Collage is a separate project on github.

First this means that it’s much more lightweight to use since it has a small source code and repository size (<10MB) and less dependencies compared to the full Equalizer project. Say hello fast compilation times (less than a minute on my slowish laptop), easy setup and simpler directory layout.

Second this means that the next version will have a well-defined, stable API, similar to what Equalizer already has. I’m steadily working toward this with good progress and lots of cleanup. This means that building and maintaining distributed applications based on Collage will be painless.

Third and last my hope is that this gives the project more visibility and credibility, and therefore more outside contributions. We’ve already had quite a few awesome ones in the past, such as InfiniBand RDMA and UDT support.

Edit: API documentation can be found on eyescale.github.com.

Programming and User Guide for Equalizer 1.4

29. June 2012

Equalizer Programming and User Guide

Equalizer Programming and User Guide


I’ve just uploaded the review version of the Programming and User Guide for the upcoming 1.4 release of Equalizer, and to a certain extent, Collage.

This one packs 111 pages of content (118 total) and 63 figures. In a couple of weeks I’ll create the final hardcopy version on Amazon/CreateSpace.

What’s New?

This edition has, among the customary full review pass, a lot of new content. Starting with a full new chapter on Sequel and the associated polygonal rendering example, continuing with new section for application-specific scaling factors in immersive environments, region of interest for compositing and load-balancing and zeroconf discovery in Collage, finally finishing with a substantial rewrite of the section on distributed objects in Collage.

What?

The book is structured in two parts: The User Guide, laying the foundation on parallel rendering and scalability algorithms, and then explaining the configuration of visualization systems for Equalizer applications. The appendix contains a full reference on the file format.

The second part, the Programming Guide, gradually introduces programming parallel rendering applications. Starting with the basics in eqHello, the complexity is gradually increased with a chapter on Sequel and Equalizer using the respective example application. After this, an advanced features section focuses on introducing and demonstrating on specific features in isolation. It finishes of with a chapter on the Collage network library.

Why?

The Programming and User Guide is the ‘OpenGL red book’ of Equalizer. It consolidates all the documentation available in various places (Equalizer website, mailing list, github feature issues, my head) into a single document. Apart from gathering this information, through the format emerges a bigger picture, putting design decisions in context.

Introducing lunchbox::Servus

22. June 2012

Yes, we’re back to Lunchbox. During the 1.4 beta release (see previous post) we decided to merge a separate library in Lunchbox, since keeping it separate would have been overkill. In the end, it’s just a single class: Servus.

For the 999‰ of you who can’t get the reference: It’s a play on words to Bonjour. This class implements a simple C++ interface to announce and discover key-value pairs over zeroconf networking (aka. Bonjour).

The usage is really simple: You instantiate the class with the service name you’re interested in, e.g., “_collage._tcp”. Then you can register key-value pairs and start announcing them, and you can discover existing key-value pairs announced by other processes, typically within the subnet. It’s also legal to update key-value pairs on an announced service. Sounds simple enough, but if you’ve ever used the callback-driven dns_sd C API, you’ll know it’s much easier to use.

We use this class for two things right now: Collage node discovery and in GPU-SD.

In Collage, each co::LocalNode announces his node identifier and connection descriptions. This information is used as a fallback path when performing a LocalNode::connect( NodeID ) to identify and to connect to a previously unknown node. The Servus handle is exposed to applications through co::Zeroconf, which is a wrapper to ensure thread-safety. Applications can then use additional key-value pairs specific to their implementation.

In GPU-SD, Servus is used to announce and discover graphics cards in a visualization cluster, which in turn is used by Equalizer for auto-configuration.

Lunchbox Folly

8. June 2012

Facebook recently released folly, which caught my eye due to its similarity to Lunchbox. It’s definitely a library to watch.

I don’t think we’ll be using it right now, since it’s missing some of the stuff we’re using, e.g, the LFVector, and since the implementation so far seems to be mainly tested on Linux + gcc 4.6. On the other hand, it has some interesting components which we might need in the future, e.g, atomic hash containers. It also contains optimized version of some standard components such as vector and string, which haven’t really shown up as hotspot during profiling our code.

Interesting enough, folly also has some components which are almost identical to the Lunchbox counterparts, such as r/w spin locks and lock-free queues. It’s always good to see when ideas converge to a common design.

Equalizer 1.0 released

20. May 2011

It’s been only 21 months since the last post, but both Equalizer and me are still alive. This month we finally released the version 1.0, which was looong overdue.

Most notably, this release defines the stable API for all Equalizer 1.x releases. This means that all the functions marked with version 1.0 will be source-code compatible until we’ll release an Equalizer 2.0. Parts of the API are still undefined and unstable, in particular for the also-new Collage network library. However, 99% all of the functions used by the examples are stable.

Since the last major version, 0.9, there have been plenty of improvements and new features, e.g., subpixel compounds, reliable multicast for data distribution, runtime mono/stereo switch and many more. A comprehensive list is in the Release Notes.

Since this month I’ve started working on a new project based on Equalizer, and hopefully I’ll update this blog more regularly. More about this in another post…

Equalizer 0.9 Released!

11. August 2009

Cross-Segment Load-Balancing
We are pleased to announce the release of Equalizer 0.9, the standard framework to create and deploy parallel, scalable OpenGL applications. The most notable new features in this release are:

Please check the release notes on the Equalizer website for a comprehensive list of new features, enhancements, optimizations and bug fixes. A paperback book of the Programming and User Guide is available from Lulu.com.

We would like to thank all individuals and parties who have contributed to the development of Equalizer 0.9.

Cross-Segment Load-Balancing

30. July 2009

The upcoming Equalizer release will have another advanced scalability feature: Load-balancing across all resources used for the multi-display system.

This video should explain it all, if not ask in the comments below:

I hope to have benchmarks of this feature soon.

Dynamic Frame Resize

13. March 2009

We’ve been busy working on a new performance feature for Equalizer: Dynamic Frame Resize (DFR).

DFR automatically adapts the size of the rendering to achieve a constant framerate, which works very well for fill-limited applications such as eVolve. Of course, true to our mix-and-match strategy, it can also be combined with other scalable rendering features.

Enjoy: