1.1.1. What is a ring buffer?

This section defines a ring buffer. First, for simplicity, we'll describe a single producer, single consumer ring buffer. Finally we'll show how this can be generalized to a single producer multi-consumer ring buffer.

Suppose we have a chunk of memory which is shareable between processes and within which we are going to use modulo addressing. Using modulo addressing means that if we have a pointer sequentially accessing this memory, when the pointer would run off the end of this memory region instead it returns to the beginning.

You can therefore think of this memory as an 'endless' ring of addresses. A ringbuffer..

For a ring buffer to be useful as a mechanism for exchanging data we need a bit more information. Specifically, we need a put pointer and a get pointer. As we will see we also need some concept of flow control.

Here's how this all works; Define the ring buffer as being empty if the put and get pointers are equal. Define the ring buffer as full if advancing the put pointer one storage unit would make it equal to the get pointer (we don't want an ambiguity between full and empty ring buffers). A producer will ensure it has space in the ring buffer for whatever data it wants to put by calculating the modulo distance between the put and get pointers. When space becomes available (producers wait if necessary for space), data are transferred to the ring buffer starting at the location indicated by the put pointer (using modulo addressing again). When the data are transferred, the put pointer is advanced to point to the location just following the message.

Similarly, a consumer that wants to get data from the ring ensures there is enough data to get by waiting until the distance between the put and get pointers is at least the size of the data transfer it wants to perform. Once there is sufficient data in the ring buffer it transfers the data out of the ring and, when the data have been read, updates the get pointer so that it points to the next unread unit of memory.

This business of the produceer waiting for space to be available and the consumer waiting for data to be available is called flow control.

Ring buffers therefore can be thought of as objects with the following simple to implement set of functions:

In a data acquisition system, we usually want to have several consumers. This can be accomodated by having more than one get pointer. This slightly complicates the flow control logic of the producer. Now the producer must consider the free space to be the minimum distance between the the put pointer and all of the get pointers.

Ring buffers are a very low overhead mechanism to transfer data between processes in a single shared memory computer system. Messages can be atomically put without any requirement to negotiate locks. Furthermore, since data are in shared memory, it's not necessary to transfer buffers to kernel space and then back out to application space. For more information about ring buffers see http://en.wikipedia.org/wiki/Circular_buffer

While theoretically the data transferred through the ring buffer is just a stream of bytes, in practice and in the NSCL Ring buffer Data Acquisition system, we send messages that have structure. The messages have a structure defined by the <DataFormat.h> header.

In the next section we'll look at how to use proxy ring buffers to transfer data across the network to computers that don't share memory.

1.1.2. What are proxy rings and how are they used?

The previous section introduced ring buffers as a data transfer mechanism within the memory shared by processes running in one or more processors that can share memory. In a data data acquisition system we usually want several such computer systems to join the party. Data taken in one system must be made visible in near real-time to analysis software in other computer systems. This section describes the mechanism used by the NSCL Ring Buffer data acquisition system to accomplish this feat.

Each system that runs the ring buffer data acquisition system has a simple server process called the RingMaster. We'll talk more about the RingMaster and its role in the next section.

One role the ring master performs is to assist in hoisting data out of an local ring, sending it across the network to another system. This is done through a mechanism called a proxy ring which makes the semantics of getting data from a remote system identical to the semantics of getting data from a local system.

First we need to talk about ring buffer naming. Each ring buffer in the system has a unique name that is of the form of a URL. The URL components are tcp://hostname/ringname. Where hostname is the name of the system in which the ring lives (use localhost for local rings), and ringname is the name of the ring within the system. By convention, unless you have special needs, the ringname is your logged in user name.

If a process attempts to open a ring buffer whose URL does not specify locahost as the hostname, the ring buffer DAQ system contacts the RingMaster in the target host and collaborates with it to create a local ring and a network pipeline that ships data from the ring in the remote host to the local ring.

Such rings are called proxy ringbuffers because from the consumer standpoint they cannot be distinguished from ring buffers that have local producers. Note as well that only the first consumer goes through the gymnastics of creating a proxy ring. Subsequent consumers simply connect to the proxy as an additional consumer. In this way, network traffic between rings and their proxies are aggregated.

A proxy ring has the local name hostname.remote-ringname where hostname is the host in which the 'real ring' is located and remote-ringname is the name of the real ring in hostnanme. Thus the proxy ring for tcp://spdaq42/fox will be tcp://localhost/spdaq42.fox.

1.1.3. What is the RingMaster server and what does it do?

All systems that run the Ring buffer data acquisition system also run a simple server called the RingMaster. The RingMaster performs the following functions:

• Collaborates with remote clients to set up a pipeline to produce data into proxy rings as described in the previous section.

• Allocate ring resources for local consumers.

• Cleans up when local consumer exit or release their ring resources.

The Ring buffer DAQ system has two types of clients. Producers and consumers. Recall that each ring can have at most one producer, and many consumers.

The ring master keeps track of which local processes are attached to a ring and whether or not a process is the producer or a consumer (actually a consumer could be consuming data from several rings, or even be more than one consumer on one ring).

When a client wants to obtain the put or a get pointer, it asks the ring master for one. It does so by opening a TCP/IP connection to the ring master and sending it a pointer request message. The ring master identifies the pointer it provides to the client. The client is then required to hold the TCP/IP connection open. If the TCP/IP connection closes, as it will normally if a process exits, the RingMaster releases the pointer that was associated with that connection.

In this way, ring buffers are immune to stalls that could occur if a pointer got orphaned.