10.3. Worked example for supporting a new data format

In this section we are going to build support for a simple, fictional data acquisition system data format. The internal format of the data bears some resemblence to the ring buffer data format used by NSCLDAQ 10.x and later, however that resemblance is purely coincidental and you should not conclude from it that any of this code is actually used by SpecTcl to decode NSCLDAQ data.

Before writing any code we need to have a clear understanding of the format of the data we're going to decode. This is as true for a real case as it is for this imaginary case.

1. Data will come to us in fixed length blocks. No item will extend across block boundaries (this simplifies things' tremendously).

2. Each block will have a header that consists of several uint32_t. These are, in order, the number of 8 bit bytes occupied by useful data, the number of items in the block, a sequence number that tells us the number of the block in the event file or data taking run and a 32 bit SpecTcl byte order signature.

3. The block is divided int items. Each item consists of a self inclusive size in bytes and a type (both are uint32_t) followed by a body whose structure depends on the type value.

4. Type 1 indicates a run start, type 2 indicates a run end, Type 3 indicates scaler data and type 4 indicates data taken in response to a physics trigger.

5. The structure of type 1,2 data are the same and look like this:

                       struct {
                           header    s_header;
                           uint32_t  s_runNumber;
                           char      s_title[];
                       };
                   

   Where header is the item header described above. The title is a null terminated variable length string.

6. Scaler data (type 3), has the following structure:

                       struct {
                           header    s_header;
                           uint32_t  s_numScalers;
                           uint32_t  s_counters[];
                       };
                   

   s_counters are the array of scaler values. s_numScalers tells us how many of those there are.

7. Type 4 (physics event data) is just an array of bytes whose structure and meaning depend on the actual experiment performed using this data acquisition system.

Let's first create a header, expdata.h that captures the structure of the data described above. In order to minimize the chances for name collission, we're going to put all of those definitions in a namespace (SomeDaqSystem):

#ifndef EXPDATA_H
#define EXPDATA_H

#include <cstdint>



namespace SomeDaqSystem {
  static const unsigned BEGIN(1);
  static const unsigned END(2);
  static const unsigned SCALER(3);
  static const unsigned EVENT(4);
  
  typedef struct _Header {
    std::uint32_t    s_size;
    std::uint32_t    s_type;
  } Header;

  typedef struct _StateChange {
    Header          s_header;
    std::uint32_t   s_runNumber;
    char            s_title[];
  } StateChange;

  typedef struct _Scalers {
    Header         s_header;
    std::uint32_t  s_numScalers;
    std::uint32_t  s_counters[];
  } Scalers;

  typedef struct _Event {
    Header    s_header;
    char      s_body[];
  } Event;

  typedef struct _BufferHeader {
    std::uint32_t s_bytesUsed;
    std::uint32_t s_itemCount;
    std::uint32_t s_sequence;
    std::uint32_t s_signature;
  } BufferHeader;
    
};

#endif
                
            

The use of static const unsigned above fulfils the same purpose as #define but places those definitions inside the namespace, rather than potentially conflicting with other #defines the preprocessor knows about.

Let's look at the header for our decoder class. We're going to dispatch various types to type specific handlers. We're going to hand events one at a time to the analyzer rather than trying to We're also going to ensure we are able to deal with new item types without failing:

#ifndef SOMEDAQBUFFERDECODER_H
#define SOMEDAQBUFFERDECODER_H


#include <BufferDecoder.h>
#include "expdata.h"
#include <string>

class SomeDaqBufferDecoder : public CBufferDecoder
{
private:
  
  SomeDaqSystem::BufferHeader   m_lastHeader;
  std::string                   m_lastTitle;
  std::uint32_t                 m_lastRunNumber;    
  unsigned                      m_entities;
  SomeDaqSystem::Header*        m_pCurrentItem;

public:                                             
  virtual BufferTranslator* getBufferTranslator();
  virtual const Address_t getBody();
  virtual UInt_t getBodySize() ;
  virtual UInt_t getRun() ;
  virtual UInt_t getEntityCount() ;
  virtual UInt_t getSequenceNo() ;
  virtual UInt_t getLamCount() ;
  virtual UInt_t getPatternCount() ;
  virtual UInt_t getBufferType() ;
  virtual void getByteOrder(Short_t& Signature16,
			    Int_t& Signature32) ;
  virtual std::string getTitle() ;

  virtual void operator() (UInt_t nBytes,        
			   Address_t pBuffer,
			   CAnalyzer& rAnalyzer);

private:
  void stateChange(CAnalyzer& rAnalyzer);  
  void scaler(CAnalyzer& rAnalyzer);
  void event(CAnalyzer& rAnalyzer);
  void other(CAnalyzer& rAnalyzer);        
};
                
            

In spite of the ordering of the methods in the header, the best place to start is probable with the operator() method. Once we've coded that and the type specific handlers, the getter methods should fall into place fairly easily.

void
SomeDaqBufferDecoder::operator()(
    UInt_t nBytes, Address_t pBuffer, CAnalyzer& rAnalyzer
)
{
  SomeDaqSystem::BufferHeader* pHeader =                       
    reinterpret_cast<SomeDaqSystem::BufferHeader*>(pBuffer);
  std::memcpy(&m_lastHeader, pHeader, sizeof(SomeDaqSystem::BufferHeader));

  pHeader++;                                                    
  m_pCurrentItem = reinterpret_cast<SomeDaqSystem::Header*>(pHeader);

  BufferTranslator* pT =                                        
    BufferFactory::CreateBuffer(pBuffer, m_lastHeader.s_signature);
  
  for (int i = 0; i < pT->TranslateLong(m_lastHeader.s_itemCount); i++) {  

    switch(pT->TranslateLong(m_pCurrentItem->s_type)) {
    case SomeDaqSystem::BEGIN:
    case SomeDaqSystem::END:
      stateChange(rAnalyzer);
      break;
    case SomeDaqSystem::SCALER:                               
      scaler(rAnalyzer);
      break;
    case SomeDaqSystem::EVENT:
      event(rAnalyzer);
      break;
    default:
      other(rAnalyzer);
      break;
    }
                                                             
    std::uint8_t* pBytes = reinterpret_cast<std::uint8_t*>(m_pCurrentItem);  
    pBytes += pT->TranslateLong(m_pCurrentItem->s_size);
    m_pCurrentItem = reinterpret_cast<SomeDaqSystem::Header*>(pBytes);
  }

  delete pT;                                                
  
}

            

Let's look at the state transition method. What that needs to do is:

• Set the entity count to 1.

• Extract the run number (remember byte order translation).

• Extract the title.

• Invoke the analyzer's OnStateChange method with the appropriate item type. We're going to provide a helper method called translateItemType that will translate an item type into one of the types expected by SpecTcl.

void
SomeDaqBufferDecoder::stateChange(CAnalyzer& rAnalyzer)
{
  
  BufferTranslator* pT =
    BufferFactory::CreateBuffer(m_pCurrentItem, m_lastHeader.s_signature);
  SomeDaqSystem::StateChange* pItem =
    reinterpret_cast<SomeDaqSystem::StateChange*>(m_pCurrentItem);
  
  m_entities = 1;
  m_lastRunNumber = pT->TranslateLong(pItem->>s_runNumber);
  m_lastTitle    = pItem->s_title;

  rAnalyzer.OnStateChange(
     translateItemType(pT->TranslateLong(pItem->s_header.s_type)),
     *this
  );
  delete pT;
				     
}
            

The code above pretty much is what you'd expect to see from the specification provided.

The only differences between scaler and stateChange are that we fill in the entity count from the number of scalers in the scaler item, and there's no run number or title to extract from the item.

void
SomeDaqBufferDecoder::scaler(CAnalyzer& rAnalyzer)
{
 BufferTranslator* pT =
    BufferFactory::CreateBuffer(m_pCurrentItem, m_lastHeader.s_signature);
  SomeDaqSystem::Scalers* pScaler =
    reinterpret_cast<SomeDaqSystem::Scalers*>(m_pCurrentItem);

  m_entities = pT->TranslateLong(pScaler->s_numScalers);

  rAnalyzer.OnScaler(*this);

  delete pT;
  
}             
            

Again I think the code is pretty well self explanatory.

For events, we're just going to set the entity count to 1 and invoke OnPhysics. While OnPhysics can handle a block of events, we're not going to try to do that in this simple example.

void
SomeDaqBufferDecoder::event(CAnalyzer& rAnalyzer)
{
  m_entities = 1;
  rAnalyzer.OnPhysics(*this);
}
                
            

other is similarly simple, however, once more we need to get the item type.

void
SomeDaqBufferDecoder::other(CAnalyzer& rAnalyzer)
{
 BufferTranslator* pT =
    BufferFactory::CreateBuffer(m_pCurrentItem, m_lastHeader.s_signature);

 m_entities = 1;
 
 rAnalyzer.OnOther(
    translateItemType(pT->TranslateLong(m_pCurrentItem->s_type)),
    *this
  );

 delete pT;
}
                
            

Let's implement translateItemType before we turn to the getter menthods. The only choice we really need to make is how to handle item types that are not yet defined. For now we'll add the type to MAXSYSBUFTYPE turning it into a user item type. In doing so, we're making the implicit assumption item types won't be zero.

std::uint32_t
SomeDaqBufferDecoder::translateItemType(std::uint32_t type)
{
  static std::map<uint32_t, uint32_t>  typeMap = {
    {1, BEGRUNBF}, {2, ENDRUNBF}, {3, SCALERBF}, {4, DATABF}
  };

  auto p = typeMap.find(type);
  if (p != typeMap.end()) {
    return p->second;
  } else {
    return MAXSYSBUFTYPE + type;
  }
}
            

The only thing interesting is the use of an std::map. This is an associative container. It maps keys to values supporting an efficent find method. The auto keyword declaring p allows the compiler to infer the type to be std::map<std::uint32_t, std::uint32_t>::iterator. This is a pointer like object to an std::pair whose first member is a map index and whose second is the value of the map entry associated with that index.

Now we can look at the trivial, one-liner getter methods:

const Address_t
SomeDaqBufferDecoder::getBody()
{
  return m_pCurrentItem + 1;
}
UInt_t
SomeDaqBufferDecoder::getRun()
{
  return m_lastRunNumber;
}
UInt_t
SomeDaqBufferDecoder::getEntityCount()
{
  return m_entities;
}
UInt_t
SomeDaqBufferDecoder::getLamCount()
{
  return 0;
}
UInt_t
SomeDaqBufferDecoder::getPatternCount()
{
  return 0;
}
void
SomeDaqBufferDecoder::getByteOrder(Short_t& sig16, Int_t& sig32)
{
  sig32 = m_lastHeader.s_signature;
  sig16 = (m_lastHeader.s_signature & 0xffff);
}

std::string
SomeDaqBufferDecoder::getTitle()
{
  return m_lastTitle;
}

            

The next getters take a bit more work, but not much:

BufferTranslator*
SomeDaqBufferDecoder::getBufferTranslator()
{
  return  BufferFactory::CreateBuffer(
    m_pCurrentItem, m_lastHeader.s_signature
  );
}

UInt_t
SomeDaqBufferDecoder::getBodySize()
{
  BufferTranslator* pT = getBufferTranslator();
  UInt_t size  = pT->TranslateLong(m_pCurrentItem->s_size);

  delete pT;
  return size;
}
UInt_t
SomeDaqBufferDecoder::getSequenceNo()
{
  BufferTranslator* pT = getBufferTranslator();
  UInt_t seq = pT->TranslateLong(m_lastHeader.s_sequence);
  
  delete pT;
  return seq;
}

UInt_t
SomeDaqBufferDecoder::getBufferType()
{
  BufferTranslator* pT = getBufferTranslator();
  UInt_t            rawType = pT->TranslateLong(m_pCurrentItem->s_type);
  delete pT;

  return translateItemType(rawType);
}

            

Next we need to write a creator for this that can be passed to CAttachCommand::addDecoderType. This is pretty trivial. Almost the hardest part is to choose a -format value to associate with this decoder. Let's use blockring to indicate that these are essentially ring items but in fixed sized blocks.

Here's the header, SomeDaqDecoderCreator.h:

#ifndef SOMEDAQDECODERCREATOR_H
#define SOMEDAQDECODERCREATOR_H

#include <AttachCommand.h>
#include <CCreator.h>

class SomeDaqDecoderCreator : public CAttachCommand::CDecoderCreator
{
public:
  virtual CBufferDecoder*  operator()();
  virtual std::string describe() const;
};


#endif

            

As previously described, we need to implement both pure virutal methods. The operator() just needs to create a new SomeDaqBufferDecoder object and describe just has to return a description of the decoder.

Here's the entire class implementation:

#include "SomeDaqDecoderCreator.h"
#include "SomeDaqBufferDecoder.h"

CBufferDecoder*
SomeDaqDecoderCreator::operator()()
{
  return new SomeDaqBufferDecoder;
}

std::string
SomeDaqDecoderCreator::describe() const
{
  return "Decode buffers from blocked 'ring items'";
}                
            

In CMySpecTclApp::AddCommands, we can then add a line that looks like:

CAttachCommand::addDecoderType("blockring", new SomeDaqDecoderCreator);            
        

To make the new decoder we wrote available to SpecTcl