9.6. Developing a Tailored SpecTcl

9.6.2.1. CRawUnpacker

Without further ado, let's start by defining our CRawUnpacker header file, CRawUnpacker.h

#ifndef CRAWUNPACKER_H
#define CRAWUNPACKER_H    

#include <config.h> 
#include "CRawADCUnpacker.h"
#include <EventProcessor.h>
#include <TreeParameter.h>
#include <cstdint>  
#include <cstddef>  

class CEvent;
class CAnalyzer;
class CBufferDecoder;     

class CRawUnpacker : public CEventProcessor 
{
  private:
    CRawADCUnpacker      m_unpacker;
    CTreeParameterArray  m_values;    

  public:
    CRawUnpacker();
    virtual ~CRawUnpacker();

    virtual Bool_t operator()(const Address_t pEvent,
                              CEvent& rEvent,
                              CAnalyzer& rAnalyzer,
                              CBufferDecoder& rDecoder);  
  private:
    Bool_t unpack(TranslatorPointer<std::uint32_t> begin, 
                  std::size_t nLongWords);   
};

#endif 
        

The implementation of our CRawUnpacker class will be found in the source file CRawUnpacker.cpp. This is naturally a bit longer of a file, because it includes the implementation code. The constructors and destructors are nearly trivial and will be dealt with in one fell swoop.

#include "CRawUnpacker.h"
#include <BufferDecoder.h>
#include <TCLAnalyzer.h>
#include <iostream>
#include <stdexcept>

using namespace std; 

CRawUnpacker::CRawUnpacker()
  : m_unpacker(),
    m_values("t",4096,0.0,4095.0,"channels",64,0) 
{
}

CRawUnpacker::~CRawUnpacker()
{
}
        

The more interesting code occurs in the operator() and unpack() methods. The operator() method is responsible for the actual processing of the data as well as telling SpecTcl what the size of the event is. It will just handle the calculation and reporting of the event size and then delegate the parsing and handling of the results to the unpack() method.

At least one event processor is required to set the event size and to avoid introducing confusing bugs into your SpecTcl, it is best practice to only do this in the first event processor of the pipeline.

Bool_t
CRawUnpacker::operator()(const Address_t pEvent, 
                        CEvent& rEvent, 
                        CAnalyzer& rAnalyzer, 
                        CBufferDecoder& rDecoder)
{
  TranslatorPointer<uint16_t> p(*rDecoder.getBufferTranslator(), pEvent);  

  CTclAnalyzer& a(dynamic_cast<CTclAnalyzer&>(rAnalyzer)); 

  size_t  size = (*p++ & 0x0fff); 

  // the event header is exclusive so the actual size of the full event
  // is actually the 
  a.SetEventSize((size+1)*sizeof(uint16_t)); 

  size_t nLongWords = size*sizeof(uint16_t)/sizeof(uint32_t); 

  return unpack(p, nLongWords); 
}
        

The unpack() method implementation will look like this:

Bool_t 
CRawUnpacker::unpack(TranslatorPointer<uint32_t> begin,
                     size_t nLongWords)
{
  auto end = begin+nLongWords;  

  try {  
    vector<ParsedADCEvent> events = m_unpacker.parseAll(begin, end);  

    int offset = 0;
    for (auto& event : events) {  
      offset = (event.s_geo-10)*32;  
      for (auto& chanData : event.s_data) {  
        m_values[chanData.first+offset] = chanData.second;
      }
    }
  } catch (exception& exc) {  
    cout << "Parsing Failed! Reason=" << exc.what() << endl;
    return kfFALSE;  
  }
                                    
  return kfTRUE;  
}
        

That is it for the implementation ofthe CRawUnpacker class. Before moving on, I would like to explain the motivation for the unpack() method. It is not obvious why its contents could not have just been included as the body of the operator() method. They very well could have and it would have worked perfectly fine for our specific setup. However, splitting it off makes our event processor much more reusable.

One of the reasons NSCLDAQ 11.0 was created was for better support of event building. The very presence of the capability makes it likely that we might want to use it. Separating the the unpack() from the operator() is aimed at supporting this scenario. The event builder outputs data that looks a bit different than the input data it receives, because it appends extra information to each item and then groups those together. In the lingo of the event builder, PHYSICS_EVENT ring items that enter the event builder are transformed into fragments that are glommed into built ring items of type PHYSICS_EVENT. The built ring item has a body stuffed with fragments. If we tried to parse the body of one of these using our operator() as it is currently implemented, SpecTcl would fail miserably. However, the same data that this parser understands is present in that built ring item. It is just buried deeply in a fragment. So the logic in this unpacker is still useful if we can traverse through the structure of the built body. By separating the unpack() method from the operator() method, we can call it once the right part of the data has been found. You might be wondering why we don't call the operator() method. The reason is that the operator method is responsible for reporting the event size. In a built ring item, the value this would extract would be incorrect. The unpack() method allows us to bypass it.

9.6.2.2. The CRawADCUnpacker Class

The CRawUnpacker depends on the presence of a parser class for the V785 and V775 class. That will be implemented in the CRawADCUnpacker class. I had stated that I would do this in a framework independent way. I will do so by showing a simpler example that depends on the TranslatorPointer<uint32_t> type for simplicity and then will explain how to generalize this afterwords. This will keep the example accessible to more people that don't need their code 100% independent but will also provide a clear path forward for those who do need it.

We will begin by describing the problem this class will solve. In the most fundamental sense, we have to work our way sequentially through the data, identify each piece, store the information in a framework independent entity, and identify when we are done. By doing this we will in essence validate the structure of the data. If it is different than we expect, then we cannot gaurantee that we understand it and will fail. If we do not encounter an error, we will return our platform independent type filled with the parsed information. Let's start by defining that type, which we will call ParsedADCEvent.

#ifndef CRAWADCUNPACKER_H
#define CRAWADCUNPACKER_H

#include <vector>
#include <utility>
#include <cstdint>
#include <TranslatorPointer.h>

struct ParsedADCEvent
{
  int s_geo;
  int s_crate;
  int s_count;
  int s_eventNumber;
  std::vector<std::pair<int, std::uint16_t> > s_data; 
};

#endif
        

The ParsedADCEvent contains all of the information that we care to keep from a complete event. The data for each channel will be stored as a pair in a vector. The first element of each pair will be the channel number while the second element will be the digitized value.

The next thing we need to do is define the CRawADCUnpacker class itself. Two entry points will be provided for public use. One of them is used by the CRawUnpacker class, parseAll(), and it iteratively calls the second, parseSingle(). We will focus primarily on the implementation of parseSingle() because it is the true parsing routine. The parseSingle() method depends on a handful of helper methods that will be explained as they come up. Here is the class declaration of the CRawADCUnpacker. In CRawADCUnpacker.h, it would follow the end of the ParsedADCEvent definition but precede the #endif preprocessor directive.

class CRawADCUnpacker 
{
  public:
    using Iter=TranslatorPointer<std::uint32_t>; 

  public:
    std::vector<ParsedADCEvent> 
      parseAll(const Iter& begin, const Iter& end); 

    std::pair<Iter, ParsedADCEvent> 
      parseSingle(const Iter& begin, const Iter& end);

  private:
    // Utility methods
    bool isHeader(std::uint32_t word);
    bool isData(std::uint32_t word);
    bool isEOE(std::uint32_t word);

    void unpackHeader(std::uint32_t word, ParsedADCEvent& event);
    void unpackDatum(std::uint32_t word, ParsedADCEvent& event);
    Iter unpackData(const Iter& begin, const Iter& end, ParsedADCEvent& event);
    void unpackEOE(std::uint32_t word, ParsedADCEvent& event);
};
          
        

We know that the data from the ADC must come in a specific order. The event header should come first followed by a sequence of data words, and then ultimately an end of event or trailer word. Furthermore, the header word is descriptive about how many data words exist so by reading it we will know when to expect the end of event word. If the data we are parsing violates this structure, then something bad has happened. Each stage will check whether the data is valid and throw if it is not. Here is the code for the parseSingle() method and includes that we need to add to our CRawADCUnpacker.cpp file:

#include "CRawADCUnpacker.h"
#include <string>
#include <stdexcept>
#include <iostream>

using namespace std;

static const uint32_t TYPE_MASK (0x07000000);
static const uint32_t TYPE_HDR  (0x02000000);
static const uint32_t TYPE_DATA (0x00000000);
static const uint32_t TYPE_TRAIL(0x04000000);

static const unsigned GEO_SHIFT(27);
static const uint32_t GEO_MASK (0xf8000000);

static const unsigned HDR_COUNT_SHIFT(8);      
static const uint32_t HDR_COUNT_MASK (0x00003f00);
static const unsigned HDR_CRATE_SHIFT(16);
static const uint32_t HDR_CRATE_MASK (0x00ff0000);

static const unsigned DATA_CHANSHIFT(16);
static const uint32_t DATA_CHANMASK (0x001f0000);
static const uint32_t DATA_CONVMASK (0x00003fff);

static const uint32_t TRAIL_COUNT_MASK(0x00ffffff);
static const uint32_t BERR(0xffffffff); 


pair<CRawADCUnpacker::Iter,ParsedADCEvent> 
  CRawADCUnpacker::parseSingle(const Iter& begin, const Iter& end)
{
  
  ParsedADCEvent event;             

  auto iter = begin;                
  if (iter<end) {                
    unpackHeader(*iter++, event); 
  } else {
    string errmsg("CRawADCUnpacker::parseSingle() ");
    errmsg += "Incomplete event found in buffer.";
    throw runtime_error(errmsg);
  }

  int nWords = event.s_count;
  auto dataEnd = iter+nWords;

  if ((dataEnd > end) || (dataEnd == end)) { 
    string errmsg("CRawADCUnpacker::parseSingle() ");
    errmsg += "Incomplete event found in buffer.";
    throw runtime_error(errmsg);
  } else {
    iter = unpackData(iter, dataEnd, event); 
  }
 
  if (iter<end) {              
    unpackEOE(*iter++,event);     
  } else {
    string errmsg("CRawADCUnpacker::parseSingle() ");
    errmsg += "Incomplete event found in buffer.";
    throw runtime_error(errmsg);
  }

  return make_pair(iter,event);     
}
        

As you can see, there are many supporting pieces. We will look at them one by one, beginning with the unpackHeader() method.

bool CRawADCUnpacker::isHeader(uint32_t word) 
{
  return ((word&TYPE_MASK)==TYPE_HDR);   
}


void CRawADCUnpacker::unpackHeader(uint32_t word, ParsedADCEvent& event)
{
  if (! isHeader(word) ) {
    string errmsg = "CRawADCUnpacker::parseHeader() ";
    errmsg += "Found non-header word when expecting header. ";
    errmsg += "Word=";
    errmsg += to_string(word);
    throw runtime_error(errmsg);
  }   

  event.s_geo   = ((word & GEO_MASK)>>GEO_SHIFT);
  event.s_crate = ((word & HDR_CRATE_MASK)>>HDR_CRATE_SHIFT);
  event.s_count = ((word & HDR_COUNT_MASK)>>HDR_COUNT_SHIFT);  
}
        

The data words are unpacked in a bit more fancy method called unpackData(). It loops through a range of words designated by its arguments, calling unpackDatum() on each one. The unpackDatum() is very similar to unpackHeader() method in that it extracts specific pieces of data from the data word it is passed. The difference is that the values it extracts for the channel and value are paired together and stored in the vector of the event. This is what those look like:

bool CRawADCUnpacker::isData(uint32_t word)
{
  return ((word&TYPE_MASK)==TYPE_DATA);  
}


void CRawADCUnpacker::unpackDatum(uint32_t word, ParsedADCEvent& event)
{
  if (! isData(word) ) {
    string errmsg = "CRawADCUnpacker::unpackDatum() ";
    errmsg += "Found non-data word when expecting data.";
    throw runtime_error(errmsg);
  }

  int channel   = ((word & DATA_CHANMASK)>>DATA_CHANSHIFT);
  uint16_t data = (word & DATA_CONVMASK);   

  auto chanData = make_pair(channel,data);
  event.s_data.push_back(chanData);             
}


CRawADCUnpacker::Iter 
CRawADCUnpacker::unpackData(const Iter& begin, 
                            const Iter& end,
                            ParsedADCEvent& event)
{
  // only allocate memory once because we know how much we need already
  event.s_data.reserve(event.s_count);       

  auto iter = begin;
  while(iter != end) {

    unpackDatum(*iter, event);                

    ++iter;
  }

  return iter;   
}
        

Finally, we reach the unpackEOE() method. Once again, this is extremely similar to the unpackHeader() method, moreso than the unpackDatum() method. Here is the code for that.

bool CRawADCUnpacker::isEOE(uint32_t word)
{
  return ((word&TYPE_MASK)==TYPE_TRAIL);             
}

void CRawADCUnpacker::unpackEOE(uint32_t word, ParsedADCEvent& event)
{
  if (! isEOE(word) ) {
    string errmsg = "CRawADCUnpacker::unpackEOE() ";
    errmsg += "Found non-data word when expecting data.";
    throw runtime_error(errmsg);  
  } 
  
  event.s_eventNumber = (word & TRAIL_COUNT_MASK);   
}
        

Alright, now that we have the parseSingle() method defined and understood, we can go back to the parseAll() method. It is very simple and resemble the unpackData() method in that it just calls parseSingle() until it is done. The only differences are that it stores each ADC event into a list for returning and has to protect itself from processing the BERR words (0xffffffff) that follow the more meaningful data. Here is the parseAll() implementation:

vector<ParsedADCEvent> 
CRawADCUnpacker::parseAll(const Iter& begin,
                          const Iter& end)
{
  vector<ParsedADCEvent> parsedData;  

  auto iter = begin;
  while (iter != end) {      

    if (*iter != 0xffffffff) {              
      auto result = parseSingle(iter,end);   

      parsedData.push_back(result.second);   
      iter = result.first;                   
    } else {
      ++iter;                                
    }
  } 

  return parsedData;
}
        

That is it for the unpacker. Let me now explain what would need to be done to make this fully independent of SpecTcl. The only dependency that this has on SpecTcl are some header files and the TranslatorPointer<uint32_t> type. What may not be terribly clear is that the TranslatorPointer<uint32_t> behaves almost identically to a uint32_t*. In that case, one could just as well use a uint32_t* pointer in its stead. You may wonder why I didn't do this already. Well, the kicker is that a TranslatorPointer is able to automatically handle byte-order swapping if the data requires it. A plain old pointer is not so smart. You will only be bitten by this if the data in the data stream was generated in such a way that swapping bytes is necessary. Only you are the one who knows this if you venture outside the world of SpecTcl. If you have to swap bytes, this becomes a bit less portable and you will need to create your own byte-swapping utility. The good news is that it is little difficulty to accomodate any solution you have concerning the TranslatorPointer. Because we hid this behind a typedef, we only need to change the Iter alias to be a different type. Arguably the more general solution would be to transform the class into a template itself where the Iter is the template parameter. Doing so requires more changes to the code but would not have to be recompiled whenever you wanted to change the type. The most important thing here is that these changes never require a change to the actual implementation code. The only requirement is that whatever you choose to use for the iterator implements the operators used. Any random access iterator will fulfill that requirement, but you need not go so far to implement all requirements of a random access iterator. As for the included headers, you can use preprocessor conditionals for this. If you are building with SpecTcl, then you will need the proper headers, otherwise, you should not include them.