SpecTcl Programming Guide.
PrevNext

Chapter 3. The SpecTcl event processing pipeline.

Most people programming SpecTcl will only need to write code in the SpecTcl event processing pipeline. In this chapter we will:

A simplified version of much of this material can be found in the SpecTcl user guide. That document also describes how to modify the SpecTcl skeleton Makefile to incorporate your code in the your tailored SpecTcl executable.

The event processing pipeline is responsible for analyzing a raw event and turning it into a set of parameters that are passed to the event sink pipeline. The histogrammer is one of the elements of the event sink pipeline. To first order, you can view the purpose of the event analysis pipeline as preparing raw event data for histogramming.

The event processing pipeline consists of an ordered set of Event processors. Each event processor has access to the raw event as well as to the parameters that have been unpacked by prior elements of the pipeline.

Event pipelines are usually defined statically at compilation time by registering the pipeline stages in the CreateAnalysisPipeline method of MySpecTclApp.cpp. The SpecTcl API also allows you to dynamically manipulate the pipeline.

Due to the historical development of SpecTcl, there are two possible representation of parameters. The first, and original/native version is as a flat array like object called a CEvent. A CEvent object is passed in to the each event processor. It's a dynamically expanding array whose slots are ValidValue objects. ValidValue objects are double precision real values that know when if they've been given a value since some reset time.

The second version is a set of CTreeParameter and CTreeParameterArray objects. This version was introduced by Daniel Bazin to attach additiona meta data to parameters and to allow parameters to be organized into structures that better reflect the organization of the experiment being analyzed. The tree parameter subsystem was adopted by SpecTcl and is now supported software.

Currently we recommend the use of tree parameters over the flat representation of the CEvent array as they are much easier to manage. This manual is written with tree parameters in mind.

3.1. Event Processors

Event processors are objects that are instances of classes derived from the CEventProcessor base class. This class is defined in the SpecTcl header: EventProcessor.h

The class provides the interface between the high level parser and the event processing pipeline. The high level parsing subsystem will invoke methods of each event processor as determined by its high level analysis of the contents of the data. Let's have a look at the important parts of the EventProcessor.h header.

Example 3-1. CEventProcessor definition


#ifndef MYEVENTPROCESSOR_H
#define MYEVENTPROCESSOR_H
#include <config.h>
#include <EventProcessor.h>


class MyEventProcessor : public CEventProcessor
{
public:
  MyEventProcessor();

  virtual Bool_t operator()(const Address_t pEvent,
                            CEvent& rEvent,
                            CAnalyzer& rAnalyzer,
                            CBufferDecoder& rDecoder); // Physics Event.

  // Functions:
  virtual Bool_t OnAttach(CAnalyzer& rAnalyzer); // Called on registration.
  virtual Bool_t OnBegin(CAnalyzer& rAnalyzer,
                         CBufferDecoder& rDecoder); // Begin Run.
  virtual Bool_t OnEnd(CAnalyzer& rAnalyzer,
                       CBufferDecoder& rBuffer); // End Run.
  virtual Bool_t OnPause(CAnalyzer& rAnalyzer,
                         CBufferDecoder& rDecoder); // Pause Run.
  virtual Bool_t OnResume(CAnalyzer& rAnalyzer,
                          CBufferDecoder& rDecoder); // Resume Run.
  virtual Bool_t OnOther(UInt_t nType,
                         CAnalyzer& rAnalyzer,
                         CBufferDecoder& rDecoder); // Unrecognized buftype.

  virtual Bool_t OnEventSourceOpen(std::string name);
  virtual Bool_t OnEventSourceEOF();
  virtual Bool_t OnInitialize();

};

First note that none of the virtual methods are pure virtual. All have default implementations that do nothing. This allows you to only implement the methods you need.

Second, note that all methods return a Bool_t value. This is an enumerated type that can have the values kfTRUE or kfFALSE. The return value indicates whether SpecTcl should continue processing the event that caused the method to be called. If kfTRUE is returned, processing continues and, for a physics event, the event sink pipeline will be entered if all elements returned that. If kfFALSE is returned, event processing is halted immediately. No further stages of the event processing pipeline will be called and the event sink pipeline won't be invoked for physics events.

Before looking at these methods; we can see a couple of data types are passed to many of these methods; an instance of a CBufferDecoder and an instance of a CAnalyzer

These objects are components of the high level parsing subsystem. The CBufferDecoder is responsible for picking apart the raw data into elements of known types. The CAnalyzer is passed elements or groups of elements and is responsible for running the event processing pipeline. For most event processor methods, you can ignroe these parameters. The physics event processor, however has some requirements that must be met. We will describe those requirements in a bit.

A quick point about these high level parsing subsystem objects. They do provide services that may be useful to some event processing methods. The CAnalyzer used in SpecTcl by default is actually a CTclAnalyzer object. When we describe the event processor's responsibilities we will come back to this latter fact.

Let's look at each event processor method in turn. The order below is the approximate order in which these methods will be called.

virtual Bool_t OnAttach(CAnalyzer& rAnalyzer);

For an event processor to be called it must be attached to the analyzer. This is done either statically, in CreateAnalysisPipeline of MySpecTclApp or by dynamic registration after SpecTcl is started.

This method is called when the event processor is attached to the analyzer. rAnalyzer is a reference to the analyzer to which the processor is attached.

virtual Bool_t OnInitialize();

If an event processor is statically added to the event processor pipeline, it will be added p rior to SpecTcl's complete initialization. This means there are some SpecTcl facilities that are not available.

Once SpecTcl is initialized, and once a dynamically attached event processor is attached, SpecTcl invoked OnInitialize. This method can perform any initialization that requires full SpecTcl initialization as a prerequisite.

virtual Bool_t OnEventSourceOpen(std::string name);

SpecTcl analyzes data from event sources. This method is called to indicate to the event processor that a new event source has been opened. This is one of the few methods not invoked by the analyzer object.

The name parameter describes the event source that's being opened. The first words of this will be one of File: for a file data source or Pipe from: for pipe data sources. The remainder of the string depends on the data source type.

For File: data sources the remainder of the string is the name of the file attached, as it was supplied to the attach command.

For Pipe from: data source, the remainder of the string are the command words that define the program and command line parameters of the program that is on the other end of the pipe.

The string is essentially what the attach -list SpecTcl command returns.

virtual Bool_t OnEventSourceEOF();

Called when an end of file is encountered on an event source. This happens for a pipe when the program attached to the pipe exits. It happens for file data sources when all of the data in the file have been processed by SpecTcl.

virtual Bool_t OnBegin( CAnalyzer& rAnalyzer, CBufferDecoder& rDecoder);

Called when a begin run item is encountered on the data source. Note that this may not get called because there are data acquisition systems that don't produce begin run items. Furthermore if SpecTcl uses a pipe data source to access online data in the middle of a run, there won't be a begin item for that run.

The rAnalyzer and rDecoder parameters are references to the analyzer and buffer decoder objects respectively. The services these objects offer will be described in Interfacing SpecTcl with data from other data acquisition systems.. Their use at this time is beyond the scope of this chapter.

virtual Bool_t OnPause( CAnalyzer& rAnalyzer, CBufferDecoder& rDecoder);

Called if a run has been paused. Note that not all data acquisition systems support temporarily pausing runs. The rAnalyzer and rDecoder are once more the analyzer and buffer decopder components of the high level parsing stage of pipeline that processes data from the data source.

virtual Bool_t OnResume( CAnalyzer& rAnalyzer, CBufferDecoder& rDecoder);

For systems that support temporary pauses of data taking runs, this is called when the data from the run indicates one of those pauses has resumed. The rAnalyzer and rDecoder are once more the analyzer and buffer decopder components of the high level parsing stage of pipeline that processes data from the data source.

virtual Bool_t OnEnd( CAnalyzer& rAnalyzer, CBufferDecoder& rDecoder);

Called at the end of run indication from the data source.

virtual Bool_t operator()(const Address_t pEvent, CEvent& rEvent, CAnalyzer& rAnalyzer, CBufferDecoder& rDecoder);

Called for each physics event. In addtion to the rAnalyzer and rDecoder parameters the pEvent parameter is a pointer to the start of the event body. For data formats, that provide headers to the event, the decoder object will typically provide methods to access those headers.

The rEvent parameter is a reference to the event array that is the ultimate output of the event processing pipeline for physics events. rEvent is an array-like object of ValidValue objects. You can think of rEvent as an array that automatically expands as needed to fit the largest index it's passed.

On the left hand side of an assignment, you can think of elements of rEvent as double precision reals. In an expression, however the additional properties of a ValidValue object become visible. Specifically these objects can be reset to an undefined state and will throw an exception if referenced prior to having been assigned a new value (having become valid).

At the beginning of event processing; and as elements are added to rEvent, the valid values in it are reset to be invalid by SpecTcl. Prior to using an element of rEvent within an expression, you need to ensure that it is valid. This can be done either by reasoning about the flow of the computation or by invoking the isValid method which returns true if the element was assigned. Performing a read reference on a ValidValue that has not yet been assigned a value results in an exception.

Using tree parameters allows you to avoid direct referencing of the rEvent array, referencing instead specific CTreeParameter objects or elements of specific CTreeParameterArray objects that have been bound to rEvent elements. The comments about validity checking are still relevent however.

virtual Bool_t OnOther( UInt_t nType, CAnalyzer& rAnalyzer, CBufferDecoder& rDecoder);

Called when some other type of data is encountered from the data source. In NSCLDAQ, for example, this is called for scaler data. In addition to the rAnalzer and rDecoder parameters, nType is an integer code that indicates the type of data being provided.

This method and the decoder object must interact in order to get the data in the item. Type codes are data acquisition system (and even maybe version) specific.

Processing this method requires an understanding of the decoder for your data set and the formats of the data types you want ot handle.

In the remainder of this section, we're going to look at an event processor that is a bit more involved than the one shown in the SpecTcl User guide. We'll do this to show specifically how the event processor may interact with the analyzer and the decoder. This example is given in the context of NSCLDAQ version 11.x

As we now recommend, this example will use Tree Parameters rather than directly accessing the rEvent object.

Example 3-2. Our event processor header. 


#ifndef MYEVENTPROCESSOR_H
#define MYEVENTPROCESSOR_H
#include <config.h>
#include <EventProcessor.h>
#include <TreeParameter.h>

class MyEventProcessor : public CEventProcessor
{
private:
  CTreeParameter sum;
  CTreeParameterArray raw;

public:
  MyEventProcessor();
  
  virtual Bool_t operator()(const Address_t pEvent,
                            CEvent& rEvent,
                            CAnalyzer& rAnalyzer,
                            CBufferDecoder& rDecoder); // Physics Event.

  // Functions:
  virtual Bool_t OnAttach(CAnalyzer& rAnalyzer); // Called on registration.
  virtual Bool_t OnBegin(CAnalyzer& rAnalyzer,
                         CBufferDecoder& rDecoder); // Begin Run.
  virtual Bool_t OnEnd(CAnalyzer& rAnalyzer,
                       CBufferDecoder& rBuffer); // End Run.
  virtual Bool_t OnPause(CAnalyzer& rAnalyzer,
                         CBufferDecoder& rDecoder); // Pause Run.
  virtual Bool_t OnResume(CAnalyzer& rAnalyzer,
                          CBufferDecoder& rDecoder); // Resume Run.
  virtual Bool_t OnOther(UInt_t nType,
                         CAnalyzer& rAnalyzer,
                         CBufferDecoder& rDecoder); // Unrecognized buftype.

  virtual Bool_t OnEventSourceOpen(std::string name);
  virtual Bool_t OnEventSourceEOF();
  virtual Bool_t OnInitialize();

};

#endif

Since our example will show some (possibly trivial) examples of all of the methods an event processor can implement, we needed to show the compiler that we're going to override all possible virtual methods.

Constructor. We could initialize the tree parameters in the constructor, but in order to illustrate OnInitialize we'll do it there. Therefore our constructor is empty:

Example 3-3. Constructor for MyEventProcessor


MyEventProcessor::MyEventProcessor()
{}

OnAttach. OnAttach is called quite early in SpecTcl's initialization. We're going to use it to check an assumption that will be made in operator(), namely that the analyzer is actually an instance of a CTclAnalyzer.

CTclAnalyzer is the analyzer used by SpecTcl for typical analysis cases. It manages the flow of control through the event processing pipeline. It also imposes some requirements on the event processor's operator() that we will talk about when we get to that method's implementation.

Example 3-4. OnAttach


Bool_t
MyEventProcessor::OnAttach(CAnalyzer& rAnalyzer)
{
  try {                                     (1)
    CTclAnalyzer& rTclAnalyzer(dynamic_cast<CTclAnalyzer&>(rAnalyzer));
  }
  catch (std::bad_cast e) {                 (2)
    std::cerr << "MyEventProcessor::OnAttach - not the right type of analyzer ";
    std::cerr << e.what() <<std::endl;
    return kfFALSE;
  }
  catch (...) {                             (3)
    std::cerr << "MyEventProcessor::OnAttach - dynamic_cast failed\n";
    return kfFALSE;
  }
  return kfTRUE;                           (4)

}
(1)
As indicated we're going to check that the analyzer is an instance of CTclAnalyzer. We could compare typenames but, in fact, it's just fine if the analyzer is a subclass of CTclAnalyzer too.

dynamic_cast attempts to perform a type cast between pointers or references of differing actual types of a class hiearchy. If the cast is permissible it succeeds. If not it fails. If casting pointers, the result of a falure is a null pointer. If casting references, as in our code, the result of failure is an exception of type std::bad_cast.

(2)
If a bad cast exception was thrown, this block of code emits an error message that includes the error message stored in the std::bad_cast object (accessible via the what method).

Since this element of the pipeline failed, kfFalse is returned.

(3)
If the cast throws any other exceptions, these two are caught and reported.
(4)
If no exceptions were thrown, the dynamic_cast worked and we can return kfTRUE to indicate to the analyzer it can continue processing with the next element of the event processing pipeline.

OnInitialize. We will use OnInitialize to intialize our tree parameter members. Tree parameters and tree parameter arrays an be constructed with initialization or constructed with e.g. a default constructor and then initialized later.

At some point in the life of a tree parameter it must be bound to an underlying SpecTcl parameter. You can think of this as a process that makes a correspondence between the tree parameter and a slot number in the CEvent array like object passed to operator().

In the code that follows, we don't make any assumptions about whether SpecTcl has already done its mass binding of Tree parameters during its initialization.

Example 3-5. OnInitialize - Set up the tree parameters.


Bool_t
MyEventProcessor::OnInitialize()
{
  sum.Initialize("myevp.sum", 8192, 0.0, 8199.0, "arbitrary");  (1)
  raw.Initialize("myevp.raw", 4096, 0.0, 4096.0, "arbitrary", 16, 0); 

  if (!sum.isBound()) {                                 (2)
    sum.Bind();
  }
  for (int i = 0; i < 16; i++) {
    if (!raw[i].isBound()) {
      raw[i].Bind();                                  (3)
    }
  }
  return kfTRUE;                                    (4)
}
(1)
This pair of statements completes the initialization of the tree parameter and array characteristics. The array has 16 elements whose values are in the range [0, 4096). The single parameter has a range of [0, 8192).
(2)
This section of code binds the individual parameter to the event array like object if not.
(3)
There are no aggregate isBound or Bind methods for tree parameter arrays at the time this is being written. This section of code iterates over the elements of the array (which are after all tree parameters). All unbound parameters are bound.

Note that binding parameters if necessary, creates a SpecTcl parameter making the use of the parameter SpecTcl command largely unecessary.

OnBegin, OnEnd, OnPause, OnResume. These methods monitor what are collectively called Run state transitions. We'll emit a message to stdout describing the transition. We are going to use the facilities all buffer decoders must supply to get information about the run.

In our case, we're just going to report the state transition. This allows us to factor out the code for handling a state transition into a utility method. In more realistic cases you can't do this:

Example 3-6. State transition processing

Added to MyEventProcessor.h: 


private:
  Bool_t describeStateTransition(const char* type, CBufferDecoder& rDecoder);

The utility method implementation in MyEventprocessor.cpp: 


Bool_t
MyEventProcessor::describeStateTransition(const char* type, CBufferDecoder& rDecoder)
{
  UInt_t runNumber = rDecoder.getRun();
  std::string title = rDecoder.getTitle();

  std::cerr << "Run number " << runNumber << " just " << type
	    << " title: " << title << std::endl;

  return kfTRUE;
}

Implementation of the Onxxx methods for state transitions: 


Bool_t
MyEventProcessor::OnBegin(CAnalyzer& rAnalyzer,CBufferDecoder& rDecoder)
{
  return describeStateTransition("began", rDecoder);
}
Bool_t
MyEventProcessor::OnEnd(CAnalyzer& rAnalyzer, CBufferDecoder& rDecoder)
{
  return describeStateTransition("ended", rDecoder);
}
Bool_t
MyEventProcessor::OnPause(CAnalyzer& rAnalyzer, CBufferDecoder& rDecoder)
{
  return describeStateTransition("paused", rDecoder);
}
Bool_t
MyEventProcessor::OnResume(CAnalyzer& rAnalyzer, CBufferDecoder& rDecoder)
{
  return describeStateTransition("resumed", rDecoder);
}

The describeStateTransition utility illustrates using a pair of the buffer decoder service methods to obtain the run title and run number. Each specific state transition handler then passes in the text describiung the state transition that actually occured so that it can be plugged into the method.

We push the choice of whether or not to return kfTRUE or kfFALSE to the utility as well.

operator(). This method is called once per physics triggered event. This is the only method that needs to know something about the format of data from the experiment. This method has the following responsibilities:

Since operator() depends on the internal format of the event, we need to know what that's going to be for our "experiment" before we can write it. We're going to assume that events consist of the following:

  1. A uint32_t containing the event size. This size will be the number of uint16_t items in the event and will include the count itself (which is two uint16_t items long).

  2. A fixed format block of parameters. There will be at most 16 parameters.

  3. Each parameter will occupy one uint16_t item and fill consecutive elements of a tree parameter array named myevp.raw.

  4. If elements 0 and 3 of myevp.raw have been defined by the event, they will be summed and stored in myevp.sum.

  5. The event size in bytes will be passed to the analyzer.

NoteNOTE
 

The code is going to assume that the data are being analyzed on a system with the same byte ordering as the system on which the data were acquired. If that assumption is not valid, production code should use the Translating pointer classes described in the programmer reference rather than raw pointers. Those pointers use byte order information in NSCLDAQ data to transparently perform any ordering transformations required.

Example 3-7. Event processor operator() method.


Bool_t
MyEventProcessor::operator()(
    const Address_t pEvent, CEvent& rEvent,
    CAnalyzer& rAnalyzer, CBufferDecoder& rDecoder
)
{
  uint32_t* pSize = reinterpret_cast<uint32_t*>(pEvent);
  uint32_t  nWords = *pSize++;
  CTclAnalyzer& rTclAnalyzer = dynamic_cast<CTclAnalyzer&>(rAnalyzer); (1)
  rTclAnalyzer.SetEventSize(nWords * sizeof(uint16_t));

  uint16_t* pData = reinterpret_cast<uint16_t*>(pSize);   (2)
  nWords         -= sizeof(uint32_t)/sizeof(uint16_t);

  for (int i = 0; i < nWords; i++) {
    raw[i] = *pData++;                                         (3)
  }

  if (raw[0].isValid() && raw[3].isValid()) {          (4)
    sum = raw[0] + raw[3];
  }


  return kfTRUE;
}
(2)
The pEvent is an Address_t type. This is an opaque pointer (void*). In order to use it it must first be cast to a pointer to a specific type.

This section of code casts it to point to an unsigned 32 bit integer and uses the resulting pointer to extract the event size in uint16_t units. We then fulfil our obligation to tell the CTclAnalyzer the event size so that, if necessary, it can locate the next event in the block of events its iterating over.

(2)
pSize now points to the data in the event, but we need a pointer to the uint16_t items we want to extract. The cast takes care of this. The number of words of data is also computed from the event size. The expression sizeof(uint32_t)/sizeof(uint16_t) is the number of 16 bit words required to store a 32 bit word. We could use the value 2, but this expression makes it a bit clearer what that 2 means.
(3)
The body of this loop stores each parameter into the appropriate tree parameter arrary element. The tree parameter array has been bound to a set of slots in rEvent by SpecTcl and takes care of setting the appropriate elements of that array-like object for us.

Note that each assignment also transparently sets that element's validity so that it's isValid method will return true.

(4)
The sum an only be computed if both of the parameters it depends on have been given values in this event. It's a common problem in SpecTcl event processors not to perform this sort of check. If we attemted to perform this sum and either addend had not been given a value, the reference to the un-set value would result in an exception.

Uncaught exceptions in SpecTcl event processors, obviously abort that pipeline element, but they also abort the remainder of the pipeline (if any).

OnOther. This method is called when something other than the named item types is encountered. You will need to know something about the format of each item type you'd like to handle. You will also need to interact with the buffer decoder to obtain pointers to the actual data.

In our example we will:

Example 3-8. Event processor OnOther


Bool_t
MyEventProcessor::OnOther(UInt_t nType, CAnalyzer& rAna, CBufferDecoder& rDecoder)
{
  std::cout << "Got an 'OnOther' item type is: " << nType << std::endl;

  CRingBufferDecoder* pRingDecoder = dynamic_cast<CRingBufferDecoder*>(&rDecoder); (1)


  if (pRingDecoder) {
    if (nType == PERIODIC_SCALERS) {                       (2)
      CRingItem* pItem = CRingItemFactory::createRingItem(pRingDecoder->getItemPointer());  (3)
      CRingScalerItem* pScaler = dynamic_cast<CRingScalerItem*>(pItem);                  (4)
      if (pScaler) {
        std::cout << "Scaler item: \n";
        std::cout << pItem->toString() << std::endl;                               (5)
      } else {
        std::cerr << "OnOther - type was a scaler but could not make the ring item\n";
      }
      delete pItem;                                            (6)
    }
  } else {
    std::cerr << "Data does not come from nscldaq 10+ - can't do any more\n";
  }

  return kfTRUE;                                          (7)
}
(1)
In order to be able to do all of the processing we want to do, we need to invoke members of the decoder that are specific to the decoder for ring buffers; CRingBufferDecoder. This dynamic cast takes the generic decoder and attempts to coerce its address into a pointer to a CRinBufferDecoder.

Dynamic casts make use of run time type information (RTTI) embedded in objects to ensure that the cast being attempted is legal. If the rDecoder object actual type cannot be treated as a CRingBufferDecoder, the cast returns a null pointer.

If a null pointer is returned, a message indicating the data doesn't come from a ring buffer based system is regturned.

(2)
We only want to do something other than to output a message if the data is a periodic scaler item. Note that this also matches the data type for nscldaq-10.x incremental scalers.
(3)
The CRingItemFactory is a class with several static methods that create ring item objects. The version of the creational method we use takes a pointer to the raw data in a ring item. The CRingBufferDecoder::getItemPointer method returns a pointer to the ring item the buffer decoder is processing, has handed off to the analyzer which in turn invoked our OnOther method.

The factory gives us a pointer to a dynamically allocated CRingItem, though the actual underlying object is of the appropriate ring item class type.

(4)
This dynamic cast turns the CRingItem object into a CRingScalerItem pointer. We know that's what we should get by our analysis of the ring item type.

Although we should get a CRingScalerItem, we ensure that this is the case. The dynamic_cast operator will return a null pointer if the cast fails.

(5)
Once we've gotten this far we can use the toString method every CRingItem derived class implements to turn the ring item into a human readable dump string which we print out.
(6)
Since the factory dynamically allocated the ring item, we need to delete it in all code paths through the method.
(7)
Since this is an event processor method, we need to return kfTRUE so that other pipeline elements that follow ours will be run as well.

The OnEventSourceOpen and OnEventSourceEOF. These two methods are called, when SpecTcl is connected to a new event source (via the attach command), and when an end file indication is encountered on the active event source. In our event processor, we'll just output some text to indicate these events have occured.

Example 3-9. OnEventSourceOpen and OnEventSourceEOF


Bool_t
MyEventProcessor::OnEventSourceOpen(std::string name)
{
  std::cout << "SpecTcl has connected to a new event source : " << name << std::endl;
  return kfTRUE;
}
Bool_t
MyEventProcessor::OnEventSourceEOF()
{
  std::cout << "SpecTcl's event source processing reached and end-file\n";
}

Notes

[1]

Well for ringbuffer data this is actually a lie since blocks of events contain exactly one event and the size of that event is known from the ring item size. For nscldaq-8.0 and earlier data this is required.

For data from non NSCLDAQ systems, the necessity of this depends on the system.

PrevHomeNext
The SpecTcl processing model. Setting up an event analysis pipeline (the simple version)