46.1.1. The containers package.

The containers package is provided to manipulate the definitions of containers and to start and stop persistent containers defined in a configuration database. A common parameter to many of the exported procs is an sqlite3 database command. The example below shows what you need to do to create this command.

package require sqlite3
...

sqlite3 db mydbfile.db
            

This code snippet pulls in the sqlite3 Tcl package and later uses the sqlite3 command to associate the command db with the database file mydbfile.db. The sqlite3 command, by default will create a new database file, however since the database must have a specific schema before it can be used by this package you should not rely on this capability but use mg_mkconfig to create configuration databases instead.

The remainder of this section will briefly describe the capability of the package along with examples. Reference material on the package can be found at: containers.

Containers are represented to have the following attributes:

Containers can be created, removed, listed, tested for, activated, and deactivated. A program can be run in an activated container.

Here's an example of a typical container creation we might have at the FRIB. We want a container that will run the image /user/opt/buster.img and, for each program run in it will have the environment variables set up for NSCLDAQ-12.0. This will require an initialization script (to set up those variables) and the following code:

#---------------- Contents of ~/daq12.0.sh   --------------- 
#!/bin/bash
                                      
. /usr/opt/daq/12.0/daqsetup.bash
#------------------------------------------------------------
#------------------------ contents of mkcontainr.tcl -----------

lappend auto_path $env(DAQTCLLIBS)  

package require sqlite3             
package require containers

sqlite3 db myconfig.db              

   
container::add db buster /usr/opt/buster.img ~/daq12.0.sh \
    [list [list /usr/opt/opt-buster /usr/opt]]
    
db close                         
            

Let's pick this apart.

Defining a container does nothing more than that. To use the containerized environment it defines, you must activate the container and subsequently run programs in that container. You can then shutdown a container by deactivating it. The example below makes a ringbuffer in the system spdaq99 named fox. We override the default ring buffer data size and create a 16Mbyte ringbuffers.

lappend auto_path $env(DAQTCLLIBS)  

package require sqlite3             
package require containers

sqlite3 db myconfig.db

set containerFd [container::activate db buster spdaq99]  

set programFd   [container::run buster spdaq99 \        
     {$DAQBIN/ringbuffer create fox --datasize=16m}]
puts [gets $programFd]                                  
close $programFd

container::deactivate spdaq99 buseter               
close $containerfd                                  

            

Note that the package's ideas about which containers are active depend on internal data rather than any system information. Therefore If you activate a container in one program and then run a program in it in another, you may fail.

Note also that activating a container and running programs in it, depends on being able to run the ssh command. It is therefore important that your ~/.ssh/authorized_keys file has a public key for your ssh identity. See the results of google searches for SSH without password for information about how to set this up.

46.1.2. The programs package

The programs package provides support for making and manipulating program definitions, for running and killing programs in hosts (either in or without containers), and for capturing program output. The package, if needed, activates containers using the containers package we just described to run programs within a containerized environment.

Reference material on the programs package can be found at: programs

A program is defined by the following mandatory data:

Several optional bits of data can be associated with a program and define the environment in which the program runs. Not all of these items are used at this implementation of the system.

These bits of optional data are used to construct the command used to run the program.

Note that since many of these items are used to construct a script to run the command environment substitutions are supported e.g. if daqsetup has been run in the context in which the program was run $DAQBIN can be used to specify the path to an NSCLDAQ program.

Let's look at some of the operations you can perform on programs. We're going to assume that there's a container defined named buster whose initialization script runs daqsetup.bash.

lappend auto_path $env(DAQTCLLIBS)    
package require sqlite3               
package require programs

sqlite3 db myconfig.db                

program::add db readcrate1 \$DAQBIN/VMUSBReadout Critical spdaq99 \
     [dict create                                                  \
        container buster                                           \
        options [list {--daqconfig ~/spdaq99/daqconfig.tcl}        \ 
           {--ctlconfig /dev/null} {--ring=crate1} {--sourceid 99} \
           {--initscript ~/spdaq99/startrest.tcl}                   \
        ]                                                          \
        environment {TCLLIBPATH $DAQTCLLIBS}                       \
        service     ReadoutREST                                    \
     ]

db close


            

The contents of the ~/spdaq99/startrest.tcl script are simple:

package require ReadoutREST
close stdin
            

The example below lists the name of each program, where it runs and the image it runs. See programs for a full description of the dict used to describe program definitions.

lappend auto_path $env(DAQTCLLIBS)
package require sqlite3
pacakge require programs

sqlite3 db myconfig.db

set inf [program::listDefinitions db]
foreach p $inf {
   set name [dict get $p name]
   set host [dict get $p host]
   set path [dict get $p path]
   puts "$name runs $path in $host"
}
            

This example should be relatively self explanatory listDefinitions only requires a database command ensemble connected to a properly formatted database.

program::run establishes an output handler for the pipe. Optionally the caller can supply an output handler as well. Regardless, it's important, for those output handlers, that at some point an event loop is entered to allow those file handlers to be dispatched.

In the example below, we run two programs: readcrate1 and unimportant. We'll catch and output the output/error of readcrate1 and let the default handler take care of unimportant.

When readcrate1 exits (detected by an eof on the pipe), we'll exit as well.

lappend auto_path $env(DAQTCLLIBS)
package require sqlite3
package require programs

sqlite3 db myconfig.db

set done 0
proc ReadCrate1OutputHandler {name fd} {                
   if {![eof $fd]} {
      puts "$name: [gets $fd]"
   } else {
      incr ::done
   }
}

program::run db readcrate1 ReadCrateOutputHandler      
set fd [program::run db unimportant]

vwait done                                             

if {[program::isActive unimportant]} {
   program::kill db unimportant                       
   catch [close $fd]
}

db close
            

A note about program::kill. This operates by locating the program image in the remote system and using ssh to issue a kill command. Sadly, there are cases where this is unsufficient and I don't know how to help that. Specifically, if the image is a script that runs a program, even though the script gets killed, the program my continue to run.

46.1.3. The sequence package

The sequence package provides support for two concepts within the manager:

1. The manager implements a state machine. While there is a default state machine loaded into the manager it is possible to define any state machine as long as its initial state is called SHUTDOWN and any state can transition to SHUTDOWN

2. The manager associates sequences to transitions into a specific state. A sequence is a list of programs (see previous section) that are run to properly transition to that state.

Reference information on the sequence package is at sequence

The remainder of this section will describe:

• What a state machine is, and the default state machine that's implemented when a configuration database is created. This is described in State Machines, and the Default State Machine

• Parts of the sequence package that can be used to create other state machines than the default state machine. Note that in most cases it is not necessary to create a custom state machine. This is described in The State machine API

• The Sequence API sequences and the API section of the sequence package that manage sequences.

• The Transition API describes transitions and the parts of the API that manage them.

• Miscellaneous API Entries. describes miscellaneous parts of the API that don't neatly fit into any of these categories.

lappend auto_path $env(DAQTCLLIBS)
package require sqlite3
package require sequence                       

sqlite3 db myconfig.db

sequence::newState db PAUSED                 
sequence::newTransition db BEGIN PAUSED
sequence::newTransition db PAUSED END        
sequence::newTransition db PAUSED BEGIN
sequence::newTransition db PAUSED SHUTDOWN

db close

               

lappend auto_path $env(DAQTCLLIBS)
package require sequence
package require sqlite3

sqlite3 db myconfig.db

sequence::add db InitiateDataFlow BOOT  

::sequence::addStep db InitiateDataFlow readout1
::sequence::addStep db InitiateDataFlow readout2    
::sequence::addStep db InitiateDataFlow eventbuilder 0 2 
::sequence::addStep db InitiateDataFlow feeder1
::sequence::addStep db InitiateDataFlow feeder2

db close
               

lappend auto_path $env(DAQTCLLIBS)
package require sqlite3
package require sequence

sqlite3 db myconfig.db

set status ""

proc completion {db manager completionStatus} {   
   set ::status $completionStatus
}

if {[::sequence::currentState db] eq "SHUTDOWN"} {  
    ::sequence::transition db BOOT completion       
    vwait status                                    
    puts "Transition to BOOT completed in $status
} else {
    puts "Not in SHUTDOWN state."
}
db close

               

...
package require sequence
package require snit       
...

set manager [::sequence::getCurrentTransition]   
if {$manager ne ""} {
   set db [$manager cget -database]
   set t  [$manager cget -type]
   set allSequences [::sequence::listSequences $db]   
   foreach seq $allSequences {
      set name [dict get $seq transition_name]
       if {[dict get $seq transition_name] eq $t} {
         set steps [::sequence::listSteps $db $name]  
         foreach step $steps {
            set aMonitor [MyMonitor %AUTO% \     
              -database $db -sequence $seq -step $stepno \
            ]                          
            set stepno [dict get $step step]
            ::sequence::addMonitor $db $name $stepno $aMonitor 
         }
       }
   }
   
}
                           
snit::type MyMonitor {
   option -step
   option -database
   option -sequence
   constructor args { $self configurelist $args}
   
   method onOutput  {db program fd} {  
      set line [gets $fd]
      set name [dict get $program name]
      ::sequence::relayOutput "Step $options(-step); $name: $line" 
   }
   method onExit {program fd} {     
      ::sequence::relayOutput "[dict get $program name] exited!!"
      ::sequence::addMonitor $options(-database) $options(-sequence) \
         $options(-step) ""
      after 0 [mymethod destroy]
   }
}
                  
... # More code that eventually enters the event loop.
               

46.1.4. The kvstore package

A key value store can be thought of as an array whose keys are strings. The manager key value store is string valued. Associated with each key is a string value. A kvstore package provides the abiliy directly access the key value store in the configuration SQLite3 database file.

Using the API, you can create a new key/value pair. Your application can read existing keys and modify their values. Finally, you can also remove an existing key/value pair from the store.

The key value store can be used by you as you wish. However, note that the readout control packages create and use a pair of keys: run and title to hold the run number and title of the next run. As the manager evolves it is possible that additional facilities will create and use other standard key/value pairs.

See kvstore for reference information on the key value store API.

The example below shows a few of the features of the kvstore package. It assumes that the DAQ environment for a version 12.0 or higher NSCLDAQ has been setup or, alternatively that the TCLLIBPATH environment variable includes the TclLibs subdirectory of such a version.

if {[array names env DAQTCLLIBS] ne ""} {
   lappend auto_path $env(DAQTCLLIBS)
}
package require kvstore        
package require sqlite3

sqlite3 db myconfig.db

kvstore::create db mykey myvalue  

kvstore::modify db mykey {A new value} 

puts "Contents of kvstore:"
dict for {key value} [::kvstore::listAll db] {  '
   puts "$key => $value"
}

db close

            

46.1.5. The auth package.

While it is not yet used as of NSCLDAQ-12.0, the manager supports an authorization database. In the future, this will be used to limit the things individual users can request of the manager. Understanding this part of the API and database requires understanding the following basic terms:

Full reference material for the authorization package (auth)) is provided in auth.

The example below adds a new user to the experiment, creates a few roles and grants a few of those roles to the new user.

if {[array  names env DAQTCLLIBS] ne ""} {
   lappend auto_path $env(DAQTCLLIBS)
}
package require auth      
package require sqlite3

sqlite3 db myconfig.db

auth::adduser  db fox    

auth::addrole db manager
auth::addrole db analyzer 
auth::addrole db operator

auth::grant fox operator  

puts "People and the roles they have:"

dict for {person roles} [auth::listAll db] {     
   puts "$person has been granted [join $roles ", "]
}

db close

            

46.1.6. The eventloggers Package.

The eventloggers package provides the ability to define and control event loggers. Event logger definitions in the manager configuration database subsume all of the capabilities of the NSCLDAQ-11 ReadoutShell's multilogger and primary event logger.

An arbitrary number of event loggers of both types can be defined, enabled, disabled, and marked as critical. If a critical logger fails, a SHUTDOWN state transition is forced. In this sense, this package is dependent on the sequence package.

As you build the set of event loggers you will use, be careful to consider the bandwidth required to run them.

Event loggers have the following attributes:

Destination directories, and data source URI's must be unique.

Full loggers, on the other hand, behave like the primary event logger in the NSCLDAQ ReadoutShell. The destination directory is the top level directory of the directory tree managed by the logger and its wrapper. The structure of this tree is shown below. The intent is to provide an experiment view and a per run view. The experiment view provides access to all event files while the per run view also provides access to associated run metadata stored in the experiment/current subdirectory when the run ended.

The example below illustrates some of the simple capabilities of the eventloggers package. Full reference documentation can be found at eventloggers

if {[array names env DAQTCLLIBS]} {
   lappend auto_path $env(DAQTCLLIBS)
}

package require eventloggers
package require sqlite3     

sqlite3 db myconfig.db

set dest [file normalize ~/stagearea/evlog1] 
file mkdir $dest

set id [::eventlog::add db /usr/opt/daq/12.0-000 \ 
  tcp://spdaq99/fox $dest                        \
  [dict create host evloghost critical 1 enabled 1 container buster] \
]

set killdest [file normalize ~/stagearea/evlog2]

                    

foreach logger [eventlog::listLoggers db] {     
    if {[file normalize[dict get $logger destination]] eq $killdest} {
      eventlog::rm db [dict get $logger id]     
    }
}


db close