Data Flow Network Syntax Specification version 1.0

Written by Andre Charbonneau
    Revision Dominic Letourneau, Jean-Marc Valin
August 17 1999


Contents

Adding comments to a Data Flow network text file.
The Data Flow network block.
Populating a Data Flow network block with nodes.
Adding nodes to a Data Flow network.
Setting the type of a node in a Data Flow network.
Setting the parameters of a node.
The 13 different parameter value's type.
How to use environment variables
Passing parameters to subnets.
Using external parameters.
Setting the input and output nodes of a Data Flow network.
Data Flow subnets.
Data Flow Iterators.


This document describes in details the syntax for writting Data Flow Network files. Data Flow Network files are simple text files that the FileParser object will parse to create data structures that will be used by the NetworkBuilder to acctually build the Data Flow network.  There can exist more than one Data Flow network in a single text file and some of them can be included into other Data Flow network as what we will call 'subnets.'. This 'subnets' feature allows a Data Flow network to be reused.

The Data Flow network text file format is very flexible.  By 'flexible' is meant that space characters, tabs, newlines can be inserted anywhere in the Data Flow network text file without affecting the final network defitions. Lets start by the syntax for defining a Data Flow network.

Adding comments to a Data Flow network text file

Comments can be added to Data Flow network text file at will.  The comment format follows the one used by C programmers: /* Comment goes here */.  Anything in the Data Flow network and will be ignored by the Data Flow network file parser.  Be careful though not to have nested comments because this can lead to undesired results.

The Data Flow network block.

Each network in the Data Flow network text file will appear in what we will call a 'network block'.  A network block starts by the network:keyword.  The name given to Data Flow network will appear after this keyword.  Make sure that all network names are unique.  Following the network name will be the 'body' of the network enclosed between '<' and '>' brackets.  What will appear between those brackets will describe each node contained in this particular network.  Here is an example of a Data Flow network block:
 
/* This is a sample Data Flow network text file containing an empty net block. */

Network: Network1
{
        /* node information will go here... */
}

Note that a single Data Flow network text file can contain more than one network block, as long as they each have a unique name.

Populating a Data Flow network block with nodes.

Inside the Data Flow network block will appear lines of the following format:

<keyword: rightside>

These strings enclosed between '<' and '>' are called node blocks and will appear only inside network blocks.  Node blocks are used to specify all the information about the nodes such as the node's name, type, input variables and so on...  The following sections will describe how to write the node blocks for each possible node attribute.

Adding nodes to a Data Flow network.

Adding a node to a Data Flow network block is done with a node block having name: as the keyword and then followed by the new node's name on the rightside.  Lets call this special type of node block a node creation block.  Any node block (that is not a node creation block) that appears after a node creation block will have effect on the node which is defined by the closest node creation block above.  Here is an example to clear things up:
 
 

/* Sample Data Flow network text file. */

Network: SAMPLE_NET
{
        <node: NODE1>
        /* Anything here will relate to NODE1 only. */

        <node: NODE2>
        /* Anything here will relate to NODE2 only. */

        <node: NODE3>
        /* Anything here will relate to NODE3 only. */

}

In the above example, 3 nodes (NODE1, NODE2, NODE3) where created inside the Data Flow network named SAMPLE_NET.  Note that the identation in the above example is purely arbitrary.  The following Data Flow text will have the same effect as the above one, although it might be a bit more difficult to read by a user. Again note that each node within a Data Flow network block must each have its own unique name.
 

Setting the type of a node in a Data Flow network.

Each node in a Data Flow network must be given a type.  The node block used to specify the type of a node is formatted the following way:

<type: the_type_of_the_node_goes_here>

In this case the keyword of the node block is type: and the rightside is the type of the node.  Here is an example:
 

/* Sample Data Flow network text file. */

Network: SAMPLE_NET
{
  <node: NODE1> <type: AND>
  /* Other stuff for NODE1 will go here... */

  <node: NODE2> <type: PROC1>
  /* Other stuff for NODE2 will go here... */
}
 

Setting the parameters of a node.

Each node in a Data Flow network can have 0, 1 or more parameters.  Each parameter for a node is defined with a node block of the type:

<param: the_param_name,param_type:the_param_value>

In this case the keyword of the node block is param: and the rightside contains the name of the parameter variable, its type and its initial value, seperated by a comma and a ":", respectively.  Both the name and its value must appear in each parameter definition node block.  If a node has more than one parameter, then there will be one parameter node block for each parameter.  Make sure that no two parameter names are identical.  Here is an example of a Data Flow network with 2 nodes, the first one with a single boolean parameter initialized to true and the second node having two parameters of numeric type.
 

/* Sample Data Flow Network text file. */

Network: SAMPLE_NET
{
  <node: NODE1> <type: TYPE1>
  <param: BOOL1, bool:true>

  <node: NODE2> <type: TYPE2>
  <param: NUM_VAR1, float:10.0>
  <param: NUM_VAR2, 20>
}
 

The 13 different parameter value's type.

The value of a node's parameter can be of 13 types: char, unsigned char, int, unsigned int, short, float, double, long, unsigned long, string, boolean, env, subnet_param, ext.

Parameter values are preceded by its type keyword.  If there is no type keyword preceding the parameter's value, then for numerical values, int or float will be used, depending if the value contains a decimal point or not, and for non-numerical values, string will be used for anything that is double quoted and char will be used for single quoted characters.  Boolean are written as true or false and the case must be respected.  Writting down True, TRUE or TrUe will not work.

Here is a table representing the type keywords to use for each parameter variable type:
 
 
Parameter keyword  Meaning
int: Int (int)
short: Short (short)
long: Long (long)
char: Char (char)
u_char: U_Char (unsigned char)
u_int: U_Int (unsigned int)
u_long U_Long (unsigned long)
bool: Bool (bool)
float: Float (float)
double: Double (double)
string: String (string)
ext: external parameter specified by the program to the builder
subnet_param: parameter taken from the network parameters
env: string from an environment variable
... ...

How to use environment variables

Environment variables can be used in a node as a string parameter which value is set to an environment variable at run-time.  To use environment variable simply create a parameter of type env with its value set to the name of the environment variable.  At run-time, the env: ENV_VAR_NAME pair will be replaced by string: "the_value_of_the_environment_variable_ENV_VAR_NAME".

Here is an example of a node containing an environment variable:
 
<node: NODE1> <type: A_NODE_TYPE>
<param: PATH_PARAM, env:PATH>

In the above example, the line <param: PATH_PARAM, env:PATH> will be replaced (at run-time) by: <param: PATH_PARAM, string:"./:/opt/bin/:/home/username/.....">

Another way of using environment variable is to use the $(VAR) syntax. This feature allows the user to include environment variables anywhere into the description file of the network.

Here is an example of a node containing this type of environment variable:
 
<node: NODE1> <type: A_NODE_TYPE>
<param: PATH_PARAM, "$(MY_ENV_PATH)/script_to_run.csh">

Note: Make sure the environment variable exists, if not, it will cause an exception to be thrown.

Passing parameters to subnets

When including a subnet into a data flow network parameters can be passed to the subnet.  With this feature, a subnet can be 'parameterized' and be used in more than one data flow network without having to write several versions of the same subnet.  A parameterized subnet will have parameters of the following type:

<param: variable_name, subnet_param: parameterized_variable_name>

To better understand this feature, here is an example of a simple data flow network that includes a parameterized subnet:
 
 

Network: A_DATA_FLOW_NET
{

        ...
 

        <node: FILENAME_NODE> <type: CONST>
                <param: VALUE, "/home/a_user/a_file.dat">

        <node: PROCESSING_NODE> <type: LOADING_NET>
                <param: PARAM1, float:180.0>
                <input: FILENAME, LOADER_NODE, OUTPUT>

        ...

}

/* Parameterized subnet */
Network: LOADING_NET
{
        <node: LOADER_NODE> <type: LOADER>

        <node: PROC1_NODE> <type: PROC1>
        <param: VAL1, subnet_param: PARAM1>
        <input: INPUT, LOADER_NODE, OUTPUT>
 

        <netInput: LOADER_NODE>
        <netOutput: PROC1_NODE>

}

 


 

At run-time, the above subnet example will be translated the following way:
 
Network: A_DATA_FLOW_NET
{

        ...
 

        <node: FILENAME_NODE> <type: CONST>
        <param: VALUE, "/home/a_user/a_file.dat">

        <node: PROCESSING_NODE> <type: LOADING_NET__1>
        <param: PARAM1, float:180.0>
        <input: FILENAME, LOADER_NODE, OUTPUT>

        ...

}

/* Parameterized subnet */
Network: LOADING_NET__1
{
        <node: LOADER_NODE> <type: LOADER>

        <node: PROC1_NODE> <type: PROC1>
        <param: VAL1, float:180.0>
        <input: INPUT, LOADER_NODE, OUTPUT>
 

        <netInput: LOADER_NODE>
        <netOutput: PROC1_NODE>

}
 

We can see that a "copy" of the original parameterized subnet is made with the appropriate changes made to the node parameters.  A unique name is given to this copy, which is this case is "LOADING_NET__1.  This "copy creation" process is completely transparent to the user of the data flow network library and is illustrated here only to give the user a general idea of how parameterization of subnets is done.

Using external parameters.


External parameters are node parameters that are set at run-time to a value and type both defined in the client application.  An example of where this feature can be very useful is for an application that needs to pass command-line arguments to the node of its data flow network.  To add an external parameter to a node, the following syntax is used:
 

<param: param_name, ext: external_param_name>

Here is the previous example but now using an external parameter for the filename:
 

Network: A_DATA_FLOW_NET
{

        ...
 

        <node: FILENAME_NODE> <type: CONST>
                <param: VALUE, ext:EXTERNAL_FILENAME>

        <node: PROCESSING_NODE> <type: LOADING_NET>
                <param: PARAM1, float:180.0>
                <input: FILENAME, LOADER_NODE, OUTPUT>

        ...

}

/* Parameterized subnet */
Network: LOADING_NET
{
        <node: LOADER_NODE> <type: LOADER>

        <node: PROC1_NODE> <type: PROC1>
        <param: VAL1, subnet_param: PARAM1>
        <input: INPUT, LOADER_NODE, OUTPUT>
 

        <netInput: LOADER_NODE>
        <netOutput: PROC1_NODE>

}

 

In the above example, the VALUE parameter of the FILENAME_NODE node will be set at runtime.  The way external variables are accessed is throughout a ParameterSet object, constructed in the client application, that is passed by pointer to the NetworkBuilder object when calling the 'build' method.  Here is an example:
 

int main()
{
        NeworkBuilder nb;
        ParameterSet ps;

        ...

        ps.add("EXTERNAL_FILENAME",ObjectRef(new String("/home/a_user/a_file.dat")));

        ...

        nb.build(net_info, "A_DATA_FLOW_NET", &ps);

        ...

        return 0;
}
 


 

In the above example, the value of the filename node's parameter will be set to "/home/a_user/a_file.dat" at run-time when the data flow network is built.

Setting the input variables of a node.


Setting the input variables of a Data Flow network node is very similar to setting the parameters.  To set the inputs of a node, a node block of the following format is used:
 

<input: input_var_name, source_node, source_node_output_var>
 

Here the node block key is input: and the right side contains 3 fields, seperated by commas.  The first field, input_var_name, is the name to give to the input variable.  The second field, source_node, is the name of the node to read the input from... this is why its called the 'source' node.  The third field, source_node_output_var, is the name of the variable from the source node to accept as input.  For example, suppose that our Data Flow network contains 2 nodes named NODE1 and NODE2.  NODE1 has 2 output variables named OUT1 and OUT2 and NODE1 has 1 input variable named IN1 that will be connected to NODE1's output variable named OUT1. Here is the RGB text that will accomplish this:
 
 
Network: SAMPLE_NET
{
  <node: NODE1>
    <type: TYPE1>
 

  <node: NODE2>
    <type: TYPE2>
    <input: IN1, NODE1, OUT1>

}


 
 

Note that the output variable names are not explicitly specified in the Data Flow text file.  The name of the output variables are defined in the actuall C/C++ code that implements the functionallity of each node.  A node can have more than one input variable, and these may also come from different nodes.  It's up to the user to make sure that all the variable names correspond to the actual implementation of each node type.

Setting the input and output nodes of a Data Flow network.

There is a way of explicitly specifying the input and output nodes of a Data Flow network.  This is necessary when a network can possibly be used as a subnet in another Data Flow network.  There exist two network attribute blocks (network attribute blocks because they relate to the network itself, not to the node's attributes) which have the following format:

<netInput: input_node_name>
<netOutput: output_node_name>

The first block with key netInput: is used to set the input node of the Data Flow network.  The second one, with key netOutput: is used to set the output node of the network.  For the 2 blocks, the right side is the name of the node to be set as input or output for the Data Flow network.

Data Flow subnets.

To allow reuse of Data Flow network code the Data Flow network text file syntax includes the notion of a 'Data Flow subnet'.  A Data Flow subnet is simply a net that is inserted into another network as a node.  This way, a network that is used often can be written only once and included in one or more other networks.  Also, this 'subnet' feature allows complex Data Flow networks to be broken down into smaller and simplier networks.

To include a subnet into a Data Flow network simply use the name of the subnet as the type of the node where the subnet should be inserted.  A net that can be included as a subnet must have its input and output node explicitly specified in the Data Flow network text file using the <netInput: ...> and <netOutput: ...> network attribute blocks.

Important: A Data Flow network does not support cycles in the net inclusion (subnets) so make sure that there are no cycles in the inclusion of subnets. Here is an example of Data Flow network that includes a simple 2 node subnet:
 
Network: BIGNET
{
   <node: AUDIO> <type: AUDIOFILE>
   <param: FILENAME, string:"/home/aUser/test.au">

   <node: FRAME> <type: FRAMING>
   <input: INPUT, AUDIO, OUTPUT>
   <param: LENGTH, int:256>

   /* Subnet node */
   <node: HANNING> <type: WINDOW>
   <input: INPUT1, FRAME, OUTPUT>

   <node: FFT> <type: FFT>
   <input: INPUT, HANNING, OUTPUT>
   <param: LENGTH, int:256>

   <node: PS> <type: POWERSPECTRUM>
   <input: INPUT, FFT, OUTPUT>
   <param: LENGTH, int:256>

   <node: FILTER> <type: FILTER1>
   <input: INPUT, PS, OUTPUT>

   <node: COND> <type: CONST>
   <param: VALUE, bool:true>

   <node: SWITCH> <type: SWITCH>
   <input: DATA, FILTER, OUTPUT>
   <input: CONDITION, COND, OUTPUT>

   <netOutput: SWITCH>
}

Network: WINDOW
{
   <node: HANNING> <type: WINDOW>
   <param: LENGTH, int:256>
   <param: TYPE, string:"HANNING">

   <node: WINDOW> <type: PRODUCT>

   /* INPUT1 is not specified here because it will get it from the 'wrapping' node. */
   <input: INPUT2, HANNING, OUTPUT>
   <param: LENGTH, int:256>

   <netOutput: WINDOW>
   <netInput: WINDOW>
}
 

Data Flow Iterators


In the data flow network there exists a special network that acts like a loop (while, for, etc.) and is specified by the Iterator: keyword. An Iterator Network must have the regular netInput and netOutput nodes plus the netCondition that enables us to verify if a certain condition holds. When this condition is false, the iteration stops and the execution continues at the output of the iterator network. You can specify the network parameter DOWHILE if you want the condition to be tested after each iteration.
 
 
Iterator: MY_ITERATOR_NET
{
<node: NODE1> <type: TYPE1>

<node: NODE2> <type: TYPE2>
<input: NODE1, INPUT, OUTPUT>

/* We are checking for the validity of the output of NODE1 */
<node: CONDITION_NODE> <type: COND1>
<input: NODE1, INPUT, OUTPUT>

<netInput: NODE1>
<netOutput: NODE2>
<netCondition: CONDITION_NODE>

}