Multiple Stream Input

Multiple stream input works as follows:-

1. The input processor uses EIO_EVREAD to read events. When it is called for the first time, it reads the first events for all defined input streams and places them in an input buffer (it is the ZEBRA structural link LEIO_I).

2. A selection process decides which event in the buffer to process next and takes it from the input buffer and defines it to be the current event (by hanging it on the ZEBRA structural link LMAST). At the same time, it is possible that other, unwanted events, are removed from the input buffer.

3. Event processing and output proceeds as required.

4. The current event is removed from memory.

5. On the next call to EIO_EVREAD the input buffer is replenished so that it again holds an event for every input stream and the process repeats.

Warning: Multiple stream input carries a performance penality. With a single input stream, dropping the event after processing is very fast as it involves little more than reseting a few pointers (MZWIPE). However, if the input buffer is not empty, this option is not available, and so the current event is dropped and a garbage collection (MZGARB) is used to win back space. So, multiple input streams require:-

• More memory (there is a copy of an event of every stream in memory at once).

• More CPU (it takes longer to garbage collect than to wipe and event output involves writing a selected structure rather than a complete division).

The way events are selected from the input buffer is determined by a word in the TEIO bank that can be modified by the $MULTI_INPUT_SEL symbolic command e.g.:-

$multi_input_sel $merge_all

The following input selection schemes are currently available:-

$merge_all

The input files are all assumed to be ordered by universal date time. The selection scheme merges the streams together so as to retain this ordering. Should two or more streams have indentical date times, they will all be processed, in stream order number. Processing continues until all streams are exhausted; only then is an EOF generated.

$merge_first

This is identical to $MERGE_ALL except that in the case of two or more events on different streams having the same date time, only the one from the lowest stream number is kept; the others are discarded. Note: this scheme does not discard two events with the same date and time if they appear on the same event stream.

$by_weight

This scheme can be used to mix streams randomly according to a set of associated weights. The following rules apply:-

• Each stream should have a WEIGHT option. If omitted, WEIGHT=1. is assumed.

• Should multiple files be concatenated to a single stream, the WEIGHT option from the first file is used on all subsequent files; any further WEIGHT options are ignored.

• Events are selected randomly according to their relative weights.

• As soon as any stream is exhausted, an EOF is generated, as to continue would bias the mixture in favour of larger event files.

$user_sel

In this case the routine EIO_I_SELECT_USER is called and it is up to the user to nominate the next event to process. An EOF is only generated when all streams are exhausted. However, the user routine can also signal EOF if required.