The same way as the FnReturns can be used to get back the direct results of the operations on the tables, can be also used on the templates in general. Indeed, it's a good idea to have a method that would create an FnReturn in all the templates. So I went ahead and added it to the LookupJoin, JoinTwo and Collapse.
For the joins, the resulting FnReturn has one label "out". It's created similarly to the table's:
my $fret = $join->fnReturn();
And then it can be used as usual. The implementation of this method is fairly simple:
sub fnReturn # (self)
{
my $self = shift;
if (!defined $self->{fret}) {
$self->{fret} = Triceps::FnReturn->new(
name => $self->{name} . ".fret",
labels => [
out => $self->{outputLabel},
],
);
}
return $self->{fret};
}
All this kind of makes the method lookup() of LookupJoin redundant, since now pretty much all the same can be done with the streaming function API, and even better, because it provides the opcodes on rowops, can handle the full processing, and calls the rowops one by one without necessarily creating an array. But it could happen yet that the lookup() has some more convenient uses too, so I didn't remove it yet.
For Collapse the interface is a little more complicated: the FnReturn contains a label for each data set, named the same as the data set. The order of labels follows the order of the data set definitions (though right now it's kind of moot, because only one data set is supported). The implementation is:
sub fnReturn # (self)
{
my $self = shift;
if (!defined $self->{fret}) {
my @labels;
for my $n (@{$self->{dsetnames}}) {
push @labels, $n, $self->{datasets}{$n}{lbOut};
}
$self->{fret} = Triceps::FnReturn->new(
name => $self->{name} . ".fret",
labels => \@labels,
);
}
return $self->{fret};
}
It uses the new element $self->{dsetnames} that wasn't present in the code shown before. I've added it now to keep the array of data set names in the order they were defined.
Use these examples to write the fnReturn() in your templates.
Showing posts with label templates. Show all posts
Showing posts with label templates. Show all posts
Saturday, November 10, 2012
Friday, September 28, 2012
Streaming functions introduction
Now for a moment let's take a break from the C++ API description (especially that it's a good spot, with all the types described), and talk about something new for version 1.1. I've been working on it in the background.
This new thing is the streaming functions. It's a cool and advanced concept, I've never seen it anywhere before, and for all I know I have invented it.
First let's look at the differences between the common functions and macros (or templates and such). Please turn your attention to the illustration below:
What happens during a function call? Some code (marked with the light bluish color) is happily zooming along when it decides to call a function. It prepares some arguments and jumps to the function code (reddish). The function executes, computes its result and jumps back to the point right after it has been called from. Then the original code continues from there (the slightly darker bluish color).
What happens during a macro (or template) invocation? It starts with some code zooming along in the same way, however when the macro call time comes, it prepares the arguments and then does nothing. It gets away with it because the compiler has done the work: it has placed the macro code right where it's called, so there is no need for jumps. After the macro is done, again it does nothing: the compiler has placed the next code to execute right after it, so it just continues on its way.
So far it's pretty equivalent. An interesting difference happens when the function or macro is called from more than one place. With a macro, another copy of the macro is created, inserted between its call and return points. That's why in the figure the macro is shown twice. But with the function the same function code is executed, and then returns back to the caller. That's why in the figure there are two function callers with their paths through the same function. But how does the function know, where should it jump on return? The caller tells it by pushing the return address onto the stack. When the function is done, it pops this address from the stack and jumps there.
Still, it looks all the same. A macro call is a bit more efficient, except when a large complex macro is called from many places, then it becomes more efficient as a function. However there is another difference if the function or macro holds some context (say, a static variable): each invocation of the macro will get its own context but all the function calls will share the same context. The only way to share the context with a macro is to pass some global context as its argument.
Now let's jump to the CEP world. The Sybase or StreamBase modules are essentially macros, and so are the Triceps templates. When such a macro gets instantiated, a whole new copy of it gets created with its tables/windows and streams/labels. Its input and output streams/labels get all connected in a fixed way. The limitation is that if the macro contains any tables, each instantiation gets a copy of it. Well, in Triceps you can use a table as an argument to a template In the other systems I think you still can't, so if you want to work with a common table in a module, you have to make up the query-response patterns, like the one described in the manual section "Comparative modularity".
In a query-response pattern there is some common sub-model, with a stream (in Triceps terms, a label, but here we're talking the other systems) for the queries to come in and a stream for the results to come out (both sides might have not only one but multiple streams). There are multiple inputs connected, from all the request sources, and the outputs are connected back to all the request sources. All the request sources (i.e. callers) get back the whole output of the pattern, so they need to identify, what output came from their input, and ignore the rest. They do this by adding the unique ids to their queries, and filter the results. In the end, it looks almost like a function but with much pain involved.
To make it look quite like a function, one thing is needed: the selective connection of the result streams (or, returning to the Triceps terminology, labels) to the caller. Connect the output labels, send some input, have it processed and send the result through the connection, disconnect the output labels. And what you get is a streaming function. It's very much like a common function but working on the streaming data arguments and results.
The next figure highlights the similarity and differences between the query patterns and the streaming functions.
The thick lines show where the data goes during one concrete call. The thin lines show the connections that do exist but without the data going through them at the moment (they will be used during the other calls, from these other callers). The dashed thin line shows the connection that doesn't exist at the moment. It will be created when needed (and at that time the thick arrow from the streaming to what is now the current return would disappear).
The particular beauty of the streaming functions for Triceps is that the other caller's don't even need to exist yet. They can be created and connected dynamically, do their job, call the function, use its result, and then be disposed of. The calling side in Triceps doesn't have to be streaming either: it could as well be procedural.
This new thing is the streaming functions. It's a cool and advanced concept, I've never seen it anywhere before, and for all I know I have invented it.
First let's look at the differences between the common functions and macros (or templates and such). Please turn your attention to the illustration below:
What happens during a macro (or template) invocation? It starts with some code zooming along in the same way, however when the macro call time comes, it prepares the arguments and then does nothing. It gets away with it because the compiler has done the work: it has placed the macro code right where it's called, so there is no need for jumps. After the macro is done, again it does nothing: the compiler has placed the next code to execute right after it, so it just continues on its way.
So far it's pretty equivalent. An interesting difference happens when the function or macro is called from more than one place. With a macro, another copy of the macro is created, inserted between its call and return points. That's why in the figure the macro is shown twice. But with the function the same function code is executed, and then returns back to the caller. That's why in the figure there are two function callers with their paths through the same function. But how does the function know, where should it jump on return? The caller tells it by pushing the return address onto the stack. When the function is done, it pops this address from the stack and jumps there.
Still, it looks all the same. A macro call is a bit more efficient, except when a large complex macro is called from many places, then it becomes more efficient as a function. However there is another difference if the function or macro holds some context (say, a static variable): each invocation of the macro will get its own context but all the function calls will share the same context. The only way to share the context with a macro is to pass some global context as its argument.
Now let's jump to the CEP world. The Sybase or StreamBase modules are essentially macros, and so are the Triceps templates. When such a macro gets instantiated, a whole new copy of it gets created with its tables/windows and streams/labels. Its input and output streams/labels get all connected in a fixed way. The limitation is that if the macro contains any tables, each instantiation gets a copy of it. Well, in Triceps you can use a table as an argument to a template In the other systems I think you still can't, so if you want to work with a common table in a module, you have to make up the query-response patterns, like the one described in the manual section "Comparative modularity".
In a query-response pattern there is some common sub-model, with a stream (in Triceps terms, a label, but here we're talking the other systems) for the queries to come in and a stream for the results to come out (both sides might have not only one but multiple streams). There are multiple inputs connected, from all the request sources, and the outputs are connected back to all the request sources. All the request sources (i.e. callers) get back the whole output of the pattern, so they need to identify, what output came from their input, and ignore the rest. They do this by adding the unique ids to their queries, and filter the results. In the end, it looks almost like a function but with much pain involved.
To make it look quite like a function, one thing is needed: the selective connection of the result streams (or, returning to the Triceps terminology, labels) to the caller. Connect the output labels, send some input, have it processed and send the result through the connection, disconnect the output labels. And what you get is a streaming function. It's very much like a common function but working on the streaming data arguments and results.
The next figure highlights the similarity and differences between the query patterns and the streaming functions.
The particular beauty of the streaming functions for Triceps is that the other caller's don't even need to exist yet. They can be created and connected dynamically, do their job, call the function, use its result, and then be disposed of. The calling side in Triceps doesn't have to be streaming either: it could as well be procedural.
Friday, July 6, 2012
more updates
Some more stuff has been getting cleaned up:
now confesses on errors, so the problem with its error checking is fixed.
The tables now allow to create rowops of rows of all matching types, not only of the equal types. The approach with matching types was not consistent with what the labels did, so I've changed it.
The TableType now has the method
that has the name consistent with the tables and labels. The old method rowType() also still exists.
The Opt::ck_ref() now also accepts the subclasses of the defined classes.
The new helper Fields::isStringType() has been added.
There also have been some major addition of the examples:
I've added a pretty big example of the main loop that includes the full socket handling in the chapter on Scheduling. It's not exactly production-ready but gives some idea.
Another addition in the chapter on Scheduling is an example of a topological loop that computes the Fibonacci numbers.
Many new examples have been added to the chapter on Templates.
$unit->setTracer($tracer);
now confesses on errors, so the problem with its error checking is fixed.
The tables now allow to create rowops of rows of all matching types, not only of the equal types. The approach with matching types was not consistent with what the labels did, so I've changed it.
The TableType now has the method
$tt->getRowType()
that has the name consistent with the tables and labels. The old method rowType() also still exists.
The Opt::ck_ref() now also accepts the subclasses of the defined classes.
The new helper Fields::isStringType() has been added.
There also have been some major addition of the examples:
I've added a pretty big example of the main loop that includes the full socket handling in the chapter on Scheduling. It's not exactly production-ready but gives some idea.
Another addition in the chapter on Scheduling is an example of a topological loop that computes the Fibonacci numbers.
Many new examples have been added to the chapter on Templates.
Thursday, February 23, 2012
Sorted index initialization, a simple ordered index template
To specify the sorting order in a more SQL-like fashion, Triceps now has the class SimpleOrderedIndex. It's implemented entirely in Perl, on top of the sorted index. Besides being useful by itself, it shows off two concepts: the initialization function of the sorted index, and the template with code generation on the fly.
First, how to create the ordered indexes:
The constructor takes a list of pairs fieldName => order, where the order is either "ASC" for ascending or "DESC" for descending.
The comparison function gets generated automatically. It's smart enough to generate the string comparisons for the string and uint8 fields, and the numeric comparisons for the numeric fields. It's not smart enough to do the locale-specific comparisons for the strings and locale-agnostic for the unit8, it just uses whatever you have set up in cmp for both. It treats the NULL field values as numeric 0 or empty strings. It doesn't handle the array fields at all but can at least detect such attempts and flag them as errors.
An interesting artifact of the boundary between C++ and Perl is that when you get the index type back from the table type like
the reference stored in $sortIdx will be of the base type Triceps::IndexType. That's because the C++ internals of the TableType object know nothing about any derived Perl types. But it's no big deal, since there are no other useful methods for SimpleOrderedIndex anyway.
If you call $sortIdx->print(), it will give you an idea of how it was constructed:
I'm not sure if I mentioned it yet, but all the index types have the method getKey() that for the hashed index types returns an array of key field names, and for the all other index types returns nothing. This includes the sorted index, and the simple ordered index that is derived from it. In the future I plan to allow returning the key list from the sorted indexes too, but haven't got around to do it yet.
The usage of the tables with these indexes is as with any other indexes. Since the PerlSortedIndex can be used in both leaf and non-leaf position, so can the SimpleOrderedIndex. Nothing special there.
Now the interesting part, the implementation of the sorted index. It's a little biggish for a blog post but not too huge:
Sorry, but I'm too lazy to wrap the long lines manually, and the @#%^ blog engine doesn't wrap them automatically either. They should really use some less brain-damaged formatting.
The class constructor simply builds the sort name from the arguments and offloads the rest of logic to the init function. It can't really do much more: when the index type object is constructed, it doesn't know yet, where it will be used and what row type it will get. It tries to enquote nicely the weird characters in the arguments when they go into the sort name. Not that much use is coming from it at the moment: the C++ code that prints the table type information doesn't do the same, so there still is a chance of misbalanced quotes in the result. But perhaps the C++ code will be fixed at some point too.
The init function is called at the table type initialization time. By this time all this extra information is known, and it gets the references to the table type, index type (itself, but with the class stripped back to Triceps::IndexType), row type, and whatever extra arguments that were passed through the newPerlSorted(). Now the actual work can begin.
By the way, the sorted index type init function is NOT of the same kind as the aggregator type init function. The aggregator type could use an init function of this kind too, but at the time it looked like too much extra complexity. It probably will be added in the future. But more about aggregators later.
The init function's return value is kind of backwards to everything else: on success it returns undef, on error it returns the error message. It could die too, but simply returning an error message is somewhat nicer.
It goes through all the arguments, looks up the fields in the row type, and checks them for correctness. It tries to collect as much of the error information as possible. The returned error messages may contain multiple lines separated by "\n", and the ordered index makes use of it. The error messages get propagated back to the table type level, nicely indented and returned from the table initialization. If the init function finds any errors, it appends the printout of the row type too, to make finding what went wrong easier. A result of a particularly bad call to a table type initialization may look like this:
Also as the init goes through the arguments, it constructs the text of the compare function in the variable $compare. Here the use of quotemeta() for the user-supplied strings is important to avoid the syntax errors in the generated code. If no errors are found in the arguments, the compare function gets compiled with eval. There should not be any errors, but it's always better to check. Finally the compiled compare function is set in the sorted index with
This method works only on the PerlSorted index types (it knows how to check internally) and would fail on all others. It replaces any previous compare function set in newPerlSorted(), as well as the extra arguments for it. So really if you use an init function, you would always set the compare function in newPerlSorted() to undef because it will be replaced anyway. If you want to pass extra arguments, you do that as setComparator($cmpfunc, @args). But in this class all the information from the arguments is already compiled into the body of the comparator, and there is no more use for them. The init function absolutely must set the compare function. If the comparator is still undef after the init returns, the initialization will see it as an error.
If you uncomment the debugging printout line (and run "make", and maybe "make install" afterwards), you can see the auto-generated code printed on stderr when you use the simple ordered index. It will look somewhat like this:
That's it! An entirely new piece functionality added in a smallish Perl snippet. This is your typical Triceps template: collect the arguments, use them to build Perl code, and compile it. Of course, if you don't want to deal with the code generation and compilation, you can just call your class methods and whatnot to interpret the arguments. But if the code will be reused, the compilation is more efficient.
First, how to create the ordered indexes:
my $tabType = Triceps::TableType->new($rowType)
->addSubIndex("sorted",
Triceps::SimpleOrderedIndex->new(
a => "ASC",
b => "DESC",
)
) or die "$!";
The constructor takes a list of pairs fieldName => order, where the order is either "ASC" for ascending or "DESC" for descending.
The comparison function gets generated automatically. It's smart enough to generate the string comparisons for the string and uint8 fields, and the numeric comparisons for the numeric fields. It's not smart enough to do the locale-specific comparisons for the strings and locale-agnostic for the unit8, it just uses whatever you have set up in cmp for both. It treats the NULL field values as numeric 0 or empty strings. It doesn't handle the array fields at all but can at least detect such attempts and flag them as errors.
An interesting artifact of the boundary between C++ and Perl is that when you get the index type back from the table type like
$sortIdx = $tabType->findSubIndex("sorted") or die "$!";
the reference stored in $sortIdx will be of the base type Triceps::IndexType. That's because the C++ internals of the TableType object know nothing about any derived Perl types. But it's no big deal, since there are no other useful methods for SimpleOrderedIndex anyway.
If you call $sortIdx->print(), it will give you an idea of how it was constructed:
PerlSortedIndex(SimpleOrder a ASC, b DESC, )
I'm not sure if I mentioned it yet, but all the index types have the method getKey() that for the hashed index types returns an array of key field names, and for the all other index types returns nothing. This includes the sorted index, and the simple ordered index that is derived from it. In the future I plan to allow returning the key list from the sorted indexes too, but haven't got around to do it yet.
The usage of the tables with these indexes is as with any other indexes. Since the PerlSortedIndex can be used in both leaf and non-leaf position, so can the SimpleOrderedIndex. Nothing special there.
Now the interesting part, the implementation of the sorted index. It's a little biggish for a blog post but not too huge:
package Triceps::SimpleOrderedIndex;
use Carp;
our @ISA = qw(Triceps::IndexType);
sub new # ($class, $fieldName => $direction...)
{
my $class = shift;
my @args = @_; # save a copy
# build a descriptive sortName
my $sortName = 'SimpleOrder ';
while ($#_ >= 0) {
my $fld = shift;
my $dir = shift;
$sortName .= quotemeta($fld) . ' ' . quotemeta($dir) . ', ';
}
$self = Triceps::IndexType->newPerlSorted(
$sortName, \&init, undef, @args
) or confess "$!";
bless $self, $class;
return $self;
}
sub init # ($tabt, $idxt, $rowt, @args)
{
my ($tabt, $idxt, $rowt, @args) = @_;
my %def = $rowt->getdef(); # the field definition
my $errors; # collect as many errors as possible
my $compare = "sub {\n"; # the generated comparison function
my $connector = "return"; # what goes between the comparison operators
while ($#args >= 0) {
my $f = shift @args;
my $dir = uc(shift @args);
my ($left, $right); # order the operands depending on sorting direction
if ($dir eq "ASC") {
$left = 0; $right = 1;
} elsif ($dir eq "DESC") {
$left = 1; $right = 0;
} else {
$errors .= "unknown direction '$dir' for field '$f', use 'ASC' or 'DESC'\n";
# keep going, may find more errors
}
my $type = $def{$f};
if (!defined $type) {
$errors .= "no field '$f' in the row type\n";
next;
}
my $cmp = "<=>"; # the comparison operator
if ($type eq "string"
|| $type =~ /^uint8.*/) {
$cmp = "cmp"; # string version
} elsif($type =~ /\]$/) {
$errors .= "can not order by the field '$f', it has an array type '$type', not supported yet\n";
next;
}
my $getter = "->get(\"" . quotemeta($f) . "\")";
$compare .= " $connector \$_[$left]$getter $cmp \$_[$right]$getter\n";
$connector = "||";
}
$compare .= " ;\n";
$compare .= "}";
if (defined $errors) {
# help with diagnostics, append the row type to the error listing
$errors .= "the row type is:\n";
$errors .= $rowt->print();
} else {
# compile the comparison
#print STDERR "DEBUG Triceps::SimpleOrderedIndex::init: comparison function:\n$compare\n";
my $cmpfunc = eval $compare
or return "Triceps::SimpleOrderedIndex::init: internal error when compiling the compare function:\n"
. "$@\n"
. "The generated comparator was:\n"
. $compare;
$idxt->setComparator($cmpfunc)
or return "Triceps::SimpleOrderedIndex::init: internal error: can not set the compare function:\n"
. "$!\n";
}
return $errors;
} Sorry, but I'm too lazy to wrap the long lines manually, and the @#%^ blog engine doesn't wrap them automatically either. They should really use some less brain-damaged formatting.
The class constructor simply builds the sort name from the arguments and offloads the rest of logic to the init function. It can't really do much more: when the index type object is constructed, it doesn't know yet, where it will be used and what row type it will get. It tries to enquote nicely the weird characters in the arguments when they go into the sort name. Not that much use is coming from it at the moment: the C++ code that prints the table type information doesn't do the same, so there still is a chance of misbalanced quotes in the result. But perhaps the C++ code will be fixed at some point too.
The init function is called at the table type initialization time. By this time all this extra information is known, and it gets the references to the table type, index type (itself, but with the class stripped back to Triceps::IndexType), row type, and whatever extra arguments that were passed through the newPerlSorted(). Now the actual work can begin.
By the way, the sorted index type init function is NOT of the same kind as the aggregator type init function. The aggregator type could use an init function of this kind too, but at the time it looked like too much extra complexity. It probably will be added in the future. But more about aggregators later.
The init function's return value is kind of backwards to everything else: on success it returns undef, on error it returns the error message. It could die too, but simply returning an error message is somewhat nicer.
It goes through all the arguments, looks up the fields in the row type, and checks them for correctness. It tries to collect as much of the error information as possible. The returned error messages may contain multiple lines separated by "\n", and the ordered index makes use of it. The error messages get propagated back to the table type level, nicely indented and returned from the table initialization. If the init function finds any errors, it appends the printout of the row type too, to make finding what went wrong easier. A result of a particularly bad call to a table type initialization may look like this:
index error:
nested index 1 'sorted':
unknown direction 'XASC' for field 'z', use 'ASC' or 'DESC'
no field 'z' in the row type
can not order by the field 'd', it has an array type 'float64[]', not supported yet
the row type is:
row {
uint8 a,
uint8[] b,
int64 c,
float64[] d,
string e,
}
Also as the init goes through the arguments, it constructs the text of the compare function in the variable $compare. Here the use of quotemeta() for the user-supplied strings is important to avoid the syntax errors in the generated code. If no errors are found in the arguments, the compare function gets compiled with eval. There should not be any errors, but it's always better to check. Finally the compiled compare function is set in the sorted index with
$idxt->setComparator($cmpfunc)
This method works only on the PerlSorted index types (it knows how to check internally) and would fail on all others. It replaces any previous compare function set in newPerlSorted(), as well as the extra arguments for it. So really if you use an init function, you would always set the compare function in newPerlSorted() to undef because it will be replaced anyway. If you want to pass extra arguments, you do that as setComparator($cmpfunc, @args). But in this class all the information from the arguments is already compiled into the body of the comparator, and there is no more use for them. The init function absolutely must set the compare function. If the comparator is still undef after the init returns, the initialization will see it as an error.
If you uncomment the debugging printout line (and run "make", and maybe "make install" afterwards), you can see the auto-generated code printed on stderr when you use the simple ordered index. It will look somewhat like this:
sub {
return $_[0]->get("a") cmp $_[1]->get("a")
|| $_[1]->get("c") <=> $_[0]->get("c")
|| $_[0]->get("b") cmp $_[1]->get("b")
;
}
That's it! An entirely new piece functionality added in a smallish Perl snippet. This is your typical Triceps template: collect the arguments, use them to build Perl code, and compile it. Of course, if you don't want to deal with the code generation and compilation, you can just call your class methods and whatnot to interpret the arguments. But if the code will be reused, the compilation is more efficient.
Sunday, January 29, 2012
a simple extension for a table
When I wrote the example for the last post, I've got a bit annoyed that to look up a row in a table I had to make a pattern row manually and then search for it. It looked easy to fix: just add a method findBy() that would take the (fieldNam, fieldValue) pairs for the keys, create the row and call find(). Then the code in "Hello, table" example
becomes
Naturally, it's not in version 0.99 but will be available in 1.00. The implementation is fairly simple. There is no reason why a class can't mix the XS methods and plain Perl methods. So I've added the file lib/Triceps/Table.pm, added it to be imported in lib/Triceps.pm, and put the Perl method in there:
Carp::confess() is a better kind of die(), I'll will discuss it in more detail later. Fairly simple and straightforward. If you see something missing, you can also always extend Triceps in the same way.
However if you change the Triceps code directly like I did, you'll have an issue with the next Triceps release: it woudl overwrite your file and your change would be lost. This can be solved in one of two ways. The first way is to write me an e-mail, describe your new useful change, and send me a context diff with its code. If I like it, I'll include it into the Triceps code base.
The second way comes useful if you want to keep the change to yourself, or if you sent it to me and I didn't like it: just make your own wrapper of the Table class and add the new method there. Then use your class instead of Triceps::Table. For example:
That's also a simplest template: a modifying wrapper for one class.
my $pattern = $rtCount->makeRowHash( address => $data[1] ) or die "$!"; my $rhFound = $tCount->find($pattern) or die "$!";
becomes
my $rhFound = $tCount->findBy( address => $data[1] ) or die "$!";
Naturally, it's not in version 0.99 but will be available in 1.00. The implementation is fairly simple. There is no reason why a class can't mix the XS methods and plain Perl methods. So I've added the file lib/Triceps/Table.pm, added it to be imported in lib/Triceps.pm, and put the Perl method in there:
package Triceps::Table;
use Carp;
sub findBy # (self, fieldName => fieldValue, ...)
{
my $self = shift;
my $row = $self->getRowType()->makeRowHash(@_) or Carp::confess "$!";
return $self->find($row);
}
Carp::confess() is a better kind of die(), I'll will discuss it in more detail later. Fairly simple and straightforward. If you see something missing, you can also always extend Triceps in the same way.
However if you change the Triceps code directly like I did, you'll have an issue with the next Triceps release: it woudl overwrite your file and your change would be lost. This can be solved in one of two ways. The first way is to write me an e-mail, describe your new useful change, and send me a context diff with its code. If I like it, I'll include it into the Triceps code base.
The second way comes useful if you want to keep the change to yourself, or if you sent it to me and I didn't like it: just make your own wrapper of the Table class and add the new method there. Then use your class instead of Triceps::Table. For example:
package MyTable;
our @ISA = qw(Triceps::Table);
sub new # (class, unit, args of makeTable...)
{
my $class = shift;
my $unit = shift;
my $self = $unit->makeTable(@_);
return undef unless defined $self;
bless $self, $class;
return $self;
}
sub myFindBy { ... }
package main;
...
my $tCount = MyTable->new(
$hwunit, $ttCount, &Triceps::EM_CALL, "tCount") or die "$!";
That's also a simplest template: a modifying wrapper for one class.
Thursday, January 5, 2012
Labels, part 1
In each CEP engine there are two kinds of logic: One is to get some request, look up some state, maybe update some state, and return the result. The other has to do with the maintenance of the state: make sure that when one part of the state is changed, the change propagates consistently through the rest of it. If we take a common RDBMS for an analog, the first kind would be like the ad-hoc queries, the second kind will be like the triggers. The CEP engines are very much like database engines driven by triggers, so the second kind tends to account for a lot of code.
The first kind of logic is often very nicely accommodated by the procedural logic. The second kind often (but not always) can benefit for a more relational, SQLy definition. Also, when every every SQL statement executes, it gets compiled first into the procedural form, and only then executes as the procedural code.
The Triceps approach is tilted toward procedural execution. That is, the procedural definitions come out of the box, and then the high-level relational logic can be defined with templates and code generators.
These bits of code, especially where the first and second kind connect, need some way to pass the data and operations between them. In Triceps these connection points are called Labels.
The streaming data rows enter the procedural logic through a label. Each row causes one call on the label. From the functional standpoint they are the same as Coral8 Streams, as has been shown earlier in the introduction. Except that in Triceps the labels get not just rows but operations on rows, as in Aleri: a combination of a row and an operation code. The name is "labels" because Triceps has been built around the more procedural ideas, and when looked at from that side, the labels are targets of calls and GOTOs.
If the streaming model is defined as a data flow graph, each arrow in the graph is essentially a GOTO operation, and each node is a label.
A Triceps label is not quite a GOTO label, since the actual procedural control always returns back after executing the label's code. It can be thought of as a label of a function or procedure. But if the caller does nothing but immedially return after getting the control back, it works very much like a GOTO label.
Each label accepts operations on rows of a certain type.
Each label belongs to a certain execution unit, so a label can be used only strictly inside one thread and can not be shared between threads.
Each label may have some code to execute when it receives a row operation. The labels without code can be useful too.
A Triceps model contains the straightforward code and the mode complex stateful elements, such as tables, aggregators, joiners (which may be implemented in C++ or in Perl, or created as user templates). These stateful elements would have some input labels, where the actions may be sent to them (and the actions may also be done as direct method calls), and output labels, where they would produce the indications of the changed state and/or responses to the queries. The output labels are typically the ones without code ("dummy labels"). They do nothing by themselves, but can pass the data to the other labels. This passing of data is achieved by chaining the labels: when a label is called, it will first execute its own code (if it has any), and then call the same operation on whatever labels are chained from it. Which may have more labels chained from them in turn. So, to pass the data, chain the input label of the following element to the output label of the previous element.
The execution unit provides methods to construct labels. A dummy label is constructed as:
It takes as arguments the type of rows that the label will accept and the symbolic name of the label. The name can be any but for the ease of debugging it's better to give the same name as the label variable.
The label with Perl code is constructed as follows:
The row type and name arguments are the same as for the dummy label. The following arguments provide the references to the Perl functions that perform the actions. execSub is the function that executes to handle the incoming rows. It gets the arguments:
Here $label is this label, $rowop is the row operation, and args are the same as extra arguments specified at the label creation.
The row operation actually contains the label reference, so why pass it the second time? The reason lies in the chaining. The current label may be chained, possibly through multiple levels, to some original label, and the rowop will refer to that original label. So the extra argument lets the code find the current label.
The clearSub deals with the destruction. Remember that the Triceps memory management uses the reference counting, which does not like the reference loops. The reference loops cause the objects to be never freed. It's no big deal if the data structures exist until the program exit anyway but becomes a memory leak if they are created and deleted dynamically.
If the labels are arranged in a cyclic graph, they refear to each other and create a reference loop. So the execution unit keeps track of all its labels, and when it gets destoryed, clears them up, breaking up the loops.
The clearing of a label drops all the references to execSub, clearSub and arguments, and clears all the chainings. But before anything else is done, clearSub gets a chance to execute and clear any application-level data. It gets as argument the label reference all the args from the label constructor:
A typical case is to keep the state of a stateful element in a hash:
Then the clearing function can wipe out the whole state of the element by undefining its hash:
Either of execSub and clearSub can be specified as undef. Though a label with an undefined execSub is essentially a dummy label, only more heavyweight.
Another potential for reference loops is between the execution unit and the labels. A unit keeps a reference to all its labels. So the labels can not keep a reference to the unit. And they don't. Internally they have a plain pointer. Note however that in the example shown the labels have a Perl reference to the object where they belong. If that object is to have a Perl reference to the unit, it would create a reference loop, and the object will never be destroyed and never clear the labels. So generally the objects should never keep references to the unit. The unit also provides another way around this situation: it has a way to force the label clearing when a helper object gets destroyed. It will be described later.
P.S. The original published version of this post had a few paragraphs lost, the updated version from 01/06/12 has them re-added.
The first kind of logic is often very nicely accommodated by the procedural logic. The second kind often (but not always) can benefit for a more relational, SQLy definition. Also, when every every SQL statement executes, it gets compiled first into the procedural form, and only then executes as the procedural code.
The Triceps approach is tilted toward procedural execution. That is, the procedural definitions come out of the box, and then the high-level relational logic can be defined with templates and code generators.
These bits of code, especially where the first and second kind connect, need some way to pass the data and operations between them. In Triceps these connection points are called Labels.
The streaming data rows enter the procedural logic through a label. Each row causes one call on the label. From the functional standpoint they are the same as Coral8 Streams, as has been shown earlier in the introduction. Except that in Triceps the labels get not just rows but operations on rows, as in Aleri: a combination of a row and an operation code. The name is "labels" because Triceps has been built around the more procedural ideas, and when looked at from that side, the labels are targets of calls and GOTOs.
If the streaming model is defined as a data flow graph, each arrow in the graph is essentially a GOTO operation, and each node is a label.
A Triceps label is not quite a GOTO label, since the actual procedural control always returns back after executing the label's code. It can be thought of as a label of a function or procedure. But if the caller does nothing but immedially return after getting the control back, it works very much like a GOTO label.
Each label accepts operations on rows of a certain type.
Each label belongs to a certain execution unit, so a label can be used only strictly inside one thread and can not be shared between threads.
Each label may have some code to execute when it receives a row operation. The labels without code can be useful too.
A Triceps model contains the straightforward code and the mode complex stateful elements, such as tables, aggregators, joiners (which may be implemented in C++ or in Perl, or created as user templates). These stateful elements would have some input labels, where the actions may be sent to them (and the actions may also be done as direct method calls), and output labels, where they would produce the indications of the changed state and/or responses to the queries. The output labels are typically the ones without code ("dummy labels"). They do nothing by themselves, but can pass the data to the other labels. This passing of data is achieved by chaining the labels: when a label is called, it will first execute its own code (if it has any), and then call the same operation on whatever labels are chained from it. Which may have more labels chained from them in turn. So, to pass the data, chain the input label of the following element to the output label of the previous element.
The execution unit provides methods to construct labels. A dummy label is constructed as:
$label = $unit->makeDummyLabel($rowType, "name");
It takes as arguments the type of rows that the label will accept and the symbolic name of the label. The name can be any but for the ease of debugging it's better to give the same name as the label variable.
The label with Perl code is constructed as follows:
$label = $unit->makeLabel($rowType, "name", \&clearSub, \&execSub, args...);
The row type and name arguments are the same as for the dummy label. The following arguments provide the references to the Perl functions that perform the actions. execSub is the function that executes to handle the incoming rows. It gets the arguments:
execSub($label, $rowop, args...)
Here $label is this label, $rowop is the row operation, and args are the same as extra arguments specified at the label creation.
The row operation actually contains the label reference, so why pass it the second time? The reason lies in the chaining. The current label may be chained, possibly through multiple levels, to some original label, and the rowop will refer to that original label. So the extra argument lets the code find the current label.
The clearSub deals with the destruction. Remember that the Triceps memory management uses the reference counting, which does not like the reference loops. The reference loops cause the objects to be never freed. It's no big deal if the data structures exist until the program exit anyway but becomes a memory leak if they are created and deleted dynamically.
If the labels are arranged in a cyclic graph, they refear to each other and create a reference loop. So the execution unit keeps track of all its labels, and when it gets destoryed, clears them up, breaking up the loops.
The clearing of a label drops all the references to execSub, clearSub and arguments, and clears all the chainings. But before anything else is done, clearSub gets a chance to execute and clear any application-level data. It gets as argument the label reference all the args from the label constructor:
clearSub($label, args...)
A typical case is to keep the state of a stateful element in a hash:
package MyElement;
sub new # (class, unit, name...)
{
my ($class, $unit, $name) = @_;
my $self = {};
...
$self->inLabel = $unit->makeLabel(..., \&clear, \&handle, $self);
$self->outLabel = $unit->makeDummyLabel(...);
...
bless $self, $class;
return $self;
}
Then the clearing function can wipe out the whole state of the element by undefining its hash:
sub clear # (label, self)
{
my ($label, $self) = @_;
undef %$self;
}
Either of execSub and clearSub can be specified as undef. Though a label with an undefined execSub is essentially a dummy label, only more heavyweight.
Another potential for reference loops is between the execution unit and the labels. A unit keeps a reference to all its labels. So the labels can not keep a reference to the unit. And they don't. Internally they have a plain pointer. Note however that in the example shown the labels have a Perl reference to the object where they belong. If that object is to have a Perl reference to the unit, it would create a reference loop, and the object will never be destroyed and never clear the labels. So generally the objects should never keep references to the unit. The unit also provides another way around this situation: it has a way to force the label clearing when a helper object gets destroyed. It will be described later.
P.S. The original published version of this post had a few paragraphs lost, the updated version from 01/06/12 has them re-added.
Wednesday, December 28, 2011
a little about templates
Since people have started commenting about templates, let me show a bit more, what do I mean by them on a simple example.
Coral8 doesn't provide a way to query the windows directly, especially when the CCL is compiled without debugging. So you're expected to make your own. People at DB have developed a nice pattern that goes approximately like this:
To query the window, a program would select a unique query id, subscribe to result_my with a filter (qqq_id = unique_id) and send a record of (unique_id) into query_my. Then it would sit and collect the result rows. Finally it would get a row with qqq_end = TRUE and disconnect.
This is a fairly large amount of code to be repeated for every window. What I would like to to instead is to just write
and have the template make_queryable expand into the rest of the code (obviously, the schema definitions would not need to be expanded repeatedly, they would go into an include file).
To make things more interesting, it would be nice to have the query filter the results by some field values. Nothing as fancy as SQL, just by equality to some fields. Suppose, s_my includes the fields field_c and field_d, and we want to be able to filter by them. Then the query can be done as:
It would be nice then to create this kind of query as a template instantiation
If there weren't already an entrenched tradition at DB, I would not write directly in CCL at all. I would have made a macro language that would generate CCL. Of course, then the IDE would see only the results of the code generation and could not be used directly to write code in it, but who cares, IDEs are useless for this purpose anyway.
Interestingly, there already are people who do that kind of things. Some people actually prefer the Aleri XML format because it's easier for them to generate the code in XML. (I don't exactly see why generating the code in XML would be easier but there are all kinds of weird XML-based infrastructures out there).
Coral8 doesn't provide a way to query the windows directly, especially when the CCL is compiled without debugging. So you're expected to make your own. People at DB have developed a nice pattern that goes approximately like this:
// some window that we want to make queryable create window w_my schema s_my keep last per key_a per key_b keep 1 week; // the stream to send the query requests // (the schema can be shared by all simple queries) create schema s_query ( qqq_id string // unique id of the query ); create input stream query_my schema s_query; // the stream to return the results // (all result streams will inherit a partial schema) create schema s_result ( qqq_id string, // returns back the id received in the query qqq_end boolean, // will be TRUE in the special end indicator record ); create output stream result_my schema inherits from s_result, s_my; // now process the query insert into result_my select q.qqq_id, NULL, w.* from s_query as q, w_my as w; // the end marker insert into result_my (qqq_id, qqq_end) select qqq_id, TRUE from s_query;
To query the window, a program would select a unique query id, subscribe to result_my with a filter (qqq_id = unique_id) and send a record of (unique_id) into query_my. Then it would sit and collect the result rows. Finally it would get a row with qqq_end = TRUE and disconnect.
This is a fairly large amount of code to be repeated for every window. What I would like to to instead is to just write
create window w_my schema s_my keep last per key_a per key_b keep 1 week; make_queryable(w_my);
and have the template make_queryable expand into the rest of the code (obviously, the schema definitions would not need to be expanded repeatedly, they would go into an include file).
To make things more interesting, it would be nice to have the query filter the results by some field values. Nothing as fancy as SQL, just by equality to some fields. Suppose, s_my includes the fields field_c and field_d, and we want to be able to filter by them. Then the query can be done as:
create input stream query_my schema inherits from s_query ( field_c integer, field_d string ); // result_my is the same as before... // query with filtering (in a rather inefficient way) insert into result_my select q.qqq_id, NULL, w.* from s_query as q, w_my as w where (q.field_c is null or q.field_c = w.field_c) and (q.field_d is null or q.field_d = w.field_d); // the end marker is as before insert into result_my (qqq_id, qqq_end) select qqq_id, TRUE from s_query;
It would be nice then to create this kind of query as a template instantiation
make_query(w_my, (field_c, field_d));
If there weren't already an entrenched tradition at DB, I would not write directly in CCL at all. I would have made a macro language that would generate CCL. Of course, then the IDE would see only the results of the code generation and could not be used directly to write code in it, but who cares, IDEs are useless for this purpose anyway.
Interestingly, there already are people who do that kind of things. Some people actually prefer the Aleri XML format because it's easier for them to generate the code in XML. (I don't exactly see why generating the code in XML would be easier but there are all kinds of weird XML-based infrastructures out there).
Saturday, December 17, 2011
surveying the landscape
What do we have in the CEP area now? The scene is pretty much dominated by Sybase (combining the former Aleri and Coral8) and StreamBase.
There seem to be two major approaches to the execution model. One was used by Aleri, another by Coral8 and StreamBase. I'm not hugely familiar with StreamBase, but that's how it seems to me. Since I'm much more familiar with Coral8, I'll be calling the second model the Coral8 model. If you find StreamBase substantially different, let me know.
The Aleri idea is to collect and keep all the data. The relational operators get applied on the data, producing the derived data ("materialized views") and eventually the results. So, even though the Aleri models were usually expressed in XML (though an SQL compiler was also available), fundamentally it's a very relational and SQLy approach.
This creates a few nice properties. All steps of execution can be pipelined and executed in parallel.For persistence, it's fundamentally enough to keep only the input data (what has been called BaseStreams and then SourceStreams), and all the derived computations can be easily reprocessed on restart (it's funny but it turns out that often it's faster to read a small state from the disk and recalculate the rest from scratch in memory than to load a large state from the disk).
It also has issues. It doesn't allow loops, and the procedural calculation aren't always easy to express. And keeping all the state requires more memory. The issues of loops and procedural computations have been addressed by FlexStreams: modules that would perform the procedural computations instead of relational operations, written in SPLASH - a vaguely C-ish or Java-ish language. However this tends to break the relational properties: once you add a FlexStream, usually you do it for the reasons that prevent the derived calculations from being re-done, creating issues with saving and restoring the state. Mind you, you can write a FlexStream that doesn't break any of them, but then it would probably be doing something that can be expressed without it in the first place.
Coral8 has grown from the opposite direction: the idea has been to process the incoming data while keeping a minimal state in variables and short-term "windows" (limited sliding recordings of the incoming data). The language (CCL) is very SQL-like. It relies on the state of variables and windows being pretty much global (module-wide), and allows the statements to be connected in loops. Which means that the execution order matters a lot. Which means that there are some quite extensive rules, determining this order. The logic ends up being very much procedural, but written in the peculiar way of SQL statements and connecting streams.
The good thing is that all this allows to control the execution order very closely and write things that are very difficult to express in pure un-ordered relational operators. Which allows to aggregate the data early and creatively, keeping less data in memory.
The bad news is that it limits the execution to a single thread. If you want a separate thread, you must explicitly make a separate module, and program the communications between the modules, which is not exactly easy to get right. There are lots of people who do it the easy way and then wonder, why do they get the occasional data corruption. Also, the ordering rules for execution inside a module are quite tricky. Even for fairly simple logic, it requires writing a lot of code, some of which is just bulky (try enumerating 90 fields in each statement), and some of which is tricky to get right.
The summary is that everything is not what it seems: the Aleri models aren't usually written in SQL but are very declarative in their meaning, while the Coral8/StreamBase models are written in an SQL-like language but in reality are totally procedural.
Sybase is also striking for a middle ground, combining the features inherited from Aleri and Coral8 in its CEP R5 and later: use the CCL language but relax the execution order rules to the Aleri level, except for the explicit single-threaded sections where the order is important. Include the SPLASH fragments for where the outright procedural logic is easy to use. Even though it sounds hodgy-podgy, it actually came together pretty nicely. Forgive me for saying so myself since I've done a fair amount of design and the execution logic implementation for it before I've left Sybase.
Still, not everything is perfect in this merged world. The SQLy syntax still requires you to drag around all your 90 fields into nearly every statement. The single-threaded order of execution is still non-obvious. It's possible to write the procedural code directly in SPLASH but the boundary where the data passes between the SQLy and C-ish code still has a whole lot of its own kinks (less than in Aleri). And worst of all, there is still no modular programming. Yeah, there are "modules" but they are not really reusable. They are tied too tightly to the schema of the data. What is needed, is more like C++ templates.
There seem to be two major approaches to the execution model. One was used by Aleri, another by Coral8 and StreamBase. I'm not hugely familiar with StreamBase, but that's how it seems to me. Since I'm much more familiar with Coral8, I'll be calling the second model the Coral8 model. If you find StreamBase substantially different, let me know.
The Aleri idea is to collect and keep all the data. The relational operators get applied on the data, producing the derived data ("materialized views") and eventually the results. So, even though the Aleri models were usually expressed in XML (though an SQL compiler was also available), fundamentally it's a very relational and SQLy approach.
This creates a few nice properties. All steps of execution can be pipelined and executed in parallel.For persistence, it's fundamentally enough to keep only the input data (what has been called BaseStreams and then SourceStreams), and all the derived computations can be easily reprocessed on restart (it's funny but it turns out that often it's faster to read a small state from the disk and recalculate the rest from scratch in memory than to load a large state from the disk).
It also has issues. It doesn't allow loops, and the procedural calculation aren't always easy to express. And keeping all the state requires more memory. The issues of loops and procedural computations have been addressed by FlexStreams: modules that would perform the procedural computations instead of relational operations, written in SPLASH - a vaguely C-ish or Java-ish language. However this tends to break the relational properties: once you add a FlexStream, usually you do it for the reasons that prevent the derived calculations from being re-done, creating issues with saving and restoring the state. Mind you, you can write a FlexStream that doesn't break any of them, but then it would probably be doing something that can be expressed without it in the first place.
Coral8 has grown from the opposite direction: the idea has been to process the incoming data while keeping a minimal state in variables and short-term "windows" (limited sliding recordings of the incoming data). The language (CCL) is very SQL-like. It relies on the state of variables and windows being pretty much global (module-wide), and allows the statements to be connected in loops. Which means that the execution order matters a lot. Which means that there are some quite extensive rules, determining this order. The logic ends up being very much procedural, but written in the peculiar way of SQL statements and connecting streams.
The good thing is that all this allows to control the execution order very closely and write things that are very difficult to express in pure un-ordered relational operators. Which allows to aggregate the data early and creatively, keeping less data in memory.
The bad news is that it limits the execution to a single thread. If you want a separate thread, you must explicitly make a separate module, and program the communications between the modules, which is not exactly easy to get right. There are lots of people who do it the easy way and then wonder, why do they get the occasional data corruption. Also, the ordering rules for execution inside a module are quite tricky. Even for fairly simple logic, it requires writing a lot of code, some of which is just bulky (try enumerating 90 fields in each statement), and some of which is tricky to get right.
The summary is that everything is not what it seems: the Aleri models aren't usually written in SQL but are very declarative in their meaning, while the Coral8/StreamBase models are written in an SQL-like language but in reality are totally procedural.
Sybase is also striking for a middle ground, combining the features inherited from Aleri and Coral8 in its CEP R5 and later: use the CCL language but relax the execution order rules to the Aleri level, except for the explicit single-threaded sections where the order is important. Include the SPLASH fragments for where the outright procedural logic is easy to use. Even though it sounds hodgy-podgy, it actually came together pretty nicely. Forgive me for saying so myself since I've done a fair amount of design and the execution logic implementation for it before I've left Sybase.
Still, not everything is perfect in this merged world. The SQLy syntax still requires you to drag around all your 90 fields into nearly every statement. The single-threaded order of execution is still non-obvious. It's possible to write the procedural code directly in SPLASH but the boundary where the data passes between the SQLy and C-ish code still has a whole lot of its own kinks (less than in Aleri). And worst of all, there is still no modular programming. Yeah, there are "modules" but they are not really reusable. They are tied too tightly to the schema of the data. What is needed, is more like C++ templates.
Subscribe to:
Posts (Atom)