tag:blogger.com,1999:blog-3195352510924611504.comments2017-12-30T20:02:16.245-05:00Sergey Babkin on CEP and stuffSergey Babkinhttp://www.blogger.com/profile/06686909700446011233noreply@blogger.comBlogger26125tag:blogger.com,1999:blog-3195352510924611504.post-80096425312486571522017-12-30T20:02:16.245-05:002017-12-30T20:02:16.245-05:00I'd like to make a correction:
Technically sp...I'd like to make a correction:<br /><br />Technically speaking, provided you assume your error is Gaussian with zero mean and a common variance (iid etc.), Bayesian learning will show that the sum of the squared errors is equivalent to finding the maximum likelihood - and that's what gradient descent is doing. So I suppose your right that there is a connection between NN and Bayesian models.<br /><br />As far as I can tell, sigmoid is just not used that much anymore in current NN models. Maybe they should? LOL.<br /><br />AlexMetalFihttps://www.blogger.com/profile/10911692344196588308noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-12200508607844102182017-11-14T13:53:57.296-05:002017-11-14T13:53:57.296-05:00Oops, I wanted to answer and forgot.
1) From my e...Oops, I wanted to answer and forgot.<br /><br />1) From my experiments with Tensorflow, Relu is actually not a very good function, the arctangent or sigmoid work better. Relu is good for the linear regressions but not as much for the decisions. But there are a couple of reasons for why the Relu can be used to the same effect:<br /><br />(a) NNs have the offset in each node, so with the cut-off of the negative arguments in Relu, you can add together multiple Relu functions to get a close approximation of a sigmoid. I.e. if you have two Relu functions, one with the offset of 0 and slope of 2, and another one with offset of 0.4 and slope of -1.67, you'd get a broken line that goes from (0, 0) to (0.4, 0.8) to (1, 1), because the second function would kick in in the range [0.4, 1] and the slope in this range will become the sum 2-1.67=0.33. Such addition would come naturally because that's what the second layer does: adds up the outputs of the first layer with coefficients.<br /><br />(b) The other reason why the cutting-off of the negative side in Relu is useful: basically, it naturally supports the multiple-choice values as described in http://babkin-cep.blogspot.com/2016/08/bayes-23-enumerated-events-revisited.html and its references (although I'm not sure if I wrote it clearly).<br /><br />2) I think the Bayesian equivalent is computing the conditional probabilities. If you take an one-level NN, the result of the gradient climbing would be the same as computing the averages that give you probabilities. For example, if you have 70 training cases that pull the result up and 30 cases that pull it down, you'd end up with the coefficient of 0.7, which also happens to be the probability. AdaBoost does a variation of the same thing, computes the chances from the gradient descent. But when you have 2 or more layers, the gradient-climbing does two things at once: decides what should the intermediate level-event be and computes the conditional probabilities.<br /><br />Sorry, I don't have any advice, I'm in kind of the same position only worse: I haven't taken any official grad classes :-) I've been thinking that taking such classes might provide some connections but looks like it didn't? I've had some links to Reddits on this subject that might be a good place to ask around but I haven't used Reddit much yet at all :-)<br /><br />Thanks for the links, I'll read them!Sergey Babkinhttps://www.blogger.com/profile/06686909700446011233noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-8138836909893596012017-11-10T14:21:00.165-05:002017-11-10T14:21:00.165-05:00Sergey, here is a post that talks about it:
https...Sergey, here is a post that talks about it:<br /><br />https://towardsdatascience.com/understanding-objective-functions-in-neural-networks-d217cb068138<br /><br />Maybe that is what you are saying?AlexMetalFihttps://www.blogger.com/profile/10911692344196588308noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-51270564051182008472017-10-28T10:12:59.859-04:002017-10-28T10:12:59.859-04:00Well a couple of things:
1) FYI: Sigmoid isn'...Well a couple of things:<br /><br />1) FYI: Sigmoid isn't really used anymore for modern day NN (relus or leaky relus seem more popular).<br /><br />2) I don't think you can convert them back and forth as you describe. That's my point. With NN you can arrive at different weights depending on your objective function and type of gradient descent (you can arrive at different local minima depending on how you initialize your weights and your learning rate/step size, etc.). Put simply, you're climbing down a slope using partial derivatives. What is the Bayesian equivalent of that? That's what I don't quite understand in your analysis. You are looking at one node out of context and assigning chances. Fine, but once you add multiple layers, I think this all falls apart very fast.<br /><br />I think this paper would be of interest to you (using Bayesian Backprop):<br /><br />https://arxiv.org/pdf/1505.05424.pdf<br /><br />3) Yeah, exactly. Simple NNs are easier to follow. But once your parameters expand ("curse of dimensionality" and all that), as I said, it falls apart.<br /><br />As you can tell, ML/DL has been a hobby of mine for the past few months. I took the graduate classes at Columbia via edX last year and just finished the Udacity courses on AI and DL. I love this field and am looking to break into it (even though my background is as you know in kernel and system programming). Have any advice? (you seem to be deeply involved with it).<br />AlexMetalFihttps://www.blogger.com/profile/10911692344196588308noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-14077995957116479152017-10-25T21:05:55.166-04:002017-10-25T21:05:55.166-04:00Hi Alex! What have you been up to?
The weights i...Hi Alex! What have you been up to?<br /><br />The weights in a particular neuron do represent the Bayesian probabilities of its inputs, only slightly indirectly, as logarithms of the chances. But they can be converted to the probability values, as described in http://babkin-cep.blogspot.com/2017/02/a-better-explanation-of-bayes-adaboost.html and http://babkin-cep.blogspot.com/2017/06/neuron-in-bayesian-terms-part-2.html . That's kind of the point I'm making, that you can just convert them back and forth, that they are equivalent. It's kind of funny that whatever AI approach you take: decision trees, Bayesian, boosting, neural networks, they all seem to be pretty much variations of each other.<br /><br />In the simple NNs (like those in the Tensorflow examples) you can actually get interesting insights by looking at the heatmap graphs of the functions of the individual neurons. You can see that this neuron in the first layer matches this feature, and that neuron matches that feature, and then they get combined in a second-layer neuron. But in a large NN I think it's sheer size would make it difficult to look at each single neuron.Sergey Babkinhttps://www.blogger.com/profile/06686909700446011233noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-39009090742854314612017-10-25T15:39:53.855-04:002017-10-25T15:39:53.855-04:00Sergey, long time no see! The problem with the abo...Sergey, long time no see! The problem with the above is that neurons do not have meanings in themselves. If you look at any given weights of a particular neuron it doesn't tell you anything - the same is not true about a Bayesian network. Bayesian networks impart meaning in their structure and node-to-node relationships (dependence). Neural networks do not (hidden layer X weight matrix has no real obvious relationship between hidden layer Y).<br /><br />I suppose you could think of the output layer of a fully connected NN using a softmax activation function as kinda Bayesian since the final values are indeed probabilities (beliefs) of classifications. But there is no way to walk those values back which is why it is sometimes hard to explain why a NN behaves the way it does. Again, the same is not true for Bayesian networks.<br /><br />Alex (remember SmallFoot?)AlexMetalFihttps://www.blogger.com/profile/10911692344196588308noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-27500347900720548442017-03-27T01:48:14.013-04:002017-03-27T01:48:14.013-04:00"Second writing" help not only to improv..."Second writing" help not only to improve the writing itself, but also helps to improve your writing skill so your future posts improve too.Dennis Gorelikhttps://www.blogger.com/profile/17700219093521377626noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-47877935962039847982017-03-26T21:25:08.537-04:002017-03-26T21:25:08.537-04:00Well, it could use some more editing, all the text...Well, it could use some more editing, all the texts usually look better on the second writing :-) But the main point is that 10x productivity requires getting a lot of things right to streamline the development. And getting even one or two of them very wrong derails the whole thing.Sergey Babkinhttps://www.blogger.com/profile/06686909700446011233noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-14224406484229050912017-03-23T13:09:08.201-04:002017-03-23T13:09:08.201-04:00That should be a numbered lists of tricks.
You may...That should be a numbered lists of tricks.<br />You may also consider cover one "10x improvement" item per article and give more examples demonstrating how that item work.Dennis Gorelikhttps://www.blogger.com/profile/17700219093521377626noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-53571382423464421322017-01-04T15:03:18.444-05:002017-01-04T15:03:18.444-05:00Thanks, fixed.Thanks, fixed.Sergey Babkinhttps://www.blogger.com/profile/06686909700446011233noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-84794115959686307782017-01-04T04:38:33.500-05:002017-01-04T04:38:33.500-05:00"I thing that the most important..."
thi...<br />"I thing that the most important..."<br />think<br />ZeaGatehttps://www.blogger.com/profile/10817824919381079019noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-4673869724231313802014-07-28T10:33:41.609-04:002014-07-28T10:33:41.609-04:00Thank you for this contribution Sergey!!! Thank you for this contribution Sergey!!! Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-54926858758332916062014-07-13T16:01:09.325-04:002014-07-13T16:01:09.325-04:00Man, rules are rules. C++ is known for its non-obv...Man, rules are rules. C++ is known for its non-obvious rules. On the other hand using new part of C++ language std::shared_ptr is so much easier than programmatic analogues with reference counter stored in some refcounted_base class you must derive from.<br /><br />Works for me.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-53292747504923425692013-06-12T22:40:32.091-04:002013-06-12T22:40:32.091-04:00Sure. You can write to me at sab123@hotmail.com.
...Sure. You can write to me at sab123@hotmail.com. <br /><br />If you are building on Linux, I would suggest first to try checking out the the latest version from the SVN repository. I've recently committed an improvement that should be capable of detecting the NSPR library on Linux, which is usually the source of the problems.Sergey Babkinhttps://www.blogger.com/profile/06686909700446011233noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-35072449471750998342013-06-12T18:43:47.088-04:002013-06-12T18:43:47.088-04:00hi
I'm interested by your perl module but i ...hi <br /><br />I'm interested by your perl module but i don't success to build it.<br /><br />Is it possible to contact you in order to have some help in building the perl module ?<br /><br />regards<br />sdeseilleAnonymoushttps://www.blogger.com/profile/17239836360817468304noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-62187909115243726962012-08-01T23:12:35.847-04:002012-08-01T23:12:35.847-04:00The future versions will instead have the definiti...The future versions will instead have the definition -DTRICEPS_NSPR=4 to include from , and -DTRICEPS_NSPR=0 to include from . So editing the Makefile.inc sould be enough, without changing the source code.Sergey Babkinhttps://www.blogger.com/profile/06686909700446011233noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-72397477845636327562012-08-01T21:03:59.095-04:002012-08-01T21:03:59.095-04:00Thanks! I've seen a similar report about nspr ...Thanks! I've seen a similar report about nspr when building on FreeBSD.Sergey Babkinhttps://www.blogger.com/profile/06686909700446011233noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-42090235241136376452012-07-31T23:18:13.821-04:002012-07-31T23:18:13.821-04:00I wanna report two trivial error. I built triceps...I wanna report two trivial error. I built triceps on Ubuntu 12.04 and found that.<br />1) It couldn't found nspr4 but actually the library could be found in /usr/include/nspr<br />2) It generated error message because triceps-1.0.0/cpp/common/Hash.h Line 26 , so I had to change 2166136261 to 2166136261u.<br />I just start to learn development so I don't know how to write patch or make file. Sorry about that. But I wanted to contribute as far as I can.<br />With respect.junhttps://www.blogger.com/profile/12674622370419084407noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-4552511927248037962012-03-14T20:42:14.809-04:002012-03-14T20:42:14.809-04:00Thanks, interesting to know. Well, technically, Al...Thanks, interesting to know. Well, technically, Aleri can do it too: you can run in artificial time, setting and stopping it. So you an stop the time, set to record timestamp, feed the record, wait for processing to complete, advance time, wait for any time-based processing to complete, and so on. I'm not sure if it made to Sybase R5, but it definitely worked on Aleri. However there was no tool that did it for you easily, and also all these synchronous calls present a pretty high overhead.Sergey Babkinhttps://www.blogger.com/profile/06686909700446011233noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-5984274024732393032012-03-14T09:44:18.302-04:002012-03-14T09:44:18.302-04:00Esper can give complete control over time to an ap...Esper can give complete control over time to an application and there is no limit in which events can be processed when time is controlled externallyAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-62491658840791320792012-01-06T19:07:26.637-05:002012-01-06T19:07:26.637-05:00For (1) it actually sounds like StreamBase simply ...For (1) it actually sounds like StreamBase simply doesn't need this particular pattern because it allows the direct access to tables, which are the same thing as Coral8 calls windows. Same as Aleri. <br /><br />For (2), I don't really understand how StreamBase works, but it looks like it can't really take a list of fields as a parameter value. It's the same limitation as the C++ templates have. I've been feeling for a long time that the C preprocessor and later C++ template should have the directives for conditional compilation and loops in the macros/templates. The lack of the loops leads to all kind of recursive monstrosities in the Alexandrescu book about C++ templates.<br /><br />I hope that using Perl for the macro-language can solve both the problem with loops in code generation and with debugging difficulties: provide the flexible code generation, and also the more meaningful high-level error messages. I.e. when an error is found, the code can provide a more meaningful message of what is wrong with the template parameters rather than simply dumping the call trace. And of course, it could do the explicit checks of template parameters before using them.Sergey Babkinhttps://www.blogger.com/profile/06686909700446011233noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-89878653749806983462011-12-30T00:24:21.343-05:002011-12-30T00:24:21.343-05:00Checking out the "A little about templates ar...Checking out the "A little about templates article"; I debated posting this comment there, but I think keeping the conversation in one place might make more sense.<br /><br />For the first example:<br /><br />Yes, capture fields enable doing this in a clean way. The idiom in StreamBase would be to create a module that exports a table (as queryable as any other table) and maintains the contents of that table as desired. Capture fields are what allow the exported table's schema to be based on any input schema to the module.<br /><br />For the second example:<br /><br />This kind of "templating" is exactly what we use StreamBase "module parameters" for. They work very much like you suggested your template language might work, generating a "substituted" version of a StreamBase module that uses the parameters specified at each module call site.<br /><br />Unfortunately, module parameters do suffer from all the problems you'd expect of any kind of templating system that generates code: errors get arcane and difficult to debug, static type guarantees become a lot more difficult, compilation times become longer because less work can be shared, etc. Capture fields allow us to avoid these problems of templating for a wide range of real-world cases: whenever the modules are parameterized by type, instead of by arbitrary expression.Naomi Seyferhttp://www.streambase.comnoreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-78770631365405021012011-12-29T08:54:57.213-05:002011-12-29T08:54:57.213-05:00The capture fields are definitely a step in the ri...The capture fields are definitely a step in the right direction. However they don't seem to solve the field-names-as-parameters problem. They probably could implement the first example in http://babkin-cep.blogspot.com/2011/12/little-about-templates.html, though maybe with a more cumbersome calling sequence, but I think not the second one.Sergey Babkinhttps://www.blogger.com/profile/06686909700446011233noreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-58760361539002930152011-12-28T14:39:36.871-05:002011-12-28T14:39:36.871-05:00Anonymous: StreamBase has the same properties as ...Anonymous: StreamBase has the same properties as Esper here; we try to make it as easy as possible to say "this stream gets the same data as that one" or "this stream gets the same data as that one except for these specific modifications"Naomi Seyferhttp://www.streambase.comnoreply@blogger.comtag:blogger.com,1999:blog-3195352510924611504.post-88004841867342757072011-12-28T13:04:56.138-05:002011-12-28T13:04:56.138-05:00Hello! I'm a runtime engineer over at StreamB...Hello! I'm a runtime engineer over at StreamBase. I just wanted to let you know about some of the work we're doing over here to ameliorate some of the issues you've pointed out.<br /><br />Specifically: "Yeah, there are "modules" but they are not really reusable. They are tied too tightly to the schema of the data. What is needed, is more like C++ templates."<br /><br />We've come up with a way of cleanly allowing type parameterization of StreamBase modules, released in StreamBase version 7.2. The documentation is <a href="http://docs.streambase.com/sb72/index.jsp?topic=/com.streambase.sb.ide.help/data/html/authoring/capturefields.html" rel="nofollow">here</a>; the research paper I presented at the last Distributed and Event Based Systems conference is <a href="http://www.streambase.com/wp-content/uploads/2011/12/StreamBase_WhitePaper_CaptureFields.pdf" rel="nofollow">here</a>.Naomi Seyferhttp://www.streambase.comnoreply@blogger.com