Hey, it's been almost a year since I wrote anything here. Well, I've been busy. Mostly with all kinds of stupid things. But here is something new, which is also something old.
Consider the typical Devops problem from my past experience at a Company that shall be left Unnamed, to avoid triggering the envy of the other companies where exactly the same thing is going on unsung: there is a web-based app, with user sessions. We want to monitor the user sessions experiencing the serious errors. If the number of such sessions grows above a certain threshold (say, 5% or 10%), alert the oncall person. Of course, the next thing that happens is that there are very few active sessions at night, say 3 of them, and one of them experiences an error, the percentage of failure is high, so the oncall gets paged. And gets paged night after night. This problem has been bothering me then, so I've had a workaround made for it, based on the number of active sessions. For my project only, since that Unnamed Company is famous for its Not Invented Here Syndrome, and there is basically no use trying to propose any general solutions to the groups that own these general solutions.
I've been recently reading a book on statistics, and it turns out this exact problem has been solved a very long time ago. It's the p-value. The same p-value that the scientists are required to use in their scientific papers (me guessing that most of them are doing it without much understanding). The problem becomes framed like this:
There is a certain probability for a session to fail. By observing the active sessions, we're doing a sampling of them and estimating this probability. With few samples, the estimator has a high variance, and basically we can't tell if the estimated probability really is high or we accidentally managed to get the few bad samples. The p-value is the probability that we've only accidentally seen the high probability of failure due to the unlucky sampling. When the p-value gets below 0.05 (the common boundary for the scientific papers), it meas the 95+% probability that the effect we're observing is true.
So the solution is simple: all we need to do to stop the spurious alerts is compute the p-value and alert on the p-value crossing the threshold. And in retrospect, my hack was an uneducated approximation of that.
Which brings an interesting point. The described issue has been present at the Unnamed Company for literally more than a decade or two. The Unnamed Company also prides itself on hiring the Smart and Highly Educated engineers, many with PhD degrees. How come that either none of these Smart and Highly Educated engineers paid attention, or all their efforts to solve the issue got thwarted (just like mine did)?
