There’s some ongoing discussions in blogland as a result of a recent blog posting here by my good friend Tim Bass. In his post, he applauds the use of analytics to the point of damning the current crop of CEP vendors for being made up of companies that don’t “support or advocate advanced analytics”.
Opher responded here by saying that most of today’s CEP problems do not require advanced analytics, and used the metaphor of blind men feeling an elephant.
For me, I confess that I fail to understand the debate. I’m curious as to the term “analytics”. Just what is “Advanced Analytics” as applied to CEP? Is it advanced situation detection – in other words, will I use advanced analytical techniques to detect a situation? Or is it advanced visualisation – will I be able to produce a real-time updating graph or chart based on a series of complex mathematical calculations?
Here at WestGlobal, we tend to focus on detecting whatever situations need to be detected, in the lowest latency required. We can use a variety of techniques, depending on the situation. We might find that one situation may be detected with a series of very simple rules, but that another requires enrichment from a data source, while joining with a variety of other data streams, and a dependency on the occurance of a series of other situations within the past 2 minutes. But is this advanced analytics?
For me, analytics is not a requirement – at most it’s an implementation detail for a specific problem. As Tim points out, there are probably many examples of event processing that require very sophisticated processing techniques (I’m avoiding the term analytics). As Opher points out, most applications today don’t require it. And I’d like to point out, if the requirement existed and someone could make lots of $$ doing it, then chances are there’s somebody doing it already (and they’re keeping it a secret for as long as they can).
I’d love for Tim (or anyone else) to post an example or two of specific problems that exist that require advanced analytics. Otherwise, it may be that I, and many other people, conclude that the trumpeting of the requirement of advanced analytics is just another type of snake oil.
June 26th, 2008 at 10:16 pm
Hi Brian, advanced analytics refer to the use of statistical techniques, machine learning, bayesian networks in order to mine patterns, predict future events, infer statistical causality etc… there are certainly tpyes of applications that require it, and some others can work perfectly well with determinstic situations.
June 27th, 2008 at 12:05 am
Hi Opher, thanks for the response.
Perhaps I was being too unclear.
Traditional analytics is often used to determine a value. For example, what is the likelyhood of this transaction being fraudulent? Answer = 0.76. Etc, etc.
How is this used in CEP? Answer, it isn’t …. or at least I don’t believe it will be used in this way.
In CEP, we will instead determine a value that needs to be detected, and react accordingly. For example, when probability for a fraudulent transaction reaches 0.85 or greater, then shut down the account.
In my view, this is not analytics (although the maths might be pretty similar). This is pattern/situation detection.
Furthermore, an interesting problem, is that the situation detection may be “distributed” in such a way that the description of the final pattern or situation is an abstraction of earlier patterns and situations.
This causes problems when trying to maintain definitions – each abstraction must also be aware of changes to its dependencies…
Brian