The BTM Blog

The EPTS Use Case working group met on Monday in Rome before the DEBS 08 conference got underway.  It was hot stuff – temperatures touched 40C or 104F, laptops overheated, but great progress was made.  The new use case template should be available on the EPTS website in about four weeks.  After that, work should begin on trying to produce a reference architecture.  It was great to see Alex from Betfair, an end user of CEP software, with great views and great ideas.   Alex had some great ideas and great views on the benefits of CEP, and it benefits everyone to have real customers describe the value the derive, and the areas they wish to see progress.

The EPTS seems to be achieving some real momentum – no small part in thanks to the tireless Opher who pushes and organises everything behind the scene.  My thanks to Opher and to the DEBS 08 organisers for a great conference, but it reminds me that everybody has a part to play.  Those with an interest in CEP also have a responsibility to be heard and to play a part.

“Ambition should be made of sterner stuff”

There’s some ongoing discussions in blogland as a result of a recent blog posting here by my good friend Tim Bass.  In his post, he applauds the use of analytics to the point of damning the current crop of CEP vendors for being made up of companies that don’t “support or advocate advanced analytics”.

Opher responded here by saying that most of today’s CEP problems do not require advanced analytics, and used the metaphor of blind men feeling an elephant.

For me, I confess that I fail to understand the debate.  I’m curious as to the term “analytics”.  Just what is “Advanced Analytics” as applied to CEP?  Is it advanced situation detection – in other words, will I use advanced analytical techniques to detect a situation?  Or is it advanced visualisation – will I be able to produce a real-time updating graph or chart based on a series of complex mathematical calculations?

Here at WestGlobal, we tend to focus on detecting whatever situations need to be detected, in the lowest latency required.  We can use a variety of techniques, depending on the situation.  We might find that one situation may be detected with a series of very simple rules, but that another requires enrichment from a data source, while joining with a variety of other data streams, and a dependency on the occurance of a series of other situations within the past 2 minutes.  But is this advanced analytics? 

For me, analytics is not a requirement – at most it’s an implementation detail for a specific problem.  As Tim points out, there are probably many examples of event processing that require very sophisticated processing techniques (I’m avoiding the term analytics).  As Opher points out, most applications today don’t require it.  And I’d like to point out, if the requirement existed and someone could make lots of $$ doing it, then chances are there’s somebody doing it already (and they’re keeping it a secret for as long as they can).

I’d love for Tim (or anyone else) to post an example or two of specific problems that exist that require advanced analytics.  Otherwise, it may be that I, and many other people, conclude that the trumpeting of the requirement of advanced analytics is just another type of snake oil.

Next week, I’m attending the DEBS 2008 conference in Rome. The conference is a great occasion for people from the academic and industrial worlds to mix and share ideas. Unfortunately, I’ll be cutting my attendance short due to customer commitments, but I am looking forward to seeing how event processing is maturing as a technology on many fronts. The DEBS conference pushes large scale considerations to the fore, and focuses less on the minutiae of implementation such as event processing languages or correlation techniques. Instead, discussions will tend to focus on the distributed aspect of event processing, with issues such as security, availability and reliability, volumes, filtering, event ordering, and synchronization all being presented. And I’m especially looking forward to the software demonstrations.

Last week, I visited a Tier 1 European Telco, and spent some time with their Technical Operations director. We discussed the growing trend towards outsourcing, and in particular the difficulty that executives have in placing their trust in an external company running all of their IT systems and providing at least the same level of service performance as before the outsourcing occurred. A pretty basic requirement, I’d say. But it appears that traditional monitoring tools, focussed on infrastructure and the like, are really designed for “reactionary” organizations – that is organizations that wait until a problem has occurred and then rush to fix the problem. It struck me that this type of monitoring is binary in nature. Its alive = everything is OK. It’s not alive = react and fix it. When you think about it, traditional monitoring nearly always fits this pattern. This model also carries through to performance monitoring. Each single event is examined and is compared to a threshold – some value that equates to “Below the line = OK – but above the line = react and fix”. So I equate traditional monitoring with single-event threshold based monitoring for reactionary organizations.

The executive in question described the ideal solution as one that could predict what was going to happen next. I don’t think he meant a crystal ball for taking to the racetrack or picking next weeks Lotto numbers – but what he described as the ideal solution was one that would inherently understand what was considered “normal”, and to also understand what was considered “abnormal” – from a business activity point of view. By looking for patterns of abnormal activity over shorter periods of time, disasters can be avoided by spotting early warning signs and fixing smaller problems before they turn into giant ones. His ideal solution has a “Big Brother” feel to it, where all behaviour is analysed and examined and hit teams immediately dispatched to fix anomalies.

It made sense to me and gave me a lot of food for thought. What I like is the top-down approach – which made me ask the question “Why do we monitor bottom-up? How do I know for sure that the technical problem is causing a business impact?”.

Do I even care about a technical problem that doesn’t cause any business impact? Very Zen-like. Reminds me of trees falling in forests.

Brian Connell

CTO

The Event Processing Technical Society (EPTS) has been officially launched, and the website is at http://www.ep-ts.com. A lot of credit has to go to Opher Etzion (http://epthinking.blogspot.com/) for being the main driving force and organizer behind the scenes, but there are lots of other names too that have been involved from the start.

The EPTS is an unusual group by today’s standards. It is made up of a great mixture of individual people and companies from the academic and commercial worlds, and even includes customers and analysts. It has officially met three times, and the fourth meeting will take place later this year. Work has been ongoing for some time now, albeit in the background. As the group gathers momentum, I anticipate a busy number of years ahead.

Brian Connell

CTO