The BTM Blog

Back from beautiful Trento in Italy where the 5th EPTS Symposium was held.  Marvelous location, and we even managed to find a really beautiful Michelin-star rated restaurant.

While it was great to meet up with some friends and colleagues, and there were some very interesting nuggets at the conference, my overall impression was one of disappointment and frustration.  We are still struggling with basic concepts, arguing about the definition of Event Processing (and there are some very … different … views), and still haven’t managed to produce anything that either identifies the major components you’d find in a reference architecture, or found commonality in any of the use cases.  Actually .. working groups aside, that’s kinda what we ended up with last year.  And the year before…

And before I offend the very people I wish to praise and give credit to, let me be clear.  It’s most definitely not the fault of the people collaborating on the working groups.  They’re contributing.  Giving up their precious time.  In order to make progress, we really need to get more people involved, and we need to set clearer mandates on the deliverables for these working groups.

A new working group was proposed to promote the EPTS.  This will also involve publishing the public deliverables from the working groups, and encouraging new members to join and contribute.

Here’s hoping that this time next year, I won’t feel another Bill Murray moment coming on!

Having read Opher’s excellent blog posting on describing CEP maturity models, I found that while I agreed with Opher’s descriptions of differences between messaging and events, I disagreed with describing phase 3 of CEP as “towards looking at ‘event clouds’ instead of events one-by-one”.

The glossary notes on the definition of an event cloud says that “CEP usually refers to event processing that assumes an event cloud as input, and thereby can make no assumptions about the arrival order of events”. This implies that events “arrive” – just not necessarily in a defined order such as creation time, etc. It also implies that events may arrive one-by-one. It certainly does not preclude one-by-one processing.

The other implication of Opher’s posting is that the cloud may somehow be processed as a whole. Looking at the definition of a cloud, it is made up of many events of differing types where each event may have been created at a different time, and may have a different time-to-live value within the cloud. But in order to make the entire cloud accessible to an event processing agent as a whole, a mechanism must exist that persists the events within the event cloud and manages the cloud events according to their time-to-live values.

(An easy parallel to this view is “ordinary” data processing where sets of persisted data (i.e. events) are made available for queries. Data/Events are stored in tables and keyed by their time-to-live values. Obviously, given a large enough quantity of events, the storage and processing requirement may be considerable.)

But I disagree that this is the only way to define CEP. Indeed it has long been a fiery debate among the CEP community on how, exactly, an event cloud may be practically processed without creating a partially ordered set of events (which may be regarded as a stream of one-by-one events). I would argue that persisting an entire event cloud is fine for ad-hoc processing and analysis, but that the vast majority of CEP involves detection of predefined patterns and is efficiently performed as a form of one-by-one processing.

In Operational Intelligence, applying CEP to the enterprise event cloud is a practical application whereby predefined patterns are detected and acted-upon in real-time. It would be impractical and practically impossible to persist the entire event cloud as the volumes of events are considerable, and the rate of events would require a lot of expensive equipment to provide the required processing power. Yet Vantify Experience Center uses CEP to process an enterprise’s event cloud, and provides real-time intelligence to operational staff to meet challenges and opportunities for maximum business benefit with relatively inexpensive equipment. The events are processed one-at-a-time rather than as a single cloud for efficiency, and the value to customers has been demonstrated many times. To imply that CEP excludes one-by-one processing is inaccurate and wrong – rather it is an important and critical subset of CEP.

The evolution of enterprise monitoring has evolved greatly in recent years by focusing on the service being delivered to the Business rather than the health of the underlying IT infrastructure. We have ITIL and ISO20000 frameworks and certification to learn how to align the services to Business needs, and we also have CMDB and BSM solutions to help organize and manage our IT resources. And while these products have improved how IT delivers services and prioritizes resources, it’s only half the picture and severely limits the ability of IT Operations to detect and react to threats. So while IT has better tools to organize management of IT infrastructure resources, IT Operations is still a stressful place where most problems are still reported by users.

An analogy that’s often used is monitoring the human body. If you monitor the Key Performance Indicators (KPIs) of the human body, you probably end up a list such as heart rate, respiration, and temperature. You may even try to develop a more holistic approach by not only monitoring each KPI, but also monitoring the relationship between each KPI. Therefore if you see an increase in the heart rate, you also expect to see a corresponding increase in body temperature and respiration. And if you see such an increase, you might infer that the body is exercising – perhaps riding a bicycle.

To me, this represents a problem. Sure, the body may be biking, but is it going in the right direction?

Applying this same technique to IT system management is inaccurate and results in many business problems remaining undetected. It is just not possible to qualitatively monitor business activities from infrastructure data.

But a complete 360° view is possible. By monitoring the myriad interactions and units of work that make up the business transactions and activities, the IT organization now provides a business context for the work that the infrastructure is carrying out. We use Service Activity Monitoring (SAM) to qualitatively monitor the units of work, and Business Activity Monitoring (BAM) to qualitatively monitor the desired Business Activity or transaction. By looking qualitatively at Service Delivery from Top Down as well as Bottom Up, an IT organization can control all aspects of Service Delivery with all the benefits such as lower costs, higher revenues, and happier customers.

Note: In my previous musings, Doug McClure kindly pointed me to his excellent blog, and mentioned that he hadn’t heard of “Service Activity Monitoring” (SAM) before. It’s a term I used after I first came across it from a presentation given by CITT online describing an architecture overview in Deutsche Post. Essentially, SAM sits between BAM and infrastructure monitoring and monitors how services from applications are being delivered as units of work within a business process.

The EPTS Use Case working group met on Monday in Rome before the DEBS 08 conference got underway.  It was hot stuff – temperatures touched 40C or 104F, laptops overheated, but great progress was made.  The new use case template should be available on the EPTS website in about four weeks.  After that, work should begin on trying to produce a reference architecture.  It was great to see Alex from Betfair, an end user of CEP software, with great views and great ideas.   Alex had some great ideas and great views on the benefits of CEP, and it benefits everyone to have real customers describe the value the derive, and the areas they wish to see progress.

The EPTS seems to be achieving some real momentum – no small part in thanks to the tireless Opher who pushes and organises everything behind the scene.  My thanks to Opher and to the DEBS 08 organisers for a great conference, but it reminds me that everybody has a part to play.  Those with an interest in CEP also have a responsibility to be heard and to play a part.

“Ambition should be made of sterner stuff”