The BTM Blog

A surprisingly interesting question; are we talking about a) the processing of complex events or b) the complex processing of events? Both possibilities lead to interesting follow on questions such as what is a complex event? or what constitutes complex processing of events? This has been a source of much discussion within the event processing community and rather than present the argument in this post, I’d like to present the outcome of the discussion.

Recently, David Luckham and Roy Schulte produced an event processing glossary which defines complex event processing as the processing of complex events, and defines a complex event as an event that contains other events (known as it’s members).

Of course, the members of one complex event can be other complex events, leading to a hierarchical tree (or even graph!) structure of nested events. The processing of complex events can consist of tasks such as aggregating a set of events into a new event; deriving events from other events; filtering events based on their arbitrarily deeply nested members and so on – a rich paradigm indeed!

Obviously this hierarchy of events must start somewhere; with simple events that do not have other members. According to the event processing glossary, these simple events can be used to represent anything that happens or is perceived as happening. Starting with source events a powerful complex event processing network can be built which uses tasks such as aggregation to build intelligent models of business processes in an organisation.

One of the first tasks of the newly formed Event Processing Technical Society when they meet, will be to produce an event processing glossary which will hopefully become an industry standard. We look forward to seeing what they come up with!

In our work with IT intensive enterprises, we observe different approaches to the implementation of IT and service monitoring. These range from a fragmented model dependent on multiple and effectively isolated ‘point’ solutions through to highly unified BPM (Business Process Management) models providing excellent monitoring performance but typically demanding significant investment, technology and process re-architecting and imposing restrictions on supported technologies or vendors.

In our view, the most successful approaches in terms of maximising IT ROI, technology performance and improving Customer Experience are based on a combination of pragmatism and planned coherence. Pragmatism comes from an approach that supports highly heterogeneous IT and network environments comprising multi-vendor technology deployed often over as many as 10 or more years. Coherence recognizes the need to combine different monitoring ‘models’ to achieve a rich, timely, efficient and complete view of how IT and network systems are interacting to deliver excellent Customer Experience.

At WestGlobal, we have found a 4-Quadrant grid is useful in explaining the ‘coherence’ element. On the vertical axis, ‘Monitoring Focus’ moves from Technology in the ‘South’ to Business Activity/Customer Experience in the ‘North’. The horizontal axis measures timeliness or immediacy, with non-real-time (nrt) to the West and real-time (rt) to the East. All monitoring approaches can be positioned on this grid and we can outline the key characteristics of each quadrant as follows. . .

NW (nrt/ Business Activity Focus): Business Intelligence / Data Warehouse exceptionally rich in the ability to create marketing and businesses views of customer and business activity; but ‘after the event’.

SW (nrt / Technology Focus): Historical technology performance measurement. Typically based on log and CDR file analysis and frequently used for capacity measurement and planning of underlying IT and network elements.

SE (rt / Technology Focus): Here the focus is typically on monitoring and alerting the health of individual or groups of IT / network elements or application servers (e.g. WEB servers). But this provides limited insight into actual Customer Experience which may depend on the interaction of many elements and servers.

NE (rt / Business Activity Focus): Here the focus is on measuring the health and performance of actual business activities. It is interesting that while it is this layer that the customer interacts with and can therefore reasonably be argued as the most important in measuring Customer Experience, monitoring often remains focused on the underlying technology i.e. the ‘South Side’. The emergence of Complex Event Processing as a means to enhance the performance of solutions in the NE quadrant provides excellent opportunities for enterprises to detect patterns and events that signal potential Customer Experience degradation even before the customer is impacted.

The most effective monitoring approaches reflect the need to ‘cover the bases’ in the above grid. This implies a need for organizations to have some degree of coordinated planning across the complete monitoring architecture. With such an approach, additional benefits arise from ‘Grid Synergies’. For example, while it can often be difficult for Business Intelligence solutions to access certain network or low level data directly, the data can often be proxied via elements in the real-time side of the grid. Similarly, integration between the NE and SE quadrants combines real-time Business Activity Monitoring with the ability to ‘drill-down’ into the underlying technology only when necessary, for example to investigate a degradation or failure in a Business Activity. This is approach we have adopted with Vantify Experience Centre, in other words an ‘East side’ solution.

In many organizations, despite knowing what everyone is doing, finding out what’s actually going on can be next or nigh impossible. Operators are delivered notifications about everything that happens, good and bad, and they end up in a fog of detail.

Complex Event Processing (CEP) allows us to tackle this “wood for the trees” problem in two ways. First, it allows the unimportant details to be discarded, leaving only those that are relevant. Second, it allows those details to be tied together into a business oriented view.

It is often the case that under normal operation, things don’t always go to plan. A user’s session may take longer than normal to start or a credit card validation times-out. These once-off errors are not that interesting, particularly, as by the time they register with the support staff, the cause of the error has passed and everything is back to normal. What’s far more interesting is knowing when five users in a 10 minute window are experiencing problems or the credit card validation fails 3 times in a row. CEP’s Event pattern matching solves this problem very nicely. It allows detection of patterns like “X is followed by Y or Z in a 10 minute window”. When the pattern is detected an alert notification can appear on an operators dashboard or be sent by text message to the support staff. Only the events that indicate something of interest is happening get presented to the operators.

Another problem is that the monitoring that does take place does not give any real insight into how the business is performing. Low-level service monitoring can tell us that the web application is up and running or that the credit-card payment gateway is functioning ok but does not tell us how these services are supporting (or not) the business side and, ultimately, the customer. CEP solves this problem by allowing the low-level happenings (a user a logged in, a credit card payment was processed) to be aggregated into a more high-level event (user bought something over the web). This is known as “Event Abstraction” where a set of basic events are represented by a more high-level one. Thus, the actual business value emerges from the deluge of detail but the detail is retained should it be needed.

So CEP allows the bigger picture to emerge from overwhelming amount of detail. That’s the well informed ones! The not-so-well informed ones may be oblivious to what’s happening until they get phone calls from irate customers.

Conrad O’Dea

Chief Architect