Those Nasty “Hairball” Analytics Projects: Usually There’s A Data Warehouse At The Core

·  Share |

James Kobielus
Information Management Blogs, July 12, 2010

Sometimes I feel like Ol’ Doc Jim.

As Forrester’s lead analyst on data warehousing (DW), my core job often involves diagnosing enterprise analytics practitioners’ DW aches and pains. I try to cultivate a reassuring bedside manner and give them something for both immediate and long-term relief from their problems.

I receive all Forrester customer inquiries on DW matters, many of which are from IT practitioners who have hit a wall of intractable technical, operational, or vendor-related issues. Those sessions usually involve me probing for the source of the IT practitioner’s DW-relevant woes. If all of these issues could be isolated to the DW itself, my life would be much easier. But customers’ DW concerns are often tangled into stubborn knots of business intelligence (BI), master data management (MDM), data integration, data governance, business process management, IT service management, and other critical infrastructure, operations, and application issues. Often a seemingly DW-based problem such as poor-performance queries reveals that the root cause is somewhere else entirely, and the DW itself is the least of their problems.

I like to refer to these as DW “hairball” issues, because they are often a complex, bewildering series of intertwined problems with no clear starting point or easy solution. Often the siloed nature of the customer's own IT groups can aggravate the hairball, as various, isolated, and mutually distrusting technical specialists each propose limited solutions that have nasty consequences elsewhere. For example, reallocating DW hardware resources to speed up queries can choke off the resources needed for concurrent queries, extract transform load (ETL) jobs, and/or in-database analytics. In a similar vein, consolidating many DWs, data marts, and operational data stores into an enterprisewide DW can save money on hardware, software licenses, and database administration labor, but it often screams for a long-term, complex, administratively intricate new overlay of cross-enterprise data governance and MDM tools, roles, and workflows. Or a customer’s real-time BI applications can quickly bog down into chronic latency problems if they haven’t provisioned ample processing, storage, and transmission capacity throughout their data management infrastructure.

Like any conscientious DW doctor, I don’t necessarily wash my hands of the customer’s pain. And I don’t always tell them to seek out another specialist or two or three who might be able help them dislodge or detangle the hairball (though I sometimes suggest a follow-on joint inquiry involving the most appropriate other Forrester analyst). My approach on these tangled inquiries is simply to help the customer zero in on the likely source of the problem, to identify the projects or tasks that might be necessary to address it globally, to prioritize them, and to identify any budgetary, staff, or technical constraints that may prevent them from moving off square one.

It’s not rocket science, but it’s a level of architectural and planning assistance that an industry analyst — not privy to their pain — is equipped to provide. I can also help the customer drill down to various degrees on all these other adjacent technologies, if need be, and the fact that I cover a wide range of analytics and data management specialties helps me in that regard. But, in the context of a half-hour phone inquiry, that’s often overkill when the IT practitioner simply needs a 30,000-foot view — and when the client him or herself can do those topic-specific drilldowns as well as or better than me.

Hairballs are nasty things to choke on. Nevertheless, DW issues are at the heart of many enterprises’ increasingly complex analytics initiatives. The DW is as good a place as any to start in untangling and managing the myriad issues that practitioners confront.

Quite often, a little quality time with your friendly neighborhood DW analyst is just what the doctor ordered.

Jim also blogs at http://forrester.com/blogs/business_process.