WHY AGILE DATA WAREHOUSING?

Data warehousing efforts are enormous projects, consuming on average $13 million and 18 months of calendar time. [1] Even their smaller siblings, data mart applications, can strain the budget and time frames available to their departmental sponsors. Ironically, the large investment required for these projects drives most organizations into pursuing them in exactly the most unproductive way possible: a large, plan-driven waterfall method.

The term "waterfall method" was coined in 1970 by Dr. W. Royce in a paper where he laid out the conceptual steps to building a large software application: system requirements, application requirements, analysis, design, coding, testing, and operations. Royce states that one could finish each one of these steps before going on to the next, but he did not advocate it. Unfortunately, the readers of this paper and the folks that later crafted the development methods followed for the next thirty years somehow skipped over the warnings embedded throughout the paper:

"...The implementation described above is risky and invites failure...If [fundamental aspects of the system delivered] fail to satisfy the various external constraints, then invariably a major redesign is required... The required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated. Either the requirements must be modified, or a substantial change in the design is required. In effect the development process has returned to the origin and one can expect up to a l00-percent overrun in schedule and/or costs." [2]

Having missed this dire prediction, project managers the world over have organized software projects during the past four decades using strict watefall methods. It is not unusual for a waterfall, enterprise data warehouse project to involve a 1,000-line project plan to which the project managers demand unwavering compliance. The only problem was that the results have been for the most part extremely disappointing to the program sponsors. A pair of "Chaos Reports" published by the Standish Group detailed that 45 percent of even small waterfall projects do not deliver all they promise on time and on budget, and that this failure rate quickly climbs to 100 percent as project grow in size. [3]

It just does not make sense to try to plan out every step of delivering a large system that has never been built before. We can see the big picture of a project, yes, but we are very bad a predicting which of a thousand contingencies that might arise in requirements, analysis, design, and coding are going to seriously impact the total effort required to deliver our applications. Trying to foresee all these possible events is tantamount to "outdriving our headlights" –we are going to hit a rock or steer completely off the road if we try to drive from here to Patagonia by dictating in advance every turn and swerve we will need to get there. Intuitively, we must set a destination, sketch a high-level path to get there, and then respond to road conditions as we encounter them.

PROBLEM ALREADY SOLVED FOR NON-WAREHOUSE APPLICATIONS

At the close of the 20th century, software projects including data warehousing efforts needed a way to institutionalize into the development process this adaptive, "drive within your headlights" philosophy that works so well in the rest of life. Process innovators on the transaction-processing side of information technology hit upon an answer: iterative and incremental delivery(IID). As the more successful of the IID approaches came to be called "agile," the thought leaders behind these methods gathered in Park City Utah in 2001 to verbalize the common elements found in their frameworks. The resulting "Agile Manifesto" was simple and compelling [4]:

We have come to value:

While there is value in the items on the right, we value the items on the left more.

At last count, there are now over a dozen Agile approaches including XP, Scrum, Kanban, FDD, DSDM, and RUP. Some of these are process heavy and approach the level of fully-developed methodologies, while others are very light weight and remain only "collaboration models." Yet all of them aim at achieving the Chaos' reports' recipe for success: small teams with small scopes. The benefits these practices have delivered can be seen in a recent survey conducted among OLTP development teams: 82 percent responded that Agile methods improved productivity, 78 percent cited increases in customer satisfaction, and 77 percent reported that Agile methods lifted application quality.[4]. The only problem discernable was that no one had yet tried for the other half of IT–data warehousing and business intelligence (DW/BI).

CEREGENICS INNOVATION

To fill this gap, DW/BI professionals responded with "Agile Data Warehousing" -a method that synthesizes several Agile methods and is specifically designed to bring iterative and incremental development to the world of very large database projects. Based mostly upon the research of Ralph Hughes (MA, PMP, CSM) and the field work amassed by the data management team at Ceregenics, Inc., Agile Data Warehousing combines the best of Scrum, XP, and Kanban into a six-step maturity path for teams pursuing data warehouse and/or business intelligence projects.

Ceregenics' early work employed only the collaboration model of Scrum and several of the development techniques found in XP. This combination led to several encouraging results for projects in the telecommunication and aerospace industries. However, project initiation and close out proved to difficult during these initial efforts, prompting Ceregenics to add more "industrial-strength" techniques for requirements management (borrowed from RUP) and quality assurance (adapted from the later work of several thought leaders in the Agile community).

Scaling the fundamental Scrum approach, too, proved challenging at first when the method was utilized to coordinate the many teams required for enterprise data warehousings programs. To fill this gap, Ceregenics incorporated a new, probability-driven scheduling approach arising in the world of traditional project management called "Critical Chain," a replacement for the "critical path" taught in most project management programs. Enterprise warehousing efforts also required Agile development teams to coordinate with the multiple competancy centers such as enterprise architecture, datbase administration, and operations support. To elminate the need to perfectly cadence the work of all these intersecting disciplines, Ceregenics adapted the buffer-driven approach of Kanban to avoid the conflicts that can arise between multiple time box-driven technical teams.

The results of this fully adapted Agile data warehousing method have been notable. When run side-by-side with waterfall projects of very similar scopes, Agile Data Warehousing projects have accelerated the number of objects delivered per developer week by nearly a factor of three while reducing project costs by over 50 percent. Ceregenics' continuing research now focuses upon what has come to be the three pillars of Agile Data Warehousing -"maximizing the work not done" through

• 1. Agile methods adapted for DW/BI
• 2. next-generation back-end tools that streamline the data architectures and accelerate load times
• 3. front-end tools that allow greater self-service while maintaining a single approach to the truth of the enterprise

Agile Data Warehousing now exists as a set of ready-to-deliver training modules, ready-to-deploy templates and policies, ready-to-impelment tools and utilities, and ready-to-start assessments plans to guide organizations new to agile along the fastest path to incremental delivery of DW/BI applications.

As always, the objective of this Agile Data Warehousing suite continues to be "the delivery of high-quality business intelligence in a time frame that maximizes business value."

[1] DM Review magazine October 2004 survey of their readership. [2] Royce, Winston W. 1970. "Managing The Development Of Large Software Systems." Proceedings, IEEE WESCON, August 1970. New York: Institute of Electrical and Electronics Engineers., p 1-2. [3] The Standish Group International. 1995.The Chaos Report. http://www.standishgroup.com (accessed February 2007). The Standish Group International. 1999. Chaos: A Recipe for Success. http://www. standishgroup.com (accessed April 2006). [4] Source: http://www.agilemanifesto.org [5] Ambler, Scott. Feburary 2008. "Agile Adoption Rate Survey Results: February 2008," Dr. Dobb's Journal. www.ambysoft.com/surveys/agileFebruary2008.html (accessed March 2011).