The fourth and final suggested tenet for an agile DW/BI manifesto focuses upon the unfortunate tendency of DW/BI departments to rigidly restrict themselves to their familiar techniques and tools. Whether it be a particular data modeling style or a given set of data integration and visualization products, the tool set from previous decades is not a perfect match for teams that want to deliver incrementally. Today’s thought leaders and software vendors are rapidly innovating faster, better, cheaper ways to deliver business analytic services and these innovations must be included in our agile DW/BI strategies. Whereas the original manifesto’s fourth tenet emphasized “responding to change over following a plan,” the DW/BI adaptation of this notion encourages teams to err on the side of leveraging:

4. Appropriate technology over maintaining an infrastructural monoculture

DW/BI Teams Wary of a Certain Kind of Change
The signatories of the original manifesto cautioned application developers against persevering with any particular project plan in the face of new conditions. The typical waterfall project maintains a complex plan based upon PERT algorithms and Gantt charts that only the project manager can begin to understand. Unfortunately, these plans are drawn up at the beginning of a project, when the team knows the least about the problem they are trying to solve. As the signatories of the original manifesto pointed out over ten years ago, plans should be subordinate to change because new requirements equal business opportunity. To persist with a bad plan in the face of new opportunities leads to the worst kind of wasted effort, and cannot avoid undermining the team’s delivery speed or the customer’s satisfaction. Accordingly, high performance agile teams tend to plan light and revisit their assumptions and delivery forecasts frequently. Many agile methods allow a team’s business partners to alter the project direction as much as they would like, as long as they provide the new requirements in a lightweight format at the right point in the team’s planning cycle.

DW/BI teams with significant data integration work cannot be so sanguine about changing requirements, however. They must build what amounts to several applications end users will never see for every new metric or qualifier they place on an end-user’s analytic dashboard. Every new request can thus force upon them five to ten times the work that front-end agile teams would have to consider. Moreover, DW/BI projects have data models buried deep inside their code—models with a high level of dependencies and complexity to rival the PM’s complex project plan. A small change in requirements can require a larger change in the database, which in turn can suddenly undermine the effectiveness of months of programming on extract, transform, and load (ETL) applications. If production data has already been loaded into the warehouse, then the team must also build conversion scripts to migrate millions or billions of records to the new target schema. Such scripts cost much in both labor and money to program, test, execute, and validate—despite the fact that the team needs to run them only once.

Indeed, allowing for changing data requirements is a fast way to exhaust a project sponsor’s funding and enthusiasm for a data warehousing project. For this reason, even veteran agile data warehousing teams adopt a very non-agile approach to their database designs. They want all the requirements up front so they can design the target data schemas in one pass, thus avoiding ruinously expensive re-work. This unfortunately works against agility because it gives projects very long start-up phases and makes the team very reluctant to consider new requirements that involve target schema changes.

Agile DW/BI Teams Need Appropriate Technology
Fully agile data warehousing has a two-fold mechanism to prevent the enormous harm inconstant data requirements can inflict upon a business intelligence project:
a) Utilize an advanced data modeling technique to design an adaptable target schema, and
b) Acquire a new generation of tools that leverage these advanced data modeling techniques.

Most DW/BI departments today follow either the third-normal form data modeling techniques inherited from Inmon advocates, the dimensional modeling approach offered by the Kimball school of practitioners, or a combination of the two. Unfortunately, both Inmon and Kimball approaches eventually result in enterprise data schemas involving so many dependencies and data records, that even the slightest enhancement will require enormous re-engineering and conversion effort. Caught in this situation, developments teams—especially those funded by business departments—frequently opt to address new requirements with small, separately designed solutions rather than risk any modification to the enterprise data warehouse. This fear of touching the corporate DW/BI applications only leads to a proliferation of unaligned silos of disparate data, quickly undermining the value of the enterprise warehouse itself.

The advanced data modeling techniques utilize sixth-normal form or associative concepts to generate target schema’s that are “three-way” robust: 1) teams can progressively extend the designs without having to re-engineer or re-load existing warehouse tables, 2) when re-engineering is required, it is necessary only for a very small portion of the schema, and 3) the resources needed to load the schemas scales out linearly or better as a function of the number of tables and records involved.

Though load and retrieval performance are not serious concerns for warehouses created using this new approach, advanced modeling techniques do incur one of two other disadvantages: either they “hyper normalize” the schema and create many more tables to manage, or they “hyper generalize” the model so that that much of the design becomes expressed in meta data records rather than directly visible structures. Either way, DW/BI teams often struggle to manage this new complexity. Luckily, both hyper normalized and generalized schemas prove to be machine intelligible, and vendors have created a new generation of tools for managing these advanced warehouse schemas.

The best of these tools allow designers to model entities and data flows primarily at the business/conceptual level because the warehouse management application can infer the remaining updates that must occur with each new business requirement. These tools can analyze the delta implicated by a business model change, unload the production data, update the existing tables, and reload the data into the new schema. With this level of support, DW/BI teams no longer need to fear the labor and risk a small requirements change can inflict, allowing them to focus instead on progressively evolving the warehouse to stay aligned with business needs.

The Dangers of a Technical Monoculture
Unfortunately, the minute an agile solutions architect suggests utilizing a new tool, IT management usually raises some very strong objections. The most common is “We can’t afford to have any more tools in this shop.” A slight variant of this objection is “We can bring in any new tool that our current systems integrator supports.” Both of these objections aim to maintain a single solution set for DW/BI, that is, to maintain a technical monoculture. Regrettably, striving for this technical objective can result in a big loss for IT’s business partners and the competitive prospects of the company.
Adhering to a technical monoculture makes sense when our industry is experiencing days of static technology. Keeping fewer products in the department’s toolkit does keep down the number of skills IT needs to recruit and maintain in its labor force. Small toolkits require less cross training, and thus provide greater flexibility for managers to re-assign staff members between projects as delivery schedules might require. Small toolkits also involve fewer application upgrades, reduce the numbers of repositories to administer, and minimize the conflicts between packages sharing hosts in the machine room. Restricting a vendor list to a single systems integrator (SI) has advantages in that it makes the DW/BI department’s vendor relationships easier to manage. One SI means fewer planning meetings for IT managers, and many fewer monthly billings to review. It is also far easier to initiate and update policies regarding architectural and procedural policies when a department has one outside partner to work with.

Unfortunately, data warehousing is not in a technologically quiescent period. Given new developments such as cloud computing, warehouse appliances, and big data, we have rarely seen the set of DW/BI solutions changing as fast as they are today. Agile methods in particular have created a time of disruptive technology, enabling IT to triple its ability to provide the business analytics needed to overcome a company’s competitors. Refusing to try a new tool or filtering all tool considerations through the lens of a single, non-agile SI is to stand flat-footed before this extremely competitive business environment. Even the simplicity of a DW/BI department’s labor force cannot be maintained by DW/BI departments that are slow to innovate, as their most talented staff members will trickle away to find opportunities elsewhere to work with the new generation of tools.

In IT, we constantly automate the drudgery found within our work processes in order to invest our rare labor resources in adding value through progressively higher design activities. We want to steadily raise our operating abstraction, so that an increasing portion of design work occurs at the business/conceptual levels. The more a business conversation can be immediately translated into a business systems improvement, the better our companies will perform in the marketplace. To keep IT’s work life simple by deliberately foregoing the advantages of DW/BI tools with advanced capabilities is tantamount to insisting on building business applications in assembler because languages such as COBOL and SQL require some effort to learn. It is a strategy error the marketplace will soon set right.

IT managers fearing larger toolkits should realize that the optimal course for incorporating agile DW/BI products is actually quite free of extremes. DW/BI departments do not have to embrace each and every new tool that comes along. They can maintain the advantages of small toolkits cited early by simply blending a few new tools into a solid base of “legacy” tools. DW/BI departments need only to pursue the best of the new tools as part of a “strategic” add-on that equips their companies with the sharp “cutting edge” that fast business intelligence can provide. In fact, this solution is already incorporated into the design of many agile products. The best agile tool vendors realize they will sell more far product by offering bolt-ons to the legacy DBMS and ETL packages than by trying to overthrow the existing solutions altogether. Eventually, the legacy tool makers will incorporate the best features of the agile tools, allowing DW/BI to trim their vendor list a bit and then move on to the next wave of innovative products. Steadily advancing DW/BI’s technical infrastructure through a combination of legacy and strategic tool choices will yield far greater benefits for a company than insisting on a simple technical monoculture that cannot match the application delivery speed of one’s competitors.

New Tenets Only a Start
This series of blogs has proposed four new tenets for DW/BI professionals to consider in order to transform the agile manifesto of 2001 into a agile data warehousing manifesto for the next ten years of business analytics. The four new concepts suggested address the fundamental challenges encountered while building back-end systems for large data volumes—challenges we do not encounter while developing front-end applications. Agile data warehousing has been around for less than fifteen years at this time, so doubtless we have more to learn as a community of practitioners and the four tenets suggested will need to evolve further as we perfect our craft. The original agile manifesto in fact provides twelve principles that provide greater detail to implementing its four tenets. Perhaps the first step for the agile DW/BI community would be to take the generic manifesto’s twelve principles and apply them to data warehousing. Such an effort would provide a good test of whether the four tenets articulated in this series of blogs are currently actionable or need further refinement.

No matter what degree we keep or re-write the tenets suggested above, boiling our experiments and lessons learned down to a few, seminal philosophies will spark a conversation among agile DW/BI practitioners that will greatly deepen the understanding of our profession. The resulting reflection and innovation can only re-double the benefits that the original agile manifesto brought to our business customers, and will be therefore well worth the effort.

The third suggested tenet strives to articulate how agile teams would best approach the notion of value. It focuses the original manifesto’s “emphasize customer collaboration over contract negotiation” more sharply in order to urge teams to:
• Emphasize evolving data rather than iterating on application code.

Original Notion of Collaboration Is Too Vague

Of the four tenets of the original manifesto, the notion of “close customer collaboration” is the new behavior that allows IT to most improve upon its past performance. For over three decades waterfall methods allowed IT to sink enormous amounts of money and time into projects without regularly checking back with the sponsors to see if they felt they were getting value commensurate to expense out of the engagement. A 2004 survey in an old data warehousing trade magazine once revealed that the average data warehousing program cost $13 million and took 17 months to deliver its first workable version. Improvements in DBMS, ETL, and front-end packages have undoubtedly lowered these averages somewhat, but the DW/BI applications are still far too big and expensive to leave our customers in the dark until the first release.

In contrast, agile teams pursue frequent touch points with the folks footing the bills for the work. By regularly demonstrating to the user community the functionality achieved to date, these teams manage their business partners’ expectations and shore up business support for the project. Even more importantly, these iterative product reviews allow IT to learn with little delay whether it’s heading in the wrong direction or working off some horrible misunderstanding. The course corrections garnered from these review are particularly crucial when business conditions are changing. Agile’s notion of close customer collaboration, then, is key to a company’s ability to compete and thrive with the help of business intelligence.

The original manifesto leaves the notion of customer collaboration vague, however. Given that we are building business intelligence applications, we can be far more specific. Of all the additional facets that data warehousing adds to an application development project, by far the most salient is “data.” Whereas transactional applications must have individual widgets and menu items that behavior properly, data warehouses must have millions, even billions, of records that are darn near perfect to have any value. Given that the front-end, data visualization tools are now so powerful, getting the data right has become the primary challenge for a DWBI team. With the incremental and iterative delivery approach built into our agile methods, we do not have to get the data right on the first pass, but that data better quickly start providing insights into our company’s pressing concerns if our customers are going to view a given project as one that is on the road to success.

Thus we hone the original notion of “customer collaboration” to a point which states that the primary objective of the agile data warehousing team is to quickly evolve the data delivered so that it meets business needs.

DWBI Teams Need To Focus On Data

In the original manifesto, the antipode of “customer collaboration” is “contract negotiation.”

The signatories certainly realized that a bit of contracting would always be necessary, but they wisely encouraged us to keep it to the minimum amount necessary to facilitate a productive relationship with the business. Similarly, when business intelligence teams begin to focus predominantly on evolving the data, they too will have to manage a “necessary evil”—crafting through programming the modules needed to extract and transform source information so that it can be loaded into the analytic’s data repository. “Fiddling with the application’s ETL code is in fact the one activity where agile teams still tend to get “lost in the weeds” and disappear for months from their business partners’ view. This is a grievous mistake, because DWBI project sponsors rarely care whether the coding is done. Instead, they repeatedly ask “Is the data ready??” For data warehousing, programming the transform modules is necessary, but it is the data that represents the value. Agile warehousing teams will thrive only if they emphasize steadily moving the breadth, depth, and quality of the data forward, to the delight of their business partners and their project sponsors. In other words, agile teams succeed when they concentrate upon evolving the data rather than iterating upon the code.

Data warehouses are typically very large databases, so steadily evolving the data design necessarily involves unloading and reloading considerable volumes of data. The time required just to move these volumes in and out of target schemas can easily consume most of a team’s development time. This consideration often causes even an agile DWBI team to want to get the data design perfect before attempting to load the tables for the first time. Yet, striving for perfectly structured target schemas works against many of our best agile notions such as “good enough” specifications and “fail fast and fix quickly.” Consequently, to focus effectively on evolving data rather than iterating on code, agile warehousing teams will need the services that a new generation of DWBI tools can offer. These tools eliminate much of the work of restructuring tables and reloading data into them. Not surprisingly, then, the proper place of tools in one agile philosophy will be an important part of the next and final tenet suggested for an agile data warehousing manifesto.

Notes: 1 DM Review survey of its readership October 2004, cited in Jim Ericson, “A Simple Plan,” Information Management Magazine, April 2006, http://www.information-management.com/issues/20060401/1051182-1.html, accessed Sep 2011.

The original manifesto lists its second precept as “Working software over comprehensive documentation.” Working software typically translates into agile teams iteratively delivering the desired application in small slivers that can be used by the business. For warehouses, however, the multiple layers of their data architectures make it nearly impossible to deliver a fully working increment every sprint or two, so this agile precept is too great a stretch in its current wording.
Luckily, we need make only a small modification to make the notion right for DW/BI. In agile data warehousing, we connect quick front ends to every data layer in our applications so that we can demonstrate business information as it progresses sprint by sprint across the chasm between staging and dashboard. True, we’re not showing working software that the end users can operate. But think of these demos from a typical sponsor’s point of view. For decades now, our business partners have funded DW/BI projects for IT to build. They have learned the hard way how rarely we succeed. Perhaps they’ve become resigned to seeing us disappear after gathering requirements, only to return 10 or more months later with applications that don’t meet the needs or are out of date because the business has changed while we were off coding. Though they expect this dysfunctional approach from us, they don’t particularly like it.

For the corporate executive who has been burned by IT many times in the past, regularly seeing the data as it takes shape is a far preferable approach. Even if the front end is simple and ugly, seeing the transactions and decodes flow steadily toward the dashboard gives them a chance to check the direction IT is heading with the project. These regular demos assure them that their money is accomplishing something. It lets them catch us if we’re making a mistake. It allows them to manage the risk of the project. These regular health checks are one of the primary reasons agile becomes so popular with project sponsors and why surveys show that the iterative approach greatly improves customer satisfaction.

In the original version of the second tenet, the counterweight to working software was “comprehensive documentation.” For DW/BI, the dysfunctional practice is the much larger notion of comprehensive specifications, which we see as a futile attempt to eliminate uncertainty. When warehousing consultants take two years to develop the perfect data model for the warehouse’s integration layer, they are striving to leave nothing unanticipated. If the model completely mirrors the business reality, they tell us, then theoretically the data generated by any new policy or product aligned with the business can be stored without having to re-engineer this perfect repository.

Even if the business were to never move on to policies or products unaligned with the current reality—which is doubtful—the problem with resolving every last unknown in order to build the perfect data model is opportunity costs. Months and years go by with the business starving for intelligence, missing out on major revenue opportunities and unable to rectify excessive costs. It is too expensive to wait for complete certainty in a data model of every aspect of the application’s design. Yes, teams need a good idea where they’re going, but such insight can be gained from a series of good white board sketches. It does not require a professionally drawn blueprint.

In the agile world, corporate DW/BI no longer asks the company to wait while the DW/BI staff figures everything out. Roadmaps, data strategies, and templates are streamlined so that they constrain risk just enough to get by, and then the DW/BI department relies upon agile’s incremental approach and its close collaboration with the business to address the remaining, nasty, little surprises that will inevitably crop up as the project moves ahead. Companies that stand still, waiting for risk-free road maps, decline. Companies that manage risk, so they have resources left over to innovate and improve, thrive.

In the original manifesto, “individuals and interactions” suggests close, frequent collaboration, which we still believe in heartily. However, warehouses take so long and are so expensive; we need to strive for something greater to turn our profession around. “Sponsor-valued results” places our focus exactly where it has to be: the folks signing those enormous checks must believe they or their staff members obtained something usable in exchange for their funding. “Prompt” is a necessary modifier, too, reminding us that instead of having to wait for a year and half, say, our sponsors should receive this value within a time frame where its capabilities can still make a difference for the business.

Switching our attention to the right side of the tenet, the original tenet’s bogey man of “process and tools” can be brought into better focus for DW/BI. On waterfall projects, the greatest disservice we in corporate IT promulgated upon our customers is our pre-occupation with overly elaborate, flawless first deliverables. We insist on delivering perfect rather than timely solutions. Traditional enterprise warehouse teams envision their goals as enormous constructs called corporate information factories or conformed dimensions. We make the business wait forever while we program our type-two dimensions and struggle with complex business rules. While these objectives certainly have advantages, they all require lengthy modeling and protracted build-outs, one layer at a time. Rather than find a way to rapidly provide insight into the business challenges threatening the organization—be it falling revenue, sky rocketing cost, or departing customers—we toil over the ever tightening spirals on our specs, preaching that what the organization really needs is that distant—and ultimately unobtainable—“single version of the truth.”

Another trajectory is possible. During the months or years it takes to deliver a new, fully populated dashboard, incomplete and half-transformed data exists in intermediate layers. It is not perfect, but it still has value. Yet, rather than letting end users access that data to gain the insights for they require, a traditional DW/BI manager would lock them out, claiming the data is not clean enough or integrated sufficiently to be of use. In an agile world, we acknowledge what the business has been telling us all along—that they need business insights now. Yes, a six-lane highway bridge will be needed someday for all the vehicles we anticipate, but we can get foot traffic across the chasm now by jumping from boulder to boulder.

Once we de-emphasize perfection, a myriad of possible collaborations between corporate DW/BI and departmental BI become apparent. We find ways to deliver vertical or horizontal slices of the planned subject areas that will answer one or two burning questions at a time: Maybe simple counts based upon current data can answer half the questions. There’s no need to wait for type-two dimensions and derived columns. Maybe the department’s power users or consultants can build their own data marts on top of enterprise persistent staging. Maybe corporate IT can feed them one clean dimension at a time so they can progressively swap out the spaghetti code of their temporary front-end applications. Perhaps they can report upon regions or business units separately, drawing upon the integration layer directly until EDW can provide a conformed product dimension from the presentation layer. The answers take creativity and flexibility to find, but they are there, visible once corporate IT stops insisting on perfect deliverables.

A change in end users is part of the equation, too, when we move away from perfection towards prompt results. Formerly, end users played a passive role, impatiently waiting for final delivery, but only because corporate IT told them to expect the perfect BI application sometime in the future. In an agile world, our deliveries are instead frequent, partial, and imperfect at first. End users must actively familiarize themselves with the data flaws, build temporary workarounds to compensate at times, and willingly scuttle those workarounds when better data comes along soon after. As the six-lane highway bridge extends progressively further across the gorge, fewer boulders have to be hopped.

With this combination of collaboration and progressive intermediate solutions, our end goal is no longer a “go live” date for a complete subject area. Instead we expect the enterprise data warehouse to emerge incrementally as the result of intense, cooperative activity at both corporate and departmental levels. We accept the flaws that will exist in the data during the meantime, because they are a reasonable price to pay for getting our business partners the insights they need to keep the company successful or even out perform its competitors.

The exercise made me appreciate that the Agile Manifesto has indeed helped many of us deliver large, data integration and analytical applications faster, better, cheaper. But I also had to acknowledge how often new practitioners seem to get it wrong, ending up with disappointing project outcomes just as easily as if they had followed traditional, waterfall methods. If the tenets of the agile manifesto can be so easily misinterpreted, I wondered, could it be because they were written by transaction-application developers? Perhaps the DW/BI professionals need to take a new look at the manifesto and perhaps update it to make it apply directly to our half of information technology—the world of data-driven applications such as data warehousing.

I don’t believe we need to replace the manifesto. Its four tenets and twelve supporting principles are relevant to the software development in general, urging us to deliver a steady stream of user-tangible benefits in a time frame that has value for our business customers. Instead, I would rather ask what should be added to the manifesto’s four tenets. If we are lucky, the result might add up to a Agile Data Warehousing Manifesto that will allow us to reliably succeed with data integration and business analytic projects as well.

The place to start, of course, is with the original manifesto’s preamble. Given the dismal failure rate of traditionally-managed software projects*, the seventeen signatories of 2001 could have started with a brash comment such as: “It is delusional to believe you can completely understand in advance what a project’s sponsors and users will need from their software application, so if you don’t want your project plan to collapse like a house of cards when the first nasty surprise crops up, you’d be well advised to….”
But, they took a softer, far more diplomatic approach and began simply with: “We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value….” The non-confrontational tone of the original’s preamble allows even a skeptical audience to continue reading and consider the manifesto as a whole.

We should probably strive to match its tone as we kick off our variant for DW/BI. What about the following? “Using Agile techniques, we are finding ways to break through the problems which used to make data warehousing projects frequently fail: high-cost, slow-delivery, and excessive risk. Through this work we have come to value the following notions in, addition to the tenets of the Agile manifesto….” Such an opening would be respectful and fair, yet clear that something is very wrong with the traditional, waterfall approach to DW/BI and demands to be fixed. The four tenets that follow will then paint a better path forward for our projects.
______________________________________________________

* As documented by a survey of 2,500 transaction application projects documented in the Standish Group’s 1994 Chaos Reports, projects under $1 million had a failure rate of 67% which fell quickly to 100% as project size exceeded $10 million (2009 dollars).

BUSINESS PARTNER WHO WON’T DESIGNATE A PRODUCT OWNER WITH AUTHORITY

Another common situation that contraindicates an ADW approach is a project involving a weakly-empowered Product Owner. The Product Owner must be empowered to make important requirements and functional-design decisions for the team, because often there will be a day-for-day slip in the delivery schedule if a matter cannot be settled in real time while discussing matters with the developers. The most frequent cause of such delays is a Product Owner who must “socialize” even moderate changes in requirements with her peers and superiors on the business side. A stronger Product Owner will know the business better, so that she can divine the right choice from her own experience. She will also have the authority within the business group to set the direction and functional design of the application on her own.

If the current Product Owner does not have these strengths, the team would be smart to request a more empowered Product Owner. The role of the original Product Owner can evolve into a “Proxy Product Owner,” so that the new Product Owner can spend less time with the team because the Proxy can fill-in some details when the new Product Owner is not in the project room. In essence, by requesting a more empowered Product Owner, the team has simply gotten the business to provide senior support for a business member who was not quite ready for the full Product Owner assignment.

POOR FACILITIES THAT PREVENT SMOOTH COLLOCATION

Frequently, even organizations that are absolutely enthusiastic to introduce Agile Data Warehousing fail to provide the work space a team of collocated developers require to work productively. Most commonly, IT management says “we’re going to try Agile, so grab you laptops—you’ll all be working in a conference room on the sixth floor for the next 30 weeks.” In that work room, the developers unfortunately find little more than a conference table large enough for ten people–people simply having a meeting, that is. The room or table is nowhere large enough for people who are going to be working on computers with stacks of paper on both sides of them.

Such improper facilities cause huge stress for the team from day one, and if it is a struggle for the developers to overlook their new, inadequate workspace, the company is not going to get good productivity out of them.

Agile Data Warehousing teams need good facilities, including:

• “Caves and commons”: the team has a common conference table where they can work eye-to-eye when the work requires collaboration, and then private “caves” where they can retreat to nearby when they have to make a phone call or have heavy coding to do that requires some quiet concentration. The conference rooms that serve as “commons” will need to isolate the noise from the cave area, and have tables and white boards large enough that some real work can be done.
• Sufficient connectivity: Wi-Fi that frequently crashes, disappears, selectively drops connections, or runs slow will thoroughly prevent the team from developing work patterns where they can consistently arrive on the job, sit down, and start getting a ton of work done.
• Sufficiently sized monitors: Laptops are great for portability, but it is very difficult to develop ETL mappings (which come to resemble microcircuit layouts by the middle of a project) using a laptop’s 15-inch screen. If the team is really going to perform, managers need to give them the visual real-estate to see the details of what they are trying to create, fix, and improve.

INSUFFICIENT PROJECT TIME

An organization with a top-notch Agile team already established can throw a quick little effort to that team and have it delivered smoothly. However, a company just starting Agile methods will do better with a project that involves a significant amount of time. To fully introduce such a radically-simplified engineering process as Agile and successfully adapt it to a company’s particular culture requires three to six months, and it is best to start with a project that has something like that amount of work to do.

Agile teams are self-organized teams that work repeatedly with small chunks of a project’s requirements in order to establish a steady pace that delivers high-quality, potentially-shippable code. Management needs to give the new Agile teammates a project where they’ll have the time to self organize, which includes a) figuring out all the details of the many roles on the team and b) streamlining their design and documentation artifacts now that they are working eye-to-eye.

Moreover, organizations need to know how long a project will take and how much it will cost. New Agile teams need two or three iterations using their new method to learn how to define and package the requirements of the project, so that time and all staffing needs can be accurately estimated.

Forcing all this learning to take place on a short project of four to six weeks will give the team no time to identify any of the patterns that the Agile method will require, and probably lead to a highly improvised effort where the developers cut too many corners because they must work at breakneck speed with no time to reflect on their work methods. The results will be less than spectacular.

In this series of blog postings, we present the conditions which pertain to the most common cause when agile fails: bad initial setups. By this we mean situations with one or more of these handicaps: no support from neither business nor IT, business partner who won’t designate a product owner with enough bandwidth, business partner who won’t designate a product owner with authority, poor facilities that prevent smooth collocation, and insufficient project time.

NO SUPPORT FROM BUSINESS NOR IT

Agile is a disruptive technology, mainly because it introduces simplicity where most orgs have layers of complexity. This radical simplicity is of course where it gets much of the time and speed savings, but it does leave those who are interfacing with it or providing program- and departmental-level management many gaps to fill as the project progresses. Every gap is a headache, and may even suggest large changes to many, established organizational roles, and thereby a widely-sensed threat to the status quo. For that reason, Agile projects quickly amass more detractors than supporters, so if you’re going to introduce Agile, be sure you have clear support from either the business or IT, otherwise you will face your detractors alone and there will be nobody defending you in the executive meetings that will take place without you.

To be effective, your business or IT defenders will have to understand not only the large cost advantages that Agile offers, but also how complex it will be to adapt Agile to the culture and wrapper processes of an actual corporation. The team members will need a few months before they sort through all this complexity to arrive at a streamlined, triply effective method that everyone can share. Your champions will have to cover for you as the pilot team makes a few false starts and has a couple of mediocre iterations.

A business champion is usually the most effective for a new Agile team. Typically, business champions are seriously looking at outsourcing DWBI work, because the warehousing department is just too slow, and expensive, and/or delivering very poor-quality product. The Agile pilot will find solid support here because finding an internal development solution is still instinctively preferable to using outside vendors who have a an additional profit motive and lengthy contracting requirements, and who are not controlled by the company, so that they may very well change personalities one or two years in the future.

IT departments can also be effective champions. They are often aware of growing frustration in the business with their slow, expensive, and flaw-prone deliveries to date. They are open to Agile for multiple reasons: a) they have been hearing about iterative development for at least 15 years now, b) it has gathered plenty of mind share among their colleagues and there are now plenty of literature sources and training programs for Agile DWBI, and c) they truly fear that the business will start sending work to outside vendors.

Getting solid support from business and/or IT will be the wind in your sails you will need during your first couple of pilot Agile projects when the waters get rough. It you have neither, stay in port, wait for conditions worsen and for a steady wind of “we’ve got to do something different” to blow.

BUSINESS PARTNER WHO WON’T DESIGNATE A PRODUCT OWNER WITH ENOUGH BANDWIDTH

Even in companies where both business and IT were enthusiastic to try a new, streamlined software development approach, our first pilot projects sometimes do not go so well. One of the most common causes for such a false start is a distracted Product Owner from the business group sponsoring the project. She is either too uninterested or just too loaded with other work to spend significant time in the project room, keeping the team provisioned with accurate and business-valuable requirements.

The Product Owner plays a central, leadership role on the team and must truly “own” the project if it is to be a success. She must consider it as her top first or second priority for the current year. If the project is, say, number eight on her priority list, she will balk at spending countless hours in the project room, she will take too long to read or understand the requirements or high-level dilemmas that the team brings her to settle, and she will tend to forget decisions made just the iteration before, all because the Agile project is just one among many crisis burning constantly in her professional life.

In this situation, the team needs to insist on working with a Product Owner that can spend at least half of her time in the project room. There are three possible solutions the team can suggest: a) The business needs to clear the non-project work load off the current Product Owner’s queue, b) replace the current Product Owner with another person who has the time and interest to work extensively with the team, or c) send a business analyst from IT into the business environment for a few weeks to learn the business well enough that he can perform the role of a Proxy Product Owner. (He’ll still need some backup from the original Product Owner, but the time commitment for the latter will be greatly reduced.)

— — — — — — — — — — — — — — — —

See later postings for the continuation of this thread.