All successful business intelligence / analytics endeavors are based on a formal strategy and use a best-practices based methodology, even in agile environments.
Many companies do not have a data strategy, which is a basic component for successful management of this valuable asset. Additionally, many companies do not require their staff to use a methodology for project planning or application development, especially for business intelligence / analytics initiatives.
The complaints against methodologies are loud and the list is long: methodologies are a thing of the past; methodologies belong with the IT-dinosaurs back in the 1970’s; methodologies are rigid and so inflexible that nimble organizations cannot adopt them with confidence; the paperwork is onerous; dynamic teams don’t need no stinkin’ methodology; and so on.
Many of these complaints have been made with a grain of truth, since many methodologies were paperwork-intensive, and many organizations did not know how to adapt the activities for a fluid situation. However, those issues were and are not reasons to abandon the fact that every successful initiative has been completed with the use of a formal process (aKa “methodology”).
Following is a true story of a DW/BI/analytics team that operated without any guidance of a methodology, and the lessons learned from that experience.
Agility versus Rigor
The goal was to conduct a methodology/project management seminar for a small software development organization’s very young and very enthusiastic staff. When the expert consultant arrived one hour prior to the seminar, the workbooks were not ready yet, but “on their way” from the external printer. Once the workbooks arrived 15 minutes prior to class, they were collated and bound incorrectly.
Rather than causing panic or even mild concern, this situation raised the level of excitement among the staff. They jumped on their skateboards and rolled in “to the rescue.” They manually re-collated and re-bound all 50 workbooks in a record time of 25 minutes. When they were done, there was cheering in the hallways, and “high fives” were exchanged among the “heroes” who “pulled off the impossible” one more time. It was immediately clear that “constant firefighting” was how they worked on their projects, and that this was the very reason why the expert consultant had been engaged to conduct this training seminar. It was delightful to see their teamwork, their enthusiasm to solve the problem, their ability to reorganize themselves intuitively, and to accomplish what seemed impossible. There was a lesson here about the approach that should be explored.
This experience demonstrates the potential productivity that could come from the “thrills of chaos” (agility) if channeled effectively, and the devastating effects of killing enthusiasm and morale that could result from the “dregs of structure” (rigor). The consultant abandoned the lesson plan for this group and started with an exercise of two questions: “What are the thrills of chaos?” and “What are the dregs of structure?”
The first question, “What are the thrills of chaos?” produced these answers:
- It’s like a “high”
- Unlimited creativity
- You are free, unrestrained
- Spontaneity
- Camaraderie
- Be a hero
- Stand out and be noticed
- Boost to self-esteem
- Excitement, fun
- Defiance
- Sense of accomplishment
- You can hide your incompetence
There is an indisputable payback for operating in chaos. Whether the return is a boost to one’s self-esteem or the ability to hide one’s incompetence, there are distinct personal benefits, which spur people on to accomplish “missions impossible.” It would be counter-productive to try to squelch these perceived advantages. On the contrary, these benefits must be considered and carefully architected into any structure or discipline, such as a methodology.
The second question, “What are the dregs of structure?” produced an even longer list of answers:
- Oppressive
- Big brother
- Being under somebody’s thumb
- Kills creativity
- Boring
- Never works anyway
- Wastes time
- Suffocating
- Military
- The establishment
- Being old and stale
- Dictatorship
- No individualism
- Choking, controlling
- Un-American
These fears and anxieties are even greater motivators to defend the “thrills of chaos.” It is no wonder that any type of structure or discipline (rigor) is rejected when it results in such strong personal feelings that include choking or working under a dictatorship.
While it appeared that the students had found bliss in their chaos, it seemed many had not slept much during the previous week because they had to finish a project. Their manager reported they would not get much sleep in the future because portions of their work had to be redone due to errors; and that this was a common occurrence. Some students commented they had family problems because they were never home. It was time to ask two more questions: “What is bad about chaos?” and “What benefits can structure provide?”
Surprisingly, there were some honest answers to the first question, “What is bad about chaos?”:
- You don’t have a life
- Not enough sleep
- Lost concentration
- It’s frustrating to keep redoing work
- High error rate
- You run 200 mph but cover less than 100 [miles]
- Management is displeased
- Spouses threaten divorce
- Sets precedents for unreasonable expectations
Equally surprising were the answers to the second question, “What benefits can structure provide?”:
- Calmness
- Being organized feels good
- Sense of being in control
- Being able to plan activities outside of work
- Time to think
- Time to learn and develop skills
- Be creative
- Balance
- Rest and vacations
- Sense of accomplishment, job well done
- Recognition
- Learning
- Raised self-esteem
It was striking that some students’ responses were the same as to the previous questions, such as “be creative” and “raised self-esteem.” They did not reject structure and methodologies per se, only the “rigor” that accompanied it and only when imposed by their manager.
The lesson was clear: agility and rigor not only can complement each other, they should. The other lesson, of course, was that agility without rigor is chaos, and chaos has extremely high risk and high cost. The purpose of a methodology is to reduce risks and costs. The more complex a project, the higher its risks and costs, and the more rigor is required to manage it.
DW/BI/Analytics methodologies
Data warehouse (DW) Business intelligence (BI) and Analytics projects are particularly complex for several reasons. Their scope is often very large and the project deadline may be less-than-realistic. The requirements are frequently changing or expanding without adjustments to the deadline. Data is integrated from multiple legacy source systems that were never designed with integration in mind. Dirty data, duplicates, and inconsistencies are everywhere and add to the complexity. Most project teams are understaffed and insufficiently trained, etc…
Traditional methodologies can be too rigid for the dynamic nature of data warehouse (DW) Business intelligence (BI) and Analytics projects. Traditional methodologies prescribe a sequence to their tasks and deliverables. This sequence presumes that all requirements can be identified and documented before analysis starts, that analysis can be completed before commencing the design, that the design can be approved before coding starts, and that testing can be performed at the completion of coding. None of these assumptions are true for analytical projects. Therefore, many organizations struggle to identify the right type of methodology for their DW/BI/analytics initiatives.
In one comprehensive methodology, the Business Intelligence Roadmap, the author compiled almost 1,000 tasks that are potentially applicable to BI and DW projects. Of course, not all tasks are applicable to all projects, but they must be considered and either selected or rejected, according to the project’s scope. Using a formal methodology allows a project leader or manager to focus on the initiative and select the necessary activities, without expecting the team or leader to remember 1,000 tasks and their contents / dependencies / artifacts / etc.
For analytics projects, a spiral methodology is recommended, since agility and rigor are combined to maximum effect. The rigor should be documented in the methodology as entry and exit criteria for all development steps, things to consider, activities and tasks, deliverables, dependencies, roles and responsibilities, do’s and don’t, tips and rules of thumb. The agility should be reflected in the methodology in two ways: extreme scoping, an agile approach to scope management and project management, and self-organizing project teams, which are a modified version of those “kids on skateboards.”
Conclusion
Despite resistance from many practitioners, successful data warehouse / business intelligence / analytics efforts are based on the use of a formal methodology. The best methodology for a DW/BI/analytics environment is one designed to take advantage of two competing forces: agility and rigor.
