Data warehouse / business intelligence / analytics environments require business and IT cooperation for successful architecture and design
In working with many large corporations across a variety of industries, it is clear that we all face the reality of re-architecting data warehouses, whether we want to admit it or not. Since data warehouses are business driven (rethink your strategy if you have IT driving your warehouse), it is very important that the business leaders championing your effort understand the perils of avoiding a well thought out architecture and design.
You have probably heard the now famous words from Benjamin Franklin: “The definition of insanity is doing the same thing over and over and expecting different results.” Yet, we continue to see corporations of many sizes having to throw out their existing solutions and spend significant amounts of time and resources to build new ones. Look at the number of data warehouse vendors and business intelligence (BI) tools vendors that have emerged in the last several years. Too often, organizations look to technology vendors to solve their problems, thinking it is the technology itself that is the issue. Rarely, is that the case.
Many individuals that attend data warehousing seminars, conferences, and classes have told me that they can’t understand why there is so much focus on data warehousing theory. My thought is that there is still so much unnecessary failure in data warehousing and it all starts with understanding the right leadership (business and IT), approach, and learning how to design a successful data warehouse architecture.
Business Sponsors / Champions
Recently, I spoke at a large conference about an enterprise data warehouse effort I managed for a large healthcare provider. It was amazing how many attendees stayed to chat or contacted me via email. Most of them had one consistent theme: they insisted that the project was extremely fortunate to have executive sponsorship. Indeed, we would not have had the funding and direction required to deliver to the business needs without that sponsorship.
However, sponsors don’t find you and tell you the right way to build a sound data warehouse solution. Often, we need to find them, and we always need to work with them to repeat their needs back and explain the solution that is required to meet their real needs. So next time a business sponsor/champion tells you exactly what they need, or insists on cutting corners, use that as an opportunity to work with them to come up with approaches/solutions that hit their pain points, but enable a timely/valuable business solution. Generally, leaders/executives are in that role because of their ability to reason, rationalize, and strategize. If you are knowledgeable about the topic, they will likely listen. If this is not one of your strengths, consider bringing in an outside expert (again, make sure they are not a technology vendor, but a solution vendor).
What do these business executives need to know? Start here: Regardless of the business need, the challenge with building a successful data warehouse is directly related to the fact that most corporations have transactional systems with little to no intelligent integration. That makes it very difficult to bring data together into a data warehouse. In most cases, this is not a technology problem, although many software vendors insist they have tools that can alleviate the problem. Perhaps one day a silver bullet technology solution will arrive, but I am not holding my breath. Let us consider, if your data was standardized in terminology, definition, and quality across systems, building a data warehouse would be downright simplistic. Transformations would only occur for repositioning data into the most efficient database structure. Users would already be comfortable with the terminology and quality of the data. All of these are inherent in well designed information management. Yet, for most large corporations, this isn’t reality. Your sponsors need to understand this.
Two Simple Approaches to Data Warehouses
First, as simple, quick solutions (90 – 120 days), that answer a specific question. Many consultants and teachers preach this today, but the consultants don’t tell you the requirements and ramifications of this type of approach. This definition of a data warehouse has significant value for a specific need, but more often than not this is an easy out for someone that has many needs and wants at least one problem solved at some point in their life. Can you blame them?
Secondly, as a larger enterprise structure for large corporations that can afford to spend a lot of money, and wait many years to get any results. This misconception of an enterprise approach stems from those that have likely read various horror stories about implementations that had no business value or were never delivered. In all likelihood, this was not due to the design or architecture, but was due to demands and shortcuts that led to getting something quickly rather than well. More often than not, an enterprise approach is what is needed, but those that don’t know how to implement it iteratively fall back to the excuse that it is the monolithic, boiling of the ocean solution. We all know that never works.
Often, when we design something as complex and yet valuable as a data warehouse, we are pressed by needs or priorities that cause us to repeat the exact same problem that made this so difficult to begin with. We don’t have time to properly define the data of our business. Yet the companies that have been successful with data warehousing / business intelligence have all taken the time and resources to plan, architect, and design the data, approach, and delivery.
You can be assured that your business leaders hear about successes and failures and want to make sure their business is on the success side of that equation. Make sure you share some of this background with them so they can navigate all of the vendors and information they see with what is really needed for your organization.
Where do you start?
The only way to avoid repeating chaos in your data warehousing efforts is to work to educate the business leaders you work with so they understand the cost, decreased value, and limited lifespan of efforts that meet a single need, but not the needs of the business. This doesn’t mean that they can’t get what they want in short order. Whether you are building an Enterprise Data Warehouse or a specific data warehouse/business intelligence solution, it can be built in stages, incrementally, but it has to be designed with the end in mind. It sounds pretty simple, yet most businesses continually look at redesigning / architecting / building their data warehouses and when they do, they repeat the exact same mistakes again.
The Keys to Successful Data Warehouse Architecture
Conclusion
Data warehouse / business intelligence / analytics architecture or re-architecture requires a focus on two things initially – Data and Usage. Your business users will be much more successful and thus so will you, if you deliver a solution they use than if you deliver exactly what they ask for.
