Affiliated with:

Agile Framework for Managing and Measuring Enterprise Business Intelligence

Warehousing 2

To be agile, business intelligence and analytics systems need frameworks and metrics to enable stable evolution.


Enterprise Business Intelligence and analytics solutions are complex implementation efforts because of the Develop – Support (Growth-Sustain) cycles followed concurrently. Every enterprise wide BI system continuously evolves over a period with new business functionality added at regular intervals, and they need to be in conformance with existing capabilities. In addition, with continuous evolution of functionality comes the question – “How does one measure the progress of the environment”?

There are two major problems in managing Enterprise Business Intelligence and analytics initiatives, namely:

  1. Sustaining concurrent Develop-Support cycles
  2. Calibrating the evolution of business functionality

The solution to the vexing problem in development and maintenance of large data warehouses lies in the adaptation of an agile methodology. Agility in the data warehousing context is an approach that “cycles” through the different phases, with the ultimate aim of adding new functionality and stabilizing what is already present. An agile methodology also provides the platform for measuring / calibrating the progress of Business Intelligence initiatives.


Business intelligence (BI) is a business management term which refers to applications and technologies that are used to gather, provide access to, and analyze data and information about company operations. Business intelligence systems can help companies have a more comprehensive knowledge of the factors affecting their business, such as metrics on sales, production, and internal operations, and they can help companies to make better business decisions.

Data Warehousing can be considered as the technology domain that facilitates Business Intelligence and analytics in any organization. This technology environment has 3 major components:

  1. Back-room architecture – Technology components that are used to extract data from source transactional systems, integrate them, transform the data using business rules and load into target data repositories that aid decision making.
  2. Front-room architecture – Technology components that help the business users analyze the information present using pre-built & ad-hoc reports and utilize the whole range of analytical solutions.
  3. Data Storage – Typically called a Data Warehouse / Data Mart / Operational Data store, this layer models the data in a subject oriented, integrated, non-volatile, time-variant fashion that enables the back-room & front-room architectures to work seamlessly.

Warehousing 1 1

Figure 1: Business Intelligence Architecture Landscape

Problem Domain

Enterprise Business Intelligence and analytics solutions are complex from an implementation standpoint because of the Develop – Support (Growth-Sustain) cycles that are followed concurrently. Every enterprise-wide BI system continuously evolves with new business functionality added at regular intervals; the systems’ new functionality must be in conformance with existing capabilities. In addition, with continuous evolution of functionality comes the question – “How does one measure the progress of the environment”?

The key challenges involved in managing enterprise BI and analytics systems are:

  • Evolution in functionality over a period of time
  • BI systems drive business decisions – Needs “Total alignment” with the corporate vision
  • Development and Support to be managed concurrently
  • Very highly user-centric in nature than compared to other technology areas
  • Robust measurement techniques are not in place to calibrate such systems

Solution Domain

The solution domain has two parts:

  1. Managing the evolution
  2. Measuring the evolution

Focus Area 1 – Managing the Evolution of BI systems

In this section, we will focus on the different implementation methodologies and establish the fact that an agile framework provides the best solution for managing the evolution of BI systems. In addition, we will present the different phases of “agility” in the BI and analytics context.

Implementation Methodologies
Data warehouse implementation methodologies can be one of the following:

  • Waterfall Model
  • Spiral Development Model
  • Iterative Development Model
  • Agile Methodology

Warehousing 2 1

Table 1: Data Warehouse Methodologies Criteria Matrix

Agile frameworks provide the best value for managing data warehouse implementations since they satisfy the key criteria shown in Table 1.

As defined by Webopedia, agile development is a software development approach that “cycles” through the development phases, from gathering requirements to delivering functionality into a working release.

The ultimate goal of any bottom-up development project should be to deliver new functionality on a regular and rapid basis with a high degree of conformance to what was already there. By adopting specific practices from rapid application development methodologies, data warehouse developers can facilitate the bottom-up, frequent release approach and, even more importantly, change the project team culture and associated behaviors to create better, more customer-focused applications than with the traditional waterfall approach.

Some salient points of rapid, agile development are:

  • Shared vision and small teams working on a specific functionality
  • Deliver frequent releases – Agile development strives to deliver small units of functionality that make good business sense.
  • Relentlessly manage scope – Meeting a fixed release schedule will not happen unless the resource triangle is managed actively. As defined by the Project Management Institute, the resource triangle is the three-way combination of requirements, time and resources. Any change to one leg of the triangle (misunderstood requirement, less time or fewer people) requires a corresponding change to at least one other leg.
  • Create a multi-release framework – Agile development stresses that there must be a master plan (e.g., methodology or framework) and a supporting architecture. Use releases to add more customer functionality, not constantly rework what was done in the past

Agility in Business Intelligence

Two phases to the agile framework implementation are:
1) Planning Phase
2) Execution Phase

Agile Framework – Planning Phase
Typically, planning is done at the end of a particular year for the subsequent year, once the business plans & budgets are finalized.

Assumptions / Pre-requisites:

  1. Enterprise BI infrastructure is already present in the organization
  2. Technology Architecture (BI Tools/Technologies) and Process Architecture (Standards, Policies, Procedures) are already defined

Warehousing 3

Table 2: Agile BI Planning Phase Chart

Agile Framework – Execution Phase
The Execution Phase implements the monthly release. This has the following components:

Warehousing 4

Table 3: Agile BI Execution Phase Chart

Agile Framework – Putting Everything Together

The diagram below illustrates the implementation of the agile framework in the real-world.

Warehousing 5

Figure 2: Illustration of Agile Implementation

Some of the salient points in the illustration are:

  • Two stories / business functionality envisaged in the planning phase are:
  1. a) Integrating Sales & Marketing Data,
  2. b) Project Accounting Analytics
  • The story related to Integrating Sales and Marketing data has three (3) phases associated with it: loading Dimension, Facts and Forecast data. Similarly the Project Accounting analytics story also has three (3) phases
  • Each of the phases is divided into multiple cycles. There are two (2) types of cycles. Development (D1,D2 etc.) and Stabilization (S1,S2 etc.) cycles
  • Development cycles involve addition of business functionality while Stabilization cycles ensure that the functionality added is in conformance to the standards enforced in the environment
  • Multiple cycles from different phases and different stories are combined in periodic releases.

Key Challenges and Mitigation Strategy

Warehousing 6

Focus Area 2 – Measuring the Evolution of BI systems

An Enterprise BI / analytics system can be measured by calibrating it against pre-set goals. At a high level, measurement can be viewed as the alignment of process to certain calibration factors so the health of the process can be compared to those factors.

Business Intelligence and analytics systems are measured with the following objectives:

  • Strategic tools to prioritize and align the Enterprise Data Warehouse (EDW) with the corporate vision
  • Measure the evolution of EDW against pre-set goals
  • Mechanism to identify technology issues and take appropriate actions
  • Objectively communicate the progress of EDW to business stakeholders
  • Enable the DW project manager to plan for the immediate future

The Measurement Framework

The Measurement Framework for Enterprise Business Intelligence combines the practical implementation power of the Agile Methodology and the statistical robustness of the Analytic Hierarchy Process (AHP).

There are three levels of scorecards that are part of the measurement framework:

  • Level 1 (Highest Level) – Actual and Planned rating of the environment shown on a periodic basis (Figure 3 illustration is for periodicity of one month)
  • Level 2 – For each period, the rating for different components (“Stories” in the Agile terminology) are accomplished.
  • Level 3 – For each component, the score at the end of that particular period is calculated using appropriate calibration factors.

Level 1 Scorecard – Salient Points

  • Captures the Planned rating, Revised rating and Actual rating for each period
  • Captures the key comments for each point and provides the management a quick snapshot of the progress
  • Planned rating is arrived at during the planning phase of the Agile methodology, and documented as part of the managed metadata
  • Ratings are revised due to changes in priorities of different projects

Warehousing 7

Figure 3: Measuring BI evolution: Level-1 Scorecard

Level 2 Scorecard – Salient Points

  • Level 2 scorecard is a drill-down from the Level 1 scorecard and is focused on the ratings for a particular period. (In this case it is October 07)
  • At Level 2, the planned and actual ratings are provided for each component. These components are analogous to projects / stories
  • Weight for each project/story is derived using the Analytic Hierarchy process (AHP) during the Planning stage
  • The score for each component is multiplied by its corresponding weight to arrive at a final rating for the component
  • For example, October 07 had an actual rating of 78% in the Level-1 scorecard (Figure 3). The scorecard shown below (Figure 4) illustrates the way by which the score for October (78%) is arrived at.
  • October Rating = ∑ i=1-8 (Component Rating for i * Weight for i) = (98*0.05 +85*0.15 +86*0.2 + …….+67*0.1) = 78%

Warehousing 8

Figure 4: Measuring BI evolution: Level-2 Scorecard

Level 3 Scorecard – Salient Points

  • Level 3 scorecard is a drill-down from the Level 2 scorecard and is focused on component ratings (Figure 5 depicts the HR Analytics component)
  • Sub-components in the Level 3 scorecard relate to the different phases for a particular project/story in the agile framework.
  • For each sub-component, a list of calibration factors is derived. Examples of calibration factors can be Business Functionality, Performance, Data Quality, Stability etc.
  • Weight for each sub-component and calibration factors are derived using the Analytic Hierarchy Process (AHP)
  • Calibration Factors play a very important role in arriving at the rating. These are derived based on organizational needs. Some sample ways of quantifying them can be:
    • Functionality (Dimensions) = Number of Dimension tables created / Total number of targeted dim tables
    • Functionality (Facts) = Number of use cases completed / Total number of identified use cases
    • Performance (Facts) = (1 / Actual Time taken by Fact load) / (1 / Targeted load time as per standards)
    • Data Quality (Facts) = Actual ‘System of Record’ identified for HR measures / Total number of measures
  • Sub-component scores (Row scores) are arrived at by multiplying the calibration factor ratings with the calibration factor weights. Similarly the calibration factor scores (columns) are arrived at by multiplying the calibration factor ratings with sub-component weights.

For example:
Developing Facts: (Sub-component Score) =
(75% * 0.4 + 47% * 0.3 + 10% * 0.3) * 0.4 = 18.8%
Data Quality: (Calibration Factor Score) =
(12% * 0.2 + 10% * 0.4 + 10% * 0.2 + 10% * 0.2) * 0.3 = 3.1%

  • The total score for a particular component (Ex: HR Analytics) is the sum of sub-component scores (row total addition) or sum of calibration factor scores (column total addition). In this example, the total score of 56% is equal to (13.6%+18.8%+11.9%+11.6%) or (32.8%+20%+3.2%)

Warehousing 9

Figure 5: Measuring BI evolution: Level-3 Scorecard

To summarize, the scores that are measured for each component in the level-3 scorecard are taken for further aggregation at level 2 to arrive the monthly (any periodicity can be defined) scorecard. Level 1 scorecard provides a snapshot of the trend across different periods. The expectation is that all the stories would achieve a score of 100% at the end of the pre-defined time frame.

Analytic Hierarchy Process (AHP) – A Primer
AHP is one of the powerful methods used in Multi-Criteria Decision Making (MCDM). MCDM is a discipline aimed at supporting decision makers faced with multiple conflicting alternatives. AHP is a systematic procedure that helps to:

  • Represent the elements of any problem, breaking it down into smaller constituents
  • Assign weights to each constituent by following a pair-wise comparison technique
  • Leverages expert judgment and intuitive feel into a coherent framework for problem solving

In this paper, AHP is used at 2 levels:

  1. a) Assign Weight to each component (Stories) that forms part of the EDW
    Illustration in this paper: MDM has a weight of 0.15, Informatica Upgrade has 0.2 and CEO dashboard has a weight of 0.3
  2. b) Within each component (Stories) assign weights to the Sub-components (Phases) and Calibration Factors

Illustration in this paper: For the HR Analytics component, sub-component of “Develop dimensions” had a weight of 0.2, Developing Facts had 0.4 etc. Also each of the calibration factors had their weights assigned: Functionality – 0.4, Performance – 0.3, Data Quality – 0.3


In conclusion, there are many reasons to develop and maintain an agile framework / methodology for an enterprise business intelligence and analytics environment. Some of these reasons include:

  • Enterprise Business Intelligence systems are complex to manage as they constantly evolve over time
  • An agile framework provides an elegant way for managing the concurrent “Develop-Support” cycles required for Business Intelligence and analytics projects.
  • AHP based measurement techniques provide a powerful way for calibrating and enhancing BI application performance
  • AHP is a simple yet comprehensive way of determining relative importance / weights among sub-projects that compose complex systems.

Karthikeyan Sankaran

Karthikeyan Sankaran is a skilled business intelligence analytics professional and data scientist, with over 20 years’ experience in information technology and management. As a consultant he has assisted clients in achieving their business goals for improved data capabilities. Karthik is a regular blogger, forum contributor and author of many white papers on Business Intelligence & Analytics.

© Since 1997 to the present – Enterprise Warehousing Solutions, Inc. (EWSolutions). All Rights Reserved

Subscribe To DMU

Be the first to hear about articles, tips, and opportunities for improving your data management career.