Agile project management requires self-organizing teams that can adapt to a variety of projects, including data warehousing, business intelligence, and analytics initiatives.
Extreme Scoping is an agile project management approach for building data warehouse and business intelligence (DW and BI) applications. This new development approach requires a new dynamic project team structure. Before exploring the new team structure, it is important to review how teams are organized traditionally.
Traditional teams depend on the project manager to coordinate and assign tasks to individual project team members and to track and report the progress of the project. The project manager also reviews the work deliverables and makes project-related decisions. The individual project team members are responsible for their own task deliverables, which they hand off at the appropriate time.
For example, the requirements analyst gathers the application requirements, which are then handed off to the data modeler who creates the logical data model, which is then delivered to the database administrator who produces the physical data model and builds the database structures, which are then given to the developer who constructs the application.
The only collaborative interactions that involve the entire team happen once a week during the status review meetings. This approach works well on very large, multi-year projects that use traditional methodologies and have 20 to 30 people on a team, but it is ineffective for DW and BI projects that must deliver in 90 to 120 days.
Project Team Structure
In “Extreme Scoping” there is a recommended the software release approach for building applications. In extreme scoping the project management function is performed by the entire core team and not by a single project manager. The following figure (Figure 1 – DW/BI Team Organization) illustrates how a self-organizing project team is organized.
Figure 1 – Self Organizing DW/BI Team Organization
Core Team
The core team should be staffed – at a minimum – by a seasoned project manager; a business representative (end-user or subject matter expert) who has the authority to make decisions and to set policies; an enterprise information management (EIM) person who is trained in data management disciplines such as data governance, logical data modeling (not database design), normalization rules, data and metadata standards, taxonomy; and at least one senior technical person (lead developer or technical architect) with strong programming and technology skills. All core team members together comprise the project management team, which replaces the single project manager function. The core team members manage all software releases of an application, thus ensuring continuity and knowledge transfer.
Core teams are self-organizing SWAT teams where all members of the team (together as a team) coordinate and assign tasks to each other. They review one other’s work, collaborate on project issues, and make project-related decisions. The entire core team meets every day for a status review, and the core team members take over each other’s tasks when needed (for example, when a core team member is out of the office due to illness or vacation).
A core team must be very small; a team size of 4 to 6 people (per project) is optimal. Project core teams should never exceed 7 people.
Development Track Team
The most common development tracks are ETL and front-end application development, and metadata repository. If metadata is not a deliverable, then at least two development track teams work in parallel, namely ETL and front-end application development. These development track teams are extensions of the core team. However, the development track team members are not part of the project management team.
The development track teams are led by the lead developer or technical architect from the core team. An organization may choose to have a lead developer or technical architect per development track, if that supports the organization more effectively and if the resources exist. Development track team members should be developers (and potentially a senior analyst) that have the skills needed for their particular track. They only participate in track-specific tasks for the duration of their track activities.
A development track team is usually smaller than a core team; a team size of 3 to 4 people (per track and per project) is optimal. Development track teams should never exceed 5 people.
Extended Team
The extended team members play traditional roles and participate on the DW and BI projects on an as-needed basis, not on a full-time or daily basis. Members of the extended team may be heavily involved only at certain times during the project, or they may be called intermittently for their expertise and contributions. Extended team members include technicians and business people who contribute in much the same way they do on traditional projects. Extended teams include support roles such as technical support, operations, the security officer, IT auditor, the business sponsor, and other stakeholders.
The core team, the development track teams, and the extended team comprise the complete DW/BI project team.
BI Steering Committee
The BI steering committee is an advisory body of business executives and senior business managers, who understand BI and DW, who support enterprise-wide activities, and who provide collective sponsorship. They meet on a regular basis to discuss, plan, staff, and fund business initiatives, such as DW, BI, MDM, CRM, CDI, and so on. They communicate necessary data integration principles to the organization. They stand behind a BI strategy that supports the business drivers. They fund an EIM group to perform enterprise-wide data management and data governance. They identify data owners and direct the data owners to identify data stewards in their line of business. They also separate business people from their operational responsibilities so they can participate as full-time members on the project core teams.
BI Program Manager
The BI program management office is led by a BI program manager (director level) who works directly with – if not for – the BI steering committee. The BI program manager performs periodic readiness assessments to identify new information needs and to ascertain end-user satisfaction with the DW and the BI applications. With the backing of the BI steering committee, the BI program manager creates a BI strategy and enforces the common technical and non-technical infrastructure components in the BI environment. Working with the BI steering committee, the BI program manager prioritizes BI and DW projects, determines the project dependencies, and coordinates the project resources and activities around these dependencies.
Team Interaction and Communication
The dynamics of the core team or the development track teams are not the same as that of the extended team or a traditional project team. The core team is like the SWAT team at NASA in Houston, Texas. Although they have their assigned cubicles and offices, they do not spend most of their time in them. Instead they collaborate, brainstorm, solve problems, and make decisions together in a “war room” dedicated to their project. For example, the core team meets every day to review status and deliverables and to discuss roadblocks and solutions. Sometimes these meetings last ten minutes, at other times working sessions could be scheduled for half a day. During these sessions, individual assignments are distributed to the appropriate team members who will work on them after the meeting and report back the following day. If major roadblocks are encountered, they are addressed immediately and alternative solutions and contingency plans are prepared. The project manager and the business representative will take the alternative solutions and contingency plans to the sponsor for negotiation and/or a decision.
The development track teams function similarly to the core team, only on a smaller scale. They also meet every day to review the status and deliverables within their own track. And, they collaborate, brainstorm, solve problems, and make decisions together on their track-specific issues. In fact, they function similar to extreme programming (XP) teams sharing the prototyping activities of analysis, design, coding, and testing. Communication with the core team is established through the lead developer or technical architect, who is managing the development track team(s).
The extended team meets with the core team once a week, usually on the same day and same time of the week. The core team members establish the meeting schedule at the beginning of the project, and the sponsor sends out the notification to the extended team members. The meetings are scheduled for one hour, but could be as short as ten minutes if no major issues are to be discussed. The purpose of the weekly meetings is to keep all team members current with activities, status, and issues of the project.
Core Team Roles
There are a number of roles that are played by the members of a core team. Which specific role is applicable to a project depends on the scope of the work and the type of deliverable. Roles do not equate one-for-one to people. A person can, and probably will, play multiple roles, but one role can also be played by multiple people. Some roles are IT roles, while other roles are business roles. Note that one business person must be assigned to the project as an active full-time participant and to whom project tasks are given.
- Application Architect
The application architect is responsible for designing and delivering the front-end access and analysis BI application. He/she is the lead developer or technical architect on the core team (or one of them), who represents and mentors the other front-end application developers, and who writes and tests the more complex programs. - BI Infrastructure Architect
The BI infrastructure architect is responsible for the technical infrastructure components (configuration management), such as hardware, system software, tools, and network components. He/she aligns the technical infrastructure components with the overall strategic IT infrastructure of the company. He/she also participates in product evaluation, selection, installation, and testing. And, he/she monitors all platform components for usage, trends, and performance. - Business Representative
The business representative is the primary business person on the project. This person must be familiar with business policies and business rules, and must have some authority to make decisions about the data and the project requirements. He/she must have extensive industry knowledge and must be respected by other business people in order to be able to facilitate dispute resolutions among the business units. He/she should participate in the planning, business analysis, logical data modeling, and prototyping activities as well as in most testing activities. - Data Management Specialist
The data management specialist is responsible for data standardization, data governance, logical data modeling, and normalization. Data management is a cross-organizational business analysis function and is not to be confused with database administration, which is a technical function. - Data Quality Analyst
The data quality analyst is responsible for bottom-up source data analysis, finding the dirty data, and writing the data cleansing specifications. He/she must have a technical background and must be able to easily navigate through the operational source systems, get data dumps, use a data profiling and data cleansing tool, and write simple reports. This person must also have a good rapport with business people and with the operational IT staff. - Database Administrator (DB Architect)
The most common responsibility given to a DBA is to maintain application databases (take backups, perform database reorgs, etc.) The senior DBA is often known as the system administrator who maintains the various DBMS instances. The third and most important responsibility of a DBA is to be the database architect who designs and creates databases. This last role is especially important on data warehouse projects because of the different types of database design schemas applicable to different types of DW databases. In some organizations, the title of the DBA who performs the database design function is officially separated into the title database architect. - ETL Architect
The ETL architect is responsible for designing and delivering the back-end ETL process. He/she is the lead developer or technical architect on the core team (or one of them), who mentors the other ETL developers, and who writes and tests the complex ETL programs. He/she also works closely with the database designer to create the ETL process flow. - Metadata Administrator
The metadata administrator is responsible for creating, loading, and maintaining the metadata repository. He/she works closely with the ETL lead developer to capture load statistics, data quality statistics, and reconciliation totals (tallies) produced by the ETL programs during the initial, historical, and ongoing database loads. He/she also extracts, links (relates), and loads metadata from various metadata sources, such as the CASE tool, ETL tool, OLAP tool, RDBMS system tables, EXCEL macros, and so on. - Project Manager
In addition to the traditional project management responsibilities of defining, planning, controlling, and reviewing project activities (as a member of the core team), the project manager is responsible for enabling the core team and the development track teams to work on project activities with minimum interruptions. He/she has to negotiate with the business sponsor, report the progress of the project to stakeholders, negotiate with vendors, resolve technical issues, and measure the success of the project. - Subject Matter Expert
The subject matter expert is a business person who provides in-depth business knowledge for a subject area or business domain, and who participates in planning, business analysis, design, and testing activities. This is often staffed by a senior business analyst or a business liaison person who has a cross-functional view over many departments. He/she can also help facilitate data dispute resolutions among the business units.
Development Track Team Roles
The roles played on the development track teams are mostly technical roles specific to a track. Which specific role is applicable to a project depends on the scope of the work and the type of deliverable. Typical development tracks include ETL, front-end application, and metadata repository. Depending on project organization and available resources, one person may play multiple roles or one role can be assigned to multiple people.
- Application Developer
Additional application developers join the project during the prototyping, construction, and testing periods. They work under the direction of the lead developer or technical architect on the core team. They are responsible for building the end-user deliverables, such as reports, queries, cubes, scorecards, dashboards, and so on. - ETL Developer
Additional ETL developers join the project during the construction and testing periods. They work under the direction of the lead developer or technical architect on the core team. They are responsible for building and maintaining the extract, transform, and load processes. ETL developers work closely with the metadata repository developers to collect and store load statistics, data quality statistics, and reconciliation totals. - Metadata Repository Developer
Additional metadata repository developers join the project during the construction and testing periods. They work under the direction of the metadata administrator. They build and maintain the metadata repository solution. They extract metadata from the various tools where it is collected. They link the business metadata with the technical metadata, process metadata, and usage metadata. They provide access to the metadata in the metadata repository to technicians as well as to business people. - Extended Team Roles
There are a number of other roles played by the members of the extended team. Again, which specific type of role will participate on a project depends on the scope of the work and the type of deliverable. Extended team roles usually do equate one-to-one to a person, such as business sponsor, auditor, or security officer. - Data Miner
A data miner is usually a statistician who builds the analytical models, interprets the data mining results, and conveys these results to business managers and executives. - IT Auditor or QA Analyst
An IT auditor or QA analyst is tasked with the responsibility of identifying risks and exposures of the DW initiative and the BI applications. - Network Services
Network services staff is responsible for managing and maintaining the network components. - Operations and Services
Operations staff is responsible to run the scheduled BI application jobs, including the periodic ETL load cycle as well as canned queries and reports. - Security Officer
A security officer is responsible to ensure that security issues are considered, and that appropriate security measures are implemented. - Sponsor
The executive sponsor spearheads the DW/BI initiative and approves the direction and changes of the projects. - Stakeholders (Other Business People)
Other stakeholders are individuals who are interested in the projects for their own future plans to participate in the organization’s DW/BI initiative. - Support Staff (Helpdesk)
End-user support can be in the form of hot-line trouble-shooting personnel or an individual designated to mentor the end-users. - Technical Services
Technical services staff is responsible for preparing, testing, managing, and maintaining the technology platform components. - Testers
Additional testers participate during system or integration testing, regression testing, and performance or stress testing activities. - Tool Administrator
Tool administrators are charged with installing, testing, and maintaining developer tools (e.g., data modeling tool, data profiling tool, ETL tool) and end-user BI tools (OLAP, report writer, query tool).
Conclusion
It is impossible to have an agile development environment without converting traditional project teams into agile, self-organizing SWAT teams. The key is to keep all project teams small. Transfer the project management activities to the core team members collectively. Have a business representative on the core team, and assign the various core team roles and responsibilities to those core team members who have the most appropriate skills to play those roles. Development track teams can have as few as one developer in each team (per project) or as many as three or four. Finally, include the extended team members on each project as-needed to ensure agility.
