Organizations are changing their decision-making process from an expertise and intuition basis to data and information foundation. Doing so points to the need for effective data governance.
A book by a Yale University econometrician, Ian Ayers, looks at a trend in organizations that is changing the decision-making process from one based on expertise and intuition to a data-based effort. This change is possible due to an almost inexhaustible supply of data on every topic, gathered from many sources and made available by the development of huge databases and the tools to manipulate the data in a variety of ways.
The book, Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart, calls the data set analysts “super crunchers” and discusses the changes they are making to industries as varied as medical diagnostics, air travel pricing, screenwriting and online dating services. Although the author presents both sides of this debate (intuition versus use of data), clearly he is convinced that the use of large amounts of data for “objective” decision-making is the better approach.
Difference in Approaches
The difference in the two approaches is not just a matter of managerial preference according to the author: “We are in an historic moment of horse vs. locomotive competition where intuitive and experiential expertise is losing out time and time again to number crunching.” Ayers shows that some older industries, such as wine-making, still rely more on feeling and experience than on the quantitative method. He believes that the data-based approach is needed to improve performance in every operation, using the incredible volumes of data accumulated in every organization, regardless of field.
This trend, which started with the development of data warehouses and other large databases for decision support in the late 1980’s and early 1990’s, is increasing due to the availability of enormous amounts of raw data, the relatively inexpensive data storage mechanisms and the creation of many sophisticated data mining and artificial intelligence software systems. As these factors continue their inexorable progression, the use of very large data sets to make “objective” decisions will increase.
This trend shows the need for improved and consistently applied data governance, so that the decisions are made with accurate, timely and valid data. Humans can overcome data anomalies with experience and intuition (“that data just doesn’t look right”, “I don’t think those values are accurate”, etc.) but software is programmed to accept the data as it is presented and is expected to use it according to rules and routines instantiated in the code. Without well-governed data for super-crunching applications to use, the decisions made by the “machines” will be flawed, and could result in loss of revenue, loss of market share, loss of lives. Without well-governed processes that represent accurately the business activities and rules, the analysis software will not perform as Dr. Ayers expects, and will provide inaccurate or false or misleading results. The governance of data and process becomes increasingly important as the trend toward data-based decision-making permeates organizations from every field.
Need for Human Interaction with Data
The author still wants both human and machine to be in a mutually supportive relationship, with more weight given to machine predictions as time proceeds. Dr. Ayers answers the fundamental question of what place humans are to have in this “new world order” by identifying the need for humans to lay the foundations that enable super-crunching to occur. Humans must still “hypothesize,” he states; they must make the decisions about the variables to be used, while the computers actually perform the statistical analysis.
Humans govern data and process, humans act as data stewards, humans make the decisions about the data to be used in a data set or with an analytical application; all of these actions can fall under the role that Dr. Ayers describes for people: “laying the foundations” that enable super-crunching. This foundation must be solid, using accepted best practices for developing data governance and executing the role of stewardship, using data quality approaches and relevant software to ensure the accuracy and validity of the data. These foundations are also important for processes and analysis methods, since it is essential to use good data with good analysis methods to ensure good results.
Any organization that uses data-based decision-making or is contemplating it, should institute a data governance program to ensure their business processes provide the right data for the “super-crunchers” to load into their very large databases for their statistical packages to operate against. Since Dr. Ayers’ research shows that most types of organizations are using or planning to develop the data-based decision-making capabilities, the book Super Crunchers can be viewed as a testament for the need to develop a data governance program in all organizations.