Self-service Business Intelligence (BI) is taking off. According to Market Research Future, the global self-service analytics market is expected to grow approximately 20% per year through 2022, with a total value of over $10B USD at the end of that timeframe.
There are a multitude of BI tools that can enable self-service analytics. But while many of these tools have fantastic capabilities, no tool is better than the analyst using it. In fact, if improperly used, the blessing of these tools’ visual appeal and ease-of-use becomes a curse, as they can spread bad insights far and wide. Carlie J. Idoine, Director of Research at Gartner, put it this way:
“If data and analytics leaders simply provide access to data and tools alone, self-service initiatives often don’t work out well. This is because the experience and skills of business users vary widely within individual organizations.”
What’s needed is a way to train business users to think about and develop Business Intelligence products that account for the business problem, available data, and the need for customer validation. Here’s how business users can start their journey toward effective, self-service analytics – and better decision-making.
Write Down the Business Question
Before writing a single line of code, examining a single row of data, or building a single visualization, first articulate the decision that must be made. For example, if you want to decide which customer segment to target for a new product, simply write down, “Which customer segment should we target with our new product?”
This question should be posted to an easily-accessible, public place like an intranet wiki or even a white board in a high-traffic area of the office. Putting it in writing gives stakeholders and developers the opportunity to develop a shared understanding of the problem, and provides meaningful direction for the BI solution.
A publicly-available and publicly-stated business question tends to initiate a problem-solving virtuous cycle: developers start writing down pieces of the solution that others can see, test, improve, and emulate. The solution can take shape in a white board drawing, a spreadsheet, bits of SQL code, or a working prototype of a dashboard. In all these cases, actually seeing a solution in action helps business stakeholders better understand both the business question, and what will and will not work in the solution.
Find the Data
With the decision framed in writing, the next step is to get the data required to answer the question. For the question posed above, for example, customer data from a CRM tool would be helpful, as well as information on competitive offerings.
With any data source, be sure to address the following questions:
- How is the data collected or generated? Was the data generated by sensors and automatically recorded? Was it manually entered by humans? Was it a simple recording of a transaction? Knowing the answers to these questions can help with data profiling and quality efforts later on.
- How can it be obtained? Can you query a database for the data? Or does it live in a spreadsheet somewhere on your internal network? Is it in a text file sent via SFTP? The format and acquisition method will impact your initial processing of the data.
- How frequently can it be updated? If you have a business decision to make on a recurring basis, knowing how often the data is updated impacts how often you can rerun the analysis.
- Are Subject Matter Experts (SMEs) available? Is there anyone who understands both the business context and technical constraints of the data? If so, get their contact information, as they can be a great help in understanding the data.
Understand the Data
With data in hand, the real nuts-and-bolts work of Business Intelligence begins. The analyst must thoroughly explore the data to understand what it shows, what it doesn’t show, data gaps, potential data quality issues, and possible remediation strategies. Data dictionaries, data exploration tools (which often come with BI software), and access to data stewards can all help strengthen the analyst’s understanding of the data.
- Data Dictionary: The owner of the data source may provide a data dictionary, which is a document that explains the file format, technical definitions of the columns (like data type and field length), business definition of the columns, and formulas for calculated fields.
- Data Exploration and Profiling: This is the practice of getting to know the data set. It reveals the columns, range of values, row count, incidence of junk data, and possible data quality issues.
- Data Stewards: These SMEs can answer in-depth business and technical questions about the data set, and can often help with remediation of data quality issues.
Prep the Data
With a firm understanding of the data in mind, the analyst is ready to engage in the most time-consuming aspect of BI: data preparation. This best practice involves taking the data through a journey of shaping, aggregating, blending, and calculating on the way toward a usable end state that will help answer the documented business question. (Surprisingly, writing down the business question is incredibly valuable for this step, as it helps keep the preparation work from veering off into interesting, but unneeded, tangents.)
Closely related to data preparation is data transformation. This is the practice of developing data architecture to house prepared data and writing code to load the data into the data architecture. Read more about data transformation here.
When faced with a new business question and/or data source(s), there are a lot of “unknown unknowns” that could trip up even the most astute BI analyst. For example, different stakeholders may have a different understanding of the business question (again, writing it down helps here). Sometimes, the meanings of new columns or tables in an unfamiliar data source may not mean what they seem to at first. Starting with a smaller-scale deliverable for a smaller, preferably data-savvy audience can help tease out these nuances while reducing the risk of lost time on the project.
Another benefit of small, incremental BI solutions is that they can provide some early value to the product stakeholders. Early ROI of any BI product helps earn additional funding, engage stakeholders in the next phase of development, and set the stage for early validation of the data.
Validate Early and Often
It’s critical to have a set of trustworthy, expected values that can help validate the results of a BI product. It may be useful to validate an early BI product against an existing report. Also, stakeholders and SMEs can sometimes provide expected values.
Documenting the business question and incrementally developing the solution set the stage for successful BI validation. The business question guides the testers in developing test scripts and finding test data sets. In addition, the incremental approach reduces testing time and finds BI defects early in product development while they’re relatively easy to fix.
Iteratively Improve With Stakeholders
Related to the practice of frequent validation are iterative reviews with stakeholders. Following the Agile principles of daily stakeholder engagement and continuous delivery, it’s important to show business stakeholders incremental versions of the BI product. Frequent interaction gives them the chance to provide early feedback and adjust the direction of the product based on new business conditions. A handy best practice for this principle is to do frequent “dark releases” to internal users and a small set of trusted external users.
By engaging stakeholders early and often, BI developers can gain early feedback on source data, calculations, aggregations, and BI presentation, which will ultimately lead to higher-quality insights and business decisions. When working with stakeholders, keep another Agile principle in mind: “Simplicity – the art of maximizing the amount of work not done – is essential.”
Business Intelligence, combined with Big Data, opens up a world of possibilities for generating business value. The computing power and storage capacity of Big Data provides an unprecedented amount of data to a wide array of users. Data Science can find timely insights in that data. But it’s Business Intelligence that makes those insights accessible to a large audience of business decision makers.
The key to making better decisions with BI is clearly understanding and documenting the business question. This question drives the sourcing and understanding of the data needed to make the business decision. Once that data has been prepped and worked into a prototype, early stakeholder engagement and validation can create a compelling BI product that earns the trust of business decision makers.
How have you used BI tools to make business decisions? Let us know in the comments: