Notes

A Brief History of Business Intelligence

2021-05-05

About

A quick look at how business intelligence fits within business analytics and business software in general.

This piece features a business analytics software classification scheme, a high level historical overview of business intelligence, and a timeline of key companies, trends, and related things in this space.

Introduction

Competition drives business. Whether its new entrants disrupting an old industry or established players with a century of operations, every company is in perpetual competition with other players in the industry. Some companies create uncontested market space, but every company competes with others at least on their understanding of the industry, the marketplace, and their own operations. The best informed companies outplay the competition.

Businesses have used statistics and data analysis to inform their operations for as long as these disciplines have been around. Analysing business data was manual work for a long time, well into the 20th century, but equally businesses have always been quick to try out new methods for their number crunching. Leading companies have always looked to new technology for an edge over their competition.

After electronic computers showed their utility in science and public sector projects, various commercial computers found customers in the business decision support market. Over time, users and uses of business data have changed dramatically, but the overall mission of informing business decisions has remained. The history of business data analysis is dominated by innovations in automation and ease of use.

With every computer generation, business analysts have been able to do more and to get their work done faster. From mainframes to personal computing and from on-premise deployments to the cloud, business analytics software has evolved rapidly over the years. Today, more people than ever make some use of business data. Everyone, across a wide range of abilities, is looking for valuable data-driven insights that could drive business outcomes.

Given the long history and rapid evolution of the field of business data analysis, it's not that surprising that there's lots of overlapping terminology and confusion. I generally use the term business analytics in a broad sense to refer to any craft of using data to inform business operations. Sometimes it makes sense to distinguish different categories of business data analysis systems, in which case business analytics is distinct from, for example, customer relationship management (CRM) — even though both essentially involve data analysis for business gain.

Some people use business analytics and business intelligence interchangeably. Others think one is a subset of the other — and it varies which way around. With this piece, I hope to clarify my own view on this scene.

The point is that historically, and still today, a variety of terms have been used to describe more or less the same products and services and workflows.

I'm trying to find some order in this space, because I'm trying to formulate an idea that I have about a new approach to business intelligence, namely data storytelling. For now, data storytelling is probably best understood as the craft and practice of communicating data-driven insights to decision makers effectively using compelling narratives.

This piece is a quick attempt to get some clarity on some of the terms in the business analytics space through classification and a high level historical overview. The idea is that I can expand this page as I figure things out for myself, plus I can link here from elsewhere, whenever I need to introduce the topic.

A Taxonomy of Business Analytics

In 2019, the Competition and Markets Authority (CMA), a non-ministerial department of HM Government, decided on the market impact of the Salesforce Tableau acquisition. As part of that decision, the CMA provided a handy classification chart that positions Tableau and Salesforce within the software industry. We'll adapt that classification here to form an overview of the business analytics landscape and see how data storytelling fits in that.

UK Gov, BI classification under software
Image: BI classification, as relating to the Salesforce/Tableau case, from the CMA decision (pdf, via gov.uk).

Business Software

The CMA classification divides software into two top level categories: consumer and business software. Consumer software is sold directly to individual users, while business software is typically procured for groups of people at a time or through an organisation-wide licence. Naturally, business software helps the enterprise in carrying out its business activities, while consumer software can provide value in many areas of life from social media and online dating to video processing tools and all kinds of entertainment.

Business software splits into individual-use software and enterprise application software (EAS). Individual-use software is concerned with personal, private workflows and processes, while EAS is more commonly a solution for the whole business. For example, the Microsoft Office suite — Excel, PowerPoint, etc. — is a family of individual-use software tools. In contrast, EAS systems see use for example in corporate finance, HR, and customer relations management (CRM): one system for many users. The European Commission defines EAS as "software that supports major business functions needed to manage a business effectively".

Enterprise applications span several software categories, in a sense aligned with corporate functions. For example, there are many large software vendors for CRM alone. In many cases enterprise systems are sold as a bundle, meaning that a single vendor supplies the organisation with a horizontally integrated solution that spans multiple software categories — multiple business functions.

These software bundles can be delivered as a suite of independent, but connected tools, or through some kind of module scheme within a unified host application. The integrations, connections between software systems, are a central concern in business software deployment.

For the present discussion, we are interested in business analytics, one of the software categories under enterprise software. This category splits in three in the CMA classification: data science platforms, analytics applications, and business intelligence.

Data science platforms support workflows around large scale data processing and analysis. Many contemporary platforms focus on machine learning and AI applications, and are often based on workbook powered workflows. Some platforms specialise in data mining or predictive analytics, others focus on data engineering, model deployment operations and more. In broad terms, it is perhaps fair to say that data science platforms are focused on model building.

Business intelligence (BI) software is all about data-driven business insight generation. These insights could be based on data on internal business activities, customers and the competitive marketplace, suppliers, or some third party data — or all of it at the same time. Analysis software that doesn't fit the BI description, including specialist industry specific solutions and other more bespoke software, can be classified as general analytics applications.

The rest of the picture is a little bit fuzzy, the categories are not very well defined. The whole business analytics sector has evolved considerably over the last 10 years: many of the old players are still around, but there are lots of new players as well.

Business Intelligence

Business intelligence, our main focus here, can be divided into modern business intelligence and traditional business intelligence. All BI is concerned with feeding insights into the decision-making process, but modern and traditional approaches have some key differences. Both support some variation of a Query-Reporting-Analysis workflow (QRA).

Modern BI is frequently about self-service and empowering users; traditional BI often requires IT department involvement for any changes. Modern BI workflows are typically powered by interactive dashboards and customisable reports, while traditional BI offers a relatively static set of outputs and little interactivity. Modern BI often features modern application features such as support for collaboration, powerful searching capabilities, intuitive drag-and-drop UIs, rich integration options, and more. In contrast, traditional BI is more fixed and focused in its operation.

In the traditional model, raw data undergoes heavy transformations and is carefully packaged into warehouses and relatively inflexible data views. These stiff resources can then by queried for historical review, but without capabilities for dynamic exploration. In a sense the the interface reveals the underlying technical limitations. As a business user, changing the question in a traditional setup requires talking to database people or other specialists.

One key difference between modern and traditional BI is the delivery mechanism. Modern BI is often browser based, and powered by cloud services and cloud data storage, though many vendors support on-premise setups as well. Cloud/on-premise hybrids are also a viable option. Traditional BI, built for the pre-cloud era, is primarily on-premise. In recent years, cloud BI solutions have won market share from on-premise alternatives, and the trend is definitely towards more cloud solutions in the future. Some recent surveys suggest the split is about 50-50 today.

It is clearly the case that all of the innovation in the BI space happens in the modern BI category. At the same time, we have seen a kind of a homogenisation in the BI world, with most vendors having the same core set of features. Yesterday's differentiating selling points are mere table stakes today. After ten years of Big Data, some even claim that business intelligence is mature technology — BI is a solved problem. It's just that the users need to catch up!

This is where data storytelling enters the picture. Data storytelling is a new take on business intelligence, focused on how data-driven insights get communicated.

Business analytics has been around for a long time and business intelligence is as old as computing itself. Given where we are and the legacy landscape, it may be difficult to see how things could be different. At the same time, many feel that the Big Data revolution has failed to deliver on the hype and there's lots of room for innovation.

Personally, I strongly believe that there is demand for entirely new approaches to business intelligence and for revisiting what business analytics even means.

"By 2025, data stories will be the most widespread way of consuming analytics, and 75% of stories will be automatically generated using augmented analytics techniques." — Gartner research, 2021 (via TechTarget)

A Brief History of BI

The history of business intelligence is best understood as a progression towards ever greater analytical capability. Businesses have always wanted to make informed decisions: analytical capabilities are a long-standing competitive advantage. This competition in making sense of company and market data — the business analytics arms race — has always pushed the industry forward. Over time analytical methods have become highly sophisticated, and the user facing technology has become easier and easier to use.

Business analytics has been around for at least as long as electronic computers have been part of the story. In the beginning, in the age of mainframes and heavy duty systems, the only viable focus was on descriptive analytics, answering the question "What has happened?". It could take weeks to get an answer.

As technology and data processing scale improved, analysts could move deeper, towards diagnostic analytics and ad hoc queries. Analysts could start asking questions such as "Why did something happen?".

These backwards looking questions, the domain of traditional BI, are still the foundation for many reporting pipelines. But business has evolved. People increasingly want to look forward rather than back.

Beyond interactive historical queries, we may be interested in real-time, or streaming analytics, where the volume and velocity of digested data is so great that any analytics is best done on-line as soon as the data arrives. This can result in a powerful picture of the current state of a system or of a given business function, or even of the entirety of a business. Real-time analytics can provide answers to questions such as "What is happening?".

Taking another step forward we arrive in the modern business intelligence sphere. In the Big Data and infinite cloud compute era we can ask ever more sophisticated questions. Modern BI is focused on predictive analytics ("What will happen?") as well as prescriptive analytics ("What could happen, if we make certain choices?"). Answers to these uncertain queries can come from AI systems, machine learning models, or other augmented analytics tools.

Today, the biggest challenges in business intelligence are not so much technical as they are social and interpersonal. What are the best ways to communicate analysis results? What are the best ways of turning insights into action?

Timeline

Key companies, terms, solutions, and trends in business analytics, and some related things for context.


Further Reading

References

Anderson, Nate, et al. 2019. Bain & Company. What’s Hot and What’s Not.

Avidon, Eric. 2021. TechTarget. Gartner predicts data storytelling will dominate BI by 2025.

Competition and Markets Authority. 2019. Salesforce.com, Inc. / Tableau Software Inc merger inquiry.

Evelson, Boris. 2019. Forrester. Business Intelligence Market Consolidation — What To Expect Next.