GPT represents a significant breakthrough in large language models and the application of AI to knowledge work across almost all domains.
Over the next few weeks, we will have a series of blog posts that assess the incumbent natural language technologies that are part of the current BI ecosystem and analyze the strengths and limitations of large language models as they relate to performing BI Analyst activities. We will also provide a set of recommendations for effectively leveraging this revolutionary new technology in the BI ecosystem to provide meaningful improvements in engagement and end-user experience.
The Promise of Automated Analysis
Long before Open AI irrevocably changed the world with the release of GPT-3 and GPT-4, many voices were making a case for integrating Natural Language with BI. The case for bringing these technologies together is simple and compelling. Data is complex, and business users struggle to answer seemingly simple questions with their existing BI Tools. The reports and dashboards available to them are often difficult to use and don’t consistently produce helpful insights.
As a result, organizations turn to analysts to interpret the trends in the data and communicate key findings to the business. However, this approach doesn’t scale because analyst resources are scarce and expensive, and business users want answers to their questions immediately. The proponents of natural language solutions point to these problems and suggest that what is needed is Natural Language BI Bots.
AI agents field questions posed in natural language and respond via a narrative that provides a simple and coherent answer to the user’s question. This type of “virtual analyst” would answer the user’s questions without the cost of consulting a human analyst. There would be no delay in answering the question or the need for a human analyst to prioritize the work needed to answer the question against other urgent projects. At its core, the promise of an AI Analyst is the ability to deliver true self-service analytics at scale with a dramatically reduced cost.
Capabilities & Limitations
To date, the reality of natural language agents has yet to reach the lofty promise of this technology. There are two ways in which Natural Language plays a role in the current BI Landscape. These are through tools that use Natural Language Generation (NLG) and those that generate queries and create visualization through Natural Language Queries (NLQ)
Natural Language Generation tools like Arria and Narrative Insights (recently acquired by Tableau) provide a mechanism to generate text narratives from datasets or dashboards. These tools often present compelling demos but are challenging to deploy and configure. Over time, the narratives generated by these tools are increasingly ignored by users, and in the end, these technologies seldom add sustained business value.
Companies like Tellius and ThoughtSpot dynamically generate visualizations as an answer to natural language inquiries posed by the user. These solutions have had some success in displacing traditional dashboards, forcing leading BI Tool vendors like Tableau to develop their own solutions in this area.
While these solutions represent a significant improvement in usability compared to the rigidity of traditional dashboards and reports, they are expensive and require massive computing resources. Also, they are limited because the user must select a specific data model before asking his questions and can only frame the question using terms the model understands. Depending on the data contained in the selected model, the user’s question may not be answerable by the model. Even when the model does contain all the data needed to answer the question, the output is generated as a set of visualizations that require considerable analytical skills for correct interpretation.
Why Natural Language BI Solutions Fall Short
Let’s explore why the current crop of technologies has failed to meet the promise of integration of BI with Natural Language.
Natural Language Generation
Time-Consuming Setup
The current crop of NLG systems requires an analyst to select and pre-configure the model that should be applied to generate the narrative. This technology is intended to reduce the time spent by analysts generating narrative insights. Still, ironically its wide-scale deployment is often limited by the time required of analysts to configure and optimize the system to create valuable narratives from the data.
Lack of Flexibility
An analyst must often apply different lenses to the data to generate useful insight from a dataset. For example, in some cases, an insight might be drawn from simply looking at trends within the data (e.g., sales increased last month). In other cases, the notable insight comes from a change in the underlying data (e.g., we have a new customer in our top 10 list). The rigid nature of existing NLG technologies means that the narrative around any given dataset always follows a pre-set pattern (e.g., highlight trends), and other types of insights may be completely missing from the generated text.
Poor Attention ROI
The biggest limitation of NLG systems is that business users must exert the effort to read and process the generated narrative to determine if the text contains valuable insights. In many cases, the narrative is simply presenting an analysis that the user already knows as the fundamental trend in the data has stayed the same. Users find themselves spending time reading text that provides them with no value. In a world of diminishing attention spans, users quickly start focusing their attention elsewhere.
Natural Language Queries
Several significant limitations with Natural Language Query systems become evident once customers undertake enterprise-wide implementations:
Limited Scope and Extensive Data Prep Requirements
Natural Language query tools can only work effectively against datasets with well-structured and clean data with a well-defined semantic model. This data typically requires extensive data-prep pre-processing and the corresponding data model must be designed to anticipate the types of inquiries the user will pose. These solutions are effective only when they strike just the right “goldilocks” sweet spot, carefully balancing functional and performance considerations. If the dataset has too many attributes, it will grow too large, and performance will become an issue, and if it is too constrained in scope, it will lack the key dimensions needed to answer critical questions posed by users.
Fragmented Solution Space
Because of the need to keep language query datasets limited in size, an enterprise-wide implementation of these systems requires the creation of many models. In addition, these systems tend to augment rather than replace existing BI Tools. This means that users are presented with a fragmented patchwork of a solution. To obtain an answer to a question, the user must know to look to the natural language system rather than their dashboards and must then select the correct prepared model which can answer this question.
A highly fragmented solution space creates significant challenges to wide-scale effective adoption. Users can’t simply ask a question via a single Google-style prompt and receive an answer. Instead, a human analyst must often be consulted to guide the user to determine the tool to answer the question.
Limitations of mapping natural language to SQL
Natural Language Query systems are purpose-built to respond to questions that explicitly reference a set of dimensions, metrics, and constraints into SQL to retrieve and visualize the answer. This approach works for well-structured questions. For example, suppose a user asks, “Show me my sales by product in AMEA?” to a model built on a dataset containing sales data dimensioned by product and geography. In that case, the application can retrieve and chart the results. However, the tool would be ineffective at answering more complex and nuanced questions. Questions such as “What are the most significant trends in our financial results last quarter?” or “Why are we having trouble sustaining our gross margin?” cannot be translated into a set of simple SQL statements and are beyond the capabilities of these tools. This constraint that only questions that can be converted to SQL can be answered restricts the utility of these solutions and means that users must still turn to human analysts to answer many important questions.
In conclusion, the above recommends effectively leveraging large language models like GPT in the BI ecosystem to address the limitations of current natural language technologies. It emphasizes the need for improved usability, flexibility, attention return on investment, and broader scope in natural language BI solutions. By overcoming these challenges, organizations can achieve meaningful improvements in engagement and end-user experience in the field of business intelligence.
