Using Predictive Analytics within Business Intelligence: A Primer

Predictive analytics has helped drive business intelligence (BI) towards business performance management (BPM). Traditionally, predictive analytics and models have been used to identify patterns in consumer oriented businesses, such as identifying potential credit risk when issuing credit cards, or analyzing the buying habits of retail consumers. The BI industry has shifted from identifying and comparing data patterns over time (based on batch processing of monthly or weekly data) to providing performance management solutions with right-time data loads in order to allow accurate decision making in real time. Thus, the emergence of predictive analytics within BI has become an extension of general performance management functionality. For organizations to compete in the market place, taking a forward-looking approach is essential. BI can provide the framework for organizations focused on driving their business based on predictive models and other aspects of performance management.

We'll define predictive analytics and identify its different applications inside and outside BI. We'll also look at the components of predictive analytics and its evolution from data mining, and at how they interrelate. Finally, we'll examine the use of predictive analytics and how they can be leveraged to drive performance management.

Overview of Analytics and Their General Business Application

Analytical tools enable greater transparency within an organization, and can identify and analyze past and present trends, as well as discover the hidden nature of data. However, past and present trend analysis and identification alone are not enough to gain competitive advantage. Organizations need to identify future patterns, trends, and customer behavior to better understand and anticipate their markets.

Traditional analytical tools claim to have a 360-degree view of the organization, but they actually only analyze historical data, which may be stale, incomplete, or corrupted. Traditional analytics can help gain insight based on past decision making, which can be beneficial; however, predictive analytics allows organizations to take a forward-looking approach to the same types of analytical capabilities.

Credit card providers offer a first-rate example of the application of analytics (specifically, predictive analytics) in their identification of credit card risk, customer retention, and loyalty programs. Credit card companies attempt to retain their existing customers through loyalty programs, and need to take into account the factors that cause customers to choose other credit card providers. The challenge is predicting customer loss. In this case, a model which uses three predictors can be used to help predict customer loyalty: frequency of use, personal financial situations, and lower annual percentage rate (APR) offered by competitors. The combination of these predictors can be used to create a predictive model. The predictive model can then be applied and customers can be put into categories based on the resulting data. Any changes in user classification will flag the customer. That customer will then be targeted for the loyalty program. Financial institutions, on the other hand, use predictive analytics to identify the lifetime value of their customers. Whether this translates into increased benefits, lower interest rates, or other benefits for the customer, classifying and applying patterns to different customer segmentations allows the financial institutions to best benefit from (and provide benefit to) their customers.

What's Really Driving Business Intelligence

If you follow the logic of the major analysts covering the business intelligence (BI) market, the market drivers for business intelligence software are based on fairly simple environmental factors. The most commonly cited market drivers are the following:

1. Increasing Regulation—New laws in both the US and Europe are requiring companies to make their external reporting more transparent, forcing business to develop better systems for storing and retrieving the most current and detailed information on operations.

2. Information Overload—Having invested heavily in CRM, ERP, and SCM systems, many businesses are awash in data, but short on actionable intelligence. Being able to aggregate, mine, and analyze data in order to prepare for and respond to business and market events is the next step in making IT investments pay off.

3. Demand for Accountability and Metrics—A slow economic recovery has forced many businesses to continue trimming budgets, while requiring greater accountability for every area of spending. Business intelligence, and its associated data-mining, analytics and scorecards, provides the tools necessary to track performance metrics tied directly to strategic corporate goals.

4. Need to Improve Competitive Responsiveness—With markets exposed to increasing competition, customer demand and pricing pressure, businesses need to reduce cycles by accelerating processes that support aggressive competitive strategies. BI initiatives provide real-time information that can help businesses eliminate process delays and streamline management to improve decision-making and market response.

There's nothing wrong with these descriptions of existing market conditions. Each of them is a correct and compelling reason for businesses to support BI initiatives. However, they don't tell the whole story. In fact none of these market drivers, taken individually or taken as a whole, are enough to explain the level of investment being made in business intelligence software.

Think about it. Businesses have found ways to skirt or delay the impact of increasing regulations for decades. Why would they suddenly respond so rapidly to new regulations today? While information overload is acute, many businesses took a soaking in IS investments over the past few years. What smart CEO would throw good money after bad to try and rescue a previous investment? Metrics and accountability are certainly in high demand while budgets are tight, but how many businesses would invest millions of dollars just to be confident their million-dollar investments are sound? That kind of long-term thinking doesn't move markets in our quarterly-driven world. And finally, yes, businesses are being compelled to be more efficient and effective in order to compete, but that battle has been shaping up for decades among TQM, Six-Sigma, lean production methodologies—does anyone really believe the end of the rainbow is only a dashboard away?

While, each of these market drivers is accurate, they're only symptoms of a much deeper drive—a drive that is shaped by a concern far greater than the threat of regulation, information overload, accountability or competitive response. It's a drive that reflects the deepest fears of a CFO. It's a drive that shapes the search for CEOs that can move the businesses that move markets. It's a drive that cuts straight to the bottom line of the corporation, because it's about the single, all-important factor that defines the success of every business today.

It's all about how the value of a business is measured.

How BI Addresses the Needs of SOX Compliance

Traditionally, BI software has targeted the needs of financial decision makers. BI tools initially enabled organizations to analyze financial data, to identify trends, and to drill down on report data to reveal operational transactions, as well as to assign tasks to individual employees, in order to give management the ability to implement robust auditing processes. The driver behind these functions is the ability to capture data from several data sources across an organization, and to centralize them in a data warehouse. Aside from data centralization across the organization, data warehouses allow organizations to implement and monitor data quality activities to ensure accurate data. This reduces the potential for accidental data errors.

BI tools help vendors to meet the demands of organizations that need to comply with SOX regulations, scorecards, and business activity monitoring (BAM). General reporting and analysis functionality permits organizations to take a top-down approach to management, yet still meet SOX compliance. CEOs and CFOs who are responsible for assuring compliancy and who are accountable to the SEC often aren't directly responsible for actual report generation or in-depth budgeting. Task assignment and management of processes are internal driving forces within BI, and help companies manage employee tasks and responsibilities for each financial report and function, as well as ensuring data quality. Basically, BI allows the CEO to manage internal processes and data to meet SOX compliance, and gives CEOs the ability to micromanage tasks at each level to ensure compliance, and to identify any potential errors (as well as identifying who made them, and when they were made within the process). If proper data quality processes are implemented, organizations can guarantee that data errors do not occur within the data warehouse itself and that any key stroke errors and the like are cleansed as they enter the data warehouse, before financial analyses and reporting functions are performed to meet SOX requirements.

Although, as mentioned above, BI software can help organizations meet SOX compliancy, vendors have also taken SOX issues into account when upgrading their product suites to make sure that required standards can be met on an ongoing basis. Even though many other forms of financial reporting software meet SOX compliancy, BI solutions have the added bonus of built-in workflow processes and data integration features to ensure long-term compliancy. Data within spreadsheets can be changed, and structures are not always put in place to manage those changes. However, BI software suites have built-in task assignment and audit functions for managing, distributing, and auditing data (based on where the data comes from, who has ownership of the data, and how the data has been processed).

Using Business Intelligence Infrastructure to Ensure Compliancy with the Sarbanes-Oxley Act

The US Sarbanes-Oxley Act (SOX) of 2002 was established to protect investors from the potential for fraudulent accounting. After the exposure of several corporate scandals, such as the Enron and WorldCom affairs, the US government was compelled to pass legislation ensuring accurate financial reporting and auditing from organizations publicly traded in the United States. SOX affects any public corporation competing in the American marketplace. As a result of SOX, not only have financial controls and reporting schedules become stricter, but responsibility for accurately reporting financial results has been placed in the hands of organizational heads, namely the chief executive officers (CEOs) and chief financial officers (CFOs), to provide accurate financial and auditing data.

This means that financial departments have had to reevaluate the way they manage their controls and reporting. It is no longer possible for organizations to change data without accounting for these changes to shareholders. Now that the responsibility for accurate financial reporting has been placed on upper management, with heavy fines and potential prison terms being imposed for noncompliance, financial analysis tools, such as those provided by business intelligence (BI) vendors, are becoming increasingly important to the financial auditing process. Ensuring proper data controls, proper reporting and auditing structures, and the accurate capture of the ensuing data, are important aspects of SOX compliance and make up the essential elements of BI solutions.

There are three sections of SOX that deal directly with the use of information technology (IT). Section 302 requires management certification that procedures have been put in place to address accurate financial conditions and disclosure controls for all financial statements. Section 404 requires management certification that effective internal controls and procedures have been developed for financial report preparation. Finally, section 409 requires that timely reports be provided to investors, the US Securities and Exchange Commission (SEC), and other corporate stakeholders.

How Predictive Analytics Are Used within BI, and How They Drive an Organization's BPM

Data mining, predictive analytics, and statistical engines are examples of tools that have been embedded in BI software packages to leverage the benefits of performance management. If BI is backward looking, and data mining identifies the here and now, predictive analytics and their use within performance management is the looking glass into the future. This forward-looking view helps organizations drive their decision making. BI is known for its consolidation of data from disparate business units, and for its analysis capabilities based on that consolidated data. Performance management goes one step further by leveraging the BI framework (such as the data warehousing structure and extract, transform, and load [ETL] capabilities) to monitor performance, identify trends, and allow decision makers the ability to set appropriate metrics and monitor results on an ongoing basis.

With predictive analytics embedded within the above processes, the metrics set and business rules identified by organizations can be used to identify the predictors that need to be evaluated. These predictors can then be used to shift towards a forward-looking approach in decision making by using the strengths from the areas identified above. Scorecards are one example of a performance management tool that can leverage predictive analytics. The identification of sales performance by region, product type, and demographics can be used to define what new products should be introduced into the market, and where. In general, scorecards can graphically reflect the selected sales information and create what-if scenarios based on the data identified to verify the right combinations of new product distribution.

What-if scenarios can be used within the different visualization tools to create business models that anticipate what might happen within an organization based on changes in defined variables. What-if analysis gives organizations the tools to identify how profits will be affected based on changes in inflation and pricing patterns as well as the impact of increasing the number of employees throughout the organization. Online analytical processing (OLAP) cubes can be created to identify dimensional data, and patterns within changing dimensions can be compared over time to contrast scenarios using a cube structure to automatically view the outcome of the what-if scenarios.

Components of Predictive Analytics

Data mining can be defined as an analytical tool set that searches for data patterns automatically and identifies specific patterns within large datasets across disparate organizational systems. Data mining, text mining, and Web mining are types of pattern identification. Organizations can use these forms of pattern recognition to identify customers' buying patterns or the relationship between a person's financial records and their credit risk. Predictive analytics moves one step further and applies these patterns to make forward-looking predictions. Instead of just identifying a potential credit risk, an organization can identify the lifetime value of a customer by developing predictive decision models and applying these models to the identified patterns. These types of pattern identification and forward-looking model structures can equally be applied to BI and performance management solutions within an organization.

Predictive analytics is used to determine the probable future outcome of an event, or the likelihood of a situation occurring. It is the branch of data mining concerned with the prediction of future probabilities and trends. Predictive analytics is used to analyze automatically large amounts of data with different variables, including clustering, decision trees, market basket analysis, regression modeling, neural nets, genetic algorithms, text mining, hypothesis testing, decision analytics, and so on.

The core element of predictive analytics is the predictor, a variable that can be measured for an individual or entity to predict future behavior. These predictors are based on models that are created to use the analytical capabilities within the generated predictive models. Descriptive models classify relationships by identifying customers or prospective customers, and placing them in groups based on identified criteria. Decision models consider business and economic drivers and constraints that surpass the general functionality of a predictive model. In a sense, statistical analysis helps to drive this process as well. The predictors are the factors that help identify the outcomes of the actual model. For example, a financial institution may want to identify the factors that make a valuable lifetime customer.

Multiple predictors can be combined into a predictive model, which, when subjected to analysis, can be used to forecast future probabilities with an acceptable level of reliability. In predictive modeling, data is collected, a statistical model is formulated, predictions are made, and the model is validated (or revised) as additional data becomes available. One of the main differences between data mining and predictive analytics is that data mining can be a fully automated process, whereas predictive analytics requires an analyst to identify the predictors and apply them to the defined models.

A decision tree is a variable within predictive analytics that allows the user to visualize the mapping of observations about an item and compare it to conclusions about the item's target value. Basically, decision trees are built by creating a hierarchy of predictor attributes. The highest level represents the outcome, and each sub-level identifies another factor in that conclusion. This can be compared to if-else statements, which identify a result based on whether certain factors meet specified criteria. For example, in order to assess potential bad debt based on credit history, salary, demographics, and so on, a financial institution may wish to identify multiple scenarios, each of which is likely to meet bad debt customer criteria, and use combinations of those scenarios to identify which customers are most likely to become bad debt accounts.

Regression analysis is another component of predictive analytics that allows users to model relationships between three or more variables in order to predict the value of one variable in comparison to the values of the others. It can be used to identify buying patterns based on multiple demographic qualifiers such as age and gender which can be beneficial to identify where to sell specific products. Within BI, this is beneficial when used with scorecards that focus on geography and sales.

Overview of Analytics and Their General Business Application

Analytical tools enable greater transparency within an organization, and can identify and analyze past and present trends, as well as discover the hidden nature of data. However, past and present trend analysis and identification alone are not enough to gain competitive advantage. Organizations need to identify future patterns, trends, and customer behavior to better understand and anticipate their markets.

Traditional analytical tools claim to have a 360-degree view of the organization, but they actually only analyze historical data, which may be stale, incomplete, or corrupted. Traditional analytics can help gain insight based on past decision making, which can be beneficial; however, predictive analytics allows organizations to take a forward-looking approach to the same types of analytical capabilities.

Credit card providers offer a first-rate example of the application of analytics (specifically, predictive analytics) in their identification of credit card risk, customer retention, and loyalty programs. Credit card companies attempt to retain their existing customers through loyalty programs, and need to take into account the factors that cause customers to choose other credit card providers. The challenge is predicting customer loss. In this case, a model which uses three predictors can be used to help predict customer loyalty: frequency of use, personal financial situations, and lower annual percentage rate (APR) offered by competitors. The combination of these predictors can be used to create a predictive model. The predictive model can then be applied and customers can be put into categories based on the resulting data. Any changes in user classification will flag the customer. That customer will then be targeted for the loyalty program. Financial institutions, on the other hand, use predictive analytics to identify the lifetime value of their customers. Whether this translates into increased benefits, lower interest rates, or other benefits for the customer, classifying and applying patterns to different customer segmentations allows the financial institutions to best benefit from (and provide benefit to) their customers.