Generating Business Value with Forecasting

The quantity and quality of forecasting has greatly improved in the big data era. More data, both internal and external, is available, while computation has become cheaper and algorithms better. Generating forecasts is easier than ever, but there still remains the challenge of turning this technology into business value.

Below are a few tips for approaching business challenges and generating value through forecasting.

Most forecasting is centered around rationing a resource. Begin by identifying which elements of a business operation are constrained. Constrained usually equates directly to expensive. For almost all businesses, human resources are one of the primary examples of an expensive resource.

Managing hiring and shift scheduling are two ways that actions might be taken based on forecasts to maximize efficiency and add value managing human resources. But before those actions are taken, forecasts must first be made on relevant data. The most valuable forecasting problems are those where decisions are based on just a few inputs.

An example of a forecastable input that guides decisions would be call center volume for a call center. How many representatives are needed is directly proportional to just one input, the expected call volume. In sales, the amount of foot traffic to a store, measurable directly or indirectly by hourly sales volume, will be directly proportional to the staffing required. Both of these inputs are likely to be highly forecastable. On the other hand, managing human resources like engineers or scientists likely cannot be performed based on forecasting, as the inputs that drive the demand may be tied to inputs that are too numerous or too qualitative to easily approach.

The process of goaling, while similar to forecasting, is distinct. Generally anything that is controllable by a business is a decision, not a forecast target directly. Along the same lines, inputs which are an average of many people’s actions are easier to predict than the actions of small groups.

Creativity is often important in forecasting. The ideal inputs to a problem may not be readily available but alternatives may be available. Foot traffic data may not be available, but instead another dataset, like hourly electricity consumption, may be useful as a good-enough proxy. Frequency of updates and amount of historical data available is usually more important than having the perfect data source.

The ultimate version of forecasting is to generate a fully automated system. Examples might be electricity storage or generation systems automatically activated in anticipation of near-term demand without any human intervention required. Compute resources are also often managed in this way, where compute servers can be reserved for customer traffic when needed, and then rescheduled for off-peak loads based on forecasts of traffic to the relevant web services.

Identifying business problems that are solvable with forecasting is straightforward. Any resource which can benefit from improved planning and foreknowledge and which has a few identifiable decision-guiding inputs is a prime candidate for a valuable forecasting problem.

with Probabilistic Forecasting

A business challenge has been identified and forecasted. Staffing has been efficiently allocated across an amusement park so that as many rides are open and staff are ready for the expected number of visitors. There is a problem though, today happens to be the most perfect weather, and coincidentally a festival, a competing draw for traffic, was canceled. The traffic greatly exceeds expectations and as a result the experience for both staff and customers degrades.

Perhaps the solution is to improve the forecasting process by adding more inputs, which if these inputs are important enough can be adjusted. In many cases, long term forecasts generated by models are supplemented by humans on the ground sourcing other inputs like the short term weather forecasts and knowledge of events. A skilled data team can build forecasting tools to automatically handle many inputs, but the setup and maintenance is often greater than the generated value.

However, there is a simpler way to begin. Many good forecasting models are probabilistic, meaning they are capable of anticipating to a certain degree how much they might be missing, and provide upper and lower forecast bounds for the range of events.

In many cases, using these upper and lower bounds accurately is what really maximizes the value potential of forecasting. It should be noted that many algorithms are fairly conservative in their outputs, and while a 90% forecast may sound high, that still means an event of this magnitude could be expected at least one day every two weeks.

Probabilistic forecasts are just about predicting unpleasant shocks but also ideal outcomes. In practice, this might mean always stocking the 99% percentile upper forecast worth of product for each month to assure that as many products as possible are sold without ever running out. Of course, stocking excessive amounts can lead to waste or high storage costs.

In the end, forecasting often is a combination of the efficiency of preparing for the point forecast (what is most expected) balanced with the probabilistic forecast (extreme events) as adjusted for the opportunity costs of missing or not preparing for those extremities. Accordingly, forecastable problems are not only those with constraints and limited inputs, but also those where the opportunity cost of efficiency and mistakes is well understood so that balanced decisions can be made.

Some things are simply too challenging to forecast. The stock market is the most common example. It is possible to forecast markets, but the sheer scale of the number of inputs and the combination of qualitative psychological influences makes it impractical to attempt. As much as you may want to, usually there are easier problems which can be attempted which can lead to plenty of more localized value.

Along with the impossible, there are many things which are already forecast. For example, gas prices, an important input for many businesses, are available as free forecasts from the U.S. Energy Information Agency, and are likely as good as or better than anything a business can generate internally.

Some of the greatest care must be taken when trying to predict events which have never happened before, or are rare. For example, the potential widespread adoption of all-electric self-driving cars is of interest to many automotive businesses, but there is very little precedent for such a major change to make it forecastable. Simulation forecasting (using a third variable that is known which predicts the rare events) or by comparison forecasting to related trends are possible but often difficult for these cases.

One trick that can help in tricky situations is resampling data. For example, it may be almost impossible to predict daily sales of grocery items, as there is too much randomness in demand. However, predicting the same sales for a less frequent period of time, weekly or monthly sales, is often much more manageable as the randomness averages out across time. There are only so many bananas the locals can eat.

In conclusion, the most value from business forecasting comes from problems that help optimize the use of constrained resources driven by a manageable number of inputs. The use of probabilistic forecasting is important for managing uncertainty. Finally, effort invested into solving problems should be modulated with the understanding that many problems are simply impossible to forecast. Certainly every business has challenges that may be addressed with forecasting, but not every problem has a forecasting solution.