Demand forecasting methods: Looking to the future of sales with machine learning and predictive analytics
What is the highest point of pain for executives in business? Gartner, the world’s largest IT research company, gives a clear answer: Fluctuations in demand. Too many factors – from weather fluctuations to the contributions of social media influencers – affect shoppers and cause them changes on their mind frequently.
Worse still, things that change customers’ intentions happen quite unexpectedly. Take the young climate activist Greta Thunberg, for example. Her refusal to fly on environmental grounds was the trigger for the “flight shame” movement, which led to a five percent drop in passenger numbers in Sweden.
There is no magic wand to predict scenarios like the “Thunberg effect”. But there are technologies that improve the accuracy of demand forecasting. Honestly, it will never be 100 percent accurate, but it can be accurate enough to help you achieve your business goals.
In this article we will look at the possibilities of advanced forecasting methods and outline their current limitations.
The place and role of forecasting in demand and supply planning
The demand forecast is the estimation of a probably future demand for a product or service. The term is often used in connection with demand planning, which is a more comprehensive process that begins with, but is not limited to, forecasting.
According to the Institute for Business Forecasting and Planning (IBF), demand planning “uses forecasts and experience to estimate demand for different items at different points in the supply chain”. In addition to creating estimates, demand planners participate in inventory optimization, ensure the availability of the required products and monitor the difference between forecasts and actual sales.
Demand planning serves as a starting point for many other activities, such as warehousing, shipping, price forecasts and especially supply planning, which aims to satisfy demand and requires data on customers’ anticipated needs. Here, too, we are returning to forecasts. The key for improving efficiency throughout the supply chain is to be as close to reality as possible. How do you achieve the highest possible accuracy? The answer depends on the type of business, available resources and objectives. Let’s compare existing options: traditional statistical forecasting, machine learning algorithms, predictive analysis combining both approaches, and demand-sensing as a supporting tool.
Traditional statistical predictions – good for stable markets, bad for changes
Traditional statistical methods (TSM) have been around for a long time and they remain a cornerstone of forecasting processes. The only difference compared to the previous century is that all calculations are performed automatically by modern software. For example, you can create time series forecasts for sales and trends in Excel.
Data sources: To predict the future, statistics use data from the past. Therefore, statistical forecasts are often referred to as historical. The common recommendation is to collect data on sales for at least two years.
Why use it: Traditional forecasts are still the most popular approach to sales forecasting, and with good reason. In general, demand planning solutions based on statistical techniques can be seamlessly integrated with Excel and existing enterprise resource planning (ERP) systems without the need for additional technical expertise. The most advanced systems can take into account seasonality and market trends as well as apply numerous methods to fine-tune results.
Things to consider. An important prerequisite for statistical forecasting accuracy is stability. We assume that history will repeat itself: situations that occurred two or three years ago will repeat themselves. Which is far from being true. In an ideal world, statistical methods are often unable to predict illogical changes in customer preferences or predict when market saturation will occur.
Automated statistical forecasting provides a satisfactory level of accuracy for:
- medium to long-term planning
- well-established products that enjoy stable demand, and
- Predicting total demand and not the sale of separate stockholding units (SKUs).
Does it make economic sense to invest in more sophisticated technologies? We will try to clarify things in the next section.
Machine learning for demand planning – advanced accuracy at the price of additional complexity
Increased computer performance on the one hand and increased demand volatility on the other created the conditions for a wider use of machine learning (ML) for forecasting.
Data sources: Building on statistical models, machine learning uses additional internal and external information sources to make more accurate, data-driven predictions. ML engines can work with both structured and unstructured data, including past financial and sales reports (historical data), marketing surveys, macroeconomic indicators, social media signals (retweets, stocks, peaks in supporters), weather forecasts and more.
Why use it? Machine learning uses complex mathematical algorithms to automatically recognize patterns, detect demand signals and identify complicated relationships in large data sets. In addition to analyzing huge amounts of information, intelligent systems continuously retrain models and adapt them to changing conditions to cope with volatility. These capabilities enable ML-based software to make more accurate and reliable predictions in complex scenarios.
What does more accurate actually mean? Companies that have added machine learning to their existing systems, report an increase in forecasting accuracy of 5 to 15 percent (up to 85 and even 95 percent). In addition, your team is freed from time-consuming manual adjustments and recalibrations.
Things to consider: To take advantage of the machine learning solution, you need sufficient processing power and really large amounts of high-quality data. Otherwise the system is not able to learn and make valuable predictions.
Also consider the additional complexity in terms of software maintenance and interpretation of results. While the ML mechanisms reach conclusions without human intervention, it is up to a live tech expert to determine which functions should be fed into the model, which of them have the greatest impact on the output and why the model generates a particular prediction.
All of this drives up your equipment and human resource costs, so you make sure that the revenue from a 5% improvement in accuracy covers the costs associated with it.
Best fit: The list of situations where machine learning definitely works better than traditional statistics includes
- short to medium-term planning,
- volatile demand patterns,
- the rapidly changing environment, and
- the introduction of new products.
Machine learning solutions for demand forecasting
As you can see, the use of machine learning involves some compromises. Depending on the planning horizon, data availability and task complexity you can use different statistical and ML solutions.
Predictive Sales Analytics: Modeling the future
A common business application of machine learning in combination with statistical methods results in predictive analytics. It allows not only the estimation of demand, but also the understanding of what drives sales and how customers are likely to behave under certain conditions.
To help you find out what might happen in the future, predictive analytics software performs the following operations:
- Aggregate historical and new data from various sources, including ERP and Customer Relationship Management (CRM) systems, point of sale (POS), sensors, customer demand studies, social media, marketing surveys;
- data cleansing;
- Determine which forecasting algorithm best suits your product;
- Build predictive models to identify likely outcomes and discover relationships between different factors; and
- Monitor models to measure their business performance and improve predictive accuracy.
Predictive analysis tools enable companies to combine business information with key economic indicators, promotional events, weather changes and other factors that affect customer preferences and purchasing decisions. They make it easier to identify new market opportunities and provide a more accurate view of future demand.
Drawbacks: Predictive analysis is not the simplest technique because it involves complex machine learning algorithms. It is also designed to produce predictions for at least a month, and it is unsuitable and not intended to visualize the near future.
When it comes to shorter time frames and daily granularity, demand-oriented instruments come into play.
Needs assessment: manage changes in real time
A relatively new concept in the planning process, demand assessment, uses machine learning to detect fluctuations in purchasing behavior in real time. Many experts do not see it as an independent forecasting method, but rather as a way to adjust existing forecasts. Nevertheless, the technology can be of great help to companies operating in rapidly changing markets.
Demand sensing solutions extract daily data from POS systems, warehouses, and external sources to detect increases or decreases in sales compared to historical patterns. The system automatically assesses the significance of any deviation, analyzes the influencing factors and offers adjustments to short-term plans.
The introduction of demand forecasting reduces, in average, the errors in timely forecasting by 30 to 40 percent. It enables companies to react quickly to sudden changes in customer needs and facilitates the establishment of a data-driven supply chain. Of course, you can’t make all decisions based on this technology alone, because it’s not suitable for medium- or long-term planning. But it can be a valuable addition to traditional forecasting methods.
Drawbacks: Because it relies heavily on machine learning algorithms, demand forecasting inherits all the advantages and disadvantages of ML. It requires considerable computing power, enormous amounts of data and a large library of ready-made models. In addition, some highly sensitive models can send false signals, so you need human logic to analyze the results generated by a demand determination machine.
When machine learning for demand planning works best: successful use cases
Not every company needs costly machine learning solutions to develop a reliable demand plan. However, if you are faced with a highly volatile environment, lack historical data, or need to consider a large number of variables, investing in smarter technology will pay off greatly. The following are typical scenarios where machine learning provides the greatest benefit to a forecasting process.
New Product Introduction (NPI)
Traditional forecasts require two to five years of sales data to ensure an acceptable level of accuracy. For new items you do not have a sales history. However, you cannot neglect demand forecasting because it drives several important processes, from procurement to logistics management to marketing support.
Apart from market research and obtaining expert opinions, the most common approach to forecasting launches is to identify clusters of predecessors with similar characteristics and product life cycle curves. Machine learning algorithms can be used to extract specific patterns from large amounts of unstructured data, find similarities and develop predictions, taking into account other sources of relevant information such as web analytics and social media. This provides a higher degree of accuracy and reduces the time needed to make predictions from days to hours.
Products with a short life cycle
In some industries, companies update their product range every few months, which makes the task of forecasting much more difficult. For example, fashion companies launch new products at least twice a year, and clothes should be sold quickly to make room for the next collection. In this scenario, a demand estimate must include an examination of fashion trends, seasonality and other external factors – along with historical data relating to previous collections.
Machine learning has proven effective in such complex scenarios, and the experience of the global Luxottica brand illustrates this fact. The world’s largest eyewear company uses machine learning to predict demand for 2000 new styles that will be added to the collection every year. Thanks to the intelligent engine, which analyzes data from previous launches and identifies patterns of common demand behavior, the manufacturer has improved its sales forecast accuracy by 10 percent – a significant improvement for a large number of products that quickly go out of fashion.
Weather sensitive products
Weather changes can trigger considerable fluctuations in demand, especially for seasonal products (from swimwear to umbrellas and fur coats), cosmetics, food and vehicles. Machine learning algorithms help companies to identify and measure the impact of meteorological elements on sales, and predictive analysis can be used to create “what-if” models for various scenarios.
This approach enables suppliers and retailers to effectively manage weather-related fluctuations and declines in local demand. Reports from ML forecasting users show that taking into account weather conditions (e.g. unusually high winter temperatures) improves forecasting accuracy by 5 to 15 percent for individual foods and up to 40 percent for product groups.
Companies run thousands of consumer promotions to boost sales. Unfortunately, various surveys show that 20 to 50 percent of these events do not result in a noticeable increase in demand. In addition, a study by Nielsen Holdings, the number one market research firm in the U.S., argues that 59 percent of retail promotions do not cover costs or, in other words, lead to additional spending, not profit. Ironically, 59 percent appear in a Gartner report that refers to the number of companies that still use spreadsheets to plan promotions and predict their impact. Obviously, without improved technology, companies can hardly make reliable predictions for costly marketing campaigns. The reason for this? The results of marketing campaigns depend on numerous factors with complex relationships hidden in large amounts of raw data. Fortunately, machine learning can handle this demanding task, as the world’s largest yogurt producer Danone has proven. Thanks to the use of a machine learning machine, the dairy giant has seen a 20% reduction in errors in advertising campaign forecasting and a 30% reduction in lost sales.
Overall, improvements in the predictability of transport bring two immediate benefits. First, they prevent marketing teams from spending too much money on events that do not pay off. Second, they lead to more accurate inventory management, eliminating the risk of over- or understocking.
Too many variables to analyze
This is the most common problem affecting forecast accuracy. A highly variable environment, dozens of factors influencing purchasing behavior, many types of data – all of these factors often make demand planning too complex to be performed successfully with simple tools.
The considerable complexity of the supply chain, short-term peaks in demand and the high cost of errors (where human lives are at stake) prompted the Blood and Transport Division of the UK National Health System (NHS) to move from spreadsheets and manual databases to an ML-based planning system with improved forecasting capabilities. This has enabled hospitals to reduce waste from excess blood stocks by 30 per cent without compromising the quality of service, and to respond quickly to potential shortages. “If there’s no yogurt on the supermarket shelf – well, that’s unfortunate. If there’s no blood in the hospital, the consequences are quite different,” said one NHS executive, explaining why he invested so much in the advanced solution.
Human brains are still important
Predicting demand is a challenging task that still has much room for improvement. A recent study shows that less than 30 percent of deals are accurately forecast.
As mentioned above, the introduction of machine learning tools can somewhat reduce the gap between anticipation and reality. However, this does not mean that every company should immediately switch to complex intelligent technology. You can start with small enhancements to your existing system that address problems that are difficult to solve using traditional methods. For example, use a machine learning module to make data-driven changes in short-term planning and leave long-term forecasting to old-school statistics.
No matter how smart your forecasting solution is, the most important decisions still lie with human capital. You need industry specialists to define which factors should be included in your forecasting models. Human logic is still required to assess the relevance of the results generated by digital brains and to draw final conclusions based on common sense and extensive expertise. For this reason, even ML-based demand planning systems often include a collaborative platform that allows you to involve different specialists in a forecasting process. Only by making the most of the artificial and human intelligence available can you see and plan for a better future for your business.
Please contact us if you have any further questions: