Improving Forecasting Accuracy in an Age of Retail Disruption

Are AI and machine learning the future?

Are AI and machine learning the future?

Demand forecasting is at the heart of every retailer’s supply chain, and for good reason: the demand forecast is central to sales, profitability and the customer shopping experience, and it has a ripple effect throughout the supply chain. Yet despite a wealth of established demand forecasting solutions and methodologies, retailers struggle to produce accurate, timely demand forecasts.

According to Retail Systems Research, 65% of retailers consistently stock out on fast-moving categories and products, and 63% have too much inventory in slow-moving categories and products. The result is a major drag on retailers’ performance; according to IHL Group, out-of-stocks account for $634 billion in lost sales worldwide each year, while overstocks result in $472 billion in lost revenues due to markdowns. These problems are only magnified as retail sales increase. U.S. retail growth is up 4.2%, but 77% of retailers can’t keep up. It’s little wonder that 53% of retail CXOs list demand management as the top area where AI can make improvements over the next five years.

Is it time to rethink demand forecasting for retail?

There are a number of reasons for these disconnects. For one, 55% of retailers still rely on in-house developed solutions for demand forecasting. However, most retailers can’t afford the investment and time that are required to continually update in-house solutions in response to market changes, much less re-evaluate their approach to demand forecasting as a whole. Retailers who use third-party forecasting systems struggle, too, as many of these systems are outdated and incapable of understanding today’s complex consumer behavior patterns. These are just a few of the many challenges retailers face with demand forecasting. Others include:

Bad data drives inaccurate forecasts

Retailers are overwhelmed by the sheer volume of data available to them, as well as the amount of bad data that must be cleansed to be usable. Bad data always leads to poor results, even with the best forecasting systems: the classic “garbage in, garbage out” syndrome. As a result of poor forecasts, users must manually adjust orders; Gartner estimates that 30 to 60% of orders require human intervention because of inaccurate forecasting.

Bad data drives inaccurate forecasts

Retailers are overwhelmed by the sheer volume of data available to them, as well as the amount of bad data that must be cleansed to be usable. Bad data always leads to poor results, even with the best forecasting systems: the classic “garbage in, garbage out” syndrome. As a result of poor forecasts, users must manually adjust orders; Gartner estimates that 30 to 60% of orders require human intervention because of inaccurate forecasting.

Promotions, seasonal changes and external influences

Retailers struggle to understand the impact of events and promotions on demand, as well as accurately forecast the introduction of new products and categories, since no prior history is available. Adding to this complexity are seasonal shifts and external influences such as weather, competitor promotions and store openings, and social media.

Promotions, seasonal changes and external influences

Retailers struggle to understand the impact of events and promotions on demand, as well as accurately forecast the introduction of new products and categories, since no prior history is available. Adding to this complexity are seasonal shifts and external influences such as weather, competitor promotions and store openings, and social media.

Challenges of fresh and ultra-fresh produce

Different products require vastly different forecasting strategies. Items with short shelf lives, such as dairy, meat and produce, can be more difficult to forecast than packaged food and basic stock items, since “fresh food is perishable, demand is highly variable, and lead times are often uncertain,” as McKinsey notes.

Challenges of fresh and ultra-fresh produce

Different products require vastly different forecasting strategies. Items with short shelf lives, such as dairy, meat and produce, can be more difficult to forecast than packaged food and basic stock items, since “fresh food is perishable, demand is highly variable, and lead times are often uncertain,” as McKinsey notes.

Shortage of data scientists

Retailers must also come to terms with an acute shortage of skilled personnel – especially data scientists that understand demand forecasting, as well as machine learning and artificial intelligence

Shortage of data scientists

Retailers must also come to terms with an acute shortage of skilled personnel – especially data scientists that understand demand forecasting, as well as machine learning and artificial intelligence

Unified commerce, industry convergence add complexity

Today’s fast-changing retail environment, characterized by industry convergence and new paths to the consumer, further complicates forecasting and replenishment challenges, as the lines between retailers, brands and distributors blur. Here are just a few examples.

  • Amazon continues to integrate Whole Foods more closely into its business, as it strategically builds market share in grocery. Whole Foods was a key focus of Prime Day 2018, with deep incentives for Prime customers to shop in store. Amazon complements its two-hour free delivery of Whole Foods orders for Prime customers with in-store pickup of online orders in as little as 30 minutes.
  • Grocery retailers are responding aggressively to the Amazon-Whole Foods juggernaut. In July 2018, United Natural Foods (UNFI) announced it would acquire Supervalu for $2.9 billion, helping the organic and natural foods distributor reduce its dependence on Whole Foods, which accounts for about a third of UNFI’s business, and giving UNFI access to Supervalu’s 3,437 stores 1.
  • Mega-mergers and alliances are redefining the retail landscape. In the UK, for example, Sainsbury’s proposed merger with Asda will vault the combined entity into the No. 1 position in UK grocery market share and give popular online digital retailer Argos a presence in Asda stores. In response to this, Carrefour and Tesco formed a strategic alliance that some analysts believe will touch off a price war in the UK.
  • Online grocery sales are growing exponentially. According to IGD, the Institute for Grocery Distribution, the UK’s online grocery market will grow 48% by 2022, while the U.S. online grocery market will expand by 129%.

As e-commerce sales rise and shoppers choose from new fulfillment options, retailers must consider the impact of unified commerce on demand forecasting. Gartner analyst Tom Enright notes, “Unified commerce retailers must take a more detailed approach to forecasting demand. They should supplement their existing focus on forecasting consumer demand at a product/location and time combination with a new approach to predict how consumers will use the shopping options available to them in making their purchases.”

Enright continues: “This means focusing on how much consumers buy as well as on how they make their purchases, whether from the store, ordering online and collecting from a locker location, or requesting direct shipment to home from a distribution center, as well as a number of other methods.”

With channel convergence and the emergence of new formats, understanding what your customer needs and forecasting for each ‘location’ is key. If the forecast isn’t accurate, there are adverse effects throughout the supply chain, since the demand forecast is central to every supply chain function – multi-echelon inventory, allocation and replenishment logistics.

Meeting these challenges, while overcoming the additional problems we’ve identified, requires new technologies and solutions for demand forecasting. It’s no wonder that 71% of retailers rank retail forecasting as “very important” to their success, and 51% plan major new implementations or replacements that incorporate AI and machine learning.

Deep learning is AI’s way of reasoning and understanding the total pictureso the system can give you recommendations on the steps to take

AI and machine learning overcome forecasting limitations

AI is quickly gaining ground in supply chain management, and in particular for demand forecasting. Applied correctly, AI alleviates inconsistent inventory buys, overstocks (and the resulting markdowns), out-of-stocks, margin erosion and for fresh waste optimization.

AI-based demand forecasting systems make use of machine learning and are based on the idea that when we submit data to the machines, they can learn for themselves. For forecasting, this means that the machine learning algorithms automatically detect patterns and make connections in huge batches of data that would be impossible, or take too long, for humans to recognize. Because machine learning algorithms are automated, they can analyze all the data – not just part of it – at scale, unlocking an enormous amount of business value by tackling data that rarely or never receives human attention. And because the data is automatically cleansed, it overcomes the “garbage in, garbage out” forecasting problems generated by dirty or incomplete data.

With traditional demand forecasting systems, data is fed into a computer, and then the computer applies that data to a static, pre-determined set of rules to analyze it and generate a result. With machine learning, however, the computer becomes adaptive – dynamically responding to changes in the data and updating the forecasts accordingly. This greatly improves accuracy and enables retailers to react more responsively to demand over time.

AI-based forecasting systems are a radical departure from existing systems, in which users try to cluster and define seasonal patterns, with results that are often disappointing and lead to out-of-stocks, lost sales and markdowns. Store-level forecasting is difficult with traditional systems, especially when retailers are dealing with millions of possible SKU and store combinations. AI can alleviate all the manual intervention that’s required today and deliver far more accurate forecasts.

Dr. Pallab Chatterjee, chairman and CEO of Symphony RetailAI, explains the difference between traditional and AI-based forecasting:

Everyone has analytics today, and they give you reports and dashboards that effectively tell you what happened in your business. You can then take those reports and ask analysts to go and examine patterns and root causes and try to answer the question of why these things happened. The first part of what we do with AI and machine learning is quickly identify those patterns and answer the ‘why’ question.

The next step is to figure out what to do about it: ‘What actions can I take to combat why this happened and prevent it from happening again?’ Deep learning is AI’s way of reasoning and understanding the total picture, so the system can give you recommendations on the steps to take. Machine learning tells you why it happened and deep learning tells you what you should do about it.

Dr. Pallab Chatterjee, chairman and CEO of Symphony RetailAI, explains the difference between traditional and AI-based forecasting:

Everyone has analytics today, and they give you reports and dashboards that effectively tell you what happened in your business. You can then take those reports and ask analysts to go and examine patterns and root causes and try to answer the question of why these things happened. The first part of what we do with AI and machine learning is quickly identify those patterns and answer the ‘why’ question.

The next step is to figure out what to do about it: ‘What actions can I take to combat why this happened and prevent it from happening again?’ Deep learning is AI’s way of reasoning and understanding the total picture, so the system can give you recommendations on the steps to take. Machine learning tells you why it happened and deep learning tells you what you should do about it.

With machine learning, demand forecasting moves beyond the limitations of statistical, static forecasting:

Machine learning algorithms can access all the history and understand the sales patterns as well as the anomalies (“What happens if it rains at an amusement park on the busiest day of the year?”) at a much deeper level.

Machine learning considers the past, present and future. It understands the past mistakes that should have been forecasted and responds faster as it learns over time, continually improving forecast accuracy.

Currently, 52% of retail supply chain executives say they spend too much time crunching data. AI and machine learning solutions remove the need for people to touch the data, leading to vast improvements in speed, accuracy and efficiency.

Machine learning automatically pulls in additional inputs and outside information to understand external impacts, determine their relevance, and find patterns that humans can’t detect. These include factors such as weather data, contextual data such as a food stamp calendar for the U.S. or school holidays in the EU, as well as competitive information, local demographics and events.

Machine learning automatically pulls in additional inputs and outside information to understand external impacts, determine their relevance, and find patterns that humans can’t detect. These include factors such as weather data, contextual data such as a food stamp calendar for the U.S. or school holidays in the EU, as well as competitive information, local demographics and events.

Poor inventory planning profoundly impacts logistics, and multi-channel retailing increases the complexities between forecasting, orders, channel allocation and logistics as retailers respond to customer demand. AI can analyze varied demand patterns and scenarios to provide an end-to-end view of the supply chain and become an important step in timely replenishment and more efficient logistics.

Poor inventory planning profoundly impacts logistics, and multi-channel retailing increases the complexities between forecasting, orders, channel allocation and logistics as retailers respond to customer demand. AI can analyze varied demand patterns and scenarios to provide an end-to-end view of the supply chain and become an important step in timely replenishment and more efficient logistics.

Keep in mind that machine learning is not a “one size fits all” solution. One of the challenges in adopting machine learning is knowing which algorithms and forecasting techniques to apply to each data set. A good AI-enabled forecasting system should pick the best approach without intervention.

AI turns data into a competitive advantage

AI-based forecasting with machine learning will increasingly become the new standard for retail demand forecasting. With AI-based systems, there’s no need for retailers to hire additional data scientists, which are a scarce resource. Instead, the system serves as an automated data scientist for your data, with new levels of information, alerts and insights.

AI isn’t just for the largest retailers, either. The emergence of new Software as a Service (SaaS) solutions makes AI a practical reality for retailers of all sizes, enabling them to take advantage of the power of machine learning and free up their supply chain managers for more strategic work.

What kind of impact can AI have on a retailer’s bottom line? One leading grocer had a 20% error rate in its current demand forecast, which means that items were not selling as predicted 20% of the time – resulting in overstocks, out-of-stocks and impact to the bottom line. However, an AI-based demand forecasting system with machine learning was machine learning was able to produce a more accurate picture of the error rate to just over 5% – a 75% improvement.

AI also enables retailers to have more accurate, automated pull-based replenishment and more profitable push-based replenishment, due to better forecasts. The algorithms are constantly learning, and can provide real-time views, as well as daily and intra-day forecasts, enabling supply chain managers to respond to sudden fluctuations in demand.

Most importantly, AI-based demand forecasting helps to create happy, loyal customers who keep returning due to more relevant assortments and new products, ultimately driving overall shopper satisfaction. As Gartner notes: “Retailers do not always recognize the impact that excelling at assortment planning, demand forecasting, and policies for shipping and returns can have to optimize consumer experience….” AI and machine learning are the keys to making this a reality.

Was the report interesting?

Register to download the whitepaper as a PDF and keep up to date with the latest research as it's published.