Factors of success in predictive supply chains

Joannes Vermorel

Having supply chains on autopilot through predictive technologies and achieving above human performance at scale remains a distant goal for nearly all companies, except the usual suspects (e.g. Amazon). This state of affairs is all the more surprising considering the vast number of software vendors that promise radical reductions of stocks and stock-outs - among other things. The long standing joke at Lokad has been that the only way Lokad could compete with the claims of our competitors would be to start saying that we cure cancer too.

Yet, my casual observations among the past experiences¹ from Lokad’s customer base indicate that the vast majority of predictive supply chain initiatives are failing. By failing, I specifically mean that these solutions don’t even manage to score 10 out of 12 on our 5min supply chain performance test. A more stringent criteria for success would be a lasting boost in the supply chain’s overall financial performance, but right now, our modest 5min test is enough to provide a reasonable upper bound on success rates.

It’s hard to put a number on the actual rate, successes are so few and far between that I believe the overall market success rate² to be below one out of ten. However, like the lottery, the winner (singular) makes the news, while the losers (hordes) are ignored. The problem is amplified as both parties, client and vendor, are strongly incentivized to market themselves as successful, no matter the actual outcome of the project. For the vendor, a success is obviously great PR material. For the client’s employees³, success means better job prospects⁴. Worse, letting the rest of the company realize that a multi-million investment was wasted is too frequently a recipe to get fired or sidetracked career-wise. Fortunately, quantitatively measuring the supply chain performance is a remarkably elusive goal - mostly due to network effects. Thus, it really takes an epic⁵ screw-up to not be able to cover up the mess by simply fudging a little with the numbers.

The first notable exception is the “AI” solutions⁶ - in supply chain optimization - that achieves a spectacular zero percent success rate based on my extensive observations⁷. Patrick Cousot, one of my former computer science professors, told me back in 2002 that in computer science, a “solution” was only referred to as “AI” as long as we had no clue whatsoever how to make it work. As soon as a practical path is discovered to make it work, the solution takes another name: convex optimization, static analysis, reinforcement learning, etc. Four years later, Mehryar Mohri, my research supervisor at the time, repeated the same thing to me. Two decades later, those insights proved prescient⁸, and indeed, those AI vendors don’t appear to have the slightest clue on how to make their “AI” deliver anything that would qualify as production-grade from a supply chain perspective.

If it wasn’t such a waste of resources, the situation would be perceived as comical. Let’s take the recent worldwide Walmart demand forecasting competition: out of the two dozen “notable” supply chain vendors as listed by, say, Gartner, none of them is making it to the Top 100 out of 900+ teams. The discrepancy between what objectively works, and what the market is buying or promoting is staggering. Nevertheless, free markets are incredible filters: over time, what doesn’t work well enough gets eliminated. It’s not because people come to their senses and change their mind, but merely because companies stuck with inefficient methods gradually fade away and get replaced by their competitors - the creative destruction as identified by Schumpeter.

The second notable exception is Lokad⁹. Over the last two years, our success rate has been consistently above three out of four. The risks are still there, but we are now one order of magnitude less risky than our competitors. Historically, it did not start that way. According to the same success criteria listed above during the first three years, from 2008 to 2011, we achieved exactly zero successes. It took us almost a gruesome decade to painfully earn each extra percentage of success, through dozens of gradual improvements. It would be exhausting to try to catalog the whole affair, but let’s review a cherry-picked list of notable insights.

• We encourage clients to terminate wherever dissatisfied. Period. Since 2008, Lokad has been promoting monthly subscriptions, while our competitors are still pushing yearly or multi-yearly commitments. This is no accident. When a client quits, it gives a clear signal that it isn’t working. It usually boils down to either a faulty tech, or a lack of competence (or both). There is no sugar coating it. It’s tough, but we can learn from it. In contrast, there is usually nothing to be learned from the made-up polite pain points made up one year after the events to make the story respectively look better than it really was¹⁰.

• The right forecasting technology is more important than a merely accurate one. It took us years to realize that classic naked forecasts were downright harmful. We solved this problem through probabilistic forecasts and specialized algebras to assign financial scores to decisions.

• The right data crunching platform matters more than raw capabilities. Supply chain data are complex, heterogeneous and poorly understood. There are tons of fairly mundane problems to be addressed to avoid the pitfalls of “garbage in, garbage out”. Facilitating the in-place documentation of the data is a good start and avoiding dumb typos through autocomplete rapidly becomes a must-have.

• To the greatest extent, correctness should be achieved by design. Fail fast and break things is not an option for supply chains. Purchasing or production blunders are exceedingly expensive. It’s already challenging enough to operate a supply chain in a highly chaotic world, a predictive technology should not make things worse by adding its own layer of entropy.

• Approximately correct is better than exactly wrong. Hard problems like lead time variability, competitors’ price moves, cannibalization within the assortment, self-prophetic effects, … should be embraced rather than dismissed. Furthermore, it’s easy to derail an initiative by focusing on the wrong challenges such as [factoring the weather]/tv/2019/7/19/can-you-use-the-weather-to-forecast-demand) because it’s cool, while dismissing tail risks because planning for the worse requires nerves and fortitude.

Most of the elements that played a decisive role in improving the success rate of our initiatives for predictive supply chains turned out to be basic - fundamental even - concepts, such as revisiting the very notion of what a forecast should be expected to be, and reengineering our technology and our processes from scratch, based on the new understanding as many times as necessary. We will keep doing so in the future. Our commitment is to the resolution of the problem, not to the specifics of the present day solution.