The beer game is a widely used in-class game that is played in supply chain
management classes to demonstrate the bullwhip effect. The game is a
decentralized, multi-agent, cooperative problem that can be modeled as a serial
supply chain network in which agents cooperatively attempt to minimize the
total cost of the network even though each agent can only observe its own local
information. Each agent chooses order quantities to replenish its stock. Under
some conditions, a base-stock replenishment policy is known to be optimal.
However, in a decentralized supply chain in which some agents (stages) may act
irrationally (as they do in the beer game), there is no known optimal policy
for an agent wishing to act optimally.
We propose a machine learning algorithm, based on deep Q-networks, to
optimize the replenishment decisions at a given stage. When playing alongside
agents who follow a base-stock policy, our algorithm obtains near-optimal order
quantities. It performs much better than a base-stock policy when the other
agents use a more realistic model of human ordering behavior. Unlike most other
algorithms in the literature, our algorithm does not have any limits on the
beer game parameter values. Like any deep learning algorithm, training the
algorithm can be computationally intensive, but this can be performed ahead of
time; the algorithm executes in real time when the game is played. Moreover, we
propose a transfer learning approach so that the training performed for one
agent and one set of cost coefficients can be adapted quickly for other agents
and costs. Our algorithm can be extended to other decentralized multi-agent
cooperative games with partially observed information, which is a common type
of situation in real-world supply chain problems.
In multi-echelon inventory systems the performance of a given node is
affected by events that occur at many other nodes and in many other time
periods. For example, a supply disruption upstream will have an effect on
downstream, customer-facing nodes several periods later as the disruption
"cascades" through the system. There is very little research on stock-out
prediction in single-echelon systems and (to the best of our knowledge) none on
multi-echelon systems. However, in real the world, it is clear that there is
significant interest in techniques for this sort of stock-out prediction.
Therefore, our research aims to fill this gap by using deep neural networks
(DNN) to predict stock-outs in multi-echelon supply chains.
The newsvendor problem is one of the most basic and widely applied inventory
models. There are numerous extensions of this problem. If the probability
distribution of the demand is known, the problem can be solved analytically.
However, approximating the probability distribution is not easy and is prone
to error; therefore, the resulting solution to the newsvendor problem may be
not optimal. To address this issue, we propose an algorithm based on deep
learning that optimizes the order quantities for all products based on features
of the demand data. Our algorithm integrates the forecasting and
inventory-optimization steps, rather than solving them separately, as is
typically done, and does not require knowledge of the probability distributions
of the demand. Numerical experiments on real-world data suggest that our
algorithm outperforms other approaches, including data-driven and machine
learning approaches, especially for demands with high volatility. Finally, in
order to show how this approach can be used for other inventory optimization
problems, we provide an extension for (r,Q) policies.