2 Markov Decision Process: Turning Problems intro Solvable Models

This chapter introduces Markov decision processes as a practical way to turn messy, high-dimensional business problems into structured models that can actually be solved. The central idea is to divide complexity into states, actions, transitions, and rewards, but to do so in a way that preserves the relationships between parts of the system. Rather than merely breaking a problem apart, the framework helps reconnect the pieces so that uncertainty, dynamics, and long-term strategy can be handled coherently.

A major theme is the Markov property: the future should depend only on the present state, not on the full history, because carrying all past information quickly becomes computationally infeasible. The chapter explains how good state design is what makes this possible, using examples such as software users, customer support, inventory, and maintenance. It also distinguishes passive evolution in Markov chains from active decision-making in Markov decision processes, where actions change both immediate rewards and future state probabilities, and where a policy guides choices over time with long-term gain in mind.

The chapter then grounds the ideas in a factory production planning example, showing how to define a state, choose actions, model stochastic demand, compute rewards, and compare heuristic policies through simulation. It concludes by emphasizing reward engineering and constraint handling: rewards should be stepwise when possible, constraints should be encoded into the state when feasible, soft constraints can be handled with graded penalties, hard constraints may require action masking, and reward scales must be balanced to avoid misleading incentives. Overall, the chapter sets up MDPs as the foundation for reinforcement learning in business, where thoughtful modeling is just as important as the learning algorithm itself.

Three different types on how a system can evolve moving from one state to another.

Conditional probability of dependency of next state to all previous states.

Software service transitions graph and its matrix representation.

Transition distributions in three different scenarios

Markov chain and Markov decision process.

Two important probability distributions widely used for modeling business processes.

Cumulative profit ($) of following each policy in the planning horizon.

Two sides of reward function and some examples for each part.

An example of reward normalization for a multi objective portfolio optimization

• Complex business problems require more than decomposition — they demand structured reassembly, and Markov decision processes offer just that.
• Markov decision processes break decision-making into states, actions, rewards, and transitions, enabling sequential decision making under uncertainty.
• State design is the key: a good state captures all essential information to predict what happens next (Markov property). Poor state design breaks predictive power.
• Markov chains model how systems evolve probabilistically; Markov decision processes extend this by adding control — letting agents make decisions that influence outcomes.
• A policy maps each state to an action, not for short-term gains but for long-term success — balancing current choices against future consequences.
• Markov decision process modeling requires clear thinking about business objectives and constraints, which together guide optimal decisions.
• Most real-world business problems can be modeled as Markov decision processes if states are crafted wisely.
• Reward functions in business must encode multiple objectives and penalties — they are not just feedback signals but behavioral steering tools.
• Effective reward engineering means giving feedback at the right time (stepwise rewards), balancing objective and constraint terms, and avoiding scale mismatches.
• Hard constraints should be enforced using action masking and penalty, while soft constraints can be handled just by penalties and dynamic trade-offs.
• Designing rewards requires careful monitoring to avoid unintended shortcuts, where agents optimize proxy metrics instead of real goals.

FAQ