In game theory, a stochastic game (or Markov game), introduced by Lloyd Shapley in the early 1950s,[1] is a repeated game with probabilistic transitions played by one or more players. The game is played in a sequence of stages. At the beginning of each stage the game is in some state. The players select actions and each player receives a payoff that depends on the current state and the chosen actions. The game then moves to a new random state whose distribution depends on the previous state and the actions chosen by the players. The procedure is repeated at the new state and play continues for a finite or infinite number of stages. The total payoff to a player is often taken to be the discounted sum of the stage payoffs or the limit inferior of the averages of the stage payoffs.

Stochastic games generalize Markov decision processes to multiple interacting decision makers, as well as strategic-form games to dynamic situations in which the environment changes in response to the players’ choices.[2]

Two-player games

edit

Stochastic two-player games on directed graphs are widely used for modeling and analysis of discrete systems operating in an unknown (adversarial) environment[citation needed]. Possible configurations of a system and its environment are represented as vertices, and the transitions correspond to actions of the system, its environment, or "nature". A run of the system then corresponds to an infinite path in the graph. Thus, a system and its environment can be seen as two players with antagonistic objectives, where one player (the system) aims at maximizing the probability of "good" runs, while the other player (the environment) aims at the opposite.

In many cases, there exists an equilibrium value of this probability, but optimal strategies for both players may not exist.

We introduce basic concepts and algorithmic questions studied in this area, and we mention some long-standing open problems. Then, we mention selected recent results.

Theory

edit

The ingredients of a stochastic game are: a finite set of players  ; a state space   (either a finite set or a measurable space  ); for each player  , an action set   (either a finite set or a measurable space  ); a transition probability   from  , where   is the action profiles, to  , where   is the probability that the next state is in   given the current state   and the current action profile  ; and a payoff function   from   to  , where the  -th coordinate of  ,  , is the payoff to player   as a function of the state   and the action profile  .

The game starts at some initial state  . At stage  , players first observe  , then simultaneously choose actions  , then observe the action profile  , and then nature selects   according to the probability  . A play of the stochastic game,  , defines a stream of payoffs  , where  .

The discounted game   with discount factor   ( ) is the game where the payoff to player   is  . The  -stage game is the game where the payoff to player   is  .

The value  , respectively  , of a two-person zero-sum stochastic game  , respectively  , with finitely many states and actions exists, and Truman Bewley and Elon Kohlberg (1976) proved that   converges to a limit as   goes to infinity and that   converges to the same limit as   goes to  .

The "undiscounted" game   is the game where the payoff to player   is the "limit" of the averages of the stage payoffs. Some precautions are needed in defining the value of a two-person zero-sum   and in defining equilibrium payoffs of a non-zero-sum  . The uniform value   of a two-person zero-sum stochastic game   exists if for every   there is a positive integer   and a strategy pair   of player 1 and   of player 2 such that for every   and   and every   the expectation of   with respect to the probability on plays defined by   and   is at least  , and the expectation of   with respect to the probability on plays defined by   and   is at most  . Jean-François Mertens and Abraham Neyman (1981) proved that every two-person zero-sum stochastic game with finitely many states and actions has a uniform value.[3]

If there is a finite number of players and the action sets and the set of states are finite, then a stochastic game with a finite number of stages always has a Nash equilibrium. The same is true for a game with infinitely many stages if the total payoff is the discounted sum.

The non-zero-sum stochastic game   has a uniform equilibrium payoff   if for every   there is a positive integer   and a strategy profile   such that for every unilateral deviation by a player  , i.e., a strategy profile   with   for all  , and every   the expectation of   with respect to the probability on plays defined by   is at least  , and the expectation of   with respect to the probability on plays defined by   is at most  . Nicolas Vieille has shown that all two-person stochastic games with finite state and action spaces have a uniform equilibrium payoff.[4]

The non-zero-sum stochastic game   has a limiting-average equilibrium payoff   if for every   there is a strategy profile   such that for every unilateral deviation by a player  , the expectation of the limit inferior of the averages of the stage payoffs with respect to the probability on plays defined by   is at least  , and the expectation of the limit superior of the averages of the stage payoffs with respect to the probability on plays defined by   is at most  . Jean-François Mertens and Abraham Neyman (1981) proves that every two-person zero-sum stochastic game with finitely many states and actions has a limiting-average value,[3] and Nicolas Vieille has shown that all two-person stochastic games with finite state and action spaces have a limiting-average equilibrium payoff.[4] In particular, these results imply that these games have a value and an approximate equilibrium payoff, called the liminf-average (respectively, the limsup-average) equilibrium payoff, when the total payoff is the limit inferior (or the limit superior) of the averages of the stage payoffs.

Whether every stochastic game with finitely many players, states, and actions, has a uniform equilibrium payoff, or a limiting-average equilibrium payoff, or even a liminf-average equilibrium payoff, is a challenging open question.

A Markov perfect equilibrium is a refinement of the concept of sub-game perfect Nash equilibrium to stochastic games.

Stochastic games have been combined with Bayesian games to model uncertainty over player strategies.[5] The resulting stochastic Bayesian game model is solved via a recursive combination of the Bayesian Nash equilibrium equation and the Bellman optimality equation.

Applications

edit

Stochastic games have applications in economics, evolutionary biology and computer networks.[6][7] They are generalizations of repeated games which correspond to the special case where there is only one state.

See also

edit

Notes

edit
  1. ^ Shapley, L. S. (1953). "Stochastic games". PNAS. 39 (10): 1095–1100. Bibcode:1953PNAS...39.1095S. doi:10.1073/pnas.39.10.1095. PMC 1063912. PMID 16589380.
  2. ^ Solan, Eilon; Vieille, Nicolas (2015). "Stochastic Games". PNAS. 112 (45): 13743–13746. doi:10.1073/pnas.1513508112. PMC 4653174. PMID 26556883.
  3. ^ a b Mertens, J. F. & Neyman, A. (1981). "Stochastic Games". International Journal of Game Theory. 10 (2): 53–66. doi:10.1007/BF01769259. S2CID 189830419.
  4. ^ a b Vieille, N. (2002). "Stochastic games: Recent results". Handbook of Game Theory. Amsterdam: Elsevier Science. pp. 1833–1850. ISBN 0-444-88098-4.
  5. ^ Albrecht, Stefano; Crandall, Jacob; Ramamoorthy, Subramanian (2016). "Belief and Truth in Hypothesised Behaviours". Artificial Intelligence. 235: 63–94. arXiv:1507.07688. doi:10.1016/j.artint.2016.02.004. S2CID 2599762.
  6. ^ Constrained Stochastic Games in Wireless Networks by E.Altman, K.Avratchenkov, N.Bonneau, M.Debbah, R.El-Azouzi, D.S.Menasche
  7. ^ Djehiche, Boualem; Tcheukam, Alain; Tembine, Hamidou (2017-09-27). "Mean-Field-Type Games in Engineering". AIMS Electronics and Electrical Engineering. 1: 18–73. arXiv:1605.03281. doi:10.3934/ElectrEng.2017.1.18. S2CID 16055840.

Further reading

edit
edit