Page 9
Semester 5: Stochastic Processes
Definition and classification of stochastic processes
Stochastic Processes
Item
A stochastic process is a collection of random variables indexed by time or space, representing systems that evolve over time in a probabilistic manner.
Item
Stochastic processes can be classified into various types, including discrete and continuous processes, stationary and non-stationary processes, and Markov and non-Markov processes.
Item
Discrete stochastic processes involve random variables at discrete time points, whereas continuous stochastic processes involve random variables at any time point.
Item
A stationary process has statistical properties that do not change over time, while a non-stationary process has properties that do.
Item
A Markov process is characterized by the property that future states depend only on the current state and not on the sequence of events that preceded it.
Item
Stochastic processes have applications in various fields, including finance, telecommunications, queueing theory, and statistical mechanics.
Markov chains: transition probabilities, classification of states, limiting distributions
Markov chains
Transition Probabilities
Transition probabilities represent the likelihood of moving from one state to another in a Markov chain. Each transition probability is a function of the current state and can be represented in a matrix form known as the transition matrix. This matrix captures all possible transitions and their probabilities and must satisfy the condition that each row sums to one.
Classification of States
Markov chain states can be classified into several categories: transient states, recurrent states, absorbing states, and periodic states. Transient states are those that may not be visited again once left. Recurrent states are those that are revisited infinitely often. Absorbing states are those that, once entered, cannot be left. Periodic states return to their original state after a fixed number of steps.
Limiting Distributions
A limiting distribution describes the long-term behavior of a Markov chain as the number of transitions approaches infinity. The existence of a limiting distribution depends on the characteristics of the Markov chain. If a limiting distribution exists, it can be found by solving the system of equations derived from the transition probabilities, and will provide the probabilities of being in each state in the long run.
Poisson processes and their properties
Poisson processes and their properties
Introduction to Poisson Processes
A Poisson process is a stochastic process that models random events occurring independently over a fixed interval of time or space. Key characteristics include the number of events occurring in disjoint intervals being independent and the number of events following a Poisson distribution.
Properties of Poisson Processes
The main properties include: 1. Memorylessness: The future is independent of the past. 2. Independent increments: The number of events in non-overlapping intervals is independent. 3. Poisson distribution: The number of events in a fixed interval follows a Poisson distribution, characterized by the parameter lambda, which is the average rate of occurrence.
Interarrival Times
In a Poisson process, the time between consecutive events is exponentially distributed. This aspect is crucial for understanding the timing of events in the process.
Applications of Poisson Processes
Poisson processes are widely used in fields such as telecommunications, traffic engineering, and queuing theory. Applications include modeling call arrivals at a call center or the occurrence of events like requests to a server.
Relation to Other Processes
Poisson processes are a type of continuous-time Markov process and are related to other stochastic processes such as birth-death processes. Understanding these relationships helps in analyzing more complex real-world systems.
Renewal theory basics
Renewal theory basics
Introduction to Renewal Theory
Renewal theory is a branch of probability theory that deals with events that occur continuously over time. It is fundamental in studying stochastic processes where events occur at random intervals. Renewal processes are used to model scenarios such as the time until the next customer arrives or equipment failures.
Key Concepts
Important concepts in renewal theory include the renewal function, which provides the expected number of renewals in a given time period, and the inter-arrival time distribution, which describes the time between consecutive events. The theory often uses tools such as the Poisson process for simplification.
Renewal Reward Theorem
The renewal reward theorem extends renewal theory by associating a reward or payoff with each renewal. It states that the expected long-term average reward rate can be derived from the expected time until renewals and total rewards accumulated.
Applications of Renewal Theory
Renewal theory finds applications in various fields including operations research, queuing theory, inventory management, and reliability engineering. It helps in decision-making processes where the timing of events is uncertain.
Limit Theorems in Renewal Theory
Limit theorems provide insights into the behavior of renewal processes as time approaches infinity. For example, the central limit theorem can be applied to show that the normalized sums of inter-arrival times converge to a normal distribution under certain conditions.
Martingales and their properties
Martingales and their properties
Definition of Martingales
A martingale is a sequence of random variables where the expected value of the next observation is equal to the current observation, given all prior observations. Formally, for a stochastic process {X_n}, it holds that E(X_n | X_1, X_2, ..., X_{n-1}) = X_{n-1}.
Properties of Martingales
1. Martingale Property: The fundamental property of martingales where the future expectation is equal to the present value. 2. Convergence: A martingale may converge almost surely to a limit under certain conditions. 3. Uniform Integrability: A martingale is uniformly integrable if it satisfies certain boundedness conditions.
Types of Martingales
1. Sub-martingales: A stochastic process that satisfies E(X_n | X_1, ..., X_{n-1}) ≥ X_{n-1}. 2. Super-martingales: A process where E(X_n | X_1, ..., X_{n-1}) ≤ X_{n-1}. 3. Simple Martingales: Martingales formed by simple random variables with a finite number of outcomes.
Applications of Martingales
Martingales have applications in various fields such as finance for pricing options, in gambling strategies, and in the theoretical aspects of probability theory for proving other probabilistic results.
Martingale Theorems
1. Optional Stopping Theorem: States conditions under which the expected value of a martingale at stopping time equals its value at any prior time. 2. Doob's Martingale Convergence Theorem: Provides conditions under which a martingale converges almost surely.
