Contents
    Preface
    About the Author
    About the Notation
    REPRESENTATION OF DIGITAL VIDEO
   1 BASICS OF VIDEO
    1.1 Analog Video
    1.1.1 Analog Video Signal
    1.1.2 Analog Video Standards
    1.1.3 Analog Video Equipment
    1.2 Digital Video
    1.2.1 Digital Video Signal
    1.2.2 Digital Video Standards
    1.2.3 Why Digital Video?
    1.3 Digital Video Processing
   2 TIME-VARYING IMAGE FORMATION MODELS
    2.1 Three-Dimensional Motion Models
    2.1.1 Rigid Motion in the Cartesian Coordinates
    2.1.2 Rigid Motion in the Homogeneous Coordinates
    2.1.3 Deformable Motion
    2.2 Geometric Image Formation
    2.2.1 Perspective Projection
    2.2.2 Orthographic Projection
    2.3 Photometric Image Formation
    2.3.1 Lambertian Reflectance Model
    2.3.2 Photometric Effects of 3-D Motion
    2.4 Observation Noise
    2.5 Exercises
   3 SPATIO-TEMPORAL SAMPLING
    3.1 Sampling for Analog and Digital Video
    3.1.1 Sampling Structures for Analog Video
    3.1.2 Sampling Structures for Digital Video
    3.2 Two-Dimensional Rectangular Sampling
    3.2.1 2-D Fourier Transform Relations
    3.2.2 Spectrum of the Sampled Signal
    3.3 Two-Dimensional Periodic Sampling
    3.3.1 Sampling Geometry
    3.3.2 2-D Fourier Transform Relations in Vector Form
    3.3.3 Spectrum of the Sampled Signal
    3.4 Sampling on 3-D Structures
    3.4.1 Sampling on a Lattice
    3.4.2 Fourier Transform on a Lattice
    3.4.3 Spectrum of Signals Sampled on a Lattice
    3.4.4 Other Sampling Structures
    3.5 Reconstruction from Samples
    3.5.1 Reconstruction from Rectangular Samples
    3.5.2 Reconstruction from Samples on a Lattice
    3.6 Exercises
   4 SAMPLING STRUCTURE CONVERSION
    4.1 Sampling Rate Change for l-D Signals
    4.1.1 Interpolation of l-D Signals
    4.1.2 Decimation of l-D Signals
    4.1.3 Sampling Rate Change by a Rational Factor
    4.2 Sampling Lattice Conversion
    4.3 Exercises
   5 TWO-DIMENSIONAL MOTION ESTIMATION
    OPTICAL FLOW METHODS
    5.1 2-D Motion vs. Apparent Motion
    5.1.1 2-D Motion
    5.1.2 Correspondence and Optical Flow
    5.2 2-D Motion Estimation
    5.2.1 The Occlusion Problem
    5.2.2 The Aperture Problem
    5.2.3 Two-Dimensional Motion Field Models
    5.3 Methods Using the Optical Flow Equation
    5.3.1 The Optical Flow Equation
    5.3.2 Second-Order Differential Methods
    5.3.3 Block Motion Model
    5.3.4 Horn and Schunck Method
    5.3.5 Estimation of the Gradients
    5.3.6 Adaptive Methods
    5.4 Examples
    5.5 Exercises
   6 BLOCK-BASED METHODS
    6.1 Block-Motion Models
    6.1.1 Translational Block Motion
    6.1.2 Generalized/Deformable Block Motion
    6.2 Phase-Correlation Method
    6.2.1 The Phase-Correlation Function
    6.2.2 Implementation Issues
    6.3 Block-Matching Method
    6.3.1 Matching Criteria
    6.3.2 Search Procedures
    6.4 Hierarchical Motion Estimation
    6.5 Generalized Block-Motion Estimation
    6.5.1 Postprocessing for Improved Motion Compensation
    6.5.2 Deformable Block Matching
    6.6 Examples
    6.7 Exercises
   7 PEL-RECURSIVE METHODS
    7.1 Displaced Frame Difference
    7.2 Gradient-Based Optimization
    7.2.1 Steepest-Descent Method
    7.2.2 Newton-Raphson Method
    7.2.3 Local vs. Global Minima
    7.3 Steepest-Descent-Based Algorithms
    7.3.1 Netravali-Robbins Algorithm
    7.3.2 Walker-Rao Algorithm
    7.3.3 Extension to the Block Motion Model
    7.4 Wiener-Estimation-Based Algorithms
    7.5 Examples
    7.6 Exercises
   8 BAYESIAN METHODS
    8.1 Optimization Methods
    8.1.1 Simulated Annealing
    8.1.2 Iterated Conditional Modes
    8.1.3 Mean Field Annealing
    8.1.4 Highest Confidence First
    8.2 Basics of MAP Motion Estimation
    8.2.1 The Likelihood Model
    8.2.2 The Prior Model
    8.3 MAP Motion Estimation Algorithms
    8.3.1 Formulation with Discontinuity Model,
    8.3.2 Estimation with Local Outlier Rejection
    8.3.3 Estimation with Region Labeling
    8.4 Examples
    8.5 Exercises
    III THREE-DIMENSIONAL MOTION ESTIMATION
    AND SEGMENTATION
   9 METHODS USING POINT CORRESPONDENCES
    9.1 Modeling the Projected Displacement Field
    9.1.1 Orthographic Displacement Field Model
    9.1.2 Perspective Displacement Field Model
    9.2 Methods Based on the Orthographic Model
    9.2.1 Two-Step Iteration Method from Two Views
    9.2.2 An Improved Iterative Method
    9.3 Methods Based on the Perspective Model
    9.3.1 The Epipolar Constraint and Essential Parameters
    9.3.2 Estimation ofthe Essential Pararneters
    9.3.3 Decomposition of the E-Matrix
    9.3.4 Algorithm
    9.4 The Case of 3-D Planar Surfaces
    9.4.1 The Pure Parameters
    9.4.2 Estimation ofthe Pure Parameters
    9.4.3 Estimation ofthe Motion and Structure Parameters
    9.5 Examples
    9.5.1 Numerical Simulations
    9.5.2 Experiments with Two Frames of Miss America
    9.6 Exercises
   10 OPTICAL FLOW AND DIRECT METHODS
    10.1 Modeling the Projected Velocity Field
    10.1.1 Orthographic Velocity Field Model
    10.1.2 Perspective Velocity Field Model
    10.1.3 Perspective Velocity vs. Displacement Models
    10.2 Focus of Expansion
    10.3 Algebraic Methods Using Optical Flow
    10.3.1 Uniqueness of the Solution
    10.3.2 Affine Flow
    10.3.3 Quadratic Flow
    10.3.4 Arbitrary Flow
    10.4 Optimization Methods Using Optical Flow
    10.5 Direct Methods
    10.5.1 Extension ofOptical Flow-Based Methods
    10.5.2 Tsai-Huang Method
    10.6 Examples
    10.6.1 Numerical Simulations
    10.6.2 Experiments with Two Frames of Miss America
    10.7 Exercises
   11 MOTION SEGMENTATION
    11.1 Direct Methods
    11.1.1 Thresholding for Change Detection
    11.1.2 An Algorithm Using Mapping Parameters
    11.1.3 Estimation of Model Parameters
    11.2 Optical Flow Segmentation
    11.2.1 Modified Hough Transform Method
    11.2.2 Segmentation for Layered Video Representation .
    11.2.3 Bayesian Segmentation
    11.3 Simultaneous Estimation and Segmentation
    11.3.1 Motion Field Model
    11.3.2 Problem Formulation
    11.3.3 The Algorithm
    11.3.4 Relationship to Other Algorithms
    11.4 Examples
    11.5 Exercises
   12 STEREO AND MOTION TRACKING
    12.1 Motion and Structure from Stereo
    12.1.1 Still-Frame Stereo Imaging
    12.1.2 3-D Feature Matching fbr Motion Estimation
    12.1.3 Stereo-Motion Fusion
    12.1.4 Extension to Multiple Motion
    12.2 Motion Tracking
    12.2.1 Basic Principles
    12.2.2 2-D Motion Tracking
    12.2.3 3-D Rigid Motion Ttacking
    12.3 Examples
    12.4 Exercises
   13 MOTION COMPENSATED FILTERING
    13.1 Spatio-Temporal Fourier Spectrum
    13.1.1 Global Motion with Constant Velocity
    13.1.2 Global Motion with Acceleration
    13.2 Sub-Nyquist Spatio-Temporal Sampling
    13.2.1 Sampling in the Temporal Direction Only
    13.2.2 Sampling on a Spatio-Temporal Lattice
    13.2.3 Critical Velocities
    13.3 Filtering Along Motion TRajectories
    13.3.1 Arbitrary Motion Trajectories
    13.3.2 Global Motion with Constant Velocity
    13.3.3 Accelerated Motion
    13.4 Applications
    13.4.1 Motion-Compensated Noise Filtering
    13.4.2 Motion-Compensated Reconstruction Filtering
    13.5 Exercises
   14 NOISE FILTERING
    14.1 Intraframe Filtering
    14.1.1 LMMSE Filtering
    14.1.2 Adaptive (Local) LMMSE Filtering
    14.1.3 Directional Filtering
    14.1.4 Median and Weighted Median Filtering
    14.2 Motion-Adaptive Filtering
    14.2.1 Direct Filtering
    14.2.2 Motion-Detection Based Filtering
    14.3 Motion-Compenaated Filtering
    14.3.1 Spatio-Temporal Adaptive LMMSE Filtering
    14.3.2 Adaptive Weighted Averaging Filter
    14.4 Examples
    14.5 Exercises
   15 RESTORATION
    15.1 Modeling
    15.1.1 Shift-Invariant Spatial Blurring
    15.1.2 Shift-Varying Spatial Blurring
    15.2 Intraframe Shift-Invariant Restoration
    15.2.1 Pseudo Inverse Filtering
    15.2.2 Constrained Least Squares and Wiener Filtering
    15.3 Intraframe Shift-Varying Restoration
    15.3.1 Overview ofthe POCS Method
    15.3.2 Restoration Using POCS
    15.4 Multiframe Restoration
    15.4.1 Cross-Correlated Multiframe Filter
    15.4.2 Motion-Compensated Multiframe Filter
    15.5 Examples
    15.6 Exercises
   16 STANDARDS CONVERSION
    16.1 Down-Conversion
    16.1.1 Down-Conversion with Anti-Alias Filtering
    16.1.2 Down-Conversion without Anti-Alias Filtering
    16.2 Practical Up-Conversion Methods
    16.2.1 Intraframe Filtering
    16.2.2 Motion-Adaptive Filtering
    16.3 Motion-Compensated Up-Conversion
    16.3.1 Basic Principles
    16.3.2 Global-Motion-Compensated De-interlacing
    16.4 Examples
    16.5 Exercises
   17 SUPERRESOLUTION
    17.1 Modeling
    17.1.1 Continuous-Discrete Model
    17.1.2 Discrete-Discrete Model
    17.1.3 Problem Interrelations
    17.2 Interpolation-Restoration Methods
    17.2.1 Intraframe Methods
    17.2.2 Multiframe Methods
    17.3 A Frequency Domain Method
    17.4 A Unifying POCS Method
    17.5 Examples
    17.6 Exercises
    V STILL IMAGE COMPRESSION
   18 LOSSLESS COMPRESSION
    18.1 Basics of Image Compression
    18.1.1 Elements of an Image Compression System
    18.1.2 Information Theoretic Concepts
    18.2 Symbol Coding
    18.2.1 Fixed-Length Coding
    18.2.2 Huffman Coding
    18.2.3 Arithmetic Coding
    18.3 Lossless Compression Methods
    18.3.1 Lossless Predictive Coding
    18.3.2 Run-Length Coding of Bit-Planes
    18.3.3 Ziv-Lempel Coding
    18.4 Exercises
   19 DPCM AND TRANSFORM CODING
    19.1 Quantization
    19.1.1 Nonuniform Quantization
    19.1.2 Uniform Quantization
    19.2 Differential Pulse Code Modulation
    19.2.1 Optimal Prediction
    19.2.2 Quantization of the Prediction Error
    19.2.3 Adaptive Quantization
    19.2.4 Delta Modulation
    19.3 Transform Coding
    19.3.1 Discrete Cosine Transform
    19.3.2 Quantization/Bit Allocation
    19.3.3 Coding
    19.3.4 Blocking Artifacts in Transform Coding
    19.4 Exercises
   20 STILL IMAGE COMPRESSION STANDARDS
    20.1 Bilevel Image Compression Standards
    20.1.1 One-Dimensional RLC
    20.1.2 Two-Dimensional RLC
    20.1.3 The JBIG Standard
    20.2 The JPEG Standard
    20.2.1 Baseline Algorithm
    20.2.2 JPEG Progressive
    20.2.3 JPEG Lossless
    20.2.4 JPEG Hierarchical
    20.2.5 ImplementationsofJPEG
    20.3 Exercises
   21 VECTOR QUANTIZATION, SUBBAND CODING
    AND OTHER METHODS
    21.1 Vector Quantization
    21.1.1 Structure of a Vector Quantizer
    21.1.2 VQ Codebook Design
    21.1.3 Practical VQ Implementations
    21.2 Fractal Compression
    21.3 Subband Coding
    21.3.1 Subband Decomposition
    21.3.2 Coding of the Subbands
    21.3.3 Relationship to Transform Coding
    21.3.4 Relationship to Wavelet Transform Coding
    21.4 Second-Generation Coding Methods
    21.5 Exercises
    VI VIDEO COMPRESSION
   22 INTERFRAME COMPRESSION METHODS
    22.1 Three-Dimensional Waveform Coding
    22.1.1 3-D Transform Coding
    22.1.2 3-D Subband Coding
    22.2 Motion-Compensated Waveform Coding
    22.2.1 MC Transform Coding
    22.2.2 MC Vector Quantization
    22.2.3 MC Subband Coding
    22.3 Model-Based Coding
    22.3.1 Object-Based Coding
    22.3.2 Knowledge-Based and Semantic Coding
    22.4 Exerclses
   23 VIDEO COMPRESSION STANDARDS
    23.1 The H.261 Standard
    23.1.1 Input Image Formats
    23.1.2 Video Multiplex
    23.1.3 Video Compression Algorithm
    23.2 The MPEG-l Standard
    23.2.1 Features
    23.2.2 Input Video Format
    23.2.3 Data Structure and Compression Modes
    23.2.4 Intraframe Compression Mode
    23.2.5 Interframe Compression Modes
    23.2.6 MPEG-l Encoder and Decoder
    23.3 The MPEG-2 Standard
    23.3.1 MPEG-2 Macroblocks
    23.3.2 Coding Interlaced Video
    23.3.3 Scalable Extensions
    23.3.4 Other Improvements
    23.3.5 Overview of Profiles and Levels
    23.4 Software and Hardware Implementations
   24 MODEL-BASED CODING
    24.1 General Object-Based Methods
    24.1.1 2-D/3-D Rigid Objects with 3-DMotion
    24.1.2 2-D Flexible Objects with 2-D Motion
    24.1.3 Affine Transformations with TRiangular Meshes
    24.2 Knowledge-Based and Semantic Methods
    24.2.1 General Principles
    24.2.2 MBASIC Algorithm
    24.2.3 Estimation Using a Flexible Wireframe Model
    24.3 Examples
   25 DIGITAL VIDEO SYSTEMS
    25.1 Videoconferencing
    25.2 Interactive Video and Multimedia
    25.3 Digital Television
    25.3.1 Oigital Studio Standards
    25.3.2 Hybrid Advanced TV Systems
    25.3.3 All-Oigital TV
    25.4 Low-Bitrate Video and Videophone
    25.4.1 The ITU Recommendation H.263
    25.4.2 The ISO MPEG-4 Requirements
    APPENDICES
   A MARKOV AND GIBBS RANDOM FIELDS
    A.l Definitions
    A.l.l Markov Random Fields
    A.1.2 Gibbs Random Fields
    A.2 Equivalence of MRF and GRF
    A.3Local Conditional Probabilities
   B BASICS OF SEGMENTATION
    B.l Thresholding
    B.I.l Finding the Optimum Threshold(s)
    B.2 Clustering
    B.3 Bayesian Methods
    B.3.1 The MAP Method
    B.3.2 The Adaptive MAP Method
    B.3.3 Vector Field Segmentation
   C KALMAN FILTEMNG
    C.l Linear State-Space Model
    C.2 Extended Kalman Filtering