Fast rate-distortion optimized mode decision of H.264/AVC video coding standard

Abstract

Video compression plays important rule in communications and multimedia applications. The appearance and development of various new multimedia services have need for higher coding efficiency. H.264/AVC is a newest international video coding standard that can achieve considerably highest coding efficiency than previous standards. To achieve the highest coding efficiency, H.264/AVC uses rate-distortion optimized mode selection technique. This means that the encoder has to code the video by exhaustively trying all the mode combinations including the different intra and inter prediction modes. However, this mode selection process also makes the encoding process extremely complex, especially in the computation of the rate-distortion cost function, which includes the computations of the sum of squared difference (SSD) between the original and reconstructed image blocks and context-based entropy coding of the block. Therefore, the complexity and computation load of video coding in H.264/AVC increase drastically compared to any previous standards. To reduce the complexity of rate-distortion cost computation, this thesis proposes a fast bit rate estimation technique to avoid the entropy coding method during intra and inter mode decision of H.264/AVC. The estimation method is based on the properties of context-based variable length coding (CAVLC). The proposed rate model predicts the rate of a 4× 4 quantized residual block using five different tokens of CAVLC. Additionally, this thesis also proposes a look-up table based rate estimator which is very efficient in terms of quality and bit rate. This algorithm can also be used with efficient distortion estimation algorithms to further reduce the complexity of cost function. For a macroblock in I-slice, rate-distortion optimization exhaustively searches the predefined 13 intra modes (9 modes for 4x4 block and 4 modes for 16x16 block) to produce the best encode mode for this macroblock. To reduce the complexity of 4x4 intra mode decision, this thesis proposes an efficient and fast 4x4 intra prediction mode selection scheme. The proposed method reduces the candidate of the prediction modes based on the Sum of Absolute Hadamard-Transformed Difference (SATD) between the original block and the intra predicted block. Rank of each mode is obtained based on the SATD value. The candidate modes are further reduced by using the combination of rank and most probable mode. The proposed method reduces the number of candidate mode to either one or two.