Faster Video Super-Resolution System

Abstract

Deep neural network (DNN) has shown its superior performance on image super-resolution (SR) task in the past few years. The SR technique can be applied to content delivery network to achieve a lower bandwidth requirement for high-resolution videos. Low-resolution videos will be sent to the clients and processed locally using DNN to generate corresponding high-resolution videos. The video will be decoded into frames, then DNN will be applied to them to produce frames with higher quality. In this way, the clients will get access to high-resolution videos. However, DNN is computationally expensive compared with other SR approaches, like bicubic interpolation. That makes it hard for some client devices to achieve real-time video super-resolution (usually 30 fps). To achieve both good visual quality and fast inference speed, we have managed to utilize the temporal correlation between adjacent frames. In most cases, adjacent frames have high similarity. And one frame can be predicted by transferring blocks of pixels from other frames. So instead of doing SR frame by frame, we only need to apply DNN to a subset of frames in each group of picture (GOP) and transfer the SR result to the other frames. What makes the transferring possible is High Efficiency Video Coding (HEVC), one of the latest video encoding protocol. According to motion vectors provided by HEVC encoder, we will know which pixel blocks in the previous frame can be copied to the current frame. Moreover, we can manually choose the transfer ratio such that we can have a flexible trade-off between visual quality and inference speed. Based on the above theory, we have implemented a faster video super-resolution system that provides a full set of functionalities including training, testing and video playback. The fast mode offered by the system can accelerate the SR process by at most three times with negligible visual quality loss. As a result, the final system provides a more practical solution for real-time high-quality video super-resolution with modest computation power requirement.