The Latency of Video Codecs – How Latency happens and How to Solve It?

In the process of audio and video transmission, latency/delay is often considered the most important factor. Unclear images and delay in video will directly affect the quality of video images and user experience.

Delay in video codec happens when the output image with compression, transmission, and decoding has a certain time lag with the front-end, which is also called end-to-end latency/delay. For example, in real-time on-site command and dispatch, the director guides cameraman on shooting based on the returned video. If there is too much latency/delay, the shooting will be affected. Another example is to remotely control the movement of the camera. The latency/delay in transmission will made a slow rotation of the camera. Assuming that the video delay is 1 second, it means that it takes 1 second to start the camera rotation before you can see the picture on the monitor, and the picture will continue to move for 1 second after you stop the operation – it is the picture you see. So imagine how delay has a huge impact on video shooting!

Look into the process of video transmission, there are three parts usually cause the delay – latency/delay on encoding side, delay on transmission side and delay on decoding side.

Latency/Delay on the encoding end

The encoder needs to collect and buffer two original images (two frames) for compression processing, that is to say, after the video is input, it must wait for two frames before the encoded data is output, so the encoder has a certain latency/delay of about 67ms.

Latency/Delay on transmission end

There are many kinds of transmission links, such as wired network, WIFI, 4G, satellite, etc. When video data is transmitted via the wired network, there are small areas where it buffers the data when it is forwarded through switches and routers, – the delay happens. However, this Latency/delay is not much compared to the front-end encoding and back-end decoding, usually we can consider it to be within 5ms. The Latency/delay of the external network, that is, the cross-regional Internet/WAN, is more complicated, and the latency/delay depends on your network.

Latency/Delay at the decoding end

The latency/delay time at the decoding end contributes mostly to the entire end-to-end delay, and has the greatest impact on the fluency and integrity of the back-end playback, which we must lay focus on greatly.

The normal and stable output video picture of the decoding system requires that the video data transmitted by the network be complete, uniform and continuous. Otherwise the display output picture will have jams (dropped frames) and blurred screens (Mosaic), or even disconnection, but the transmission characteristics of the computer network and the processing mechanism of the front-end and back-end software system lead to the existence of packet loss and the inconstant sending rate (jitter).

How do we solve this problem?

Increasing the buffer is one of the most effective ways. We add a buffer area between the data receiving port and the back-end decoding display to ensure that the received data is a continuous and complete piece of data – if any loss is found, it immediately notifies a retransmission. Evenly send it to the decoding end, which solves a problem, but also brings the negative impact of increased delay. The larger the buffer, the stronger the ability to resist network jitter and packet loss. Therefore, how to achieve a smoother image transmission effect under a smaller buffer is the optimization direction of various codec manufacturers.

Kiloview’s video decoding buffer is about 150ms, which can be adjusted by the user according to the application.

Kiloview’s video transmission solution is of buffer about 67ms + 5ms + 150ms, namely 220ms, which is an end-to-end delay of the video transmission system composed of the conventional H.264 encoder and decoder.

The latency/delay of NDI series products – the end-to-end delay of NDI|HX codec is about 150ms; the NDI High Bandwidth series is almost lossless compression (so the bandwidth is required), and the end-to-end delay of NDI High Bandwidth is about 80ms.