JEG Reproducible Video Quality Analysis Framework

From VQEG JEG Wiki
Revision as of 08:43, 2 June 2017 by Gvwallen (Talk | contribs)

Jump to: navigation, search


Main idea

The work on this page presents a framework to facilitate reproducibility of research in video quality evaluation. Its initial version is built around the JEG-Hybrid database of HEVC coded video sequences. The framework is modular, organized in the form of pipelined activities, which range from the tools needed to generate the whole database from reference signals up to the analysis of the video quality measures already present in the database. Researchers can rerun, modify and extend any module, starting from any point in the pipeline, while always achieving perfect reproducibility of the results. The modularity of the structure allows to work on subsets of the database since for some analysis this might be too computationally intensive. To this purpose, the framework also includes a software module to compute interesting subsets, in terms of coding conditions, of the whole database.

Software Architecture

The software consists of a pipeline architecture where each active component communicates to the other component using directories (see Fig. 1). Each individual component will be explained next.


Large scale encoding environment

The large scale encoding environment performs the Hypothetical Reference Circuit (HRC) processing. At this moment, this module consists of the HEVC standardization reference encoding package ( accompanied by a valuable set of configuration files and scripts in order to reproduce or extend the first version of the HEVC large scale database. Whenever the proposed video quality analysis framework needs improvement with more compression standards, other compression parameters, or network impairment simulations, then solely this block must be extended upon.

The correct creation or download of the large-scale database can be verified by SHA512 checksums available at: (

Subset selection

This process facilitates the research work if a proper HRCs are selected to represent a large-scale database. Therefore, this component consists of two algorithms for subset selection. The first one is optimized to cover different ranges of quality and bitrate. The second algorithm is optimized for HRCs that behave differently with different source contents. The two algorithms are demonstrated in the accompanying DSP paper.

Quality measure

The quality measurement component consists of a collection of full reference quality measurements like Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Multi-Scale Structural Similarity (MS-SSIM), and Visual Information Fidelity (VIFp). These measurements are integrated in this Reproducible Video Quality Analysis software package using the Video Quality Measurement Tool (VQMT) from Ecole Polytechnique Fédérale de Lausanne (EPFL).

Quality measure analysis

The quality measure analysis focuses on extracting the relevant data from the full reference quality measurement database and process them in order to perform the analysis. In this particular work we extracted frame-level MSE and PSNR values to compare the effect of temporal pooling through averaging either MSE or PSNR. Moreover, the variance of the frame-level PSNR is also computed and made available for the next visualization block. The module can be easily customized to handle different measures either present in the database or made available through files in the same format. Moreover, other indicators (e.g., moving averages etc., can be included in the output for the visualization block.


The visualization block is currently a set of gnuplot command files that can be used to easily visualize the data produced in the previous step. In particular, they can automatically generate scatter plots and, with the aid of some custom-developed external modules, interpolation parameters for better visualization.

Running the software

The package contains scripts that are written in different language environments but can be executed under Linux and Windows platforms.

The software package contains 2 directories `DATA' and `SoftwareLibraries'. `DATA' directory contains sub directories that contains the source data and the outputs of running the softwares that can be found in `SoftwareLibraries'.

Large Scale Encoding Environment