ML-PEA: Machine Learning-Based Perceptual Algorithms for Display Power Optimization

_{Our pipeline generates images which consume less power than the original when shown on a display, while minimizing perceptual impact. Here, we show an example of an image generated with our technique compared to the reference and its uniformly dimmed version. The corresponding dimming maps are shown in the insets, with the multiplicative scaling factor presented in the color bar on the right. Note that both the uniformly dimmed image and the image generated with our technique in this figure consume the same amount of display power: 52.1% of the reference.}

Image processing techniques can be used to modulate the pixel intensities of an image to reduce the power consumption of the display device. A simple example of this consists of uniformly dimming the entire image. Such algorithms should strive to minimize the impact on image quality while maximizing power savings. Techniques based on heuristics or human perception have been proposed, both for traditional flat panel displays and modern display modalities such as virtual and augmented reality (VR/AR). In this paper, we focus on developing and evaluating display power-saving techniques that use machine learning (ML). This pipeline was validated via quantitative analysis using metrics and through a subjective study. Our results show that participants prefer our technique over a uniform dimming baseline for high target power saving conditions. In the future, this work should serve as a template and baseline for future applications of deep learning for display power optimization.

To train a model, run the following command:

python train.py --w_vgg 0.5 --w_ssim 5 --w_power 50 --method MULT --dataset div2k

Configs:

• w_vgg: weight on the VGG loss
  • parameter value used in paper: 0.5
• w_ssim: weight on the SSIM loss
  • parameter value used in paper: 5
• w_power: weight on the power optimization loss
  • parameter value used in paper: 50
• method: dimming map modulation method [MULT, ADD]
  • parameter value used in paper: MULT (I_new = I * dimming_map)
• dataset: training dataset

Place dataset images in [dataset]/train/*.png and [dataset]/test/*.png.

To test a model, run the test.py function after training with the relevant parameters: python test.py --w_vgg 0.5 --w_ssim 5 --w_power 50 --method MULT --dataset div2k This will save images generated by our method automatically, as well as compute metrics saved as a .csv.

If you want to use the saved checkpoints from our paper, you can run the python test_single.py function on your own images.

Saved checkpoints are found at our Google Drive link.

Thank you to Tina Su for help running experiments, and to the user study participants for their time. This research is partially supported by National Science Foundation award #2225861.

We also thank the authors of the Pytorch-UNet repository for their UNet implementation.

Contact Kenneth Chen (kennychen@nyu.edu) with any questions about the codebase.

PEA-PODs: Perceptual Evaluation of Algorithms for Power Optimization in XR Displays, SIGGRAPH 2024. Kenneth Chen, Thomas Wan, Nathan Matsuda, Ajit Ninan, Alexandre Chapiro†, Qi Sun†.

Color-Perception-Guided Display Power Reduction for Virtual Reality, SIGGRAPH Asia 2022. Budmonde Duinkharjav*, Kenneth Chen*, Abhishek Tyagi, Jiayi He, Yuhao Zhu†, Qi Sun†.