Learning to Read Residual Images
for Single-Pixel Photography

Hongjun An1 Shuaizhe An1 Tong Tian3,4 Zhe Sun1,2,∗

1School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University 2Institute of Artificial Intelligence (TeleAI), China Telecom 3Institute of Optics and Quantum Electronics, Abbe Center of Photonics, Friedrich Schiller University Jena 4Helmholtz Institute Jena

∗Corresponding author

Ground truthMGV-Flow (ours)History-FMFlowDPS-inspired

Video reconstructed from 1,000 bucket values per frame, 0.24% of the 856×480 pixels. Every five-frame block is read from its own measurements with K = 5 Euler steps, and on this validation segment MGV-Flow reaches 22.41 dB against 21.85 dB for History-FM and 19.26 dB for FlowDPS-inspired.

0.24%of the pixels measured per frame
256×faster than per-clip inversion
+4.8 dBmedian PSNR over per-clip inversion
−33%bucket values with directional fidelity

Abstract

Continuous video scales in cost with camera count, resolution and recording duration, yet many monitoring tasks seek events rather than complete images and can operate on a compact single-pixel measurement stream. Recovering a chosen interval as video is then desirable, but at 0.24% sampling the scene correlation image looks like noise and per-clip optimization takes minutes. Correlating the residual of an estimate with the coding patterns nevertheless gives a residual image, the gradient of a centered measurement loss. Lifted through a pretrained video decoder, its agreement with the true error exceeds that of control decoders. Video-latent inversion (VLI) then outperforms frame-wise image-latent reconstruction at 1,000 bucket values per frame. Measurement-Gradient-Conditioned Video Flow (MGV-Flow), trained once to read the lifted residual image with preceding frames, reconstructs five frames in 1.55 s at a median peak signal-to-noise ratio of 22.9 dB, against 398 s and 18.1 dB for VLI, with the largest gains in high-motion scenes. Directional fidelity ranks 30 coding configurations and selects one needing a third fewer bucket values. On a physical system, MGV-Flow outperforms per-frame reconstruction without retraining. A sliding window over the bucket-value stream yields 3.2 frames per second on one RTX 4090 and, in theory, 20 frames per second with a 0.23 s delay on one data-centre accelerator. The correlation that founded single-pixel imaging thus becomes feedback a generative model learns to read.

How it works

Single-pixel recording with a digital micromirror device and one detector
A digital micromirror device displays binary coding patterns and a single-pixel detector records one bucket value per pattern. At 1,000 bucket values per frame, traditional correlation imaging returns noise.
Reading residual images with MGV-Flow
The residual between predicted and acquired bucket values, correlated with the coding patterns, is the residual image. The transposed Jacobian of the pretrained video decoder lifts it into the latent space, and MGV-Flow reads it with the history frames for five Euler steps.

Rolling single-pixel photography

Rows show the full frames with per-frame PSNR, a zoom on moving content, and the absolute error. Every block of five frames is reconstructed from its own 1,000 bucket values per frame with the five preceding frames as history.

Mean PSNR over the segmentMGV-FlowHistory-FMFlowDPS-inspired
Red kayak I22.29 dB20.71 dB18.59 dB
Red kayak II19.18 dB17.86 dB17.66 dB
Dinner I22.41 dB21.85 dB19.26 dB
Dinner II22.28 dB21.83 dB19.29 dB
Blue sky17.13 dB16.75 dB16.38 dB

History-FM is the same flow model conditioned on the preceding frames alone. FlowDPS-inspired adds the measurement gradient to its velocity with a fixed weight. Red kayak clips are from the training split, dinner clips from the validation split and blue sky from the test split.

Results

1

Lifting residual images into a video latent space makes inversion scalable

Through the pretrained video decoder, the noise-like residual image becomes a structured correction that agrees with the true error better than through re-initialized or weight-shuffled decoders. Video-latent inversion (VLI), which reconstructs five frames jointly in this latent space, outperforms pixel networks and frame-wise image-latent reconstruction (GILM) when bucket values are scarce.

At 1,000 bucket values per frame, averaged over four resolutions and five test videos, VLI reaches 18.02 dB against 16.44 dB for GILM and 11.59 dB for a U-Net. At 856×480 with 10,000 bucket values per frame, joint inversion needs 56.3% less computation and 72.8% less peak GPU memory than GILM.

18.02 dBVLI at 1,000 bucket values per frame
−56.3%computation versus frame-wise inversion
Residual image, decoder response and true error

Figure 1. A noise-like residual image becomes a structured video correction through the pretrained decoder.

a The residual between predicted and acquired bucket values, correlated with the coding patterns, gives the residual image gx. The transposed Jacobian of the LeanVAE decoder lifts it into the latent space as gz.

b Ground truth, scene correlation image of traditional correlation imaging (TCI), residual image, decoder response and true error for windows of highest, median and lowest motion. Corr is the Pearson correlation with the true error.

c Band coherence with the true error for the residual image and for the decoder responses of pretrained, re-initialized and weight-shuffled decoders.

d Directional fidelity along a ten-step trajectory, the cosine between the lifted residual image and the true latent correction.

Measurement, spatial and computational scaling

Figure 2. Measurement, spatial and computational scaling across reconstruction spaces.

a PSNR versus M, the bucket values per frame, for TCI, a convolutional network (CNN), a U-Net, frame-wise image-latent reconstruction (GILM) and video-latent inversion (VLI).

b PSNR advantage of VLI over the U-Net and over GILM across M and resolution.

c Computation per frame and peak GPU memory of GILM and VLI, with the reduction relative to GILM.

d Reconstructions of all methods at two values of M.

2

Reading instead of consuming residual images enables amortized video reconstruction

MGV-Flow is trained once to read the lifted residual image together with the history frames. A five-frame clip takes 1.55 s at K = 5 instead of 397.99 s with VLI, a median speedup of 256×, while median PSNR rises from 18.12 to 22.92 dB. VLI fits the measurements far more closely, yet in none of 21 cases does it reach the PSNR of MGV-Flow, because at 0.24% sampling many videos fit the measurements almost equally well.

Setting the measurement inputs to zero lowers PSNR by 1.57 dB and inputs from another video lower it by 0.68 dB, so the model reads the measured scene. The benefit of reading grows with motion (Spearman 0.738), where the history predicts the target least.

256×faster than per-clip inversion
+4.8 dBmedian PSNR over VLI
Reading versus consuming the residual image

Figure 3. Reading versus consuming the residual image.

a MGV-Flow predicts its velocity by reading the history, the normalized physical direction and a quality descriptor. History-FM omits the measurement inputs, FlowDPS-inspired adds the physical direction to the velocity with a fixed weight, and VLI optimizes a network anew for every clip.

b Latency per five-frame clip and PSNR, with the speedup over VLI.

c Measurement NMSE and PSNR during VLI optimization against MGV-Flow at K = 5. VLI fits the measurements more closely but stays below in PSNR.

d PSNR gain of the correct measurements over zeroed and shuffled measurement inputs across Euler steps K.

Benefit of reading versus motion

Figure 4. Reading the residual image pays off where the history cannot predict the target.

a PSNR difference of MGV-Flow versus History-FM against the motion score, the copy error from the last history frame to the five target frames.

b Mean PSNR difference versus History-FM in the high, middle and low motion groups across Euler steps K.

c The same versus FlowDPS-inspired. Bands are 95% bootstrap intervals over 35 validation windows with three seeds each.

3

Directional fidelity of residual images reveals coding-pattern utility

Directional fidelity measures whether the lifted residual image points toward the true latent correction, at the cost of one residual image and one decoder vector-Jacobian product and without running a reconstruction. Across 30 configurations of pattern family, grain size and measurement count, it predicts reconstruction quality with a Spearman correlation of 0.779 (95% confidence interval 0.644 to 0.862).

Ranked on calibration windows, the top configuration, a Walsh basis at 16 px, matches the median-ranked configuration at 3,000 bucket values per frame with only 2,000 and exceeds it in all eight held-out windows. The best pattern also depends on the video autoencoder.

0.779Spearman correlation with quality
−33%bucket values for the same quality
Residual images of three coding configurations

Figure 5. Residual images of three coding configurations at 3,000 bucket values per frame on one held-out window.

Rows are a Walsh basis at 16 px, phase-speckle patterns at 32 px and a low-frequency Fourier basis at 4 px. Columns show a coding pattern, the residual image, the decoder response, the decoded latent correction and the reconstruction after ten physics-only steps. C is the directional fidelity and Pearson the correlation between decoder response and decoded latent correction. The pattern whose feedback points best also reconstructs best.

Directional fidelity reveals coding-pattern utility

Figure 6. Directional fidelity reveals coding-pattern utility.

a Directional fidelity C is the cosine between the negative lifted residual image and the true latent correction at a probe latent.

b Reconstruction PSNR versus mean C over 30 configurations, with the Spearman correlation and its 95% confidence interval.

c Mean C versus M for five pattern families with LeanVAE.

d Mean C of the families with four video autoencoders, LeanVAE, VidTok, Wan and HunyuanVideo.

e PSNR versus M for the selected Walsh basis and the median-ranked phase speckle. The Walsh basis matches the comparator at 2,000 bucket values per frame.

4

On physical measurements, reading transfers without retraining

On bucket values acquired by a physical single-pixel system with structured illumination, MGV-Flow, neither retrained nor fine-tuned, predicts frames 6 to 10 of two animated film excerpts from 10,000 measured bucket values per frame and five ground-truth history frames. It reaches an equalized PSNR of 10.25 dB against 8.63 dB for GILM, the best per-frame method, and produces five frames in 7.85 s while GILM needs 44.13 s for each frame.

Bucket values of the other clip lower PSNR in all six runs, and the measurement residual falls to 0.54 to 0.61 of its initial value against 0.65 to 0.69 for the other clip, so the model reads the physical measurements.

10.25 dBequalized PSNR against 8.63 dB
7.85 sfor five frames on hardware data
Physical single-pixel photography system, top view

Figure 7. Physical single-pixel photography system, top view.

A laser beam passes two lenses and a beam expander and is folded by a mirror onto a digital micromirror device (DMD), which displays the binary coding patterns. Two lenses project the patterns onto the scene, and a fifth lens collects the light onto the single-pixel detector, which records one bucket value per pattern. The red line marks the beam path.

Physical single-pixel reconstruction

Figure 8. Physical single-pixel reconstruction with 10,000 measured bucket values per frame.

a For two clips, the last ground-truth history frame, the ground-truth frames 6 to 10 and the frames MGV-Flow predicts from the measured bucket values with K = 5.

b Equalized PSNR of each frame for MGV-Flow and for per-frame reconstruction by a CNN, a U-Net and GILM.

c RMS of the measurement residual along the Euler steps, normalized by its first value, with the measured bucket values and with those of the other clip.

Imaging rate and compute

Output frame rate versus compute

The detector delivers bucket values as a continuous stream, so reconstruction runs as a sliding window over the most recent measurements, and the reconstruction delay is the latency of one five-frame block.

HardwareK = 5Delay
RTX 4090 measured3.2 fps1.55 s
RTX 3090 scaled2.0 fps2.51 s
RTX 5090 theoretical6.4 fps0.78 s
H100 theoretical10.0 fps0.50 s
B200 theoretical21.9 fps0.23 s

Code

Code is available at github.com/cavedweller509/MGV-Flow, including the single-pixel operator, residual and lifted residual images, Euler sampling, directional fidelity and the sliding-window camera loop.

git clone https://github.com/cavedweller509/MGV-Flow.git
cd MGV-Flow && pip install -e ".[test]" && pytest

Citation

@article{sun2026mgvflow,
  title   = {Learning to Read Residual Images for Single-Pixel Photography},
  author  = {An, Hongjun and An, Shuaizhe and Tian, Tong and Sun, Zhe},
  year    = {2026}
}