X→RGB Comparison — Compared to X→RGB from RGB↔X, our method achieves more realistic lighting with high-detail illumination while maintaining temporal consistency across frames over long sequences. X→RGB produces images that appear artificially flat with uniform lighting, lacking the rich lighting variation, shadow depth, and atmospheric effects present in photorealistic rendering.

DiffusionRenderer Comparison — Unlike DiffusionRenderer, which requires complete sequences upfront and is limited to 24 frames, our autoregressive approach enables frame-by-frame generation for interactive applications. On comparable 24-frame sequences, FrameDiffuser maintains superior temporal coherence and produces more nuanced illumination with realistic shadows and atmospheric depth, while DiffusionRenderer's batch processing results in less natural lighting transitions.

Scene Editing — When objects are added to the scene through G-buffer modifications, FrameDiffuser automatically synthesizes appropriate lighting, shading, and cast shadows. This enables artists to maintain full control over scene composition while FrameDiffuser handles the computationally expensive lighting synthesis automatically.

VAE Flicker — The VAE of Stable Diffusion 1.5 introduces temporal flicker, especially in scenes with high spatial frequencies like metropolitan cityscapes. When comparing to pure VAE reconstruction, this flicker becomes apparent in generated sequences. We believe that with a different VAE—such as finetuned VAE decoders or VAEs incorporating previous frame context—this flicker could be substantially reduced or eliminated. However, training or fine-tuning a specialized VAE requires significant computational resources and falls outside the scope of this work, which focuses on the frame generation pipeline itself.

GBufferDiffuser: Inverse Rendering — While forward rendering (G-buffer to RGB) is the main focus of this work, we also developed GBufferDiffuser for inverse rendering (RGB to G-buffer), similar to RGB↔X. This system employs five independent ControlLoRA models, each specialized for reconstructing a specific G-buffer component: basecolor, depth, normals, roughness, and metallic. In contrast to generalist approaches that train a single model across multiple G-buffer types and environments, we train smaller specialized adapters for each component on a single environment. This specialization approach allows for faster training while achieving superior results within the target domain, substantially outperforming generalist baselines across all G-buffer components.

Style Transfer — We explored artistic control through style transfer by applying first-frame augmentation. Given an original rendered frame, we apply Stable Diffusion's image-to-image transformation with a style-specific text prompt to create a stylized version. This stylized frame then serves as the previous frame input for the first generated frame, and generation continues autoregressively. The experiments revealed limited style transfer capabilities due to the model's specialization on training environments with fixed prompts. After training on a single prompt-environment combination, the model's ability to respond to alternative text conditioning decreased.