Accelerating Diffusion Model via Inherent Characteristic of Diffusion Process

Sungbin Kim - eSCaL, EE, Yonsei University

This presentation provides an overview of the characteristics of diffusion models and introduces acceleration techniques and accelerator architectures that leverage their inherent properties. We analyze the iterative structure of diffusion models and observe that adjacent time steps exhibit strong value similarity, resulting in small differences between consecutive steps. By applying this observation to quantized diffusion models, we show that most of these differences can be represented using reduced bit-width—or even zero.

Building on these insights, we propose Ditto, a temporal difference–based processing algorithm that exploits temporal similarity under quantization to improve the efficiency of diffusion model execution. Ditto performs full–bit width computation only at the initial time step and processes subsequent steps using temporal differences through the distributive property of layer operations. Furthermore, we introduce an execution–flow optimization that reduces the memory overhead associated with temporal difference processing, further enhancing overall efficiency. Finally, we present Ditto Hardware, a specialized accelerator designed to fully exploit the dynamic characteristics of the proposed algorithm.

PPT CV

Catering Courtesy of ASO Lab
Invited Talk