We present Style Matching Score (SMS), a novel optimization method for image stylization with diffusion models. Balancing effective style transfer with content preservation is a long-standing challenge.
Unlike existing efforts, our method reframes image stylization as a style distribution matching problem. The target style distribution is estimated from off-the-shelf style-dependent LoRAs via carefully designed score functions. To preserve content information adaptively, we propose Progressive Spectrum Regularization, which operates in the frequency domain to guide stylization progressively from low-frequency layouts to high-frequency details. In addition, we devise a Semantic-Aware Gradient Refinement technique that leverages relevance maps derived from diffusion semantic priors to selectively stylize semantically important regions.
The proposed optimization formulation extends stylization from pixel space to parameter space, readily applicable to lightweight feedforward generators for efficient one-step stylization. SMS effectively balances style alignment and content preservation, outperforming state-of-the-art approaches, verified by extensive experiments.
Our key idea is to match the distribution of the output images with the target style distribution by leveraging powerful diffusion priors. We minimize the Kullback–Leibler (KL) divergence between the distribution of the stylized images and the target style distribution, estimated by score functions of a style LoRA-integrated pretrained diffusion model. By interpreting the denoised gradient directions that make the image more stylized, we effectively align generated images with the target style.
@misc{jiang2025balancedimagestylizationstyle,
title={Balanced Image Stylization with Style Matching Score},
author={Yuxin Jiang and Liming Jiang and Shuai Yang and Jia-Wei Liu and Ivor Tsang and Mike Zheng Shou},
year={2025},
eprint={2503.07601},
archivePrefix={arXiv},
primaryClass={cs.CV},
}