Smooth-Foley

Creating Continuous Sound for Video-to-Audio Generation Under Semantic Guidance

Paper
Code · Coming soon

Yaoyun Zhang^1,2, Xuenan Xu³, Mengyue Wu^1,2

¹X-LANCE Lab, Department of Computer Science and Engineering ²Shanghai Jiao Tong University ³Shanghai Artificial Intelligence Lab

yaoyunzhang897@gmail.com

Abstract.

The video-to-audio (V2A) generation task has drawn attention in the field of multimedia due to the practicality in producing Foley sound. Semantic and temporal conditions are fed to the generation model to indicate sound events and temporal occurrence. Recent studies on synthesizing immersive and synchronized audio are faced with challenges on videos with moving visual presence. The temporal condition is not accurate enough while low-resolution semantic condition exacerbates the problem. To tackle these challenges, we propose Smooth-Foley, a V2A generative model taking semantic guidance from the textual label across the generation to enhance both semantic and temporal alignment in audio. Two adapters are trained to leverage pre-trained text-to-audio generation models. A frame adapter integrates high-resolution frame-wise video features while a temporal adapter integrates temporal conditions obtained from similarities of visual frames and textual labels. The incorporation of semantic guidance from textual labels achieves precise audio-video alignment. We conduct extensive quantitative and qualitative experiments. Results show that Smooth-Foley performs better than existing models on both continuous sound scenarios and general scenarios. With semantic guidance, the audio generated by Smooth-Foley exhibits higher quality and better adherence to physical laws.

Method

Smooth-Foley combines frame-wise semantic guidance with a temporal adapter for continuous and synchronized sound generation.

Smooth-Foley pipeline showing video labels and frames feeding frame and temporal adapters before audio generation — The overall pipeline of Smooth-Foley.

Dataset Construction

VGGSound-Continuous is constructed through label-level semantic screening and two clip-level text-grounding passes. Invalid clips are removed before the final continuous/non-continuous split.

Swipe or scroll horizontally to inspect the full pipeline.

VGGSound-Continuous filtering pipeline: GPT-4 screens labels, the first text-grounding pass removes invalid clips, the second pass accumulates regions with similarity at least 0.5, and clips with at least three matched seconds form the continuous subset — Construction of VGGSound-Continuous. The two branches first identify candidate labels and build a quality-controlled clip set; temporal grounding then determines the final subset.

Filtering procedure

Label-level screening. GPT-4 reviews all VGGSound class labels and selects labels that may describe continuous sounds, producing the candidate set L_cont.
Grounding pass 1 — validity cleaning. The text-to-audio grounding model removes silent clips and clips whose audio does not match the class label. These invalid clips are discarded; the remaining quality-controlled clips form V_valid.
Valid candidate pool. Clips in V_valid whose labels belong to L_cont are retained for temporal verification.
Grounding pass 2 — temporal verification. For each candidate clip, the model produces an audio–text similarity sequence s(t). Regions satisfying s(t) ≥ 0.5 are accumulated over time.
Final split. A clip is assigned to VGGSound-Continuous when its accumulated matched duration is at least 3 seconds. Non-Continuous is defined as the complement of VGGSound-Continuous within V_valid, so rejected invalid clips are not included in either split.