Modern visual effects (VFX) software has made it possible for skilled artists to create imagery of virtually anything. However, the creation process remains laborious, complex, and largely inaccessible to everyday users. In this work, we present AutoVFX, a framework that automatically creates realistic and dynamic VFX videos from a single video and natural language instructions. By carefully integrating neural scene modeling, LLM-based code generation, and physical simulation, AutoVFX is able to provide physically-grounded, photorealistic editing effects that can be controlled directly using natural language instructions. We conduct extensive experiments to validate AutoVFX's efficacy across a diverse spectrum of videos and instructions. Quantitative and qualitative results suggest that AutoVFX outperforms all competing methods by a large margin in generative quality, instruction alignment, editing versatility, and physical plausibility.

We compare our method with three baseline methods, including Instruct2Nerf, DGE, and FRESCO.

We compare our method (right) with ChatSim (middle) for autonomous driving scene simulation. The original input image is shown on the left.

We compare our method (right) with the visual effect, Pikaffect, created by Pika1.5 (middle). The original input image for Pika1.5 is shown on the left. Our method generates more localized and physically accurate visual effects.

Our instruction-guided video editing framework consists of three main modules: (1) 3D Scene Modeling (left), which integrates 3D reconstruction and scene understanding models; (2) Program Generation (middle), where LLMs generate editing programs based on user instructions; and (3) VFX Modules (right), which include predefined functions specialized for various editing tasks. These components are integrated with a physically-based simulation and rendering engine (e.g., Blender) to generate the final video.