guoyww/AnimateDiff

AnimateDiff aims to learn transferable motion priors that can be applied to other variants of Stable Diffusion family. To this end, we design the following training pipeline consisting of three stages.

• In 1. Alleviate Negative Effects stage, we train the domain adapter, e.g., v3_sd15_adapter.ckpt, to fit defective visual aritfacts (e.g., watermarks) in the training dataset. This can also benefit the distangled learning of motion and spatial appearance. By default, the adapter can be removed at inference. It can also be integrated into the model and its effects can be adjusted by a lora scaler.

• In 2. Learn Motion Priors stage, we train the motion module, e.g., v3_sd15_mm.ckpt, to learn the real-world motion patterns from videos.

• In 3. (optional) Adapt to New Patterns stage, we train MotionLoRA, e.g., v2_lora_ZoomIn.ckpt, to efficiently adapt motion module for specific motion patterns (camera zooming, rolling, etc.).

SparseCtrl aims to add more control to text-to-video models by adopting some sparse inputs (e.g., few RGB images or sketch inputs). Its technicall details can be found in the following paper:

SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models
Yuwei Guo, Ceyuan Yang✝, Anyi Rao, Maneesh Agrawala, Dahua Lin, Bo Dai (✝Corresponding Author)

In this version, we use Domain Adapter LoRA for image model finetuning, which provides more flexiblity at inference. We also implement two (RGB image/scribble) SparseCtrl encoders, which can take abitary number of condition maps to control the animation contents.

1. Small fickering is noticable;
2. To stay compatible with comunity models, there is no specific optimizations for general T2V, leading to limited visual quality under this setting;
3. (Style Alignment) For usage such as image animation/interpolation, it's recommanded to use images generated by the same community model.

Input (by RealisticVision)	Animation	Input	Animation

Input Scribble	Output	Input Scribbles	Output

Release the Motion Module (beta version) on SDXL, available at Google Drive / HuggingFace / CivitAI. High resolution videos (i.e., 1024x1024x16 frames with various aspect ratios) could be produced with/without personalized models. Inference usually requires ~13GB VRAM and tuned hyperparameters (e.g., sampling steps), depending on the chosen personalized models.
Checkout to the branch sdxl for more details of the inference.

Original SDXL	Community SDXL	Community SDXL

In this version, the motion module mm_sd_v15_v2.ckpt (Google Drive / HuggingFace / CivitAI) is trained upon larger resolution and batch size. We found that the scale-up training significantly helps improve the motion quality and diversity.
We also support MotionLoRA of eight basic camera movements. MotionLoRA checkpoints take up only 77 MB storage per model, and are available at Google Drive / HuggingFace / CivitAI.

Zoom In	Zoom Out	Zoom Pan Left	Zoom Pan Right
Tilt Up	Tilt Down	Rolling Anti-Clockwise	Rolling Clockwise

Here's a comparison between mm_sd_v15.ckpt (left) and improved mm_sd_v15_v2.ckpt (right).

The first version of AnimateDiff!