F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis

Abstract

Recently Text-to-Video (T2V) synthesis has undergone a breakthrough by training transformers or diffusion models on large-scale datasets. Nevertheless, inferring such large models incurs huge costs.Previous inference acceleration works either require costly retraining or are model-specific.To address this issue, instead of retraining we explore the inference process of two mainstream T2V models using transformers and diffusion models.The exploration reveals the redundancy in temporal attention modules of both models, which are commonly utilized to establish temporal relations among frames.Consequently, we propose a training-free and generalized pruning strategy called F3-Pruning to prune redundant temporal attention weights.Specifically, when aggregate temporal attention values are ranked below a certain ratio, corresponding weights will be pruned.Extensive experiments on three datasets using a classic transformer-based model CogVideo and a typical diffusion-based model Tune-A-Video verify the effectiveness of F3-Pruning in inference acceleration, quality assurance and broad applicability.

References in corpus (19)

• Generative Adversarial Networks
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• High-Resolution Image Synthesis with Latent Diffusion Models
• Imagen Video: High Definition Video Generation with Diffusion Models
• Make-A-Video: Text-to-Video Generation without Text-Video Data
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers
• Token Merging: Your ViT But Faster
• BiBERT: Accurate Fully Binarized BERT
• Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
• Group Fisher Pruning for Practical Network Compression
• BiT: Robustly Binarized Multi-distilled Transformer
• Structural Pruning for Diffusion Models
• Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning
• Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
• Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
• Two-stream Collaborative Learning with Spatial-Temporal Attention for Video Classification
• Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer
• Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers