SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches

Abstract

2D animation is a common factor in game development, used for characters, effects and background art. It involves work that takes both skill and time, but parts of which are repetitive and tedious. Automated animation approaches exist, but are designed without animators in mind. The focus is heavily on real-life video, which follows strict laws of how objects move, and does not account for the stylistic movement often present in 2D animation. We propose a problem formulation that more closely adheres to the standard workflow of animation. We also demonstrate a model, SketchBetween, which learns to map between keyframes and sketched in-betweens to rendered sprite animations. We demonstrate that our problem formulation provides the required information for the task and that our model outperforms an existing method.

7 pages, 6 figures, ACM Conference on the Foundations of Digital Games

References in corpus (7)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Zero-Shot Text-to-Image Generation
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• Predicting Video with VQVAE
• Pixel VQ-VAEs for Improved Pixel Art Representation
• Generating Gameplay-Relevant Art Assets with Transfer Learning
• Video Frame Interpolation via Structure-Motion based Iterative Fusion