How Will It Drape Like? Capturing Fabric Mechanics from Depth Images

Abstract

We propose a method to estimate the mechanical parameters of fabrics using a casual capture setup with a depth camera. Our approach enables to create mechanically-correct digital representations of real-world textile materials, which is a fundamental step for many interactive design and engineering applications. As opposed to existing capture methods, which typically require expensive setups, video sequences, or manual intervention, our solution can capture at scale, is agnostic to the optical appearance of the textile, and facilitates fabric arrangement by non-expert operators. To this end, we propose a sim-to-real strategy to train a learning-based framework that can take as input one or multiple images and outputs a full set of mechanical parameters. Thanks to carefully designed data augmentation and transfer learning protocols, our solution generalizes to real images despite being trained only on synthetic data, hence successfully closing the sim-to-real loop.Key in our work is to demonstrate that evaluating the regression accuracy based on the similarity at parameter space leads to an inaccurate distances that do not match the human perception. To overcome this, we propose a novel metric for fabric drape similarity that operates on the image domain instead on the parameter space, allowing us to evaluate our estimation within the context of a similarity rank. We show that out metric correlates with human judgments about the perception of drape similarity, and that our model predictions produce perceptually accurate results compared to the ground truth parameters.

arXiv comment12 pages, 12 figures. Accepted to EUROGRAPHICS 2023. Project website: https://carlosrodriguezpardo.es/projects/MechFromDepth/

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Simple Framework for Contrastive Learning of Visual Representations
• Self-Attention Generative Adversarial Networks
• On the Opportunities and Risks of Foundation Models
• Squeeze-and-Excitation Networks
• Image Quality Assessment: Unifying Structure and Texture Similarity
• Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning
• Transfusion: Understanding Transfer Learning for Medical Imaging
• Learning multiple visual domains with residual adapters
• A Haar Wavelet-Based Perceptual Similarity Index for Image Quality Assessment
• Adaptively Learning the Crowd Kernel
• Mean Deviation Similarity Index: Efficient and Reliable Full-Reference Image Quality Evaluator
• Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
• Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs
• DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact
• A Similarity Measure for Material Appearance
• gradSim: Differentiable simulation for system identification and visuomotor control
• Learning icons appearance similarity

Cited by in corpus (1)

• Polar Interpolants for Thin-Shell Microstructure Homogenization