Estimating and abstracting the 3D structure of bones using neural networks on X-ray (2D) images

Abstract

In this paper, we present a deep-learning based method for estimating the 3D structure of a bone from a pair of 2D X-ray images. Our triplet loss-trained neural network selects the most closely matching 3D bone shape from a predefined set of shapes. Our predictions have an average root mean square (RMS) distance of 1.08 mm between the predicted and true shapes, making it more accurate than the average error achieved by eight other examined 3D bone reconstruction approaches. The prediction process that we use is fully automated and unlike many competing approaches, it does not rely on any previous knowledge about bone geometry. Additionally, our neural network can determine the identity of a bone based only on its X-ray image. It computes a low-dimensional representation ("embedding") of each 2D X-ray image and henceforth compares different X-ray images based only on their embeddings. An embedding holds enough information to uniquely identify the bone CT belonging to the input X-ray image with a 100% accuracy and can therefore serve as a kind of fingerprint for that bone. Possible applications include faster, image content-based bone database searches for forensic purposes.

arXiv comment13 pages, 5 figures, 1 table, submitted to Communications Biology

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• In Defense of the Triplet Loss for Person Re-Identification
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era
• Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Images
• Learning Local Image Descriptors with Deep Siamese and Triplet Convolutional Networks by Minimising Global Loss Functions