3D Hand Bones And Tissue Estimation From A Single 2D X-Ray Image Via A Two-Stream Deep Neural Network

The hand bones and soft tissue are fundamentally important for many fields such as clinical diagnosis, hand modeling and metaverse. However, their 3D reconstruction from CT or MRI data requires too much effort from human experts and computational resource from modern hardware. In this paper, we present a novel method to estimate the 3D hand bones and soft tissue from a single X-ray image via a two-stream deep neural network. One stream is for the bone estimation which successfully cooperates a module component from other modality. The other stream is for the soft tissue modeling which adopts a sub-network from hand pose estimation. After merging these two streams, we successfully construct a 3D virtual hand from a single 2D X-ray image. Several numerical experiments are conduced to validate the proposed two-stream network. Our method can be used in the hand bone and soft tissue modeling from X-ray images.