Xin Yu, Jeroen van Baar and Siheng Chen

To resolve this unknown alignment issue, we propose a two-layer graph structure, where one layer is a mesh graph for human shapes, and the other layer is a skeleton graph for body joints. Information between the two layers is exchanged via fusion graph modules. The skeleton graph can associate the 2D body joints estimated from an image with their coordinates in 3D space. 2D body joints can be well estimated from an image. This body-joint-based correspondence allows us to attach local image features to each body joint in the skeleton graph, and then propagate local image information to mesh nodes through the two-layer graph structure. An added benefit of this two-layer graph structure is multi-tasking: achieving shape recovery and pose estimation at the same time. Two layers naturally model a human body from mesh and skeleton scales, handling shape recovery and pose estimation, respectively. The cross or fusion layer is a trainable bipartite graph that connects body joints and mesh nodes. Instead of imposing any fixed connections, such a bipartite graph can adaptively adjust the relationships between the mesh nodes and body joints. It enables the feature fusion between two scales of a human body, mutually enhancing two tasks. This is related to linear blend skinning, however, where skinning provides an analytical transformation, our cross-layer learns a data-adaptive transformation between body joints and mesh nodes in the high-dimensional feature space.