Joint 3D Human Shape Recovery and Pose Estimation from a Single Image with Bilayer Graph

Authors:

Xin Yu, Jeroen van Baar and Siheng Chen

Abstract:

To resolve this unknown alignment issue, we propose a two-layer graph structure, where one layer is a mesh graph for human shapes, and the other layer is a skeleton graph for body joints. Information between the two layers is exchanged via fusion graph modules. The skeleton graph can associate the 2D body joints estimated from an image with their coordinates in 3D space. 2D body joints can be well estimated from an image. This body-joint-based correspondence allows us to attach local image features to each body joint in the skeleton graph, and then propagate local image information to mesh nodes through the two-layer graph structure. An added benefit of this two-layer graph structure is multi-tasking: achieving shape recovery and pose estimation at the same time. Two layers naturally model a human body from mesh and skeleton scales, handling shape recovery and pose estimation, respectively. The cross or fusion layer is a trainable bipartite graph that connects body joints and mesh nodes. Instead of imposing any fixed connections, such a bipartite graph can adaptively adjust the relationships between the mesh nodes and body joints. It enables the feature fusion between two scales of a human body, mutually enhancing two tasks. This is related to linear blend skinning, however, where skinning provides an analytical transformation, our cross-layer learns a data-adaptive transformation between body joints and mesh nodes in the high-dimensional feature space.

PDF (protected)


  Important Dates

All deadlines are 23:59 Pacific Time (PT). No extensions will be granted.

Paper registration July 23 30, 2021
Paper submission July 30, 2021
Supplementary August 8, 2021
Tutorial submission August 15, 2021
Tutorial notification August 31, 2021
Rebuttal period September 16-22, 2021
Paper notification October 1, 2021
Camera ready October 15, 2021
Demo submission July 30 Nov 15, 2021
Demo notification Oct 1 Nov 19, 2021
Tutorial November 30, 2021
Main conference December 1-3, 2021

  Sponsors