Michael Seeber, Roi Poranne, Marc Pollefeys and Martin R. Oswald

Estimating 3D hand meshes from RGB images robustlyis a highly desirable task, made challenging due to the nu-merous degrees of freedom, and issues such as self simi-larity and occlusions. Previous methods are generally di-vided to parametric 3D hand models and model free ap-proaches. While the former can be considered more ro-bust, e.g. to occlusions, they are less expressive. We pro-pose a hybrid approach, utilizing deep neural network anddifferential rendering based optimization to demonstrablyachieve the best of both worlds. In addition, we explore Vir-tual Reality (VR) as an application. Most VR headsets arenowadays equipped with multiple cameras, which we canleverage by extending our method to the egocentric stereodomain. This extension proves to be more resilient to theabove mentioned issues. Finally, as a use-case, we showthat the improved image-model alignment can be used toacquire the user’s hand texture, which can be for a morefaithful virtual hand representation.