Home
With four parameters I can fit an elephant, and with five I can make it wiggle its trunk. — John Von Neumann (abridged)
Differentiable vision, graphics, and physics applied to machine learning
Virtual event, 11 December 2020 (Friday)
“Differentiable programs” are parameterized programs that allow themselves to be rewritten by gradient-based optimization. They are ubiquitous in modern-day machine learning. Recently, explicitly encoding our knowledge of the rules of the world in the form of differentiable programs has become more popular. In particular, differentiable realizations of well-studied processes such as physics, rendering, projective geometry, optimization to name a few, have enabled the design of several novel learning techniques. For example, many approaches have been proposed for unsupervised learning of depth estimation [1‒6] from unlabeled videos. Differentiable 3D reconstruction pipelines [7, 8] have demonstrated the potential for task-driven representation learning. A number of differentiable rendering approaches [9‒14] have been shown to enable single-view 3D reconstruction and other inverse graphics tasks (without requiring any form of 3D supervision). Differentiable physics simulators are being built [15‒20] to perform physical parameter estimation from video or for model-predictive control. While these advances have largely occurred in isolation, recent efforts [21‒24] have attempted to bridge the gap between the aforementioned areas. Narrowing the gaps between these otherwise isolated disciplines holds tremendous potential to yield new research directions and solve long-standing problems, particularly in understanding and reasoning about the 3D world.
Hence, we propose the “first workshop on differentiable computer vision, graphics, and physics in machine learning” with the aim of:
- Narrowing the gap and fostering synergies between the computer vision, graphics, physics, and machine learning communities
- Debating the promise and perils of differentiable methods, and identifying challenges that need to be overcome
- Raising awareness about these techniques to the larger ML community
- Discussing the broader impact of such techniques, and any ethical implications thereof.
Notably, most of the advances in our focus areas have occurred in the last 3-4 years, and the interest in this nascent field seems to be increasing (as evident from the push towards the release of open-source tools such as Kaolin [25], PyTorch3D [26], Kornia [27], tiny-differentiable-simulator [28], and more. This workshop aims to bring together researchers from outside the core ML community (such people from the graphics, vision, and physics simulation communities), enabling synergies among them.
