Differentiable Path Tracing by Regularizing Discontinuities

Reviewer 1

-> I cannot understand the contribution of the paper from the abstract. This should be made clearer. -> It is hard throughout the paper to understand how the presented method is different from the other cited literature in differentiable ray tracing. This should really be made clear . -> Adding motion blur and depth of field effects is interesting! -> The results are very nice!

Reviewer 2

Contribution:

The proposed methods to make Edge discontinuity and Depth Order Discontinuity differentiable are valid contributions. In the proposed approach, the gradient can be back-propagated to vertex positions/colors(texture). The importance sampling technique for multiple bounces also makes sense
The proposed method also supports many camera effects, such as depth of field and motion blur.

Limitation: The paper requires a predefined mesh topology and optimize the geometry, it’s not clear how can it generalize to a different topology.” “1. It would be better to compare with methods from Computer Graphics, such as [9], [12] they can also optimize the geometry but using different methods,

It would also be better if the proposed method can be utilized in the deep learning pipeline, for example: reconstructing 3D shape by using the differentiable render

Reviewer 3

This paper discusses extending the ““SoftRas”” strategy of smoothing discontinuities to handle global illumination. Essentially each opaque triangle is converted into a transparent one with the transparency gradually approaches zero at the boundary. This may allow us to control the smoothness of the loss landscape and benefit optimization, at the cost of having some difficult-to-tune smoothing parameters (I believe you’ll have to adaptively smooth the scene, with different smoothing parameters spatially) and more expensive ray queries (you need all hit instead of closest hit now).

Rhodin’s ““A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation”” should be cited. Their method can also handle global illumination, even though they didn’t show it.

de La Gorce’s ““Model-based 3D hand pose estimation from monocular video”” should be cited. It’s the first general differentiable rasterizer which is actually a lot more accurate in certain circumstances than many of the followups.

This is an interesting idea, but the paper contains several false/vague statements, especially in the introduction, that will need to be fixed before the publication: Figure 1: is Russian roulette differentiable? (it probably is but would be nice to justify) L8 ““that can be handled by rasterization …””: typo, you mean can’t L22 ““Differentiable rendering involves modifying the rendering process to remove discontinuities””: No! What Li et al. and many have shown was that you DON’T need to modify the rendering process. Instead you only need to take into account the Dirac delta that appears in differentiation of discontinuities. L26 ““Rasterization uses multiple matrix multiplications to efficiently project the geometry””: The main reason rasterization is fast compared to ray tracing has NOTHING to do with matrix multiplication at all. Rasterization loops over each object, then each pixel, while ray tracing loops over each pixel, then each object. This leads to different acceleration structures and different memory behavior. Rasterization usually has better cache locality, but well-tuned ray tracing is not too far and has less overdraw. With hardware support, the performance of the two approaches are not that different, except that the acceleration data structures for ray tracing is more time-consuming to build. Furthermore, several differentiable rasterization approaches, e.g., SoftRas, do not use Z-buffer (due to the need of transparency handling) and can be significantly slower than ray tracing on high-depth-complexity scenes because of this. L31-35 One main reason these approaches used ray tracing has nothing to do with realism at all. Instead it is because we need ray tracing to obtain unbiased gradients, even if we only care about local shading. Z-buffer based rasterization requires a regular sampling pattern for resolving occlusion, so when explicitly sampling the Dirac delta that appears in differentiation, we need to resolve the irregular Dirac using ray tracing. L35 ““their high computational cost have limited their use in ML applications”” When focusing on local shading, these approaches are equally fast, if not faster, than the rasterization implementations out there. L36 ““While the rendering process is trivially differentiable with respect to the material properties””: Not really. If you consider discontinuous procedural models they are not (trivially) differentiable. Even with standard models, refraction with IOR != 1 can lead to discontinuities (e.g., total internal reflection). L82-91: Would be nice to mention how emitter sampling is done – since you blur the emitter, you will need to sample the enlarged region of the emitter. Would be nice to mention MIS as well. L93-95 ““Textures are naturally differentiable””: This is true. However, the way you smooth the 3D scene will likely screwup texture lookup at the boundaries, since you will have negative UV values outside of the triangle. I know you can clamp the UVs (this is what the SoftRas approach did), but this leads to weird artifacts. L99 ““To our knowledge, ours is the first work to include differentiable camera effects””: Li’s thesis (““differentiable visual computing””) described how to implement depth of field, but indeed they didn’t release a public implementation. L139 ““Our differentiable rendering does not produce exact gradients””: Not really. You produce exact gradients, at the cost of changing the rendering model.

Technical-wise the paper is fine. I vote for weak reject because I want the comments above to be addressed (last time I accepted a paper like this the authors ignored most of my comments). If other reviewers are enthusiastic and can help shepherding the paper, I am happy to see the paper published in this workshop.