Toward A Blurry Rasterizer: Jacob Munkberg
Toward A Blurry Rasterizer: Jacob Munkberg
Jacob Munkberg
Intel Corporation
Goals
Realistic camera model
Finite aperture: Depth of field Slow shutter: Motion blur
Ease of use
Set camera parameters and
per-vertex motion vectors
This Talk
Covers:
Stochastic rasterization for motion blur & depth of field
Not covering:
Post-processing techniques [Potmesil81, Max85, Shinya93, Sousa2008, ...] Layered depth image approaches [Lee2009, Lee2010, ...] Blurry ray tracing [Cook84, Walter2006, Wald2007, Hachisuka2008,
Overbeck2009, Hou2010, Grnshlo2011, ...]
Analytical visibility [Korein83, Sung2002, Gribel2010, Gribel2011, ...] Reconstruction & filtering [Egan2009, Soler2009, Parker2010, Lehtinen2011, ...]
For in-depth discussion of micropolygon rasterization with blur and defocus, please refer to the BPS 2009 course and Kayvon Fatahalians PhD thesis [Fatahalian2011]. Motion Blur Rendering: State of the Art [Navarro et al.2011]
Motivation
Depth of Field
Direct viewers attention Match footage with CGI
Motion Blur
Enhance effect of motion Reduce temporal aliasing
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
Temporal Aliasing
y
pixel
y x
t
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
Instead of searching for an analytical solution (very hard), we estimate the integral by Monte Carlo techniques.
pixel
lens (u,v)
focus plane
pixel t=0 image plane (x,y) lens (u,v) focus plane t=1
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
Sampling
Approximating the visibility integral
Approximate Integral
Accumulation buffering
Render scene at many fixed
times (and lens positions)
[Korein83, Haeberli90]
uniform
stochastic
t=0.3
t=0.6
t=1.0
4 spp
8 spp
16 spp
32 spp
256 spp
High quality sampling patterns available. On-the-y generation possible. Use patterns that maximizes minimum distance in 3D, and have good 2D projections (on screen space x,y) Reconstruction lters [Lehtinen2011] can help tremendously here
64 random spp
Furthermore, out-of-focus regions and high frequencies require more visibility samples per pixel to reduce noise levels.
Need large number of unique (ui,vi,ti) tuples To appear in ACM Harder to find good sample distributions
Images courtesy of Laine et al.[2011] taken from Assassins Creed by Ubisoft
InterleaveUVT 4 spp
InterleaveUVT 16 spp
Figure 1: Quality comparison between InterleaveUVT using 22 The InterleaveUVT images have 64 unique (u,v,t) positions Harderpixel tiles and a good sampling pattern generated5D space. to nd good sample distributions with stratied samples in using Sobol se-
quences. InterleaveUVT suffers from a limited number of unique (u, v, t) triplets, especially with low sample counts common in re-
Visibility
The rasterizer stage
! !"#$$%&!'("$
Traverse in an order that enables coherent texture and memory accesses Projection implies clipping. Clipping also enables xed-point rasterization
Test Samples
Tiled Traversal
Test tile of pixels for overlap
Reject samples in tiles completely outside triangle Enables coarse occlusion culling, improves coherence
Important optimization for larger triangles. Basis for a hierarchical rasterizer. Improves texture and memory coherence
Blurry Rasterization
Blurry Rasterization
Moving triangles
q2 t=0.0 r0 q0 r2 t1=0.37 q1 t=1.0
r1
Sample in (x,y,u,v,t)
Alternative 1: Use a xed set of sample times, clip triangle at each time Alternative 2: Homogeneous rasterization Linear interpolation breaks down
Screen-Space Bounds
Bound full motion and all lens positions Handle moving triangles intersecting z=0 with care
t=0 t=1 t=1
DOF BBox
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
Spatio-Temporal Visibility
Moving triangle may cover large
region in (x,y,t) More samples per pixel needed for low noise levels We need to quickly cull samples in both space and time
" "
! # #
!
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
The key is that many more samples need to be tested, and that the complexity is tied to the amount of blur applied (motion and/or DOF). Dimension on visibility query has increased from two to three (or ve) dimensions
Revert to coarser bounds if moving triangle intersects z=0. The screen space convex hull algorithm only works on projected vertices with z>0.
Interval
Use N time intervals
[Cook90, Fatahalian2009]
" "
Multiple tri setup and bounding. One tile visited multiple times
Interleaved Rasterization
Use N fixed sample times
[Keller2001, Fatahalian2009]
"
Position triangle at each time Rasterize coarsely in (x,y) Restricted sampling patterns
! #
Tile-Based Traversal
Tile-test reduces samples to test
Trivially rejects tile or returns
temporal bounds for overlap
[Laine2011, Munkberg2011b]
! # "
!
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
A tile is only visited once, enabling multisampling & coarse Z culling Extends standard GPU pipeline
5D Rasterization
Interval [Cook90, Fatahalian2009]
Many strata in (u,v,t). Find (x,y) bounds for each stratum
Tile-Overlap Tests
Find time interval when moving triangle overlaps
with the screen space tile [Laine2011, Munkberg2011b]
t=0
t=1
[0,1/4]
[0,1/2]
[1/4,3/4]
[1/2,1]
[3/4, 1]
Examples: Dual-space bounds [Laine], Tile frustum vs moving AABB & Tile vs moving triangle edges [Munkberg].
pixel
image plane
lens
focus plane
objects out of focus
Idea: Only test lens region that sees the object through a certain screen space tile
lens
focus plane
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
tile
triangle
tile
focus plane
focus plane
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
Shading
t=1/4
t=1/2
t=3/4
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
t=1
T=1
t=0
T=0
T=0
screen space
barycentric space
shading space
Reconstruction
Reduce noise in stochastic rendering
May drastically reduce required #samples per pixel Store additional information with samples [Lehtinen2011]
Images courtesy of Lehtinen et al.[2011]
Reconstructed
Culling Backface
culled Occlusion culled
t=0
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
Test entire moving triangle Gives smaller screen space bboxes (and better dual-space bounds)
p1 p0 p2
The backface test for a triangle with linear vertex motion is cubic polynomial in t. Cubic polynomial -> up to three zeros -> up to three changes in the sign of the signed area
Correct BF cull
Lens
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
Correct BF cull
Lens
Beyond Programmable Shading Course, ACM SIGGRAPH 2011
t x
Left: Solid occluder is minimal. Right: In the other extreme, a separate occlusion hierarchy per sample time. Middle ground probably most practical approach. In GPU-like pipeline, want coarse z-cull at tile level. A full spatio-temporal z-hierarchy improves culling, but is likely too expensive in terms of memory and bandwidth usage Conclusion: Tile based traversal is preferable. Efficient 5D occlusion culling is an open research eld
Conclusions
Stochastic rasterization does not come for free
Many more sample tests and shading evaluations Higher color, depth & texture bandwidth Less efficient culling and compression
Q&A
Thank You:
The Advanced Rendering Technology team at Intel The Graphics Group at Lund University Mike Doggett, Jonathan Ragan-Kelley, Samuli Laine,
and Timo Aila for fruitful discussions
References
[Akenine-Mller2007] Tomas Akenine-Mller, Jacob Munkberg, and Jon Hasselgren, "Stochastic Rasterization using Time-Continuous Triangles", Graphics Hardware, August 2007. [Akenine-Mller2011] Tomas Akenine-Mller, Robert Toth, Jacob Munkberg, and Jon Hasselgren, "Efcient Depth of Field Rasterization using a Tile Test based on Half-Space Culling", technical report, June 2011 [Boulos2010] Solomon Boulos, Edward Luong, Kayvon Fatahalian, Henry Moreton and Pat Hanrahan Space-Time Hierarchical Occlusion Culling for Micropolygon Rendering with Motion Blur High Performance Graphics 2010 [Burns2010] Burns, C.A., Fatahalian, K., Mark, W.R. A Lazy Object-Space Shading Architecture With Decoupled Sampling. High Performance Graphics, 19-28. 2010 [Cook84] Robert L. Cook, Thomas Porter, Loren Carpenter, Distributed Ray Tracing SIGGRAPH 1984 [Cook86] Robert L. Cook. Stochastic Sampling in Computer Graphics. ACM Transactions on Graphics, Volume 6, Number 1, January 1996. [Cook87] Robert L. Cook, Loren Carpenter, Edwin Catmull. The Reyes Rendering Architecture. 1987 [Cook90] Robert L. Cook, Thomas Porter, Loren Carpenter. Pseudo-random point sampling techniques in computer graphics. United States Patent 4,897,806. 1990. [Egan2009] Kevin Egan , Yu-Ting Tseng , Nicolas Holzschuch , Frdo Durand , Ravi Ramamoorthi, Frequency analysis and sheared reconstruction for rendering motion blur, ACM Transactions on Graphics (TOG), v.28 n.3, August 2009 [Fatahalian2009] Kayvon Fatahalian, Edward Luong, Solomon Boulos, Kurt Akeley, William R. Mark and Pat Hanrahan. Data-Parallel Rasterization of Micropolygons with Defocus and Motion Blur High Performance Graphics 2009 [Fatahalian2011] Kayvon Fatahalian Evolving the Real-Time Graphics Pipeline for Micropolygon Rendering. Ph.D. Dissertation, Stanford University, 2011 [Grunschlo2011] Leonhard Grunschlo, Martin Stich, Sehera Nawaz, Alexander Keller. MSBVH: An Efficient Acceleration Data Structure for Ray Traced Motion Blur High Performance Graphics 2011
References cont.
[Gribel2010] Carl Johan Gribel, Michael Doggett, and Tomas Akenine-Mller, "Analytical Motion Blur Rasterization using Compression", High Performance Graphics, pp. 163-172, June 2010 [Gribel2011] Carl Johan Gribel, Rasmus Barringer, and Tomas Akenine-Mller, "High-Quality Spatio-Temporal Rendering using Semi-Analytical Visibility", SIGGRAPH 2011. [Hachisuka2008] Toshiya Hachisuka, Wojciech Jarosz, Richard Peter Weistroffer, Kevin Dale, Greg Humphreys, Matthias Zwicker, Henrik Wann Jensen Multidimensional adaptive sampling and reconstruction for ray tracing SIGGRAPH 2008 [Hou2010] Qiming Hou, Hao Qin, Wenyao Li, Baining Guo, Kun Zhou. Micropolygon ray tracing with defocus and motion blur. SIGGRAPH 2010 [Haeberli90] Paul Haeberli , Kurt Akeley, The accumulation buffer: hardware support for high-quality rendering, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.309-318, 1990 [Korein83] Jonathan Korein , Norman Badler,Temporal anti-aliasing in computer generated animation, Computer Graphics, vol 17, no .3, p.377-388, 1983 [Keller2001] Alexander Keller , Wolfgang Heidrich, Interleaved Sampling, Eurographics Workshop on Rendering Techniques, p.269-276, 2001 [Laine2011] Samuli Laine, Timo Aila, Tero Karras and Jaakko Lehtinen. Clipless Dual-Space Bounds for Faster Stochastic Rasterization. SIGGRAPH 2011 [Lee2009] Sungkill Lee, Elmar Eisemann and Hans-Peter Seidel. Depth-of-field rendering with multiview synthesis. SIGGRAPH Asia 2009 [Lee2010] Sungkill Lee, Elmar Eisemann and Hans-Peter Seidel. Real-time lens blur effects and focus control. SIGGRAPH 2010 [Lehtinen2011] Jaakko Lehtinen, Timo Aila, Jiawen Chen, Samuli Laine and Frdo Durand. Temporal Light Field Reconstruction for Rendering Distribution Effects. SIGGRAPH 2011 [Loviscach2005] Jrn Loviscach, Motion Blur for Textures by Means of Anisotropic Filtering, Eurographics Symposium on Rendering, 2005. pp. 105-110 [Max85] Max N. L., Lerner D. M.: A two-and-a-half-d motion-blur algorithm. SIGGRAPH '85 [McGuire2010] McGuire, Enderton, Shirley, and Luebke, Real-Time Stochastic Rasterization on Conventional GPU Architectures, High Performance Graphics, June, 2010.
References cont.
[Munkberg2011a] Jacob Munkberg and Tomas Akenine-Mller, "Backface Culing for Motion Blur and Depth of Field", journal of gpu, graphics and game tools, 2011. [Munkberg2011b] Jacob Munkberg, Petrik Clarberg, Jon Hasselgren, Robert Toth, Masamichi Sugihara, and Tomas Akenine-Mller, Hierarchical Stochastic Motion Blur Rasterization, High Performance Graphics 2011. [Navarro2011] Fernando Navarro, Francisco J. Sern and Diego Guitierrez. Motion Blur Rendering: State of the Art. Computer Graphics Forum Volume 30 (2011), number 1 , pp. 3-26 [Overbeck2009] Ryan S. Overbeck , Craig Donner , Ravi Ramamoorthi, Adaptive Wavelet Rendering, ACM Transactions on Graphics (TOG), v.28 n.5, December 2009 [Potmesil81] Potmesil M. Chakravarty I: A lens and aperture camera model for synthetic image generation. SIGGRAPH 81 pp. 389-399 [Ragan-Kelley2011] Jonathan Ragan-Kelley, Jaakko Lehtinen, Jiawen Chen, Michael Doggett, Frdo Durand. Decoupled Sampling for Graphics Pipelines. SIGGRAPH 2011 [Shinya93] Shinya M.: Spatial anti-aliasing for animation sequences with spatio-temporal filtering. SIGGRAPH 93 pp. 289-296 [Shirley2011] Shirley, Aila, Cohen, Eric Enderton, Laine, Luebke, McGuire, A Local Image Reconstruction Algorithm for Stochastic Rendering, ACM Symposium on Interactive 3D Graphics and Games (I3D 2011 proceedings), February 2011 [Soler2009] Cyril Soler, Kartic Subr, Frdo Durand, Nicolas Holzschuch, Franois Sillion. Fourier depth of field. ACM Transactions on Graphics (TOG) , Volume 28 Issue 2, 2009 [Sousa2008] Sousa T.: Crysis next-gen effects. In Proceedings of the Game Developers Conference 2008. [Sung2002] K. Sung , A. Pearce , C. Wang, Spatial-Temporal Antialiasing, IEEE Transactions on Visualization and Computer Graphics, v.8 n.2, p.144-153, April 2002 [Wald2007] Wald I., Mark. W. R., Gnther J., Boulos S., Ize T., Hunt W., Parker S.G.,Shirley P.: State of the art in ray tracing animated scenes. Eurographics 2007 State of the Art Reports [Walter2006] Bruce Walter, Adam Arbree, Kavita Bala, Donald P. Greenberg. Multidimensional Lightcuts. SIGGRAPH 2006