Google Summer of Code 2009 Proposal
Seth Berrier: Advanced Lighting and Shading
Note: the 'Abstract' and 'Introduction' are identical to what was submitted to the GSoC website.
Abstract
This project proposes the modernization of the CrystalSpace shading system to employ a more sophisticated reflection model. Different industry standard models will be explored including, but not limited to, Cook-Torrance, Oren-Nayer, pre-filtered environment or reflection mapping, and numerical methods (like spherical harmonics or wavelet basis fittings). Both real-time and precomputed versions of these reflection models will be developed and incorporated in the current CrystalSpace lighting system.
Introduction
Simulation of reflection and material properties is a long and studied topic in computer graphics. Primitive ideas such as Goroud shading, the lambertion cosine law and the use of a cosine lobe for specularity date back over three decades. This ideas have withstood the test of time (particularly lambertian reflection and the models of Phong and Blinn) not so much for their accuracy but rather their efficiency.
But as with all tradeoffs they must be reevaluated as time and hardware capabilities advance. In the last five years the rapid growth of programmable graphics processing units has led to these more efficient models being replaced with higher complexity models that are more accurate. Old ideas that had been largely ignored in the real-time rendering community are see a resurgence in use. Among these are the physically accurate models of Oren-Nayaer and Cook-Torrance which simulate microfacet orientation and self-shadowing effects while remaining energy conservative. We also see newer, numerically based models that fit reflection properties of real-world objects to special basis functions like spherical harmonics, wavelets and cosine lobes. All of these have the potential to run in real-time on modern graphics hardware and even be incorporated in complex 3D simulations like the CrystalSpace engine.
CrystalSpace, however, still employs the tried and true methods proposed by Blinn and Phong and has progressed beyond their applicability. Thus, I propose the modernization of the CrystalSpace lighting and shading system. By employing physically based or numerically based reflection models both in real-time shading and in precomputing light mapping the photo-realism of the rendering can be greatly enhanced while still retaining the older, more efficient models currently in use so that scalability is maintained.
Background
While the topic of lighting and reflection has been summarized and categorized many times over (for an excellent example see 0) it is still a good idea to review the basic foundations of this topic so that the more important characteristics for this proposal can be emphasized and a focus can be made on the specific models that will be pursued.
The BRDF
Any discussion of lighting and reflection properties must begin with the BRDF as that is the glue that ties them all together. Furthermore, discussion of the BRDF nautrally begins with the Kajiya rendering equation which ties the BRDF to rendering as a whole.
In the 1986 paper 'The Rendering Equation' by Kajiya(1), the process of generating computer imagery by simulation of light was described by a single integral equation. This equation describes light leaving from a point as the sum of emitted light and reflected light. Reflected light is computed as a sum of incident light (attenuated by the incident angle) and the bidirectional reflectance distribution function (BRDF) all integrated over the hemisphere of incident directions. In other words, to compute what light is leaving a particular point in a scene we need to know how much light it creates itself (if it's a light source), how much light is hitting it (which comes from the other lights in the scene) and how the physical properties of the surface will reflect the light that hits it. This last part is the key, and is expressed precisely in the BRDF.
The BRDF describes the reflection characteristics of an ideal surface point (an infinitely small surface point) (2). It takes as input an incoming light direction (which requires at least two dimensions in 3-space) and an outgoing light direction (another two dimensions) and returns the percentage of light that would be reflected for that in and out direction. For real applications the wavelength of light must be accounted for so this often makes another dimension of the equation. In practice, a different BRDF is used for each of the three color channels; red, green and blue. The standard BRDF does not account for variance across the surface of an object (allthough variants like the spatially varying BRDF and the BTF have been proposed and studied in research). However, other rendering techniques (like texture mapping) can be employed to modulate the BRDF calculation and produce a good approximation of spatial variance.
In practice, the BRDF is rarely used directly. To fully sample every point of an object across every possible incoming and outgoing direction is impractical for complex projects like game development. Furthermore, accessing and rendering from this type of data is inefficient and memory intensive. Instead, researchers have proposed many practical analytic approximations of the BRDF that can be controlled with a only a few key parameters. Some of these analytic approximations strive only to reprodue the look-and-feal of key BRDF features (like specular highlights and diffuse attenuation). Others look closely at the microscopic features of the surface and attempt to incorporate physical laws like conservation of energy into physically accurate analytic models with higher fidelity but increased mathematical complexity. Yet a third approach tackles this problem using statistical methods for data reduction. By analyzing full BRDF data acquired with a gonioreflectometer (a device that systematically cycles through incoming and outgoing directions to measure a BRDF) it can be systematically fit to basis functions ranging from simple polynomial coefficients to complex spherical harmonics and wavelets. The reduction in data complexity allows for efficient rendering while still maintaining the fiedlity of a complete BRDF.
Here, we will briefly discuss examples from all three types of BRDF approximations with references to publications wherever possible.
Appearance Based Analytical BRDFs
There are two popular and very efficient analytical BRDF approximations that are in wide use today. The first is commonly referred to as the Lambertian model. Lambertian surfaces reflect light equally in all directions of the outgoing hemisphere. They are often called 'matte' surfaces. This type of reflectance is based on Lambert's cosine law(3) which describes mathematically the behavior of Lambertian surfaces. The Lambertian model computes the reflected light as the cosine of the angle formed between the line of sight and the line to the light source. A coefficient is used to control the strength for each color channel (or more generally, each wavelength of light).
The second BRDF approximation that falls in this category is the Phong reflectance model(4). The Phong model separates reflected light into three types: ambient, diffuse and specular. Each type is computed separately and has a coefficient to control its intensity for each color channel or wavelength. The ambient contribution models light that reaches the surface after bouncing off another diffuse surface in the scene. This light is very difficult to model accurately and its contribution is small for most scenes so Phong uses a constant value to approximate it. The diffuse component is the portion of light that is reflected evenly in all directions. While for Lambertian surfaces this is the only way light is reflected, all surfaces (except for perfect mirror surfaces) have some amount of this type of reflection. Phong approximates this component with the Lambertian model discussed above. Finally, the specular component is the portion of light that reflects mostly in the mirror direction. This type of reflection is prominent in shinny or glossy materials and as with diffuse reflection falls off relative to the cosine of the angle between the exiting direction and mirror direction. However, the falloff is exponential not linear. Phong's approximation for this effect is referred to as a cosine lobe (or sometimes a Phong lobe). The cosine of the angle between the mirror direction and viewing direction is raised to a power to achieve the exponential fall off. The higher the exponent the more distinct the specular highlight will be.
It is also worth noting that Blinn introduced an improvement to Phong's model by replacing the reflection direction (which is expensive to compute and difficult to interpolate) with the vector half-way between the viewer and the light source (which is less expensive). The cosine of the surface normal and this halfway vector is proportional to the cosine between the viewer and the mirror vector. By increasing the exponent on the specular component one can match the results achieved with the Phong model.
Physically Based Analytical BRDFs
The Phong model is not physically based and does not conserve energy. It simply attempts to reproduce the character and shape of reflected light in a form that is easy to compute. Other researchers have attempted to improve this models by closely examining the physical interactions that take place between the photons and the reflecting surface.
Perhaps the first attempt to bring a physically based model to computer graphics was undertaken by James Blinn in 1977(5). He examined the work of Torrance & Sparrow and developed a reflectance model based on their analytical model that conserves energy and gives markedly different results than the Phong model in certain viewing situations. This model is referred to as the Torrance-Sparrow model. This model was further refined into the Cook-Torrance model which accounts for the wavelength of light and can simulate effects like color shift(6).
Another such model was developed by Oren & Nayar(7). This model deals only with reflection from rough surfaces giving a nearly Lambertian reflection and as such is used as a replacement for the Lambertian model when more accuracy is needed.
A third physically based model was developed by Greg Ward and is particularly well known for its ability to account for anisotropic reflection (8). Anisotropic reflection occurs when light reflects in an asymmetric manner (i.e. with an eliptical cross-section instead of a circle). Ward's model has enjoyed longevity and wide use much like Phong's.
Statistical BRDFs
More recently we see a movement to analyze real-world reflectance and approximate the full BRDF using statistical techniques. The best known of these techniques comes from Lafortune et al.(9) and is known as the Lafortune model. This model uses different Phong lobes as basis functions and employs numerical methods to compute a set of coefficients for these bases to approximate the full BRDF.
Other approaches have employed different basis functions for a similar purpose. Spherical harmonics (the analog of sin and cos functions in spherical space) (10), wavelets(11) and even simple polynomials(12) have all been proposed for this purpose.
Controlling Reflectance Models
While accuracy and efficiency are important characteristics of a reflectance model it is also crucial to consider how easily the model can be controlled to produce a particular effect. As an example, one may try to adjust the parameters of the Cook-Torrance model to produce a metallic bronze appearance. This is very difficult. The parameters are connected to microscope features of the surface that are unnatural to adjust.
Westlund and Meyer addressed this problem by creating mappings between appearance industry standards and popular reflectance models(13). They developed controls that adjust 'gloss' and 'haze' by matching them with parameters of the Phong model, the Ward model and the Cook-Torrance model.
Statistical models present an even bigger problem. They do not have parameters that are related to physical attributes. Adjusting a particular coefficient may have unpredictable results and will vary from fitting to fitting as the underlying basis functions may change. The saving grace of these models is that they are based on real-world samples and, with the right equipment for sample a surface, new fittings can be produced for any available sample.
Finally, databases of parameters and fitting exist (like the Oregon BRDF library (14)) and are freely available on the internet to help alleviate the need to adjust these models.
Solving the Rendering Equation
Beyond approximating the BRDF, solving the rendering equation directly is (at the current state of technology) not possible. Kajiya examined many rendering techniques and explained how they can be posed as different ways of solving his equation. He discussed a Neumann series solution, the hidden surface removal solution (called the Utah approximation), the ray tracing approximation attributed to Turner Whitted(15), distributed ray tracing(16) (an application of the Monte Carlo integration technique) and the Radiosity approximation(17) (a form of finite element analysis). Kajiya goes on to describe a new way of solving his equation borrowed, like Radiosity, from the physics of heat transfer. His technique, which would be refined into the path tracing algorithm, solves the integral using Markov chains (another mathematical technique for integral approximation).
Since Kajiya's paper, other techniques have been developed that solve the equation in different ways. Many of these solutions have improved upon what Kajiya called the Utah approximation (today called scanline rendering). Scanline rendering has always opted for a local solution to the rendering equation as opposed to the global solutions that come from distributed ray tracing techniques. However, opting for global awareness at key points in the scanline calculation can greatly enhance the overall realism of the scene. Several techniques attempt to offer global awareness to the scanline algorithm. The Radiosity method is an example as are Precomputed Radiance Transfer (PRT) (18) and ambient occlusion (19).
Outline of Work
Note that I have changed the emphasis from dynamic implementation to static however I will include details about dynamic lighting as well in the event that this is a desirable proposal.
As the original GSoC project idea dealt with updating the static lighting application lighter2, this proposal will begin with static lighting. A static implementation can only effectively compute lighting for a single viewing angle. As such, lighting phenomona that depend on the viewing direction cannot be effectively computed. This includes mirror reflection phenomonea and, vicariously, specular contributions. As such, models like the Cook-Torrance and Ward models have limited applicability in a static lighting situation. Regardless, they may still be desired for completeness. To account for this I will propose two different work outlines, one with specular models and one without.
- Advanced Lighting with lighter2 (with specular models)
- The Oren-Nayer and Cook-Torrance models will be implemented for lighter2
- Internal changes in the code to indicate which model is to be used (basic/current model vs. advanced/ON & CT model)
- Simple extensions to both diffuse and specular calculations to support the new models
- Possible inclusion of some default parameters for basic material types available in Oregon BRDF lib and in publications
- Upon completion of the above, support for the Ward model will be added as well.
These three models (Oren-Nayer, Cook-Torrance & Ward) represent the current state-of-the-art in efficient, physically based rendering and are available in most commercial rendering systems. The addition of these three is most important and would constitute the bulk of work for this proposal.
If there is no desire for specular models in lighter2 then I would propose the following outline:
- Advanced Lighting with lighter2 (without specular models)
- The Oren-Nayer model will be implemented for lighter2
- Internal changes in the code to indicate which model is to be used (basic/current model vs. advanced/ON model)
- Simple extensions to both diffuse and specular calculations to support the new models
- Possible inclusion of some default parameters for basic material types available in Oregon BRDF lib and in publications
- Upon completion of the above, support for a modern globally aware rendering technique will be added (either Radiosity or Ambient Occlusion)
- Since lighter2 is a static lighting system basic ambient occlusion would be preferred to a screen space ambient occlusion algorithm which benefits dynamic, real-time implementation at the cost of reduced accuracy.
Since Oren-Nayer by itself would not be sufficient for a summer's worth of work the inclusion of Radiosity or Ambient Occlusion make this project of a reasonable scope.
Beyond the inclusion of these three models to the static lighter2 system (or perhaps in lieu of) dynamic implementations would also be possible. My own experience is with dynamic, shader-based implementations of lighting models and I would be amiss to not propose this if it is at all something the CS community desires:
- Shader based, advanced dynamic lighting modules
- The Oren-Nayer and Cook-Torrance models would be implemented using the CS shader system.
- As with lighter2, some internal changes will be needed to allow specifying the different parameters for these models.
- Again, basic material types can be included as are available in the research.
- Upon completion of the above two models one of two other improvements will be undertaken:
- Support for the LAFORTUNE model will be added.
- Support for screen space ambient occlusion will be added.
Here, the Lafortune model is desirable above the Ward model because of its rendering efficiency. A static implementation of Lafortune seems to go against its design but a dynamic implementation would exploit the flexibility of using Phong lobes (which are easily computed in the GPU) as basis functions. Similarly, screen-space ambient occlusion is ideal for dynamic implementation and is a popular component in game rendering today.
References
0) Dorsey, J., Rushmeier, H. E., & Sillion, F. X. (2008). Digital modeling of material appearance. Amsterdam ; Boston: Morgan Kaufmann/Elsevier. http://books.google.com/books?id=FCJZO9DSeXAC
1) Kajiya, J. T. 1986. The rendering equation. In Proceedings of the 13th Annual Conference on Computer Graphics and interactive Techniques D. C. Evans and R. J. Athay, Eds. SIGGRAPH '86. ACM, New York, NY, 143-150. DOI= http://doi.acm.org/10.1145/15922.15902
2) http://en.wikipedia.org/wiki/Brdf
3) http://en.wikipedia.org/wiki/Lambert's_cosine_law
4) Phong, B. T. 1975. Illumination for computer generated pictures. Commun. ACM 18, 6 (Jun. 1975), 311-317. DOI= http://doi.acm.org/10.1145/360825.360839
5) Blinn, J. F. 1977. Models of light reflection for computer synthesized pictures. In Proceedings of the 4th Annual Conference on Computer Graphics and interactive Techniques (San Jose, California, July 20 - 22, 1977). SIGGRAPH '77. ACM, New York, NY, 192-198. DOI= http://doi.acm.org/10.1145/563858.563893
6) Cook, R. L. and Torrance, K. E. 1981. A reflectance model for computer graphics. In Proceedings of the 8th Annual Conference on Computer Graphics and interactive Techniques (Dallas, Texas, United States, August 03 - 07, 1981). SIGGRAPH '81. ACM, New York, NY, 307-316. DOI= http://doi.acm.org/10.1145/800224.806819
7) Oren, M. and Nayar, S. K. 1994. Generalization of Lambert's reflectance model. In Proceedings of the 21st Annual Conference on Computer Graphics and interactive Techniques SIGGRAPH '94. ACM, New York, NY, 239-246. DOI= http://doi.acm.org/10.1145/192161.192213
8) Ward, G. J. 1992. Measuring and modeling anisotropic reflection. In Proceedings of the 19th Annual Conference on Computer Graphics and interactive Techniques J. J. Thomas, Ed. SIGGRAPH '92. ACM, New York, NY, 265-272. DOI= http://doi.acm.org/10.1145/133994.134078
9) Lafortune, E. P., Foo, S., Torrance, K. E., and Greenberg, D. P. 1997. Non-linear approximation of reflectance functions. InProceedings of the 24th Annual Conference on Computer Graphics and interactive Techniques International Conference on Computer Graphics and Interactive Techniques. ACM Press/Addison-Wesley Publishing Co., New York, NY, 117-126. DOI= http://doi.acm.org/10.1145/258734.258801
10) Westin, S. H., Arvo, J. R., and Torrance, K. E. 1992. Predicting reflectance functions from complex surfaces. SIGGRAPH Comput. Graph. 26, 2 (Jul. 1992), 255-264. DOI= http://doi.acm.org/10.1145/142920.134075
11) Claustres, L., Barthe, L., and Paulin, M. 2007. Wavelet encoding of BRDFs for real-time rendering. In Proceedings of Graphics interface 2007 (Montreal, Canada, May 28 - 30, 2007). GI '07, vol. 234. ACM, New York, NY, 169-176. DOI= http://doi.acm.org/10.1145/1268517.1268546
12) Malzbender, T., Gelb, D., and Wolters, H. 2001. Polynomial texture maps. In Proceedings of the 28th Annual Conference on Computer Graphics and interactive Techniques SIGGRAPH '01. ACM, New York, NY, 519-528. DOI= http://doi.acm.org/10.1145/383259.383320
13) Westlund, H. B. and Meyer, G. W. 2001. Applying appearance standards to light reflection models. In Proceedings of the 28th Annual Conference on Computer Graphics and interactive Techniques SIGGRAPH '01. ACM, New York, NY, 501-51. DOI= http://doi.acm.org/10.1145/383259.383318
14) http://math.nist.gov/~FHunt/appearance/obl.html
15) Whitted, T. 1980. An improved illumination model for shaded display. Commun. ACM 23, 6 (Jun. 1980), 343-349. DOI= http://doi.acm.org/10.1145/358876.358882
16) Cook, R. L., Porter, T., and Carpenter, L. 1984. Distributed ray tracing. In Proceedings of the 11th Annual Conference on Computer Graphics and interactive Techniques H. Christiansen, Ed. SIGGRAPH '84. ACM, New York, NY, 137-145. DOI= http://doi.acm.org/10.1145/800031.808590
17) Goral, C. M., Torrance, K. E., Greenberg, D. P., and Battaile, B. 1984. Modeling the interaction of light between diffuse surfaces. In Proceedings of the 11th Annual Conference on Computer Graphics and interactive Techniques H. Christiansen, Ed. SIGGRAPH '84. ACM, New York, NY, 213-222. DOI= http://doi.acm.org/10.1145/800031.808601
18) Sloan, P., Kautz, J., and Snyder, J. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In Proceedings of the 29th Annual Conference on Computer Graphics and interactive Techniques (San Antonio, Texas, July 23 - 26, 2002). SIGGRAPH '02. ACM, New York, NY, 527-536. DOI= http://doi.acm.org/10.1145/566570.566612
