Ambient occlusion vertex color

Vertex color can be used to store ambient occlusion (AO) lighting information.

Pros

Vertex color AO works well for meshes that use tiled textures, like environment models, because the tiling prevents unique lighting from being added to the textures. In this case texture-based AO would require a separate non-tiled texture using a second set of UVs, which would increase the texture memory and also increase the vertex count for the model (to store the 2nd set of UVs).

Cons

The downside of vertex color is that AO resolution is dependent on the complexity of the mesh. The less vertices there are, the less precise the shading will be. More edges can be added to the model, but it usually requires a tedious manual process.

Examples

CryEngine2 uses vertex color for baked ambient occlusion, see the Sandbox Manual.

Methods

3ds Max - GPU AO

GPU AO Vertex Color plugin by Yang Zhang is 22 times faster than CPU, but requires a Nvidia 8 series or later graphic card and 3ds Max 9 or later.

3ds Max - Pits & Peaks

Pits & Peaks Maxscript by Jeff Stewart was written for Max 3.1 but still works. May be unstable with larger meshes.

3ds Max - Radiosity

The simplest way to bake vertex color AO in 3ds Max is to use Radiosity. For additional methods, see the thread Baking Vertex Illumination without using radiosity on the Polycount forum.

1. Place skylight anywhere.

• Set the sky color to white.
• Turn on "Cast Shadows" (optional).

2. Render menu > Render

• Advance Lighting tab
• Advanced Lighting rollout, set to Radiosity
• Radiosity Processing Parameters Rollout:
  • Initial quality = 95
  • Refine Iterations (All Objects) = 1
  • Indirect Light Filter = 1
• Rendering Parameters:
  • Re-Use Direct Illumination from Radiosity Solution
• Radiosity Processing Parameters:
  • Start

3. Select all the geometry.

4. Command Panel > Utility Tab > More...

• Assign Vertex Colors
• Vertex Illum
• Lighting Only
• Color by Vertex
• Radiosity > Reuse Direct Illum. From Solution
• Assign to Selected

3ds Max - Render Surface Map

Surface map rendering tool included in 3dsmax 2010, found in the Rendering menu as "Render Surface Map". The tool was part of Polyboost but is now called Graphite Modeling Tools. This tool creates a bitmap instead of vertex color, however you can use the Vertex Paint modifier to convert any bitmap into vertex color.

FAOGen

FAOGen

Maya

Maya has a "batch bake (mental ray)" option, and you can choose to bake occlusion to a map, or directly to the vertex colors of the selected mesh. See this Digital Tutors tutorial.

xNormal

The latest versions of xNormal (3.14 or above) can bake the AO to a map or also to vertex colors. The xNormal default software renderer only renders maps. The GPU AO tool can render ao maps or per-vertex ao. If you want to bake it to vertex colors you need to look for the "Simple GPU AO Tool section".

If you press the "generate" button with the "image" tab button selected will render the AO to the output image you specify. On the other hand, if you press the "generate" button with the per-vertex tab selected it will render the AO per vertex into a .SBM or .OVB file (which can be seen in the xNormal 3D interactive viewer or imported into a 3d engine).

---