Normal Map Technical Details
Normal maps can be improved greatly by learning the implementation details.
Contents
Synched Workflow
To eliminate seams and shading artifacts, the game engine and the normal map baking tool should use the same tangent basis.
For more information about a synched workflow see the Polycount Forum thread You're making me hard. Making sense of hard edges, uvs, normal maps and vertex counts.
When you export a model from your baking tool, it is best to choose a format like FBX which can store the tangents. This insures the same tangents which were used for baking, will be used for rendering.
Handplane is a free utility that converts an object-space normal map + model into different tangent spaces. This allows a synced workflow with various tools.
- Synced tangent basis workflow from 3ds Max to Unity 5? Polycount Forum thread.
- Is my tangent basis un-synced? Polycount Forum thread, using Maya and Substance Painter.
| Renderer | Normal map baker | Handplane support |
|---|---|---|
| 3d Coat | ? | ? |
| 3ds Max | 3ds Max | Yes |
| Blender | ? | ? |
| Creation Engine | ? | Yes |
| CryEngine | ? | ? |
| Knald | ? | ? |
| Marmoset Toolbag | 3ds Max, Maya, Xnormal | ? |
| Maya | Maya | Yes |
| Source | Maya ([1]) | Yes |
| Starcraft II | ? | Yes |
| Substance Designer | ? | ? |
| Unity ([2], [3]) | Xnormal | Yes |
| Unreal Engine 4 | Xnormal ([4]) | Yes |
| Xnormal | Xnormal | ? |
Tangent-Space vs. Object-Space
Normal maps can be made in either of two basic flavors: tangent-space or object-space. World-space is basically the same as object-space, except it requires the model to remain in its original orientation, neither rotating nor deforming, so it's almost never used.
Tangent-space normal map
Image by Eric Chadwick.
Predominantly-blue colors. Object can rotate and deform. Good for deforming meshes, like characters, animals, flags, etc.
Pros:
- Maps can be reused easily, like on differently-shaped meshes.
- Maps can be tiled and mirrored easily, though some games might not support mirroring very well.
- Easier to overlay painted details.
- Easier to use image compression.
Cons:
- More difficult to avoid smoothing problems from the low-poly vertex normals (see Smoothing Groups and Hard Edges).
- Slightly slower performance than an object-space map (but not by much).
Object-space normal map
Image by Eric Chadwick.
Rainbow colors. Objects can rotate, but usually shouldn't be deformed, unless the shader has been modified to support deformation. Object-space is also called local-space or model-space.
Pros:
- Easier to generate high-quality curvature because it completely ignores the crude smoothing of the low-poly vertex normals.
- Slightly better performance than a tangent-space map (but not by much).
Cons:
- Can't easily reuse maps, different mesh shapes require unique maps.
- Difficult to tile properly, and mirroring requires specific shader support.
- Harder to overlay painted details because the base colors vary across the surface of the mesh. Painted details must be converted into Object Space to be combined properly with the OS map.
- They don't compress very well, since the blue channel can't be recreated in the shader like with tangent-space maps. Also the three color channels contain very different data which doesn't compress well, creating many artifacts. Using a half-resolution object-space map is one option.
Another way to explain these two types is to show how the spaces are oriented:
Image by Gerben Pasjes.
Converting Between Spaces
Normal maps can be converted between object space and tangent space, in order to use them with different blending tools and shaders, which require one type or the other.
Object space maps can also be converted to maps with different tangent bases, to better match the normal maps with the renderer and thus avoid lighting errors.
- Handplane by Luke Hodorowicz and Alec Moody is a tool that converts object space maps into a variety of tangent spaces: 3ds Max, Creation Engine, Maya, Source, Unity, Unreal, etc. See the Polycount Forum thread Official handplane support thread - Now freeware!!.
Image by Alec Moody.
- NSpace by Diogo "fozi" Teixeira is a tool that converts an object-space normal map into a tangent-space map, which then works seamlessly in the 3ds Max viewport. He converts the map by using the same tangent basis that 3ds Max uses for its hardware shader. To see the results, load the converted map via the Normal Bump map and enable "Show Hardware Map in Viewport". Osman "osman" Tsjardiwal created a GUI for NSpace, you can download it here, just put it in the same folder as the NSpace exe and run it.
Joe "EarthQuake" Wilson said: "[8Monkey Labs has] a tool that lets you load up your reference mesh and object space map. Then load up your tangent normals, and adjust some sliders for things like tile and amount. We need to load up a mesh to know how to correctly orient the tangent normals or else things will come out upside down or reverse etc. It mostly works, but it tends to "bend" the resulting normals, so you gotta split the mesh up into some smoothing groups before you run it, and then I usually will just composite this "combo" texture over my orig map in Photoshop."
RGB Channels
Shaders can use different techniques to render tangent-space normal maps, but the normal map directions are usually consistent within a game. Usually the red channel of a tangent-space normal map stores the X axis (pointing the normals predominantly leftwards or rightwards), the green channel stores the Y axis (pointing the normals predominantly upwards or downwards), and the blue channel stores the Z axis (pointing the normals outwards away from the surface).
Image by Eric Chadwick.
If you see lighting coming from the wrong angle when you're looking at your normal-mapped model, and the model is using a tangent-space normal map, the normal map shader might be expecting the red or green channel (or both) to point in the opposite direction. To fix this either change the shader, or simply invert the appropriate color channels in an image editor, so that the black pixels become white and the white pixels become black.
Some shaders expect the color channels to be swapped or re-arranged to work with a particular compression format. This re-arranging of the normal map axes is called swizzling. For example the DXT5_nm format usually expects the X axis to be in the alpha channel, the Y axis to be in the green channel, and the red and blue channels to be empty.
Tangent Basis
Tangent-space normal maps use a special kind of vertex data called the tangent basis. This is similar to UV coordinates except it provides directionality across the surface. It's a surface-relative coordinate system for the per-pixel normals stored in the normal map, so lighting can be applied to the normal-mapped surface.
Light rays are in world space, but the normals stored in the normal map are in tangent space. When the model is being rendered, the light rays must be converted from world space into tangent space, using the tangent basis to get there. At that point the incoming light rays are compared against the directions of the normals in the normal map, and this determines how much each pixel is going to be lit. Alternatively, instead of converting the light rays some shaders will convert the normals in the normal map from tangent space into world space. Then those world-space normals are compared against the light rays, and the model is lit appropriately. The method depends on who wrote the shader, but the end result is the same. Both methods require a tangent basis to transform the lighting.
When a triangle's vertex normals are pointing straight out, and a pixel in the normal map is neutral blue (128,128,255) this means the pixel's normal will be pointing straight out from the surface of the low-poly mesh. When that pixel normal is tilted towards the left or the right in the tangent coordinate space, it will get either more or less red color, depending on whether the normal map is set to store the X axis as either a positive or a negative value. Same goes for when the normal is tilted up or down in tangent space, it will either get more or less green color. If the vertex normals aren't exactly perpendicular to the triangle, the normal map pixels will be tinted away from neutral blue as well.
Each vertex in the tangent basis is a combination of three things: the mesh vertex's normal (influenced by smoothing), the vertex's tangent (usually derived from the V texture coordinate), and the vertex's bitangent (derived in code, also called the binormal). These three vectors create an axis for each vertex, giving it a specific orientation in the tangent space. These axes are used to properly transform the incoming lighting from world space into tangent space, so your normal-mapped model will be lit correctly.
Unfortunately for artists, there are many different ways to calculate the tangent basis: MikkTspace is the most common, but many others exist: 3ds Max, DirectX 9, Eric Lengyel, Maya, NVMeshMender, etc. This means a normal map baked in one application probably won't shade correctly in another. Artists must do some testing with different baking tools to find which works best with their output. When the renderer (or game engine) renders your game model, the shader must use the same tangent basis as the normal map baker, otherwise you'll get incorrect lighting, especially across the seams between UV shells.
The xNormal SDK supports custom tangent basis methods. When a programmer uses it to implement their renderer's own tangent basis, artists can then use Xnormal to bake normal maps that will match their renderer perfectly.
UV Coordinates
Image by Eric Chadwick.
When you look at a tangent-space normal map for a character, you typically see different colors along the UV seams. This is because the UV shells are often oriented at different angles on the mesh, a necessary evil when translating the 3D mesh into 2D textures. The body might be mapped with a vertical shell, and the arm mapped with a horizontal one. This requires the normals in the normal map to be twisted for the different orientations of those UV shells. The UVs are twisted, so the normals must be twisted in order to compensate. The tangent basis helps reorient (twist) the lighting as it comes into the surface's local space, so the lighting will then look uniform across the normal mapped mesh.
When an artist tiles a tangent-space normal map across an arbitrary mesh, like a landscape, this tends to shade correctly because the mesh has a uniform direction in tangent space. If the mesh has discontinuous UV coordinates (UV seams), or the normal map has large directional gradients across it, the tangent space won't be uniform anymore so the surface will probably have shading seams.
Common Swizzle Coordinates
3D Software capable of displaying tangent space normal maps will have a native required direction, or "handedness" for the RGB channels in a normal map, sometimes referred to as "Swizzle Coordinates," though Shaders can often be written to override this native handedness. You may hear developers refer to "flipping the green channel" in order to get a normal map to display correctly, and this simply indicates that when the normal map was baked, it was authored with the incorrect handedness in the green channel. Right handedness, which coincides with OpenGL is indicated with a plus sign (ex. +Y), whereas Left handedness, which coincides with DirectX, is indicated with a negative sign (ex. -Y)
| Software | Red | Green | Blue |
| 3ds Max | X+ | Y- | Z+ |
| Blender | X+ | Y+ | Z+ |
| CryENGINE | X+ | Y- | Z+ |
| Maya | X+ | Y+ | Z+ |
| Modo | X+ | Y+ | Z+ |
| Source | X+ | Y- | Z+ |
| Toolbag | X+ | Y+ | Z+ |
| Unity | X+ | Y+ | Z+ |
| Unreal Engine | X+ | Y- | Z+ |
