Normal map
A Normal Map texture is usually used to add detail to a polygonal surface.
The red, green, and blue channels of the texture are used to control the direction of each pixel's normal. The pixels in the normal map basically control how much lighting each pixel will receive.
A normal map is also known as a dotproduct3 bump map.
Normal maps come in two basic flavors: Object Space or Tangent Space.
RGB Channels
If you see lighting coming from the wrong angle when you're looking at your model in the game engine, and the model is using tangent-space normal maps, the game renderer might be expecting the red or green channel (or both) to point in the opposite direction. To fix this invert the appropriate channels in an image editor, so that the black pixels become white and the white pixels become black.
Games use different shader techniques to render tangent-space normal maps, but the normal map directions are usually consistent within the game. The red channel of a tangent-space normal map points the normals predominantly leftwards or rightwards (depending on the game), the green channel points the normals predominantly upwards or downwards (depending on the game), and the blue channel points the normals outwards away from the surface (all games).
UV Coordinates
Mirroring normal maps can be done but requires careful processing. You must mirror your UVs before capturing the normal map from the high-res mesh.
If you change the UV layout you should re-capture the normal map, because rotating or mirroring UVs after creating the normal map will cause the existing tangents/bitangents to become invalid. This special tangent/bitangent vertex data is calculated using the UV layout at normal map-capture time, so changing the UVs afterwards isn't a good idea.
If you want to mirror the UVs, or you want to reuse parts of the normal map by overlaying multiple bits in the same UV space, then simply move all those overlapped/mirrored bits one unit over on the U axis before you capture the normal map. Only one copy of the forward-facing UVs should remain in the 0-1 UV box at baking time.
Normal map-baking tools will only capture normals within the 0-1 UV box, any UV bits outside this area are ignored. If you move all the overlaps exactly 1 UV unit away (to the side or up or down doesn't matter) then you can leave them there after the bake and they will be mapped correctly. Or you can move them back if you want. Doesn't matter to the game engine.
Tangents and Bitangents
The normal map baking process doesn't just create a map... it also creates vertex data called tangents and bitangents. This vertex data is used along with the normal map to properly transform the incoming light, so your model will be lit correctly on both sides. This special tangent/bitangent vertex data is calculated using the UV layout, so changing the UVs after baking the map isn't a good idea.
The renderer (or game engine) must be aware of these tangents when it renders your model, otherwise you'll get incorrect lighting on one side of the model (in fact you'll get seams all over the place without it).
For example, 3ds Max can only render correct lighting in its scanline renderer if you use its own Render To Texture tool to create the normal map, because the Max renderer only understands its own tangents & bitangents.
Similarly a game engine can only render correct lighting when it uses the same tangent/bitangent data that was created during the normal map process.
3ds Max can properly light mirrored normal maps in the viewport if you use special shaders, like Ben Cloward's HLSL shaders or others.
Re-normalizing
Add info here: normal length = 1.0, specular artifacts, mipping problems.
If the game shader doesn't re-normalize the normal map, you can alter the blue channel for some interesting effects. If the blue channel is inverted, the normals will be pointing to the opposite side of the surface, which can simulate backlighting. If the blue channel is darkened towards black, the surface will recieve less light. An ambient occlusion pass can be multiplied onto the blue channel to store ambient occlusion in the normal map, because it is shortening the normals in the crevices of the surface.
Object Space vs. Tangent Space
Object-space normal map
- Rainbow colors.
- Faster performance than a tangent-space map.
- Object can rotate, but shouldn't be deformed. Good for rotating game elements (weapons, doors, vehicles, etc.).
- Unique map for each mesh shape.
- Easier to generate from a high-poly mesh than a tangent-space map. Harder to overlay painted details.
Tangent-space normal map
- Predominantly-blue colors.
- Slower performance than an object-space map, but not by much.
- Object can rotate and deform. Good for deforming meshes (characters, animals, flags, etc.)
- Map can be reused by different mesh shapes, can also be tiled.
