Normal map

From polycount
Revision as of 11:35, 6 April 2015 by EricChadwick (Talk | contribs)

Jump to: navigation, search

Out of Date

The information on this page is a bit old, it needs to be updated. See these links for more current information:

Synched Workflow

To eliminate seams and shading artifacts, the model renderer and the normal map baking tool should use the same tangent basis.

FBX model format can store tangents, and some renderers use this data to synch the incoming normal maps.

Renderer Normal map baker
3ds Max 3ds Max, Handplane, ?
Blender  ?
Creation Engine Handplane, ?
CryEngine  ?
Knald  ?
Marmoset Toolbag 3ds Max, Maya, Xnormal, ?
Maya Handplane, Maya, ?
Source Handplane, Maya ([1]), ?
Starcraft II Handplane, ?
Substance Designer  ?
Unity([2]) Handplane, Xnormal (via Asset Store), ?
Unreal Engine 4 Handplane, Xnormal([3]), ?
Xnormal Xnormal, ?

What is a Normal Map?

A Normal Map is usually used to fake high-res geometry detail when it's mapped onto a low-res mesh. The pixels of the normal map each store a normal, a vector that describes the surface slope of the original high-res mesh at that point. The red, green, and blue channels of the normal map are used to control the direction of each pixel's normal.

When a normal map is applied to a low-poly mesh, the texture pixels control the direction each of the pixels on the low-poly mesh will be facing in 3D space, creating the illusion of more surface detail or better curvature. However, the silhouette of the model doesn't change.

  • A model with a normal map.
    Image by James Ku.

  • The model without its normal map.
    Image by James Ku.

  • The high-resolution model used to create the normal map.
    Image by James Ku.

Tangent-Space vs. Object-Space

Normal maps can be made in either of two basic flavors: tangent-space or object-space. Object-space is also called local-space or model-space, same thing. World-space is basically the same as object-space, except it requires the model to remain in its original orientation, neither rotating nor deforming, so it's almost never used.

Tangent-space normal map

A tangent-space normal map.
Image by Eric Chadwick.

Predominantly-blue colors. Object can rotate and deform. Good for deforming meshes, like characters, animals, flags, etc.

Pros:

  • Maps can be reused easily, like on differently-shaped meshes.
  • Maps can be tiled and mirrored easily, though some games might not support mirroring very well.
  • Easier to overlay painted details.
  • Easier to use image compression.

Cons:

  • More difficult to avoid smoothing problems from the low-poly vertex normals (see Smoothing Groups and Hard Edges).
  • Slightly slower performance than an object-space map (but not by much).

Object-space normal map

An object-space normal map.
Image by Eric Chadwick.

Rainbow colors. Objects can rotate, but usually shouldn't be deformed, unless the shader has been modified to support deformation.

Pros:

  • Easier to generate high-quality curvature because it completely ignores the crude smoothing of the low-poly vertex normals.
  • Slightly better performance than a tangent-space map (but not by much).

Cons:

  • Can't easily reuse maps, different mesh shapes require unique maps.
  • Difficult to tile properly, and mirroring requires specific shader support.
  • Harder to overlay painted details because the base colors vary across the surface of the mesh. Painted details must be converted into Object Space to be combined properly with the OS map.
  • They don't compress very well, since the blue channel can't be recreated in the shader like with tangent-space maps. Also the three color channels contain very different data which doesn't compress well, creating many artifacts. Using a half-resolution object-space map is one option.


Converting Between Spaces

Normal maps can be converted between tangent space and object space, in order to use them with different blending tools and shaders, which require one type or the other.

Diogo "fozi" Teixeira created a tool called NSpace that converts an object-space normal map into a tangent-space map, which then works seamlessly in the Max viewport. He converts the map by using the same tangent basis that 3ds Max uses for its hardware shader. To see the results, load the converted map via the Normal Bump map and enable "Show Hardware Map in Viewport". Osman "osman" Tsjardiwal created a GUI for NSpace, you can download it here, just put it in the same folder as the NSpace exe and run it. Diogo has further plans for the tool as well.

NSpace interface.
Image by Diogo "fozi" Teixeira and Osman "osman" Tsjardiwal

Joe "EarthQuake" Wilson said: "[8Monkey Labs has] a tool that lets you load up your reference mesh and object space map. Then load up your tangent normals, and adjust some sliders for things like tile and amount. We need to load up a mesh to know how to correctly orient the tangent normals or else things will come out upside down or reverse etc. It mostly works, but it tends to "bend" the resulting normals, so you gotta split the mesh up into some smoothing groups before you run it, and then I usually will just composite this "combo" texture over my orig map in Photoshop."

RGB Channels

Shaders can use different techniques to render tangent-space normal maps, but the normal map directions are usually consistent within a game. Usually the red channel of a tangent-space normal map stores the X axis (pointing the normals predominantly leftwards or rightwards), the green channel stores the Y axis (pointing the normals predominantly upwards or downwards), and the blue channel stores the Z axis (pointing the normals outwards away from the surface).

The red, green, and blue channels of a tangent-space normal map.
Image by Eric Chadwick.

If you see lighting coming from the wrong angle when you're looking at your normal-mapped model, and the model is using a tangent-space normal map, the normal map shader might be expecting the red or green channel (or both) to point in the opposite direction. To fix this either change the shader, or simply invert the appropriate color channels in an image editor, so that the black pixels become white and the white pixels become black.

Some shaders expect the color channels to be swapped or re-arranged to work with a particular compression format. For example the DXT5_nm format usually expects the X axis to be in the alpha channel, the Y axis to be in the green channel, and the red and blue channels to be empty.

Tangent Basis

Tangent-space normal maps use a special kind of vertex data called the tangent basis. This is similar to UV coordinates except it provides directionality across the surface, it forms a surface-relative coordinate system for the per-pixel normals stored in the normal map.

Light rays are in world space, but the normals stored in the normal map are in tangent space. When a normal-mapped model is being rendered, the light rays must be converted from world space into tangent space, using the tangent basis to get there. At that point the incoming light rays are compared against the directions of the normals in the normal map, and this determines how much each pixel of the mesh is going to be lit. Alternatively, instead of converting the light rays some shaders will convert the normals in the normal map from tangent space into world space. Then those world-space normals are compared against the light rays, and the model is lit appropriately. The method depends on who wrote the shader, but the end result is the same.

Unfortunately for artists, there are many different ways to calculate the tangent basis: 3ds Max, Maya, DirectX 9, NVMeshMender, Eric Lengyel, a custom solution, etc. This means a normal map baked in one application probably won't shade correctly in another. Artists must do some testing with different baking tools to find which works best with their output. When the renderer (or game engine) renders your game model, the shader must use the same tangent basis as the normal map baker, otherwise you'll get incorrect lighting, especially across the seams between UV shells.

The xNormal SDK supports custom tangent basis methods. When a programmer uses it to implement their renderer's own tangent basis, artists can then use Xnormal to bake normal maps that will match their renderer perfectly.

The UVs and the vertex normals on the low-res mesh directly influence the coloring of a tangent-space normal map when it is baked. Each tangent basis vertex is a combination of three things: the mesh vertex's normal (influenced by smoothing), the vertex's tangent (usually derived from the V texture coordinate), and the vertex's bitangent (derived in code, also called the binormal). These three vectors create an axis for each vertex, giving it a specific orientation in the tangent space. These axes are used to properly transform the incoming lighting from world space into tangent space, so your normal-mapped model will be lit correctly.

When a triangle's vertex normals are pointing straight out, and a pixel in the normal map is neutral blue (128,128,255) this means that pixel's normal will be pointing straight out from the surface of the low-poly mesh. When that pixel normal is tilted towards the left or the right in the tangent coordinate space, it will get either more or less red color, depending on whether the normal map is set to store the X axis as either a positive or a negative value. Same goes for when the normal is tilted up or down in tangent space, it will either get more or less green color. If the vertex normals aren't exactly perpendicular to the triangle, the normal map pixels will be tinted away from neutral blue as well. The vertex normals and the pixel normals in the normal map are combined together to create the final per-pixel surface normals.

Shaders are written to use a particular direction or "handedness" for the X and Y axes in a normal map. Most apps tend to prefer +X (red facing right) and +Y (green facing up), while others like 3ds Max prefer +X and -Y. This is why you often need to invert the green channel of a normal map to get it to render correctly in this or that app... the shader is expecting a particular handedness.

When shared edges are at different angles in UV space, different colors will show up along the seam. The tangent basis uses these colors to light the model properly.
Image by Eric Chadwick.

When you look at a tangent-space normal map for a character, you typically see different colors along the UV seams. This is because the UV shells are often oriented at different angles on the mesh, a necessary evil when translating the 3D mesh into 2D textures. The body might be mapped with a vertical shell, and the arm mapped with a horizontal one. This requires the normals in the normal map to be twisted for the different orientations of those UV shells. The UVs are twisted, so the normals must be twisted in order to compensate. The tangent basis helps reorient (twist) the lighting as it comes into the surface's local space, so the lighting will then look uniform across the normal mapped mesh.

When an artist tiles a tangent-space normal map across an arbitrary mesh, like a landscape, this tends to shade correctly because the mesh has a uniform direction in tangent space. If the mesh has discontinuous UV coordinates (UV seams), or the normal map has large directional gradients across it, the tangent space won't be uniform anymore so the surface will probably have shading seams.

Modeling the Low-Poly Mesh

The in-game mesh usually needs to be carefully optimized to create a good silhouette, define edge-loops for better deformation, and minimize extreme changes between the vertex normals for better shading (see Smoothing Groups & Hard Edges).

In order to create an optimized in-game mesh including a good silhouette and loops for deforming in animation, you can start with the 2nd subdivision level of your digital sculpt, or in some cases with the base mesh itself. Then you can just collapse edge loops or cut in new edges to add/remove detail as necessary. Or you can re-toplogize from scratch if that works better for you.

See You're making me hard. Making sense of hard edges, uvs, normal maps and vertex counts on the Polycount forum.

UV Coordinates

Normal map baking tools only capture normals within the 0-1 UV square, any UV bits outside this area are ignored.

Only one copy of the forward-facing UVs should remain in the 0-1 UV square at baking time. If the mesh uses overlapping UVs, this will likely cause artifacts to appear in the baked map, since the baker will try render each UV shell into the map. Before baking, it's best to move all the overlaps and mirrored bits outside the 0-1 square.

The mirrored UVs (in red) are offset 1 unit before baking.
Image by Eric Chadwick.

If you move all the overlaps and mirrored bits exactly 1 UV unit (any whole number will do), then you can leave them there after the bake and they will still be mapped correctly. You can move them back if you want, it doesn't matter to most game engines. Be aware that ZBrush does use UV offsets to manage mesh visibility, however this usually doesn't matter because the ZBrush cage mesh is often a different mesh than the in-game mesh used for baking.

You should avoid changing the UVs after baking the normal map, because rotating or mirroring UVs after baking will cause the normal map not to match the tangent basis anymore, which will likely cause lighting problems.

In 3ds Max, W is a third texture coordinate. It's used for 3D procedural textures and for storing vertex color in UV channels (you need 3 axes for RGB, so UVW can store vertex color). Bake problems can be avoided by moving any overlapping UVs to -1 on the W axis, with the same results as moving them 1 unit on the U or V axes. The tool Render To Texture will always bake whatever UVs are the highest along the W axis. However using W can be messy... it's generally hidden unless you purposefully look for it (bad for team work), doesn't get preserved on export to other apps, and high W values can prevent selecting and/or welding UVs.

Mirroring

Normal maps can be mirrored across a model to create symmetrical details, and save UV space, which allows more detail in the normal map since the texture pixels are smaller on the model.

With object-space maps, mirroring requires specific shader support. For tangent-space maps, mirroring typically creates a shading seam, but this can be reduced or hidden altogether, depending on the method used.

Typical Mirroring Workflow

  1. Delete the mesh half that will be mirrored.
  2. Arrange the UVs for the remaining model, filling the UV square.
  3. Mirror the model to create a "whole" mesh, welding the mesh vertices along the seam.
  4. Move the mirrored UVs exactly 1 unit (or any whole number) out of the 0-1 UV square.
  5. Bake the normal map.

Sometimes an artist will decide to delete half of a symmetrical model before baking.

This is a mistake however because often the vertex normals along the hole will bend towards the hole a bit; there are no faces on the other side to average the normals with. This will create a strong lighting seam in the normal map.

It's typically best to use the complete mirrored model to bake the normal map, not just the unique half.

To prevent the mirrored UVs from causing overlaps or baking errors, move the mirrored UVs out of the 0-1 UV space, so only one copy of the non-mirrored UVs is left within the 0-1 square.

To avoid texel "leaks" between the UV shells, make sure there's enough Edge Padding around each shell, including along the edges of the normal map. None of the UV shells should be touching the edge of the 0-1 UV square, unless they're meant to tile with the other side of the map.

Center Mirroring

If the mirror seam runs along the surface of a continuous mesh, like down the center of a human face for example, then it will probably create a lighting seam.

In Epic Games' Unreal Engine 3 (UE3) their symmetrical models commonly use centered mirroring. Epic uses materials that mix a DetailMap with the normal maps; these seem to scatter the diffuse/specular lighting and help minimize the obviousness of the mirror seams. For their Light Mapped models they use a technique that can almost completely hide the mirror seam.

In UE3 a center mirror seam is reduced by using a detail normal map.
Image by "Epic Games"

GOW2 normal map seams, UDK normal map seams

Offset Mirroring

Offset mirroring is a method where you move the mirror seam off to one side of the model, so the seam doesn't run exactly down the center. For example with a character's head, the UV seam can go down along the side of the head in front of the ear. The UV shell for the nearest ear can then be mirrored to use the area on the other side of the head.

This avoids the "Rorschach" effect and allows non-symmetrical details, but it still saves texture space because the two sides of the head can be mirrored (they're never seen at the same time anyhow).

Offset mirroring doesn't get rid of the seam, but it does move it off to a place where it can either be less obvious, or where it can be hidden in a natural seam on the model.

Flat Color Mirroring

Tutorial for painting out seams on mirrored tangent-space normal maps by warby solves seams by painting a flat set of normals along the seam, using neutral blue (128,128,255). However it only works along horizontal or vertical UV seams, not across any angled UVs. It also removes any details along the mirror seam, creating blank areas.

Element Mirroring

The mirror seam can be avoided completely when it doesn't run directly through any mesh. For example if there's a detached mesh element that runs down the center of the model, this can be uniquely mapped, while the meshes on either side can be mirrors of each other. Whenever the mirrored parts don't share any vertex normals with the non-mirrored parts, there won't be any seams.

The middle part (highlighted in red) uses unique non-mirrored UVs, allowing the mesh on the right to be mirrored without any seams.
Image by "racer445"

Smoothing Groups & Hard Edges

Each vertex in a mesh has at least one vertex normal. Vertex normals are used to control the direction a triangle will be lit from; if the normal is facing the light the triangle will be fully lit, if facing away from the light the triangle won't be lit.

Each vertex however can have more than one vertex normal. When two triangles have different vertex normals along their shared edge, this creates a shading seam, called a hard edge in most modeling tools. 3ds Max uses Smoothing Groups to create hard/soft edges, Maya uses Harden Edge and Soften Edge. These tools create hard and soft edges by splitting and combining the vertex normals.

Hard edges occur where the vertices have multiple normals.
Image by Ben 'poopinmymouth' Mathis (tutorial here)

When a mesh uses all soft normals (a single smoothing group) the lighting has to be interpolated across the extreme differences between the vertex normals. If your renderer doesn't support the same tangent basis that the baker uses, this can produce extreme shading differences across the model, which creates shading artifacts. It is generally best to reduce these extremes when you can because a mismatched renderer can only do so much to counteract it.

Hard edges are usually best where the model already has a natural seam. For example, you can add a hard edge along the rim of a car's wheel well, to prevent the inside of the wheel well from distorting the shading for the outside of the car body. Mechanical models usually need hard edges where ever the surface bends more than about 45 degrees.

For most meshes, the best results usually come from adding hard edges where ever there are UV seams. There are no hard rules however, you must experiment with different approaches to find what works best in your game.

When you use object-space normal maps the vertex normal problem goes away since you're no longer relying on the crude vertex normals of the mesh. An object-space normal map completely ignores vertex normals. Object-space mapping allows you to use all soft edges and no bevels on the low-res mesh, without showing lighting errors.

Hard Edge Discussions & Tutorials

Using Bevels

Bevels/chamfers generally improve the silhouette of the model, and can also help reflect specular highlights better.

However bevels tend to produce long thin triangles, which slow down the in-game rendering of your model. Real-time renderers have trouble rendering long thin triangles because they create a lot of sub-pixel areas to render.

Bevels also balloon the vertex count, which can increase the transform cost and memory usage. Hard edges increase the vertex count too, but not when the edge also shares a seam in UV space. For a good explanation of the vertex count issue, see Beautiful, Yet Friendly.

Using hard edges with matching UV shells tends to give better performance and better cosmetic results than using bevels. However there are differing opinions on this, see the Polycount thread "Maya transfer maps help".

Edited Vertex Normals

If you use bevels the shading will be improved by editing the vertex normals so the larger flat surfaces have perpendicular normals. The vertex normals are then forced to blend across the smaller bevel faces, instead of across the larger faces. See the Polycount thread Superspecular soft edges tutorial chapter 1.

Bending normals on bevelled models.
From the tutorial Shading techniques Superspecular soft edges
Image by Paolo Oliverio

Level of Detail Models

See Problem if you're using 3point-style normals with an LOD.

Modeling The High-Poly Mesh

Subdivision Surface Modeling and DigitalSculpting are the techniques most often used for modeling a normal map.

Some artists prefer to model the in-game mesh first, other artists prefer to model the high-res mesh first, and others start somewhere in the middle. The modeling order is ultimately a personal choice though, all three methods can produce excellent results:

  • Build the in-game model, then up-res it and sculpt it.
  • Build and sculpt a high resolution model, then build a new in-game model around that.
  • Build a basemesh model, up-res and sculpt it, then step down a few levels of detail and use that as a base for building a better in-game mesh.

If the in-game mesh is started from one of the subdivision levels of the basemesh sculpt, various edge loops can be collapsed or new edges can be cut to add/remove detail as necessary.

Sloped Extrusions

Extrusions on the high-poly model should be sloped to make a better normal map.
Image by Krzysztof "Hatred" Dolas.

Floating Geometry

Normal map stores the direction the surface is facing rather than real depth information, thus allowing to save time using floating geometry.
To correctly bake AO with floating geo make it a separate object and turn off it's shadow casting.
Image by Andrew "d1ver" Maximov.

See also Modeling High/Low Poly Models for Next Gen Games by João "Masakari" Costa

Edge Thickness

When creating edges of the Highpoly, sometimes you'll need to make them rounded than in real life to
work better at the size they will be seen.
Image by "racer445" Herbert

mental ray Round Corners Bump

The mental ray renderer offers an automatic bevel rendering effect called Round Corners Bump that can be baked into a normal map. This is available in 3ds Max, Maya, and XSI. See Zero Effort Beveling for normal maps - by Robert "r_fletch_r" Fletcher.

Jeff Patton posted about how to expose Round Corners Bump in 3ds Max so you can use it in other materials.

Michael "cryrid" Taylor posted a tutorial about how to use Round Corners in XSI.

XSI is able to bake a good normal map with it, but 3ds Max seems to bake it incorrectly, and Maya isn't able to bake the effect at all. Maybe Max might be able to bake it correctly, if the .mi shader is edited to use the correct coordinate space?

Baking

The process of transferring normals from the high-res model to the in-game model is often called baking. The baking tool usually starts projecting a certain numerical distance out from the low-poly mesh, and sends rays inwards towards the high-poly mesh. When a ray intersects the high-poly mesh, it records the mesh's surface normal and saves it in the normal map.

To get an understanding of how all the options affect your normal map, do some test bakes on simple meshes like boxes. They generate quickly so you can experiment with UV mirroring, smoothing groups, etc. This helps you learn the settings that really matter.

Baking sub-sections:

  1. Anti-Aliasing
  2. Baking Transparency
  3. Edge Padding
  4. High Poly Materials
  5. Reset Transforms
  6. Solving Intersections
  7. Solving Pixel Artifacts
  8. Solving Wavy Lines
  9. Triangulating
  10. Working with Cages

Anti-Aliasing

Turning on super-sampling or anti-aliasing (or whatever multi-ray casting is called in your normal map baking tool) will help to fix any jagged edges where the high-res model overlaps itself within the UV borders of the low-poly mesh, or wherever the background shows through holes in the mesh. Unfortunately this tends to render much much slower, and takes more memory.

A bake without anti-aliasing shows artifacts where the high-poly mesh has overlaps.
Image by 'knak47'

One trick to speed this up is to render 2x the intended image size then scale the normal map down 1/2 in a paint program like Photoshop. The reduction's pixel resampling will add anti-aliasing for you in a very quick process. After scaling, make sure to re-normalize the map if your game doesn't do that already, because the un-normalized pixels in your normalmap may cause pixelly artifacts in your specular highlights. Re-normalizing can be done with NVIDIA's normal map filter for Photoshop.

3ds Max's supersampling doesn't work nicely with edge padding, it produces dark streaks in the padded pixels. If so then turn off padding and re-do the padding later, either by re-baking without supersampling or by using a Photoshop filter like the one that comes with Xnormal.

Baking Transparency

Sometimes you need to bake a normal map from an object that uses opacity maps, like a branch with opacity-mapped leaves. Unfortunately baking apps often completely ignore any transparency mapping on your high-poly mesh.

JoeWilson ivynormals error.jpg JoeWilson ivynormals rendered.jpg
3ds Max's RTT baker causes transparency errors.
image by Joe "EarthQuake" Wilson 		The lighting method bakes perfect transparency.
image by Joe "EarthQuake" Wilson

To solve this, render a Top view of the mesh. This only works if you're using a planar UV projection for your low-poly mesh and you're baking a tangent-space normal map.

The lighting setup for top-down rendering.
Image by Ben Cloward

Edge Padding

If a normal map doesn't have enough Edge Padding, this will create shading seams on the UV borders.

High Poly Materials

3ds Max will not bake a normal map properly if the high-res model has a mental ray Arch & Design material applied. If your normal map comes out mostly blank, either use a Standard material or none at all. For an example see the Polycount thread Render to Texture >:O.

Reset Transforms

Before baking, make sure your low-poly model's transforms have been reset. This is very important! Often during the modeling process a model will be rotated and scaled, but these compounded transforms can create a messy local "space" for the model, which in turn often creates rendering errors for normal maps.

In 3ds Max, use the Reset Xforms utility then Collapse the Modifier Stack. In Maya use Freeze Transformation. In XSI use the Freeze button.

Solving Intersections

The projection process often causes problems like misses, or overlaps, or intersections. It can be difficult generating a clean normal map in areas where the high-poly mesh intersects or nearly intersects itself, like in between the fingers of a hand. Setting the ray distance too large will make the baker pick the other finger as the source normal, while setting the ray distance too small will lead to problems at other places on the mesh where the distances between in-game mesh and high-poly mesh are greater.

Fortunately there are several methods for solving these problems.

  1. Change the shape of the cage. Manually edit points on the projection cage to help solve tight bits like the gaps between fingers.
  2. Limit the projection to matching materials, or matching UVs.
  3. Explode the meshes. See the polycount thread Explode script needed (for baking purposes).
  4. Bake two or more times using different cage sizes, and combine them in Photoshop.

Solving Pixel Artifacts

Random pixel artifacts in the bake.
Image by Eric Chadwick

If you are using 3ds Max's Render To Texture to bake from one UV layout to another, you may see stray pixels scattered across the bake. This only happens if you are using a copy of the original mesh in the Projection, and that mesh is using a different UV channel than the original mesh.

There are two solutions for this:

  • Add a Push modifier to the copied mesh, and set it to a low value like 0.01.

- or -

  • Turn off Filter Maps in the render settings (Rendering menu > Render Setup > Renderer tab > uncheck Filter Maps). To prevent aliasing you may want to enable the Global Supersampler in Render Setup.

See also #Anti-Aliasing.

Solving Wavy Lines

When capturing from a cylindrical shape, often the differences between the low-poly mesh and the high-poly mesh will create a wavy edge in the normal map. There are a couple ways to avoid this:

  1. The best way... create your lowpoly model with better supporting edges. See the Polycount threads Understanding averaged normals and ray projection/Who put waviness in my normal map?, approach to techy stuff, Any tips for normal mapping curved surface?.
  2. Adjust the shape of the cage to influence the directions the rays will be cast. Beware... this work will have to be re-done every time you edit the lowpoly mesh, as the cage will be invalidated. At the bottom of this page of his normal map tutorial, Ben "poopinmymouth" Mathis shows how to do this in 3ds Max. Same method can be seen in the image below.
  3. Subdivide the low-res mesh so it more closely matches the high-res mesh. Beware... this will cause the normal map not to match your lowpoly vertex normals, probably causing shading errors. Jeff "airbrush" Ross has a video tutorial that shows how to do this in Maya.
  4. Paint out the wavy line. Beware... this work will have to be re-done every time you re-bake the normal map. The normal map process tutorial by Ben "poopinmymouth" Mathis includes an example of painting out wavy lines in a baked normal map.
  5. Use a separate planar-projected mesh for the details that wrap around the barrel area, so the ray-casting is more even. Beware... this will cause the normal map not to match your lowpoly vertex normals, probably causing shading errors. For example to add tread around a tire, the tread can be baked from a tread model that is laid out flat, then that bake can layered onto the bake from the cylindrical tire mesh in a paint program.
Adjusting the shape of the cage to remove distortion.
Image by Timothy "tpe" Evison

Triangulating

Before baking, it is usually best to triangulate the low-poly model, converting it from polygons into pure triangles. This prevents the vertex normals from being changed later on, which can create specular artifacts.

When quads are triangulated in Modo, the internal edges are sometimes flipped, which causes shading differences.
Image by "Talon" O'Hare


Sometimes a baking tool or a mesh exporter/importer will re-triangulate the polygons. A quad polygon is actually treated as two triangles, and the internal edge between them is often switched diagonally during modeling operations. When the vertices of the quad are moved around in certain shapes, the software's algorithm for polygon models tries to keep the quad surface in a "rational" non-overlapping shape. It does this by switching the internal edge between its triangles.


The specular highlight is affected by triangulation. Flip edges to fix skewing. See the Polycount thread Skewed Specular Highlight? for pictures and more info.
Image by Robert "TychoVII" Kreps


Working with Cages

Cage has two meanings in the normal-mapping process: a low-poly base for subdivision surface modeling (usually called the basemesh), or a ray-casting mesh used for normal map baking. This section covers the ray-casting cage.

Most normal map baking tools allow you to use a distance-based raycast. A ray is sent outwards along each vertex normal, then at the distance you set a ray is cast back inwards. Where ever that ray intersects the high poly mesh, it will sample the normals from it.

Normalmap raycasting 1.jpg Normalmap raycasting 2.jpg
Hard edges and a distance-based raycast (gray areas) cause ray misses (yellow) and ray overlaps (cyan).
Image by Diego Castaño 		The gray area shows that using all soft edges (or hard edges and a cage-based raycast) will avoid ray-casting errors from split normals.
Image by Diego Castaño

Unfortunately with a distance-based raycast, split vertex normals will cause the bake to miss parts of the high-res mesh, causing errors and seams.

Some software allows you to use cage mesh option instead, which basically inflates a copy of the low-poly mesh, then raycasts inwards from each vertex. This ballooned-out mesh is the cage.

In 3ds Max the cage controls both the distance and the direction of the raycasting.

In Maya the cage only controls the distance; the ray direction matches the vertex normals (inverted).

This may have been fixed in the latest release...<
> In Xnormal the cage is split everywhere the model has hard edges, causing ray misses in the bake. You can fix the hard edge split problem but it involves an overly complex workflow. You must also repeat the whole process any time you change your mesh:

  1. Load the 3d viewer.
  2. Turn on the cage editing tools.
  3. Select all of the vertices.
  4. Weld all vertices.
  5. Expand the cage as you normally would.
  6. Save out your mesh using the Xnormal format.
  7. Make sure Xnormal is loading the correct mesh.

Painting

Don't be afraid to edit normal maps in Photoshop. After all it is just a texture, so you can clone, blur, copy, blend all you want... as long as it looks good of course. Some understanding of the way colors work in normal maps will go a long way in helping you paint effectively.

A normal map sampled from a high-poly mesh will nearly always be better than one sampled from a texture, since you're actually grabbing "proper" normals from an accurate, highly detailed surface. That means your normal map's pixels will basically be recreating the surface angles of your high-poly mesh, resulting in a very believable look.

If you only convert an image into a normal-map, it can look very flat, and in some cases it can be completely wrong unless you're very careful about your value ranges. Most image conversion tools assume the input is a heightmap, where black is low and white is high. If you try to convert a diffuse texture that you've painted, the results are often very poor. Often the best results are obtained by baking the large and mid-level details from a high-poly mesh, and then combined with photo-sourced "fine detail" normals for surface details such as fabric weave, scratches and grain.

Sometimes creating a high poly surface takes more time than your budget allows. For character or significant environment assets then that is the best route, but for less significant environment surfaces working from a heightmap-based texture will provide a good enough result for a much less commitment in time.

Flat Color

The color (128,128,255) creates normals that are completely perpendicular to the polygon, as long as the vertex normals are also perpendicular. Remember a normal map's per-pixel normals create offsets from the vertex normals. If you want an area in the normal map to be flat, so it creates no offsets from the vertex normals, then use the color (128,128,255).

This becomes especially obvious when mirroring a normal map and using a shader with a reflection ingredient. Reflection tends to accentuate the angles between the normals, so any errors become much more apparent.

Mirrored normal maps show a seam when (127,127,255) is used for the flat color; 128 is better.
Image by Eric Chadwick

In a purely logical way, 127 seems like it would be the halfway point between 0 and 255. However 128 is the color that actually works in practice. When a test is done comparing (127,127,255) versus (128,128,255) it becomes obvious that 127 creates a slightly bent normal, and 128 creates a flat one.

This is because most game pipelines use unsigned normal maps. For details see the Polycount forum thread tutorial: fixing mirrored normal map seams.

Blending Normal Maps Together

Blending normal maps together is a quick way to add high-frequency detail like wrinkles, cracks, and the like. Fine details can be painted as a height map, then it can be converted into a normal map using one of the normal map tools. Then this "details" normal map can be blended with a geometry-derived normal map using one of the methods below.

Here is a comparison of four of the blending methods. Note that in these examples the default values were used for CrazyBump (Intensity 50, Strength 33, Strength 33), but the tool allows each layer's strength to be adjusted individually for stronger or milder results. Each of the normal maps below were re-normalized after blending.

Nrmlmap blending methods Maps.png Nrmlmap blending methods RTTNormalMapFX.png
The blended normal maps.
Image by Eric Chadwick 		Screenshot of the 3dsmax viewport, using the RTTNormalMap.fx shader.
Image by Eric Chadwick

The four blending methods used above:

  1. CrazyBump by Ryan Clark blends normal maps together using calculations in 3D space rather than just in 2D. This does probably the best job at preserving details, and each layer's strength settings can be tweaked individually.
  2. Combining Normal Maps in Photoshop by Rod Green blends normal maps together using Linear Dodge mode for the positive values and Difference mode for the negative values, along with a Photoshop Action to simplify the process. It's free, but the results may be less accurate than CrazyBump.
  3. Making of Varga by Paul "paultosca" Tosca blends normal maps together using Overlay mode for the red and green channels and Multiply mode for the blue channel. This gives a slightly stronger bump than the Overlay-only method. Leo "chronic" Covarrubias has a step-by-step tutorial for this method in CG Bootcamp Combine Normal Maps.
  4. Normal Map Deepening by Ben "poopinmymouth" Mathis shows how to blend normal maps together using Overlay mode. CGTextures tutorial for the NVIDIA Photoshop filter by Scott Warren also shows how to create normalmaps using multiple layers (Note: to work with the Overlay blend mode each layer's Output Level should be 128 instead of 255, you can use the Levels tool for this).

The Getting good height from Nvidia-filter normalizing grayscale height thread on the Polycount forum has a discussion of different painting/blending options. Also see the 2D Tools section for painting and conversion tools.

Pre-Created Templates

A library of shapes can be developed and stored for later use, to save creation time for future normal maps. Things like screws, ports, pipes, and other doo-dads. These shapes can be stored as bitmaps with transparency so they can be layered into baked normal maps.

Re-normalizing

Re-normalizing means resetting the length of each normal in the map to 1.

A normal mapping shader takes the three color channels of a normal map and combines them to create the direction and length of each pixel's normal. These normals are then used to apply the scene lighting to the mesh. However if you edit normal maps by hand or if you blend multiple normal maps together this can cause those lengths to change. Most shaders expect the length of the normals to always be 1 (normalized), but some are written to re-normalize the normal map dynamically (for example, 3ds Max's Hardware Shaders do re-normalize).

If the normals in your normal map are not normalized, and your shader doesn't re-normalize them either, then you may see artifacts on the shaded surface... the specular highlight may speckle like crazy, the surface may get patches of odd shadowing, etc. To help you avoid this NVIDIA's normal map filter for Photoshop provides an easy way to re-normalize a map after editing; just use the Normalize Only option. Xnormal also comes with a Normalize filter for Photoshop.

The re-normalize option in the NVIDIA filter.
Image by Scott Warren

Some shaders use compressed normal maps. Usually this means the blue channel is thrown away completely, so it's recalculated on-the-fly in the shader. However the shader has to re-normalize in order to recreate that data, so any custom normal lengths that were edited into the map will be ignored completely.

Ambient Occlusion into a Normal Map

If the shader doesn't re-normalize the normal map, an Ambient Occlusion Map can actually be baked into the normal map. This will shorten the normals in the crevices of the surface, causing the surface to receive less light there. This works with both diffuse and specular, or any other pass that uses the normal map, like reflection.

However it's usually best to keep the AO as a separate map (or in an alpha channel) and multiply it against the ambient lighting only. This is usually done with a custom shader. If you multiply it against the diffuse map or normal map then it also occludes diffuse lighting which can make the model look dirty. Ambient occlusion is best when it occludes ambient lighting only, for example a diffusely convolved cubemap.

AO can be baked into a normal map, shortening the normals (lower left model).
Model by James Ku

To bake the AO into a normal map, adjust the levels of the AO layer first so the darks only go as low as 128 gray, then set the AO layer to Darken mode. This will shorten the normals in the normalmap, causing the surface to receive less light in the darker areas.

This trick doesn't work with any shaders that re-normalize, like 3ds Max's Hardware Shaders. The shader must be altered to actually use the lengths of your custom normals; most shaders just assume all normals are 1 in length because this makes the shader code simpler. Also this trick will not work with most of the common normal map compression formats, which often discard the blue channel and recalculate it in the shader, which requires re-normalization.

Back Lighting Example

You can customize normal maps for some interesting effects. If you invert the blue channel of a tangent-space map, the normals will be pointing to the opposite side of the surface, which can simulate backlighting.

Tree front.jpg Tree back.jpg
Tree simulating subsurface scattering (front view).
Image by Eric Chadwick 		Tree simulating subsurface scattering (back view).
Image by Eric Chadwick
Tree maps.jpg
The maps used for the leaves. The 2nd diffuse was simply color-inverted, hue-shifted 180°, and saturated.
Image by Eric Chadwick

The tree leaves use a shader than adds together two diffuse maps, one using a regular tangent-space normal map, the other using the same normal map but with the blue channel inverted. This causes the diffuse map using the regular normal map to only get lit on the side facing the light (front view), while the diffuse map using the inverted normal map only gets lit on the opposite side of the leaves (back view). The leaf geometry is 2-sided but uses the same shader on both sides, so the effect works no matter the lighting angle. As an added bonus, because the tree is self-shadowing the leaves in shadow do not receive direct lighting, which means their backsides do not show the inverted normal map, so the fake subsurface scatter effect only appears where the light directly hits the leaves. This wouldn't work for a whole forest because of the computational cost of self-shadowing and double normal maps, but could be useful for a single "star" asset, or if LODs switched the distant trees to a model that uses a cheaper shader.

Shaders and Seams

You need to use the right kind of shader to avoid seeing seams where UV breaks occur. It must be written to use the same tangent basis that was used during baking. If the shader doesn't, the lighting will either be inconsistent across UV borders or it will show smoothing errors from the low-poly vertex normals.

Xnormal generates accurate normals when displayed in Xnormal, and the SDK includes a method to write your own custom tangent space generator for the tool.

3ds Max Shaders

The "Render To Texture" tool in 3ds Max 2011 and older generates tangent-space normal maps that render correctly in the offline renderer (scanline) but do not render correctly in the realtime viewport with the 3ds Max shaders. Max is using a different tangent basis for each. This is readily apparent when creating non-organic hard surface normalmaps; smoothing errors appear in the viewport that do not appear when rendered.

The errors can be fixed by using "Render To Texture" to bake a tangent-space or object-space map, and using the free "3Point Shader" by Christoph 'CrazyButcher' Kubisch and Per 'perna' Abrahamsen. The shader uses the same tangent basis as the baking tool, so it produces nearly flawless results. It also works with old bakes.

You can get OK results in the Max viewport using a tangent-space map baked in Maya, loading it in a Standard material, and enabling "Show Hardware Map in Viewport". Another method is to use Render To Texture to bake an object-space map then use Nspace to convert it into a tangent-space map then load that in a DirectX material and use the RTTNormalMap.fx shader.

Autodesk is aware of these issues, and plans to address them in an upcoming release. See these links for more information:

Comparison of map baking methods in Maya and Max, and Ben Cloward's custom FX code.
Image by Ben Cloward and Eric Chadwick
3 Point Studios' normal map display fix for 3ds Max.
image by 3 Point Studios
Max2010 normalmap workarounds.png
Max2010 normalmap compare.png
Viewport methods in 3ds Max 2010.
Image by Eric Chadwick 		More baking methods in 3ds Max 2010.
Image by Eric Chadwick

3ds Max Edit Normals Trick

After baking, if you add an Edit Normals modifier to your low-poly normalmapped model, this seems to "relax" the vertex normals for more accurate viewport shading. The modifier can be collapsed if desired.

Maya Shaders

Maya seems to correctly generate normals to view in realtime, with the correct tangent basis, with much less smoothing errors than 3ds Max.

  • BRDF shader by Brice Vandemoortele and Cedric Caillaud (more info in this Polycount thread) Update: New version here with many updates, including object-space normal maps, relief mapping, self-shadowing, etc. Make sure you enable cgFX shaders in the Maya plugin manager, then you can create them in the same way you create a Lambert, Phong etc. Switch OFF high quality rendering in the viewports to see them correctly too.
  • If you want to use the software renderer, use mental ray instead of Maya's software renderer because mental ray correctly interprets tangent space normals. The Maya renderer treats the normal map as a grayscale bump map, giving nasty results. Mental ray supports Maya's Phong shader just fine (amongst others), although it won't recognise a gloss map plugged into the "cosine power" slot. The slider still works though, if you don't mind having a uniform value for gloss. Spec maps work fine though. Just use the same set up as you would for viewport rendering. You'll need to have your textures saved as TGAs or similar for mental ray to work though. - from CheeseOnToast


Normal Map Compression

see; Normal Map Compression

Links

Related Pages

3D Tools

See Category:Tools#3D_Normal_Map_Software

2D Tools

See Category:Tools#2D_Normal_Map_Software

Tutorials

Discussion

Discuss this page on the Polycount forums. Suggestions welcome.

Even though only one person has been editing this page so far, the information here was gathered from many different sources. We wish to thank all the contributors for their hard-earned knowledge. It is much appreciated!

Retrieved from "http://wiki.polycount.com/w/index.php?title=Normal_map&oldid=6203"
Personal tools
Namespaces

Variants
Actions
Navigation
Tools