Texture Baking

Texture baking is the process of transferring details from one model to another.

This is commonly used to create Normal maps, by rendering the surface curvature from a high-resolution 3d model onto a lower-resolution in-game model.

Various software tools can be used for baking: 3ds Max's Render To Texture, Maya's Transfer Attributes, Xnormal, etc.

The baking tool usually starts by projecting a certain numerical distance out from the low-poly mesh, then sending rays inwards towards the high-poly mesh. When a ray intersects the high-poly mesh, it records the mesh's surface detail and saves that into a texture map.

Baking Workflow
(work in progress)
 * 1) Re-Topo.
 * 2) * Use ReTopologyModeling to build the low-resolution in-game model.
 * 3) * Create good topology for baking. See Understanding averaged normals and ray projection/Who put waviness in my normal map?.
 * 4) * Edit the UVs for baking. See JedTheKrampus on offsetting Mirrored UVs.
 * 5) * For tangent-space normal mapping, apply hard edges to every UV seam. See You're making me hard. Making sense of hard edges, uvs, normal maps and vertex counts, and Earthquake on separating smoothing groups in the UV.
 * 6) Optimize.
 * 7) * Optimize the high-resolution model to speed up bakes, to avoid running out of RAM while baking, and to keep 3d file sizes manageable.
 * 8) * If it's a sculpt you should reduce the vertex count to a manageable file size before exporting. See Tools. Sculpting tools like Zbrush create triangles smaller than your bake pixels, which will increase baking time significantly without actually improving the bake.
 * 9) * If it's a subdivision surface you should choose an appropriate resolution. Use just enough subdivisions to get a smooth surface at your baking resolution, and no more.
 * 10) Explode.
 * 11) * Interpenetrating parts can cause baking artifacts, because neighboring surfaces will capture parts of each other.
 * 12) * Separate the parts so there is space between them. Suitable parts would be non-welded surfaces, mesh elements, Zbrush subtools, etc.
 * 13) * Use the same separation for the highpoly model and the lowpoly model. After the bake, move the pieces back together.
 * 14) * If your modeling software has animation, keyframe the explode so it can easily be unexploded. Morphing is another possibility. Or just save the model before exploding.
 * 15) Cage.
 * 16) * Create an inflated copy of the low-resolution model, which encompasses the high-resolution model.
 * 17) * A baking tool uses a cage to send rays from the low-res UVs inwards towards the high-res surface. Where those rays intersect the high-res model, the baker will render what it finds into the map. High-res normals go into a normal map, occluded surfaces go into the ambient occlusion map, etc.
 * 18) Which baker?
 * 19) * Texture Baking
 * 20) Import & Test.
 * 21) * Import the high-resolution model, the low-resolution model, and the cage.
 * 22) * Test the bake settings using lower values for faster iteration. Smaller render sizes, lower anti-aliasing samples, etc.
 * 23) * Fix errors and re-bake until solved. Look for overlaps, gaps, skewed details. Adjust the bake settings, low-poly model, cage, etc. See Skew you buddy! Making sense of skewed normal map details., and Skewmesh Tutorial.
 * 24) * Some errors can be painted out later. However any painting must be repeated if the model is re-baked. Also, painting on a normal map can introduce more artifacts.
 * 25) Render.
 * 26) * Increase the settings to full quality.
 * 27) * Use 16-bit. See Of Bit Depths, Banding and Normal Maps.
 * 28) * Object space, if converting to a specific tangent space. See Synched Workflow, and Converting Between Spaces.
 * 29) Convert.
 * 30) * Convert to a specific tangent space. See Synched Workflow, and Converting Between Spaces.
 * 31) Edit.
 * 32) * Paint out errors.
 * 33) * Combine bakes. See Ambient occlusion map#EarthQuake's Baking Method
 * 34) * Add details from photos or other bakes. See RNM Normal Map Combiner, and Combine Normal.
 * 35) * Swizzle. See Normal Map Technical Details.
 * 36) * Re-normalize.
 * 37) * Reduce to 8-bit. See Of Bit Depths, Banding and Normal Maps.

UV Coordinates
Normal map baking tools only capture normals within the 0-1 UV square, any UV bits outside this area are ignored.

Only one copy of the forward-facing UVs should remain in the 0-1 UV square at baking time. If the mesh uses overlapping UVs, this will likely cause artifacts to appear in the baked map, since the baker will try render each UV shell into the map. Before baking, it's best to move all the overlaps and mirrored bits outside the 0-1 square.



If you move all the overlaps and mirrored bits exactly 1 UV unit (any whole number will do), then you can leave them there after the bake and they will still be mapped correctly. You can move them back if you want, it doesn't matter to most game engines. Be aware that ZBrush does use UV offsets to manage mesh visibility, however this usually doesn't matter because the ZBrush cage mesh is often a different mesh than the in-game mesh used for baking.

You should avoid changing the UVs after baking the normal map, because rotating or mirroring UVs after baking will cause the normal map not to match the tangent basis anymore, which will likely cause lighting problems.

In 3ds Max, W is a third texture coordinate. It's used for 3D procedural textures and for storing vertex color in UV channels (you need 3 axes for RGB, so UVW can store vertex color). Bake problems can be avoided by moving any overlapping UVs to -1 on the W axis, with the same results as moving them 1 unit on the U or V axes. The tool Render To Texture will always bake whatever UVs are the highest along the W axis. However using W can be messy... it's generally hidden unless you purposefully look for it (bad for team work), doesn't get preserved on export to other apps, and high W values can prevent selecting and/or welding UVs.



When you look at a tangent-space normal map for a character, you typically see different colors along the UV seams. This is because the UV shells are often oriented at different angles on the mesh, a necessary evil when translating the 3D mesh into 2D textures. The body might be mapped with a vertical shell, and the arm mapped with a horizontal one. This requires the normals in the normal map to be twisted for the different orientations of those UV shells. The UVs are twisted, so the normals must be twisted in order to compensate. The tangent basis helps reorient (twist) the lighting as it comes into the surface's local space, so the lighting will then look uniform across the normal mapped mesh.

When an artist tiles a tangent-space normal map across an arbitrary mesh, like a landscape, this tends to shade correctly because the mesh has a uniform direction in tangent space. If the mesh has discontinuous UV coordinates (UV seams), or the normal map has large directional gradients across it, the tangent space won't be uniform anymore so the surface will probably have shading seams.

Triangulation
Before baking, it is usually best to triangulate the low-poly model, converting it from polygons into pure triangles. This prevents the vertex normals from being changed later on, which can create specular artifacts.



Sometimes a baking tool or a mesh exporter/importer will re-triangulate the polygons. A quad polygon is actually treated as two triangles, and the internal edge between them is often switched diagonally during modeling operations. When the vertices of the quad are moved around in certain shapes, the software's algorithm for polygon models tries to keep the quad surface in a "rational" non-overlapping shape. It does this by switching the internal edge between its triangles.



Cages
Cage has two meanings in the normal-mapping process: a low-poly base for subdivision surface modeling (usually called the basemesh), or a ray-casting mesh used for normal map baking. This section covers the ray-casting cage.

Most normal map baking tools allow you to use a distance-based raycast. A ray is sent outwards along each vertex normal, then at the distance you set a ray is cast back inwards. Where ever that ray intersects the high poly mesh, it will sample the normals from it.

Unfortunately with a distance-based raycast, split vertex normals will cause the bake to miss parts of the high-res mesh, causing errors and seams.

Some software allows you to use cage mesh option instead, which basically inflates a copy of the low-poly mesh, then raycasts inwards from each vertex. This ballooned-out mesh is the cage.

In 3ds Max the cage controls both the distance and the direction of the raycasting.

In Maya the cage only controls the distance; the ray direction matches the vertex normals (inverted).

This may have been fixed in the latest release...<> In Xnormal the cage is split everywhere the model has hard edges, causing ray misses in the bake. You can fix the hard edge split problem but it involves an overly complex workflow. You must also repeat the whole process any time you change your mesh:
 * 1) Load the 3d viewer.
 * 2) Turn on the cage editing tools.
 * 3) Select all of the vertices.
 * 4) Weld all vertices.
 * 5) Expand the cage as you normally would.
 * 6) Save out your mesh using the Xnormal format.
 * 7) Make sure Xnormal is loading the correct mesh.


 * The tutorial Modeling High/Low Poly Models for Next Gen Games by João "Masakari" Costa has more examples of ray-casting, plus how to get better results from the bake.

Anti-Aliasing
Turning on super-sampling or anti-aliasing (or whatever multi-ray casting is called in your normal map baking tool) will help to fix any jagged edges where the high-res model overlaps itself within the UV borders of the low-poly mesh, or wherever the background shows through holes in the mesh. Unfortunately this tends to render much much slower, and takes more memory.



One trick to speed this up is to render 2x the intended image size then scale the normal map down 1/2 in a paint program like Photoshop. The reduction's pixel resampling will add anti-aliasing for you in a very quick process. After scaling, make sure to re-normalize the map if your game doesn't do that already, because the un-normalized pixels in your normalmap may cause pixelly artifacts in your specular highlights. Re-normalizing can be done with NVIDIA's normal map filter for Photoshop.

3ds Max's supersampling doesn't work nicely with edge padding, it produces dark streaks in the padded pixels. If so then turn off padding and re-do the padding later, either by re-baking without supersampling or by using a Photoshop filter like the one that comes with Xnormal.

Transparency
Sometimes you need to bake a normal map from an object that uses opacity maps, like a branch with opacity-mapped leaves. Unfortunately baking apps often completely ignore any transparency mapping on your high-poly mesh.

To solve this, render a Top view of the mesh. This only works if you're using a planar UV projection for your low-poly mesh and you're baking a tangent-space normal map.


 * Make sure the Top view matches the dimensions of the planar UV projection used by the low-poly mesh. It helps to use an orthographic camera for precise placement.
 * On the high-poly mesh either use a specific lighting setup or a use special material shader:
 * 1) The lighting setup is described in these tutorials:
 * * Creating A Normal Map Right In Your 3D App by Ben Cloward
 * *Generating High Fidelity Normal Maps with 3-D Software by Dave McCoy, Graphics Techniques Consultant, Xbox Content and Design Team
 * 2) The material shader does the same thing, but doesn't require lights.
 * * NormalTexMap scripted map for 3ds Max by Dave Locke.
 * * InfoTexture map plugin for 3ds Max by John Burnett



Edge Padding
If a normal map doesn't have enough Edge Padding, this will create shading seams on the UV borders.

High Poly Materials
3ds Max will not bake a normal map properly if the high-res model has a mental ray Arch & Design material applied. If your normal map comes out mostly blank, either use a Standard material or none at all. For an example see the Polycount thread Render to Texture >:O.

Reset Transforms
Before baking, make sure your low-poly model's transforms have been reset. This is very important! Often during the modeling process a model will be rotated and scaled, but these compounded transforms can create a messy local "space" for the model, which in turn often creates rendering errors for normal maps.

In 3ds Max, use the Reset Xforms utility then Collapse the Modifier Stack. In Maya use Freeze Transformation. In XSI use the Freeze button.

Solving Intersections
The projection process often causes problems like misses, or overlaps, or intersections. It can be difficult generating a clean normal map in areas where the high-poly mesh intersects or nearly intersects itself, like in between the fingers of a hand. Setting the ray distance too large will make the baker pick the other finger as the source normal, while setting the ray distance too small will lead to problems at other places on the mesh where the distances between in-game mesh and high-poly mesh are greater.

Fortunately there are several methods for solving these problems.


 * 1) Change the shape of the cage. Manually edit points on the projection cage to help solve tight bits like the gaps between fingers.
 * 2) Limit the projection to matching materials, or matching UVs.
 * 3) Explode the meshes. See the polycount thread Explode script needed (for baking purposes).
 * 4) Bake two or more times using different cage sizes, and combine them in Photoshop.

Solving Pixel Artifacts


If you are using 3ds Max's Render To Texture to bake from one UV layout to another, you may see stray pixels scattered across the bake. This only happens if you are using a copy of the original mesh in the Projection, and that mesh is using a different UV channel than the original mesh.

There are two solutions for this:

- or -
 * Add a Push modifier to the copied mesh, and set it to a low value like 0.01.


 * Turn off Filter Maps in the render settings (Rendering menu > Render Setup > Renderer tab > uncheck Filter Maps). To prevent aliasing you may want to enable the Global Supersampler in Render Setup.

See also.

Solving Wavy Lines
When capturing from a cylindrical shape, often the differences between the low-poly mesh and the high-poly mesh will create a wavy edge in the normal map. There are a couple ways to avoid this:


 * 1) The best way... create your lowpoly model with better supporting edges. See the Polycount threads Understanding averaged normals and ray projection/Who put waviness in my normal map?, approach to techy stuff, Any tips for normal mapping curved surface?.
 * 2) Adjust the shape of the cage to influence the directions the rays will be cast. Beware... this work will have to be re-done every time you edit the lowpoly mesh, as the cage will be invalidated. At the bottom of this page of his normal map tutorial, Ben "poopinmymouth" Mathis shows how to do this in 3ds Max. Same method can be seen in the image below.
 * 3) Subdivide the low-res mesh so it more closely matches the high-res mesh. Beware... this will cause the normal map not to match your lowpoly vertex normals, probably causing shading errors. Jeff "airbrush" Ross has a video tutorial that shows how to do this in Maya.
 * 4) Paint out the wavy line.  Beware... this work will have to be re-done every time you re-bake the normal map. The normal map process tutorial by Ben "poopinmymouth" Mathis includes an example of painting out wavy lines in a baked normal map.
 * 5) Use a separate planar-projected mesh for the details that wrap around the barrel area, so the ray-casting is more even. Beware... this will cause the normal map not to match your lowpoly vertex normals, probably causing shading errors. For example to add tread around a tire, the tread can be baked from a tread model that is laid out flat, then that bake can layered onto the bake from the cylindrical tire mesh in a paint program.



Baking Tools

 * 3ds Max: Render To Texture
 * Airborn Studios Tools - Backstube (Xnormal batcher), Packstube (map packer).
 * Blender: Render Bake
 * Handplane (tangent space conversion)
 * Knald
 * Maya: Transfer Maps
 * MightyBake
 * Modo: Baking
 * Substance Designer: Bakers
 * Xnormal
 * Zbrush: Create Normal Map