Render Library References

This page is part of the Render Reference, which is a set of pages dedicated to documenting the Render Library.

Other Render References pages include:

• Beams
• Minification and Magnification Texture Filters
• Render Targets
  • Stencils

Introduction

Before learning about stencils, it helps to have some basic knowledge of Render Targets. You don’t need to be an expert, but understanding the basics will make everything on this page easier to follow.

Also, in this guide, we’ll often use the word byte to mean an integer Number between 0 and 255. This represents 8 bits of data. While Lua doesn’t use this term specifically, it’s a common concept in programming, so we’re using it to keep things simple.

What are Stencils?

Stencils are a system that lets you control (or "mask") drawing operations on Render Targets on a per-pixel basis.
Whenever something is drawn to a Render Target (like the main game view,) the Stencil system performs logical checks (we’ll cover these later) for each pixel that’s about to be modified. It then decides whether that pixel should be updated by the draw operation, or left as it is.

While Stencils are often used for 3D effects, it’s important to understand that they operate entirely in 2D. Stencils work with 2D pixels and interact with 3D objects only because those objects are drawn (or "rasterized") onto a 2D Render Target.

What are Stencils Used For?

Stencils have been used for all kinds of creative effects, including:

• Impossible rooms that look larger on the inside than the outside.
• Bullet holes that players can see through.
• Adding windows to walls that didn’t originally have them.
• Portals, like in the game Portal

What are the Parts of the Stencil System?

How to Read This Section

The following section defines and explains each of the available Stencil functions, operations, and concepts in the order they are executed during a draw operation.
Each pixel affected by a draw operation has the exact same logic applied to it and is independent of the other pixels.
You can think of this section as a for loop iterating over each affected pixel.
There are three (3) possible paths through this logic that a pixel can take. At the end of each path, the loop continues onto the next pixel.
Notes on where we are in the for loop will be shown [In this Format]

Parts of the Stencil System

[Prior to the draw operation]

• The Stencil Buffer holds a byte (Here called a "Stencil value" or "Stencil Buffer value") for every individual pixel on the Render Target it belongs to.
  • By default, every pixel's Stencil value is 0.
• The Reference Value is a byte that Stencil Buffer values are compared against.

When a draw operation is performed, each of the affected pixels has its Stencil Buffer value compared to the Reference Value.

[A draw operation has just been executed]
[At this point, the Stencil configuration cannot be changed.]
[This is the start of the loop that iterates over the pixels affected by the draw operation]

• The Compare Function determines which of the possible relationships (Greater than, less than, equal to, etc.) between the Stencil Buffer value for a pixel and the current Reference Value should be considered a "Pass" and which should be considered a "Fail".
  The full list of possible Compare Functions can be found here
• The Test Mask is a byte that will be bitwise ANDed with all values as they are read from the Stencil Buffer.
  • By default, the Read Mask is set to make no change to read values.

Depending on if a pixel Passed or Failed the Compare Function, its value is then potentially modified, depending on the current Stencil configuration.

• Operations modify the Stencil Buffer value for a given pixel depending on if it has Passed or Failed. Operations include incrementing the current value by one (1), replacing the current value with the Reference Value, making no change, and others.
  The full list of possible Operations can be found here
• The Write Mask is a byte that will be bitwise ANDed with all values before they are written to the Stencil Buffer.
  • By default, the Write Mask is set to make no change to written values.
• The Fail Operation is performed on pixels that Fail the Compare Function.
  [This pixel is done and the next one can be processed.]

For the pixels that Pass the Compare Function, there is one additional check they must Pass before they are allowed to be modified by the draw operation.

• The Depth Test (or "Z test") compares the pixel's current Depth Buffer value against the Depth value of the draw operation.

Unlike the Compare Function, this comparison is fixed and cannot be configured.

• The Z Fail Operation is performed on the Stencil Buffer for pixels where the current Depth Buffer value is less than the draw operation's proposed depth value. This indicates that the draw operation should not be visible on this pixel because it is drawing behind the current pixel.
  [The next pixel can be processed.]

If a pixel has passed both the Compare Function as well as the Depth Test, it is considered Passed and will have its contents replaced/modified by the draw operation. This will affect the Color, Depth, and Stencil Buffer values for the pixel.

• The Pass Operation is performed on the Stencil Buffer for pixels that have passed both the Compare Function and the Depth Test and are being updated by the draw operation.
  [The next pixel can be processed.]

The following flowchart outlines the same information as the text above to aid in understanding.

How Can Those Parts Be Used to Make Things?

Common Usage Pattern

In practical applications, Stencils are used in the following pattern:

1. Configure the Stencil system
2. Perform draw operations to set Stencil Buffer values
3. Re-configure the Stencil system
4. Perform draw operations that are masked by the Stencil Buffer values from steps 1-2

It's quite common for steps 1-2 and steps 3-4 to be repeated multiple times (Called "passes" and not to be confused with the term used to denote a pixel's pass/fail state) in order to configure the Stencil Buffer in ways that it cannot be with only a single pass.

Example: Simple 2D Masking

Example: Making an Impossible Space