Render Library References

These pages seek to provide helpful insight into the groups of functions provided by the Render Library.

The major function groups are:

• Beams
• Minification and Magnification Texture Filters
• Render Targets
  • Stencils

A Brief Preamble

It's going to be very helpful to have a grasp on Render Targets before learning about Stencils. It's not necessarily a hard requirement and you don't need to be an expert, but you will have a much easier time mentally "connecting the dots" if you're familiar with the terms defined there.

To avoid repetition, the shorthand term byte is used throughout this page to mean "A Number with an integer value between 0 and 255, representing 8 bits of data." While this is not a term used in Lua, it is commmon in programming.

What are Stencils?

Stencils are a system to constrain (or "mask") drawing operations on Render Targets on a per-pixel basis.
Whenever anything is drawn to a Render Target, including the main game view, the Stencil performs logical checks (More on these later.) for each pixel that is about to be affected by the draw operation and determines whether or not that pixel should be allowed to change from its current values.

It's important to understand that while Stencils are often used in 3D effects, they are fundamentally a 2D concept operating exclusively on 2D pixels. They are only able to interact with 3D objects because those 3D objects are being drawn (or "rasterized") onto a 2D Render Target.

What are Stencils Used For?

Here are just a few of the things Stencils have been used for:

• Impossible rooms that appear larger on the inside than the outside can contain
• Bullet holes that the player can see through
• Adding windows to walls that don't have them
• Portals as seen in the game Portal

What are the Parts of the Stencil System?

How to Read This Section

The following section defines and explains each of the available Stencil functions, operations, and concepts in the order they are executed during a draw operation.
Each pixel affected by a draw operation has the exact same logic applied to it and is independent of the other pixels.
You can think of this section as a for loop iterating over each affected pixel.
There are three (3) possible paths through this logic that a pixel can take. At the end of each path, the loop continues onto the next pixel.
Notes on where we are in the for loop will be shown [In this Format]

Parts of the Stencil System

[Prior to the draw operation]

• The Stencil Buffer holds a byte (Here called a "Stencil value" or "Stencil Buffer value") for every individual pixel on the Render Target it belongs to.
  • By default, every pixel's Stencil value is 0.
• The Reference Value is a byte that Stencil Buffer values are compared against.

When a draw operation is performed, each of the affected pixels has its Stencil Buffer value compared to the Reference Value.

[A draw operation has just been executed]
[At this point, no Stencil configuration can be changed.]
[This is the start of the loop over the pixels affected by the draw operation]

• The Compare Function determines which of the possible relationships (Greater than, less than, equal to, etc.) between the Stencil Buffer value for a pixel and the current Reference Value should be considered a "Pass" and which should be considered a "Fail".
  The full list of possible Compare Functions can be found here
• The Test Mask is a byte that will be bitwise ANDed with all values as they are read from the Stencil Buffer.
  • By default, the Read Mask is set to make no change to read values.

Depending on if a pixel Passed or Failed the Compare Function, its value is then potentially modified, depending on the current Stencil configuration.

• Operations modify the Stencil Buffer value for a given pixel depending on if it has Passed or Failed. Operations include incrementing the current value by one (1), replacing the current value with the Reference Value, making no change, and others.
  The full list of possible Operations can be found here
• The Write Mask is a byte that will be bitwise ANDed with all values before they are written to the Stencil Buffer.
  • By default, the Write Mask is set to make no change to written values.
• The Fail Operation is performed on pixels that Fail the Compare Function.
  [This pixel is done and the next one can be processed.]

For the pixels that Pass the Compare Function, there is one additional check they must Pass before they are allowed to be modified by the draw operation.

• The Depth Test (or "Z test") compares the pixel's current Depth Buffer value against the Depth value of the draw operation.

Unlike the Compare Function, this comparison is fixed and cannot be configured.

• The Z Fail Operation is performed on the Stencil Buffer for pixels where the current Depth Buffer value is less than the draw operation's proposed depth value. This indicates that the draw operation should not be visible on this pixel because it is drawing behind the current pixel.
  [The next pixel can be processed.]

If a pixel has passed both the Compare Function as well as the Depth Test, it is considered Passed and will have its contents replaced/modified by the draw operation. This will affect the Color, Depth, and Stencil Buffer values for the pixel.

• The Pass Operation is performed on the Stencil Buffer for pixels that have passed both the Compare Function and the Depth Test and are being updated by the draw operation.
  [The next pixel can be processed.]

The following flowchart outlines the same information as the text above to aid in understanding.

How Can Those Parts Be Used to Make Things?

Common Usage Pattern

In practical applications, Stencils are used in the following pattern:

1. Configure the Stencil system
2. Perform draw operations to set Stencil Buffer values
3. Re-configure the Stencil system
4. Perform draw operations that are masked by the Stencil Buffer values from steps 1-2

It's quite common for steps 1-2 and steps 3-4 to be repeated multiple times (Called "passes" and not to be confused with the term used to denote a pixel's pass/fail state) in order to configure the Stencil Buffer in ways that it cannot be with only a single pass.

Example: Simple 2D Masking

Example: Making an Impossible Space