<cat>Dev</cat> <title>Render Reference - Stencils</title> # Render Library References These pages seek to provide helpful insight into the groups of functions provided by the <page text="Render Library">render</page>. The major function groups are: * <page text="Beams">render_beams</page> * <page text="Minification and Magnification Texture Filters">render_min_mag_filters</page> * <page text="Render Targets">render_rendertargets</page> * **Stencils** # A Brief Preamble It's going to be very helpful to have a grasp on <page text="Render Targets">render_rendertargets</page> before learning about Stencils. It's not necessarily a hard requirement and you don't need to be an expert, but you will have a much easier time mentally "connecting the dots" if you're familiar with the terms defined there. To avoid repetition, the shorthand term `byte` is used throughout this page to mean "A <page>Number</page> with an integer value between `0` and `255`, representing 8 bits of data." While this is not a term used in Lua, it is commmon in programming. # What are Stencils? **Stencils are a system to constrain (or "mask") drawing operations on <page text="Render Targets">render_rendertargets</page> on a per-pixel basis.** Whenever anything is drawn to a Render Target, including the main game view, the Stencil performs logical checks (More on these later.) for each pixel that is about to be affected by the draw operation and determines whether or not that pixel should be allowed to change from its current values. It's important to understand that while Stencils are often used in 3D effects, they are fundamentally a 2D concept operating exclusively on 2D pixels. They are only able to interact with 3D objects because those 3D objects are being drawn (or "rasterized") onto a 2D Render Target. # What are Stencils Used For? Here are just a few of the things Stencils have been used for: * Impossible rooms that appear larger on the inside than the outside can contain * Bullet holes that the player can see through * Adding windows to walls that don't have them * Portals as seen in the game Portal # What are the Parts of the Stencil System? ## How to Read This Section The following section defines and explains each of the available Stencil functions, operations, and concepts in the order they are executed during a draw operation. Each pixel affected by a draw operation has the exact same logic applied to it and is independent of the other pixels. You can think of this section as a `for` loop iterating over each affected pixel. There are three (3) possible paths through this logic that a pixel can take. At the end of each path, the loop continues onto the next pixel. Notes on where we are in the `for` loop will be shown `[In this Format]` ## Parts of the Stencil System `[Prior to the draw operation]` * The **Stencil Buffer** holds a `byte` (Here called a "Stencil Value") for every individual pixel on the Render Target it belongs to. * The **Stencil Buffer** holds a `byte` (Here called a "Stencil value" or "Stencil Buffer value") for every individual pixel on the Render Target it belongs to. * By default, every pixel's Stencil value is `0`. * The <page text="Reference Value">render.SetStencilReferenceValue</page> is a `byte` that Stencil Buffer values are compared against. When a draw operation is performed, each of the affected pixels has its Stencil Buffer value compared to the Reference Value. `[A draw operation has just been executed]` `[At this point, no Stencil configuration can be changed.]` `[This is the start of the loop over the pixels affected by the draw operation]` * The <page text="Compare Function">render.SetStencilCompareFunction</page> determines which of the possible relationships (Greater than, less than, equal to, etc.) between the Stencil Buffer value for a pixel and the current Reference Value should be considered a "**Pass**" and which should be considered a "**Fail**". <page text="The full list of possible Compare Functions can be found here">Enums/STENCILCOMPARISONFUNCTION</page> * <page text="The Test Mask">render.SetStencilTestMask</page> is a `byte` that will be <page text="bitwise ANDed">bit.band</page> with all values as they are read from the Stencil Buffer. * By default, the Read Mask is set to make no change to read values. Depending on if a pixel Passed or Failed the Compare Function, its value is then potentially modified, depending on the current Stencil configuration. * **Operations** modify the Stencil Buffer value for a given pixel depending on if it has Passed or Failed. Operations include incrementing the current value by one (1), replacing the current value with the Reference Value, making no change, and others. <page text="The full list of possible Operations can be found here">Enums/STENCILOPERATION</page> * <page text="The Write Mask">render.SetStencilWriteMask</page> is a `byte` that will be <page text="bitwise ANDed">bit.band</page> with all values before they are written to the Stencil Buffer. * By default, the Write Mask is set to make no change to written values. * The <page text="Fail Operation">render.SetStencilFailOperation</page> is performed on pixels that Fail the Compare Function. `[This pixel is done and the next one can be processed.]` For the pixels that Pass the Compare Function, there is one additional check they must Pass before they are allowed to be modified by the draw operation. * The **Depth Test** (or "Z test") compares the pixel's current Depth Buffer value against the Depth value of the draw operation. Unlike the Compare Function, this comparison is fixed and cannot be configured. * The <page text="Z Fail Operation">render.SetStencilZFailOperation</page> is performed on the Stencil Buffer for pixels where the current Depth Buffer value is less than the draw operation's proposed depth value. This indicates that the draw operation should not be visible on this pixel because it is drawing behind the current pixel. `[The next pixel can be processed.]` If a pixel has passed both the Compare Function as well as the Depth Test, it is considered Passed and will have its contents replaced/modified by the draw operation. This will affect the Color, Depth, and Stencil values for the pixel. If a pixel has passed both the Compare Function as well as the Depth Test, it is considered Passed and will have its contents replaced/modified by the draw operation. This will affect the Color, Depth, and Stencil Buffer values for the pixel. * The <page text="Pass Operation">render.SetStencilPassOperation</page> is performed on the Stencil Buffer for pixels that have passed both the Compare Function and the Depth Test and are being updated by the draw operation. `[The next pixel can be processed.]` The following flowchart outlines the same information as the text above to aid in understanding. <image src="b2b4c/8dc4830dfd0658f.png" size="67100" name="StencilSystemFlow.png" /> # How Can Those Parts Be Used to Make Things? ## Common Usage Pattern In practical applications, Stencils are used in the following pattern: 1. Configure the Stencil system 2. Perform draw operations to set Stencil Values⤶ 2. Perform draw operations to set Stencil Buffer values⤶ 3. Re-configure the Stencil system 4. Perform draw operations that are masked by the Stencil Values from steps 1-2 4. Perform draw operations that are masked by the Stencil Buffer values from steps 1-2 It's quite common for steps 1-2 and steps 3-4 to be repeated multiple times (Called "passes" and not to be confused with the term used to denote a pixel's pass/fail state) in order to configure the Stencil Buffer in ways that it cannot be with only a single pass. <example name="Simple 2D Masking"> <description> This example shows a simple masking operation on 2D content following the Common Usage Pattern outlined above. * First, we'll configure the Stencil system, including parts we aren't using, because we can't trust other addons to clean up after themselves when they use the Stencil system and we don't want their configuration to influence our results. * Next, we'll create our mask by performing a draw operation that will change the Stencil Buffer values for the pixels we want to draw onto without actually drawing anything onto the screen. * Then, we'll re-configure the Stencil system to stop modifying the Stencil Buffer during draw operations because we don't want to mess up the mask we just created. * Finally, we'll draw the content we want to display on the screen, masked by the values in the Stencil Buffer. </description> <code> local COLOR_BLACK = Color( 0, 0, 0 ) local COLOR_BLUE = Color( 34, 19, 233 ) local MATERIAL_COLOR = Material( "color" ) -- Performs a draw operation that will be used to create our mask -- This is a separate function only to make the hook's code easier to read. local function DrawMask() -- We only need to set this color or material is so we can be sure we're not drawing a fully -- transparent shape, which would prevent any pixels from being affected and thus prevent -- the mask from being created surface.SetDrawColor( COLOR_BLACK ) surface.SetMaterial( MATERIAL_COLOR ) -- Draw the mask surface.DrawTexturedRectRotated( -- Place it in the center of the screen ScrW()/2, ScrH()/2, -- Set its size to be based on the screen size ScrW()/4, ScrH()/4, -- Make it spin based on the current time CurTime() * 100 ) end -- Performs a draw operation that we'll be masking with Stencils -- This is a separate function only to make the hook's code easier to read. local function DrawContent() -- This time, the color we choose matters visually surface.SetDrawColor( COLOR_BLUE ) surface.SetMaterial( MATERIAL_COLOR ) -- Draw the content -- This is a rectangle that moves in a circle and rotates counter/anti-clockwise surface.DrawTexturedRectRotated( -- Bounce around near the center of the screen so we can see the masking more obviously ScrW()/2 + math.sin( CurTime() * 4 ) * ScrH()/8, ScrH()/2 + math.cos( CurTime() * 4 ) * ScrH()/8, -- Set its size to be based on the screen size ScrW()/4, ScrH()/4, -- Make it spin the opposite direction of the mask CurTime() * 100 * -1 ) end hook.Add( "HUDPaint", "StencilExampleHud", function() -- If you'd like to see the mask layer, you can comment this line out render.SetStencilEnable( true ) -- First, let's configure the parts of the Stencil system we aren't using right now so we know they -- won't affect what we're doing ------ Make sure we're starting with a Stencil Buffer where all pixels have a Stencil value of 0 render.ClearStencil() ------ 255 corresponds to 8 bits (1 byte) where all bits are 1 (11111111), which is a bitmask that won't ------ change anything render.SetStencilTestMask( 255 ) render.SetStencilWriteMask( 255 ) ------ If a pixel fully passes, or if it fails the depth test, don't modify its Stencil Buffer value ------ (KEEP the current value) render.SetStencilPassOperation( STENCILOPERATION_KEEP ) render.SetStencilZFailOperation( STENCILOPERATION_KEEP ) -- Now, let's confiure the parts of the Stencil system we are going to use ------ We're creating a mask, so we don't want anything we do right now to draw onto the screen ------ All pixels should fail the Compare Function (They should NEVER pass) render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_NEVER ) ------ When a pixel fails, which they all should, we want to REPLACE their current Stencil value with ------ whatever the Reference Value is ------ We can use whatever Reference Value we want for this; They have no special meaning. render.SetStencilReferenceValue( 9 ) render.SetStencilFailOperation( STENCILOPERATION_REPLACE ) -- At this point, we're ready to perform draw operations to create our mask DrawMask() -- Now, we need to re-configure the Stencil system so we can use the mask we just created ------ Like the Pass and Z Failure operations, we don't want to change the Stencil Buffer if a pixels ------ fails the Compare Function because that would change the mask render.SetStencilFailOperation( STENCILOPERATION_KEEP ) ------ We want to pass (and therefore draw on) pixels that match (Are EQUAL to) the Reference Value render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_EQUAL ) -- We're finally ready to draw the content we want to have masked DrawContent() render.SetStencilEnable( false ) end ) </code> <output> <upload src="b2b4c/8dc48274ea360ef.mp4" size="3544394" name="SimpleStencilExampleOutput.mp4" /> To help us visualize what's happening in the clip above, I've commented out the line that enables the Stencil system so we can see both the mask (in black) and the content (in blue) at the same time. <upload src="b2b4c/8dc48271dcf0b3c.mp4" size="5533749" name="SimpleStencilExampleVisibleMaskOutput.mp4" /> </output> </example>