render.SetStencilFailOperation
Example
This is one of the most useful things stencils can do - not render something to the screen and then allow you to draw with the shape it would have been if it had rendered.
You can see in the output screenshot that a partially obscured entity will still write its entire shape to the stencil buffer.
From Lex's Stencil Tutorial.
hook.Add( "PostDrawOpaqueRenderables", "Stencil Tutorial Example", function()
-- Reset everything to known good
render.SetStencilWriteMask( 0xFF )
render.SetStencilTestMask( 0xFF )
render.SetStencilReferenceValue( 0 )
render.SetStencilCompareFunction( STENCIL_ALWAYS )
render.SetStencilPassOperation( STENCIL_KEEP )
render.SetStencilFailOperation( STENCIL_KEEP )
render.SetStencilZFailOperation( STENCIL_KEEP )
render.ClearStencil()
-- Enable stencils
render.SetStencilEnable( true )
-- Set the reference value to 1. This is what the compare function tests against
render.SetStencilReferenceValue( 1 )
-- Only draw things if their pixels are currently 1. Currently this is nothing.
render.SetStencilCompareFunction( STENCIL_EQUAL )
-- If something fails to draw to the screen, set the pixels it would have drawn to 1
-- This includes if it's behind something.
render.SetStencilFailOperation( STENCIL_REPLACE )
-- Draw our entities. They will not draw, because everything is 0
for _, ent in ipairs( ents.FindByClass( "prop_physics" ) ) do
ent:DrawModel()
end
-- If we were to re-draw our entities, we'd see them, but otherwise they're invisible.
-- If we flush the screen, we can show the "holes" they've left in the stencil buffer
render.ClearBuffersObeyStencil( 0, 148, 133, 255, false )
-- Let everything render normally again
render.SetStencilEnable( false )
end )
Output: 
Garry's Mod
Rust
Steamworks
Wiki Help
S&box