Imagine you shoot a gun in the game. Without prediction you would press the mouse button, it'd send a message to the server, the server would run the command, the gun would shoot on the server, it'd send sounds and effects down to the client, then you'd see the effects happen. We're talking about a 30ms round trip at best, half a second at worst. It would be noticeable.
With prediction it runs the same, but while it's waiting for the server it predicts what's going to happen. It runs the command clientside, so everything is instant.
When the server runs the Tick and sends the results to the client, the client can compare the entity and it's predicted variables to the actual results from the tick that ran on the server.
If any of the values are different then we have a prediction error. So the client copies all the variables from the server and re-runs any subsequent ticks to try and fix it all.
Anything that changes your pawn's position or rotation is predicted. Movement code should match and be deterministic on both client and server. The physics aren't deterministic so you can't use physics in a predictable fashion.
In your weapons, you should make things like ammo count predictable.
The main entry point for prediction is in your game class. This is the default behaviour if you don't override it.
As you can see the default behaviour is to call Simulate on the client's Pawn. In this function you should do movement and input.
If an entity has your Pawn as the owner it becomes predictable too. So, for example, you might call
Weapon.Simulate( cl ) in your pawn's Simulate method to allow the weapons to fire.
Imagine again if you shoot a gun, but this time it plays a sound. First of all, you'd hear the gunshot from the client version (which only you would hear obviously)… but then, when the server version is run, it would also play a sound for all the other clients to hear.
The predicting client running the prediction doesn't hear this sound because it's culled. The server assumes that this client has already heard it so there's no need to send it again.
The same goes for RPCs.
You might want to force the client to hear this sound. Maybe you're running some code that can only run on the server, like if you shot something and someone died. That code wouldn't run on the client - so it couldn't be predictable. In these cases you can turn prediction off, like this.