<cat>Dev.Lua</cat> <validate>This tutorial has several problems:⤶ * Doesn't explain or provide alternatives to data serialization such as <page>util.TableToJSON</page>⤶ * SteamID() during the <page>GM:ShutDown</page> hook can return an empty string and the code doesn't handle this⤶ * Saving code doesn't explicitly define argument order which can cause data to write in the wrong order potentially resulting in data corruption⤶ </validate>⤶ # Introduction Data Storage is what we use when we want data to persist through map changes and server restarts. <note>Never trust the client with data. Only store, save, change, and load data from the server. If you require data on the client - Send it from the server</note>⤶ ⤶ For the sake of understanding the tutorial, imagine data storage as a table distribution of everything we want saved. Ex:⤶ <note>Never trust the client with data. Only store, save, change, and load data from the server. If you need to give data to the client, make sure to keep a backup of it on the server!</note>⤶ ⤶ There are multiple ways of storing data in Garry's Mod. In this tutorial, we will go over File Based Storage⤶ ⤶ # Rules with data-handling⤶ ⤶ When handling any data, there are some rules that we should follow:⤶ ⤶ * **Never** trust data that has been handled by a client. Keep all data backed up on the server as this is the only way that you can be sure that the data is authentic. Let's say we started letting clients keep track of their money what's stopping them from adding a few zeros on the end?⤶ ⤶ * **Always** use identification that the client can't manipulate: If we wanted to store a client's health when they log off, and we used their steam names as identification, what's stopping them from changing it before they next log on? Nothing. This is why we must use identification such as <page>Player:SteamID</page>'s as we know that this can never be changed by the client.⤶ ⤶ In case you couldn't tell, there is an over-arching theme here: **Never trust the client**.⤶ ⤶ Unfortunately, Garry's mod servers aren't predictable - they can easily crash without warning or become overloaded. Therefore, you should deal with the data as quickly as possible to avoid losing it. Do not wait until the player leaves to save the number of tokens that they have earned whilst playing, as the server may suffer a fatal crash before then. Furthermore, you should not decide to store data during the <page>GM:ShutDown</page> hook, as it may be too late to start saving data as some player's will have become invalid and that means their precious token count is now lost. However, you may be reading this and you think that your data should be saved at the very last minute, or is really not important enough to be saved every time it is altered - in which case, you can use the <page>GM:ShutDown</page> hook at your own risk!⤶ SteamID | Time Played | Deaths |⤶ --------|--------------|--------|⤶ | STEAM_0:0:000002 | 300 | 3 |⤶ | STEAM_0:0:000003 | 80 | 1 |⤶ | Etc... |⤶ ⤶ ⤶ ⤶ Each column holds a variable for a player. The first column is the key, how we differentiate different players (A player's SteamID is a good choice for the key as it's unique). And so each row represents the data of a single player⤶ ⤶ There are multiple ways of storing data in Garry's Mod. In this tutorial, we will go over File Based Storage⤶ # About Text files are a simple and effective tool that can be used to store data. File Based Storage does not require any external modules, it relies solely on the <page>file</page>, and it's pretty straightforward. For these reasons, it is the recommended method for beginners. # Delimiter-separated values ⤶ The most common and simplest method of storing data in text files is to split the string using key-characters, separating each player with a key character, and the different variables stored with another key character. ⤶ We will now make a File Based Storage system that tracks the player's playing time on the server and their total deaths. Our key characters will be:⤶ ⤶ * **"\n"** - To separate between players. New line = new player⤶ * **;** - To separate each stored variable. ⤶ Our file, that will hold the data (e.g. player_data.txt) will look something like this:⤶ ⤶ ⤶ Text files are a simple and effective tool that can be used to store data. File Based Storage does not require any external modules, it relies solely on the <page>file</page> library, and it's pretty straightforward. For these reasons, it is the recommended method for beginners. # Storing tables - util.TableToJSON⤶ ⤶ This is perhaps the most easiest method for beginners that will circumvent frustration when converting data types such as <page>Color</page> or <page>Vector</page>, so we will start with this method first. ⤶ ## Turning tables into strings⤶ ⤶ A text file is basically a string of characters. This means that we need to convert all of our data into a string of characters...⤶ ⤶ <page>util.TableToJSON</page> is a function that will turn a Lua table into the JSON format. You do not need to know what the JSON format is, just know that it is a string of characters - something that we can directly save into a file, whereas a Lua table cannot be. ⤶ Here is an example of how we can use <page>util.TableToJSON</page>:⤶ ⤶ ```lua⤶ -- Here we have a generic table, containing different types of data that we wish to save into a file:⤶ local storeOwnerData = {⤶ pos = Vector(1000,20,3),⤶ rot = Angle(0,90,0),⤶ col = Color(255,0,0,20),⤶ text = "Dave - Store Owner"⤶ }⤶ ⤶ -- Let's see what util.TableToJSON is going to give us to work with:⤶ print(util.TableToJSON(storeOwnerData))⤶ ``` SteamID;Playtime;Deaths⤶ SteamID;Playtime;Deaths⤶ ...⤶ ⤶ Output:⤶ ``` ⤶ ⤶ And with some values:⤶ ⤶ ```⤶ STEAM_0:0:0000001;10;0⤶ STEAM_0:0:0000002;300;3⤶ STEAM_0:0:0000003;80;2⤶ {"text":"Dave - Store Owner","rot":"{0 90 0}","pos":"[1000 20 3]","col":{"r":255.0,"b":0.0,"a":20.0,"g":0.0}}⤶ ``` ⤶ ⤶ The first thing we'll do is properly load the data from the text file when the server starts. ⤶ What we're doing is creating a table with SteamIDs as keys, and whose values are playtime and deaths⤶ ⤶ ⤶ ```⤶ local PlayerData = {} local function LoadPlayerData() local data = file.Read( "player_data.txt", "DATA" ) -- Read the file⤶ if not data then return end -- File doesn't exist⤶ ⤶ data = string.Split( data, "\n" ) -- Split the data into lines (Each line represents a player)⤶ for _, line in ipairs( data ) do -- Iterate through each line⤶ local args = string.Split( line, ";" ) -- Split the line into variables⤶ if ( #args < 3 ) then continue end -- The data from the file contains unexpected amount of entries, skip the line⤶ local id = args[1] -- Store the key variable, for comfortability's sake⤶ if not id then return end -- Something is wrong⤶ PlayerData[id] = {} -- Create the table⤶ -- Update our PlayerData table⤶ PlayerData[id].playtime = args[2]⤶ PlayerData[id].deaths = args[3]⤶ end⤶ end⤶ ⤶ LoadPlayerData() -- Load the data once the server launches⤶ ⤶ Notice how the Color, Angle and Vector structures get converted into distinguishable string structures? These unique formats allows the reversing function <page>util.JSONToTable</page> to convert the formats back into their original data structures.⤶ ⤶ Let's write some further code so that we can see reversal process that will need to take place when reading the file. ⤶ ```lua⤶ -- JSONData now stores the Lua table "storeOwnerData" in JSON format.⤶ local JSONData = util.TableToJSON(storeOwnerData)⤶ ⤶ local converted = util.JSONToTable(JSONData) PrintTable(converted) ``` ⤶ ⤶ Next, we will load the data of each player as they join.⤶ ⤶ Output:⤶ ``` hook.Add( "PlayerInitialSpawn", "LoadPlayerData", function( ply )⤶ local sid = ply:SteamID()⤶ local data = PlayerData[sid]⤶ if data then -- If the player exists in the database, load his data⤶ ply.playtime = data.playtime⤶ ply.deaths = data.deaths⤶ else -- If the player doesn't exist in the database, reset the data⤶ ply.playtime = 0⤶ ply.deaths = 0⤶ end⤶ end )⤶ col:⤶ a = 20⤶ b = 0⤶ g = 0⤶ r = 255⤶ pos = 1000.000000 20.000000 3.000000⤶ rot = 0.000 90.000 0.000⤶ text = Dave - Store Owner⤶ ``` ⤶ ⤶ Lastly, update the data on the server whenever a player disconnects, and write it whenever the map changes or the server restarts.⤶ ⤶ ```⤶ local function SavePlayerData( ply )⤶ local sid = ply:SteamID()⤶ if not PlayerData[sid] then PlayerData[sid] = {} end⤶ ⤶ PlayerData[sid].playtime = ply.playtime⤶ PlayerData[sid].deaths = ply.deaths⤶ ⤶ As you can see, the JSON string has been converted back into a table format. Furthermore, we can check that the data types have been returned to their original format by using <page>Global.isvector</page> with `converted.pos ` and <page>Global.isangle</page> with `converted.rot` which tell us that they have been converted back into a <page>Vector</page> and <page>Angle</page>. Unfortunately, passing `converted.col` into <page>Global.IsColor</page> highlights that it has not been correctly converted into the <page>Color</page> structure. Fortunately, we can write some code to convert it back to normal later on.⤶ ⤶ ## Saving JSON data⤶ ⤶ Now we've learnt how to convert a table into a storable format, and also how to reverse the stored format into a normal Lua table - we can now learn how to read/write files and save the JSON data.⤶ ⤶ Believe it or not, saving data to a file only requires one function: <page>file.Write</page>. Upon looking at the documentation we learn that it only needs two things:⤶ * A filename⤶ * Data to save⤶ So let's go ahead and and write some code to finish saving the storeOwnerData table into a file called `storedata.txt`:⤶ ⤶ ```lua⤶ local storeOwnerData = {⤶ pos = Vector(1000,20,3),⤶ rot = Angle(0,90,0),⤶ col = Color(255,0,0,20),⤶ text = "Dave - Store Owner"⤶ }⤶ ⤶ -- Let's put it in a function so that when we next change the table, we can just call this function again⤶ function SaveStoreData()⤶ local converted = util.TableToJSON(storeOwnerData)⤶ file.Write("storedata.txt", converted)⤶ end ⤶ --[[⤶ We only need to write the data into the file once - when the map changes or the server restarts⤶ However, we need to update our PlayerData table (Which is a sorted copy of our text file) when a player disconnects⤶ --]]⤶ ⤶ hook.Add( "PlayerDisconnected", "SavePlayerData1", SavePlayerData )⤶ ⤶ hook.Add( "ShutDown", "SavePlayerData2", function()⤶ for _, ply in ipairs( player.GetAll() ) do⤶ SavePlayerData( ply ) -- Save the data of all players currently connected⤶ end⤶ ⤶ -- Lastly, write our txt file⤶ local str = "" -- This is the string we will write to the file⤶ for id, args in pairs( PlayerData ) do⤶ str = str .. id -- The first stored value is the identifier⤶ for _, arg in pairs( args ) do -- Add each stored variable in our table to the string⤶ str = str .. ";" .. arg⤶ end⤶ ⤶ str = str .. "\n" -- Before moving on to the next player, start a new line⤶ end⤶ ⤶ file.Write( "player_data.txt", str )⤶ end )⤶ ⤶ SaveStoreData()⤶ ``` ⤶ ⤶ Of course, if we actually want these values to be correct, we should have a code that updates them⤶ ⤶ ```⤶ hook.Add( "PostPlayerDeath", "UpdatePlayerDataDeaths", function( ply )⤶ ply.deaths = ply.deaths + 1⤶ end ) ⤶ timer.Create( "UpdatePlayerDataTime", 10, 0, function() -- Repeat every 10 seconds because accuracy isn't a necessity when counting playtime⤶ for _, ply in ipairs( player.GetAll() ) do⤶ ply.playtime = ply.playtime + 10 -- Again, accuracy isn't a necessity when counting playtime. -- If we wanted to be accurate, we'd have to use The TimeConnected function for the first occurance⤶ end⤶ end ) ⤶ That's it, that's the file saved!⤶ ⤶ If you want to view the file for your own eyes, you can go into the `garrysmod/data` folder and the file will be called "storedata.txt".⤶ ⤶ ## Reading JSON data from files, and converting it back to a table⤶ ⤶ Reading JSON data is just as easy with the key function now being <page>file.Read</page> that takes one argument... the Filename. Here we will also convert the table containing color data into a Color structure too:⤶ ```lua⤶ ⤶ local storeOwnerData = {} ⤶ function ReadStoreData()⤶ -- Make sure you use the same filename as the one in file.Write!⤶ local JSONData = file.Read("storedata.txt") ⤶ -- JSONData is currently a JSON string - let's convert that into a table:⤶ storeOwnerData = util.JSONToTable(JSONData)⤶ -- Remember how the col value does not get converted into a Color structure? Let's fix that:⤶ ⤶ local oldCol = storeOwnerData.col⤶ ⤶ storeOwnerData.col = Color(oldCol.r, oldCol.g, oldCol.b, oldCol.a)⤶ end⤶ ``` ⤶ ⤶ The final product:⤶ ⤶ ```⤶ local PlayerData = {}⤶ local function LoadPlayerData()⤶ local data = file.Read( "player_data.txt", "DATA" )⤶ if not data then return end⤶ data = string.Split( data, "\n" )⤶ for _, line in ipairs( data ) do⤶ local args = string.Split( line, ";" )⤶ if ( #args < 3 ) then continue end⤶ local id = args[1]⤶ if not id then return end⤶ PlayerData[id] = {}⤶ PlayerData[id].playtime = args[2]⤶ PlayerData[id].deaths = args[3]⤶ end⤶ ⤶ The variable `storeOwnerData` now contains all of the data and the `col` value is now a <page>Color</page>!⤶ ⤶ # Writing data types other than tables⤶ ⤶ The above method is excellent for storing tables, but what if we just want to store a number or a string?⤶ ⤶ Nothing is particularly different in this scenario, we just don't need the util functions anymore:⤶ ⤶ ## Saving the string⤶ ⤶ ```lua⤶ local myString = "This is a string that I want to save!"⤶ ⤶ function SaveString() -- We are saving the contents of myString into a file called "stringdata.txt"⤶ file.Write("stringdata.txt", myString)⤶ end⤶ ⤶ SaveString()⤶ end ⤶ LoadPlayerData()⤶ ⤶ hook.Add( "PlayerInitialSpawn", "LoadPlayerData", function( ply )⤶ local sid = ply:SteamID()⤶ local data = PlayerData[sid]⤶ if data then⤶ ply.playtime = data.playtime⤶ ply.deaths = data.deaths⤶ else ⤶ ply.playtime = 0⤶ ply.deaths = 0⤶ end⤶ end )⤶ ⤶ local function SavePlayerData( ply )⤶ local sid = ply:SteamID()⤶ if not PlayerData[sid] then PlayerData[sid] = {} end⤶ ⤶ PlayerData[sid].playtime = ply.playtime⤶ PlayerData[sid].deaths = ply.deaths⤶ ```⤶ ⤶ ## Loading the string⤶ ⤶ ```lua⤶ local myString = ""⤶ ⤶ function LoadString()⤶ myString = file.Read("stringdata.txt")⤶ end ⤶ ⤶ hook.Add( "PlayerDisconnected", "SavePlayerData1", SavePlayerData )⤶ hook.Add( "ShutDown", "SavePlayerData2", function()⤶ for _, ply in ipairs( player.GetAll() ) do⤶ SavePlayerData( ply )⤶ end⤶ ⤶ local str = ""⤶ for id, args in pairs( PlayerData ) do⤶ str = str .. id⤶ for _, arg in pairs( args ) do⤶ str = str .. ";" .. arg⤶ end⤶ ⤶ str = str .. "\n"⤶ end⤶ ⤶ file.Write( "player_data.txt", str )⤶ end )⤶ ⤶ hook.Add( "PostPlayerDeath", "UpdatePlayerDataDeaths", function( ply )⤶ ply.deaths = ply.deaths + 1⤶ end )⤶ ⤶ timer.Create( "UpdatePlayerDataTime", 10, 0, function()⤶ for _, ply in ipairs( player.GetAll() ) do⤶ ply.playtime = ply.playtime + 10⤶ end⤶ end )⤶ ⤶ LoadString()⤶ ``` ⤶ ⤶ <note>When using Delimiter-Separated Values, **do not save any data that the user can edit**. For example, in the above example, if a player with the username **;\n\n;;\n** joined the server, and we were to store his username, it would break the code</note>⤶ <note>Alternatively, you can convert your string to a series of numbers, or a sterilized string that matches your key characters. However, it is recommended to avoid doing this, and use SQL instead</note>⤶ ⤶ ⤶ ⤶ Not that difficult at all, is it?⤶ ⤶ # Alternatives to file based storage⤶ ⤶ Let's say that we needed to keep track of a player's money count, we will need to save to the file every time the player either leaves or their money count is updated and thus a 1,000 line file will need to be read, altered, and then completely overwritten with new data - which is clearly not a particularly efficient method.⤶ ⤶ Therefore, we need an alternative.⤶ ⤶ Perhaps the most efficient method of storing large amounts of data in Garry's mod is through SQL. Unfortunately, storing data in SQL is not as easy to learn as file based storage, but it is completely worth it if you do decide to learn (perhaps I'll write another tutorial for it).⤶ ⤶ You may think that PData may be a viable alternative... At the time of writing this (June 2020), I cannot recommend <page>Player:SetPData</page>, as PData has a major issue that has not been resolved yet (the issue is currently on the SetPData page, if it is ever removed, then you are safe to use it!).