Harden runner edge cases

LLM_GUIDE.md

@@ -9,7 +9,7 @@ Your job when writing tests: read the code under test first, then produce test f
 - Test files live at `lua/tests/<project>/` (any name for `<project>`, e.g. your addon's name). Subdirectories are scanned recursively; every `.lua` file found is loaded as one test group.
 - Files placed directly in `lua/tests/` (not inside a project folder) are **not** loaded.
 - Each test file must `return` a single test group table. A file whose returned table has no `cases` field is ignored with a warning. File-local helpers and constants above the `return` are fine.
-- Tests run **serverside**.
+- Tests run serverside by default. Cases marked `clientside = true` or `shared = true` can run on clients when `gluatest_client_enable 1` is set.
 
 Minimal runnable skeleton:
 
@@ -56,6 +56,8 @@ This is the complete list of case fields:
 | `cleanup` | `function( state )` | — | Runs after the case, **even if it failed or errored** |
 | `when` | boolean, function, or list of either | — | Case runs only if every condition is/returns `true` |
 | `skip` | boolean or function | — | Case is skipped if `true`/returns `true`; takes precedence over `when` |
+| `clientside` | boolean | `false` | Run on clients when clientside tests are enabled |
+| `shared` | boolean | `false` | Run serverside and on clients when clientside tests are enabled |
 
 Notes:
 - `when = false` (or a function returning anything but `true`) skips the case. A list form is supported: `when = { system.IsLinux(), function() return game.IsDedicated() end }`.

QUICKSTART.md

@@ -80,4 +80,4 @@ jobs:
 
 That's it! You have a working automated test runner for your project.
 
-Check out the README for more details about [before/after functions](https://github.com/CFC-Servers/GLuaTest/blob/main/README.md#before--after-functions) and [how to make async tests](https://github.com/CFC-Servers/GLuaTest/blob/main/README.md#async-tests-and-the-done-function).
+Check out the README for more details about [before/after functions](https://github.com/CFC-Servers/GLuaTest/blob/main/README.md#before--after-functions) and [how to make async tests](https://github.com/CFC-Servers/GLuaTest/blob/main/README.md#async-tests-and-the-donefail-functions).

README.md

@@ -21,7 +21,7 @@ GLuaTest takes a lot of inspiration from both Ruby's [RSpec](https://rspec.info/
  - [Troubleshooting](#troubleshooting-)
  - [Developers](#developers-)
 
-![GLuaLint](https://github.com/CFC-Servers/GLuaTest/actions/workflows/glualint.yml/badge.svg)
+![GLuaLint](https://github.com/CFC-Servers/GLuaTest/actions/workflows/lint.yml/badge.svg)
 
 ---
 _(Are you an impatient software developer? Check out the [quickstart](https://github.com/CFC-Servers/GLuaTest/blob/main/QUICKSTART.md) guide to go fast)_
@@ -340,8 +340,8 @@ jobs:
 | `branch`             | Which GMod branch to run your tests on                                                                                 | `public`|`prerelease`|`dev`|`x86-64`                                                                      |
 | `gluatest-ref`       | Which tag/branch of GLuaTest to run                                                                                    | `main`|`feature/new-feature-branch`                                                                     |
 | `custom-overrides`   | An absolute path with custom files to copy to the server directly. Structure should match the contents of `garrysmod/` | `$GITHUB_WORKSPACE/my_overrides`                                                                        |
-| `download-artifact`  | A URL path to a .tar.gz file that will be unpacked in the root directory                                               | `https://github.com/RaphaelIT7/gmod-holylib/releases/download/Release0.7/gmsv_holylib_linux_packed.zip` |
-| `additional-setup`   | If specificed, executes the given string as a script after all setup is complete, allowing additional setup            | `echo "Hello, this is a test!"`                                                                         |
+| `download-artifact`  | A URL path to a .tar.gz file that will be unpacked in the root directory                                               | `https://example.com/my-artifact.tar.gz`                                                                |
+| `additional-setup`   | If specified, executes the given string as a script after all setup is complete, allowing additional setup             | `echo "Hello, this is a test!"`                                                                         |
 
 </summary>
 </details>
@@ -405,13 +405,21 @@ You can do that with simple environment variables, i.e.:
 export REQUIREMENTS=/absolute/path/to/requirements.txt
 export CUSTOM_SERVER_CONFIG=/absolute/path/to/server.cfg
 export PROJECT_DIR=/home/me/Code/my_project
+export GMOD_BRANCH="public"
+export GMOD_ARTIFACT_DIR=/tmp/gluatest-artifacts
 export GAMEMODE="sandbox"
 export COLLECTION_ID="12345"
 export SSH_PRIVATE_KEY="the-entire-private-key"
 export GITHUB_TOKEN="a-personal-access-token"
 ```
 
- - You can skip the `REQUIREMENTS` and `CUSTOM_SERVER_CONFIG` if you don't need them, but you must set the `PROJECT_DIR` variable.
+ - The `REQUIREMENTS` and `CUSTOM_SERVER_CONFIG` files can be empty, but Docker Compose needs them to exist.
+
+ - The `PROJECT_DIR` variable must point to the project or override directory you want copied into the test server.
+
+ - The `GMOD_BRANCH` variable selects the Docker image tag. Use `public` unless you need `x86-64`, `prerelease`, or `dev`.
+
+ - The `GMOD_ARTIFACT_DIR` variable must point to an existing directory. It can be empty.
 
  - The `GAMEMODE` variable defaults to `"sandbox"`, so you can omit it if that's appropriate for your tests.
 
@@ -514,6 +522,8 @@ Each Test Case is a table with the following keys:
 
 <br>
 
+`clientside` and `shared` are also supported for test cases that should run on clients when `gluatest_client_enable 1` is set.
+
 ### The `expect` function
 The heart of a test is the _expectation_. You did a thing, and now you expect a result.
 
@@ -541,7 +551,7 @@ There are a number of different expectations you can use.
 | **`deepEqual`**      | Expects that two tables are deeply equal                      | `expect( {{ Entity(1) }} ).to.deepEqual( {{ Entity(1) }} )`     |
 | **`beLessThan`**     | Basic `<` comparison                                          | `expect( 5 ).to.beLessThan( 6 )`                                |
 | **`beGreaterThan`**  | Basic `>` comparison                                          | `expect( 10 ).to.beGreaterThan( 1 )`                            |
-| **`beBetween`**      | Expects the subject to be less than min, and greater than max | `expect( 5 ).to.beBetween( 3, 7 )`                              |
+| **`beBetween`**      | Expects the subject to be between the lower and upper bounds  | `expect( 5 ).to.beBetween( 3, 7 )`                              |
 | **`beTrue`**         | Expects the subject to literally be `true`                    | `expect( Entity( 1 ):IsPlayer() ).to.beTrue()`                  |
 | **`beFalse`**        | Expects the subject to literally be `false`                   | `expect( istable( "test" ) ).to.beFalse()`                      |
 | **`beValid`**        | Expects `IsValid( value )` to return `true`                   | `expect( ply ).to.beValid()`                                    |
@@ -593,7 +603,7 @@ For example, to run your test case only on the `x86-64` branch:
 Skipping is also handy if you want to disable a test but keep the code:
 ```lua
 {
-    name = "Broken test (but I'll definitely fix it some day 100%),
+    name = "Broken test (but I'll definitely fix it some day 100%)",
     skip = true,
     func = function() error() end
 }
@@ -722,7 +732,7 @@ return {
         },
         {
             name = "Should check if user exists in database",
-            func = function()
+            func = function( state )
                 local dbCheck = stub( MyProject, "UserExistsInDatabase" ).returns( true )
 
                 MyProject.CheckUser( state.validUser )

docker/base_server.cfg

@@ -31,4 +31,4 @@ sv_cacheencodedents 0
 net_compresspackets 0
 sv_stats 0
 
-gluatest_enable 1
+gluatest_server_enable 1

docker/entrypoint.sh

@@ -19,8 +19,8 @@ echo "Copying serverfiles overrides..."
 rsync --verbose --archive $home/garrysmod_override/ $server/
 
 # Any additional files
-if [ -n "$(ls -A $home/_gluatest_artifacts/_gluatest_artifacts/)" ]; then # Only execute if there are any artifacts or else tar will complain
-    cp $home/_gluatest_artifacts/_gluatest_artifacts/* $gmodroot/
+if [ -n "$(ls -A $home/_gluatest_artifacts/)" ]; then # Only execute if there are any artifacts or else tar will complain
+    cp $home/_gluatest_artifacts/* $gmodroot/
 
     for file in $gmodroot/*.tar.gz; do \
         tar --extract --verbose --ungzip --file="$file" --directory="$gmodroot" && \

lua/gluatest/expectations/negative.lua

@@ -204,9 +204,7 @@ return function( subject, ... )
 
         local success, err = pcall( subject, unpack( args ) )
 
-        if success == true then
-            i.expected( "to error" )
-        else
+        if success == false then
             if string.StartsWith( err, "lua/" ) or string.StartsWith( err, "addons/" ) then
                 local _, endOfPath = string.find( err, ":%d+: ", 1 )
                 assert( endOfPath, "Could not find end of path in error message: " .. err )
@@ -225,6 +223,8 @@ return function( subject, ... )
     --- the Positive expectation lets you specify how many times it
     --- should have been called, but this one does not
     function expectations.called()
+        assert( subject.IsStub, ".called expects a stub" )
+
         local callCount = subject.callCount
         if callCount > 0 then
             i.expected( "to not have been called, got: %d", callCount )

lua/gluatest/expectations/positive.lua

@@ -69,6 +69,7 @@ return function( subject, ... )
     --- @param tolerance? number Tolerance for the comparison
     function expectations.aboutEqual( comparison, tolerance )
         assert( TypeID( subject ) == TYPE_NUMBER, ".aboutEqual expects a number" )
+        assert( TypeID( comparison ) == TYPE_NUMBER, ".aboutEqual expects a number" )
 
         tolerance = tolerance or 0.00001
         local difference = math.abs( subject - comparison )
@@ -169,7 +170,7 @@ return function( subject, ... )
         local class = type( subject )
 
         if class ~= comparison then
-            i.expected( "to not be an '%s'", comparison )
+            i.expected( "to be an '%s'", comparison )
         end
     end
 
@@ -231,7 +232,7 @@ return function( subject, ... )
                 i.expected( "to have been called at least once " )
             end
         else
-            if callCount < n then
+            if callCount ~= n then
                 i.expected( "to have been called exactly %d times, got: %d", n, callCount )
             end
         end

lua/gluatest/expectations/utils/table_diff.lua

@@ -12,7 +12,7 @@ local function stringifyKey( key )
         key = "\"" .. key:gsub( "\"", "\\\"" ) .. "\""
     end
 
-    return "[" .. key .. "]"
+    return "[" .. tostring( key ) .. "]"
 end
 
 local function GetDiff( t1, t2, path )

lua/gluatest/loader.lua

@@ -5,10 +5,11 @@ local Loader = {}
 --- @param filePath string
 --- @param cases GLuaTest_TestCase[]
 function Loader.checkSendToClients( filePath, cases )
+    if not SERVER then return end
     if not GLuaTest.RunClientsideConVar:GetBool() then return end
 
     for _, case in ipairs( cases ) do
-        if case.clientside then
+        if case.clientside or case.shared then
             return AddCSLuaFile( filePath )
         end
     end
@@ -26,8 +27,12 @@ end
 --- @param filePath string
 --- @return table
 function Loader.simpleError( reason, filePath )
+    reason = tostring( reason or "file did not return a test group" )
+    local linePrefixStart = string.find( reason, ":%d+:" )
+    if linePrefixStart then reason = string.sub( reason, linePrefixStart + 1 ) end
+
     return {
-        reason = string.sub( reason, string.find( reason, ":" ) + 1 ),
+        reason = reason,
         sourceFile = filePath,
         lineNumber = -1,
         locals = {}
@@ -44,6 +49,10 @@ function Loader.processFile( dir, fileName, groups )
     local filePath = dir .. "/" .. fileName
     local success, result = pcall( function( givenFilePath )
         local fileContent = file.Read( givenFilePath, "LUA" )
+        if not fileContent then
+            return Loader.simpleError( "Unable to read file: " .. givenFilePath, givenFilePath )
+        end
+
         local compiled = CompileString( fileContent, "lua/" .. givenFilePath, false )
 
         if not isfunction( compiled ) then

lua/gluatest/runner/helpers.lua

@@ -30,7 +30,7 @@ function Helpers.makeHookTable()
         if not trackedHooks[event] then trackedHooks[event] = {} end
         table.insert( trackedHooks[event], name )
 
-        if not isfunction( func ) and func.IsStub then
+        if istable( func ) and func.IsStub then
             local givenStub = func
             func = function( ... )
                 givenStub( ... )
@@ -183,10 +183,10 @@ function Helpers.findStackInfo()
     -- Step up through the stacks to find the error we care about
 
     for stack = 1, 12 do
-        local info = debug.getinfo( stack, "lnS" )
+        local info = debug.getinfo( stack, "lnSn" )
         if not info then break end
 
-        local emptyName = #info.namewhat == 0
+        local emptyName = #( info.namewhat or "" ) == 0
         local notGluatest = not string.match( info.short_src, "/lua/gluatest/" )
 
         if emptyName and notGluatest then
@@ -259,33 +259,36 @@ function Helpers.MakeAsyncEnv( done, fail, onFailedExpectation )
         -- they're called in an async function
         expect = function( ... )
             local built = expect( ... )
-            local expected = built.to.expected
-            local recordedFailure = false
 
-            -- Wrap the error-throwing function
-            -- and handle the error with the correct context
-            -- (and to only record the first failure)
-            built.to.expected = function( ... )
-                if recordedFailure then return end
+            local function wrapExpected( expectations )
+                local expected = expectations.expected
+                local recordedFailure = false
 
-                local stack = debug.getinfo( 3, "lnS" )
-                local locals = Helpers.getLocals( 4 )
+                expectations.expected = function( ... )
+                    if recordedFailure then return end
 
-                local _, errInfo = xpcall( expected, Helpers.FailCallback, ... )
-                assert( errInfo )
+                    local stack = debug.getinfo( 3, "lnS" )
+                    local locals = Helpers.getLocals( 4 )
 
-                local final = {
-                    reason = errInfo.reason,
-                    sourceFile = stack.short_src,
-                    lineNumber = stack.currentline,
-                    locals = locals
-                }
+                    local _, errInfo = xpcall( expected, Helpers.FailCallback, ... )
+                    assert( errInfo )
 
-                onFailedExpectation( final --[[@as GLuaTest_FailCallbackInfo]] )
+                    local final = {
+                        reason = errInfo.reason,
+                        sourceFile = stack.short_src,
+                        lineNumber = stack.currentline,
+                        locals = locals
+                    }
 
-                recordedFailure = true
+                    onFailedExpectation( final --[[@as GLuaTest_FailCallbackInfo]] )
+
+                    recordedFailure = true
+                end
             end
 
+            wrapExpected( built.to )
+            wrapExpected( built.notTo )
+
             return built
         end,
 
@@ -323,23 +326,35 @@ function Helpers.SafeRunWithEnv( defaultEnv, before, func, state )
         return ogExpect( ... )
     end
 
-    -- Before
-    if before then
-        setfenv( before, testEnv )
-        before( state )
-        setfenv( before, defaultEnv )
+    local function cleanup()
+        for _, cleanupFunc in ipairs( cleanupFuncs ) do
+            cleanupFunc()
+        end
     end
 
-    -- Run
-    setfenv( func, testEnv )
-    local success, output = xpcall( func, Helpers.FailCallback, state )
-    setfenv( func, defaultEnv )
+    local function runWithEnv( funcToRun )
+        setfenv( funcToRun, testEnv )
+        local success, output = xpcall( funcToRun, Helpers.FailCallback, state )
+        setfenv( funcToRun, defaultEnv )
 
-    -- Cleanup
-    for _, cleanup in ipairs( cleanupFuncs ) do
-        cleanup()
+        return success, output
     end
 
+    if before then
+        local beforeSuccess, beforeOutput = runWithEnv( before )
+        if not beforeSuccess then
+            cleanup()
+
+            return {
+                result = "failure",
+                errInfo = beforeOutput
+            }
+        end
+    end
+
+    local success, output = runWithEnv( func )
+    cleanup()
+
     if success then
         -- If it succeeded but never ran `expect`, it's an empty test
         if not ranExpect then

lua/gluatest/runner/runner.lua

@@ -28,12 +28,13 @@ end
 --- @param testGroups GLuaTest_TestGroup[]
 function TestRunner:Complete( testGroups )
     local duration = SysTime() - self.startTime
+    local results = self.results
 
-    hook.Run( "GLuaTest_Finished", testGroups, self.results, duration )
+    hook.Run( "GLuaTest_Finished", testGroups, results, duration )
 
     -- Some logs may be printed after our completion message, we need to wait a bit
     timer.Simple( 0.1, function()
-        LogTestsComplete( testGroups, self.results, duration )
+        LogTestsComplete( testGroups, results, duration )
     end )
 end
 
@@ -46,6 +47,7 @@ function TestRunner:Run( testGroups )
 
     hook.Run( "GLuaTest_StartedTestRun", testGroups )
     self.startTime = SysTime()
+    self.results = {}
 
     --- @type GLuaTest_TestGroupRunner[]
     local runners = {}
@@ -55,9 +57,11 @@ function TestRunner:Run( testGroups )
         table.insert( runners, runner )
     end
 
+    local runnerIndex = 1
     local function runNext()
         --- @type GLuaTest_TestGroupRunner
-        local nextRunner = table.remove( runners )
+        local nextRunner = runners[runnerIndex]
+        runnerIndex = runnerIndex + 1
 
         if not nextRunner then
             return self:Complete( testGroups )