Skip to content

Adding more tests #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LowLevelLore wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Adding more tests #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ccce0ec
Adding more tests with the new framework
LowLevelLore May 30, 2026
2caca62
Merge remote-tracking branch 'origin/main' into comprehensive-testing
LowLevelLore May 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions generate_expected_outputs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
#!/usr/bin/env python3
import subprocess
import sys
from pathlib import Path
import tempfile
import shutil

ZPILER = Path("/home/xzist/zust/build/zpiler")
TESTS_DIR = Path("/home/xzist/zust/tests")
RUNTIME_DIR = TESTS_DIR / "runtime"
EXPECTED_DIR = TESTS_DIR / "expected" / "runtime"
Comment on lines +5 to +11

@sourcery-ai sourcery-ai Bot May 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (bug_risk): Avoid hardcoded absolute paths to make the script portable within the repo

Using /home/xzist/... makes this script unusable outside your machine and likely to fail in CI. Instead, derive these paths relative to this file (e.g. via Path(__file__).resolve() to the repo root, then into build/zpiler and tests) so it works for all environments.

Suggested change
import tempfile
import shutil
ZPILER = Path("/home/xzist/zust/build/zpiler")
TESTS_DIR = Path("/home/xzist/zust/tests")
RUNTIME_DIR = TESTS_DIR / "runtime"
EXPECTED_DIR = TESTS_DIR / "expected" / "runtime"
import tempfile
import shutil
# Derive repository-root-relative paths so the script is portable
SCRIPT_PATH = Path(__file__).resolve()
REPO_ROOT = SCRIPT_PATH
while not ((REPO_ROOT / "build").is_dir() and (REPO_ROOT / "tests").is_dir()):
if REPO_ROOT.parent == REPO_ROOT:
raise RuntimeError(
"Could not locate repository root containing 'build' and 'tests' directories"
)
REPO_ROOT = REPO_ROOT.parent
ZPILER = REPO_ROOT / "build" / "zpiler"
TESTS_DIR = REPO_ROOT / "tests"
RUNTIME_DIR = TESTS_DIR / "runtime"
EXPECTED_DIR = TESTS_DIR / "expected" / "runtime"


def run_command(cmd, check=True):
result = subprocess.run(cmd, capture_output=True, text=False)

@sourcery-ai sourcery-ai Bot May 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security (python.lang.security.audit.dangerous-subprocess-use-audit): Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'.

Source: opengrep

if check and result.returncode != 0:
print(f"Command failed: {' '.join(cmd)}")
print(f"stdout: {result.stdout.decode()}")
print(f"stderr: {result.stderr.decode()}")
sys.exit(1)
return result

def generate_outputs_for_test(test_file):
"""Generate expected outputs for a single test file"""
rel_path = test_file.relative_to(RUNTIME_DIR)
expected_dir = EXPECTED_DIR / rel_path.parent
expected_dir.mkdir(parents=True, exist_ok=True)

test_name = test_file.stem

# Compile to assembly
with tempfile.TemporaryDirectory() as tmpdir:
tmpdir = Path(tmpdir)
asm_file = tmpdir / f"{test_name}.s"
obj_file = tmpdir / f"{test_name}.o"
exe_file = tmpdir / f"{test_name}"

# Compile
print(f"Compiling {rel_path}...", end=" ")
result = run_command([str(ZPILER), "--format", "x86_64-linux", "-o", str(asm_file), str(test_file)], check=False)
if result.returncode != 0:
print(f"FAILED to compile")
return False

# Assemble
result = run_command(["as", str(asm_file), "-o", str(obj_file)], check=False)
if result.returncode != 0:
print(f"FAILED to assemble")
return False
Comment on lines +39 to +48

@sourcery-ai sourcery-ai Bot May 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Surface more diagnostic info when compile/assemble/link steps fail

Because check=False is used, failures here only show a generic FAILED to ... message and hide the underlying error output, which makes debugging test failures difficult. Please either use check=True so run_command prints stdout/stderr on failure, or explicitly log result.stdout and result.stderr in these non‑zero return cases.

Suggested implementation:

        # Compile
        print(f"Compiling {rel_path}...", end=" ")
        result = run_command(
            [str(ZPILER), "--format", "x86_64-linux", "-o", str(asm_file), str(test_file)],
            check=True,
        )
        if result.returncode != 0:
            print("FAILED to compile")
            return False

        # Assemble
        result = run_command(["as", str(asm_file), "-o", str(obj_file)], check=True)
        if result.returncode != 0:
            print("FAILED to assemble")
            return False

        # Link
        result = run_command(["gcc", str(obj_file), "-o", str(exe_file)], check=True)
        if result.returncode != 0:
            print("FAILED to link")
            return False
def run_command(cmd, check=True):
    result = subprocess.run(cmd, capture_output=True, text=False)
    if check and result.returncode != 0:
        print(f"Command failed: {' '.join(cmd)}")
        if result.stdout:
            print(f"stdout:\n{result.stdout.decode(errors='replace')}")
        if result.stderr:
            print(f"stderr:\n{result.stderr.decode(errors='replace')}")

If subprocess is not already imported at the top of generate_expected_outputs.py, you should add:

import subprocess

near the other imports.


# Link
result = run_command(["gcc", str(obj_file), "-o", str(exe_file)], check=False)
if result.returncode != 0:
print(f"FAILED to link")
return False

# Run
result = run_command([str(exe_file)], check=False)

# Save outputs
stdout_file = expected_dir / f"{test_name}.stdout"
stderr_file = expected_dir / f"{test_name}.stderr"
exitcode_file = expected_dir / f"{test_name}.exitcode"

stdout_file.write_bytes(result.stdout)
stderr_file.write_bytes(result.stderr)
exitcode_file.write_bytes(str(result.returncode).encode())

print(f"OK (exit code: {result.returncode})")
return True

def main():
# Find all .zz files in runtime directory
test_files = sorted(RUNTIME_DIR.rglob("*.zz"))

success_count = 0
fail_count = 0

for test_file in test_files:
if generate_outputs_for_test(test_file):
success_count += 1
else:
fail_count += 1

print(f"\nGenerated outputs for {success_count} tests, {fail_count} failed")

if __name__ == "__main__":
main()
Binary file added program
View file
Open in desktop
Binary file not shown.
1 change: 1 addition & 0 deletions tests/expected/runtime/conditionals/boolean_logic.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/conditionals/boolean_logic.stderr
View file
Open in desktop
Empty file.
10 changes: 10 additions & 0 deletions tests/expected/runtime/conditionals/boolean_logic.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
true
false
false
true
false
true
false
true
true
true
1 change: 1 addition & 0 deletions tests/expected/runtime/conditionals/complex_bool.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/conditionals/complex_bool.stderr
View file
Open in desktop
Empty file.
3 changes: 3 additions & 0 deletions tests/expected/runtime/conditionals/complex_bool.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
1
1
1
2 changes: 1 addition & 1 deletion tests/expected/runtime/conditionals/if-elif-else.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
2 changes: 1 addition & 1 deletion tests/expected/runtime/conditionals/if-elif.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
2 changes: 1 addition & 1 deletion tests/expected/runtime/conditionals/if-else.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
2 changes: 1 addition & 1 deletion tests/expected/runtime/conditionals/if.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
1 change: 1 addition & 0 deletions tests/expected/runtime/conditionals/nested.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/conditionals/nested.stderr
View file
Open in desktop
Empty file.
2 changes: 2 additions & 0 deletions tests/expected/runtime/conditionals/nested.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
1
6
2 changes: 1 addition & 1 deletion tests/expected/runtime/functions/basics.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
1 change: 1 addition & 0 deletions tests/expected/runtime/functions/composition.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/functions/composition.stderr
View file
Open in desktop
Empty file.
5 changes: 5 additions & 0 deletions tests/expected/runtime/functions/composition.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
10
20
27
15
20
2 changes: 1 addition & 1 deletion tests/expected/runtime/functions/recursive.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
2 changes: 1 addition & 1 deletion tests/expected/runtime/functions/scopes.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
1 change: 1 addition & 0 deletions tests/expected/runtime/functions/sequences.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/functions/sequences.stderr
View file
Open in desktop
Empty file.
3 changes: 3 additions & 0 deletions tests/expected/runtime/functions/sequences.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
5
10
17
1 change: 1 addition & 0 deletions tests/expected/runtime/loops/break_continue.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/loops/break_continue.stderr
View file
Open in desktop
Empty file.
2 changes: 2 additions & 0 deletions tests/expected/runtime/loops/break_continue.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
10
8
2 changes: 1 addition & 1 deletion tests/expected/runtime/loops/control.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
1 change: 1 addition & 0 deletions tests/expected/runtime/loops/increment_patterns.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/loops/increment_patterns.stderr
View file
Open in desktop
Empty file.
3 changes: 3 additions & 0 deletions tests/expected/runtime/loops/increment_patterns.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
3
4
5
1 change: 1 addition & 0 deletions tests/expected/runtime/loops/increments.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/loops/increments.stderr
View file
Open in desktop
Empty file.
3 changes: 3 additions & 0 deletions tests/expected/runtime/loops/increments.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
5
30
15
1 change: 1 addition & 0 deletions tests/expected/runtime/loops/nested.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/loops/nested.stderr
View file
Open in desktop
Empty file.
2 changes: 2 additions & 0 deletions tests/expected/runtime/loops/nested.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
15
9
1 change: 1 addition & 0 deletions tests/expected/runtime/loops/nested_control.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/loops/nested_control.stderr
View file
Open in desktop
Empty file.
2 changes: 2 additions & 0 deletions tests/expected/runtime/loops/nested_control.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
3
8
1 change: 1 addition & 0 deletions tests/expected/runtime/loops/sums.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/loops/sums.stderr
View file
Open in desktop
Empty file.
2 changes: 2 additions & 0 deletions tests/expected/runtime/loops/sums.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
15
30
1 change: 1 addition & 0 deletions tests/expected/runtime/loops/with_verify.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/loops/with_verify.stderr
View file
Open in desktop
Empty file.
1 change: 1 addition & 0 deletions tests/expected/runtime/loops/with_verify.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1
1 change: 1 addition & 0 deletions tests/expected/runtime/operations/basic_comparisons.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/operations/basic_comparisons.stderr
View file
Open in desktop
Empty file.
12 changes: 12 additions & 0 deletions tests/expected/runtime/operations/basic_comparisons.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
true
false
true
false
true
true
false
false
true
true
true
true
2 changes: 1 addition & 1 deletion tests/expected/runtime/operations/binary.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
1 change: 1 addition & 0 deletions tests/expected/runtime/operations/factorials.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/operations/factorials.stderr
View file
Open in desktop
Empty file.
5 changes: 5 additions & 0 deletions tests/expected/runtime/operations/factorials.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
1
2
6
24
120
1 change: 1 addition & 0 deletions tests/expected/runtime/operations/floating_point.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/operations/floating_point.stderr
View file
Open in desktop
Empty file.
7 changes: 7 additions & 0 deletions tests/expected/runtime/operations/floating_point.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
0.100000
0.200000
0.300000
0.300000
0.020000
1.234568
9.876543
1 change: 1 addition & 0 deletions tests/expected/runtime/operations/increment_decrement.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/operations/increment_decrement.stderr
View file
Open in desktop
Empty file.
6 changes: 6 additions & 0 deletions tests/expected/runtime/operations/increment_decrement.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
10
11
12
20
19
18
1 change: 1 addition & 0 deletions tests/expected/runtime/operations/mixed_types.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/operations/mixed_types.stderr
View file
Open in desktop
Empty file.
4 changes: 4 additions & 0 deletions tests/expected/runtime/operations/mixed_types.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
13.500000
15.200000
7.000000
1
2 changes: 1 addition & 1 deletion tests/expected/runtime/operations/unary.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
1 change: 1 addition & 0 deletions tests/expected/runtime/strings/basic.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/strings/basic.stderr
View file
Open in desktop
Empty file.
2 changes: 2 additions & 0 deletions tests/expected/runtime/strings/basic.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Hello
World
2 changes: 1 addition & 1 deletion tests/expected/runtime/strings/escapes.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
2 changes: 1 addition & 1 deletion tests/expected/runtime/types.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
1 change: 1 addition & 0 deletions tests/expected/runtime/types/all_types.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/types/all_types.stderr
View file
Open in desktop
Empty file.
3 changes: 3 additions & 0 deletions tests/expected/runtime/types/all_types.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-9000000
8000000
2.500000
1 change: 1 addition & 0 deletions tests/expected/runtime/types/boundaries.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/types/boundaries.stderr
View file
Open in desktop
Empty file.
7 changes: 7 additions & 0 deletions tests/expected/runtime/types/boundaries.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
255
0
100
-42
42
0
-84
2 changes: 1 addition & 1 deletion tests/expected/runtime/variables.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0
0
2 changes: 1 addition & 1 deletion tests/expected/runtime/variables.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
DO_NOT_USE_SHARED_VARIABLES_STDOUT
It is what it is
1 change: 1 addition & 0 deletions tests/expected/runtime/variables/sequential_assign.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/variables/sequential_assign.stderr
View file
Open in desktop
Empty file.
9 changes: 9 additions & 0 deletions tests/expected/runtime/variables/sequential_assign.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
0
0
0
10
20
30
50
40
2000
1 change: 1 addition & 0 deletions tests/expected/runtime/variables/shadowing.exitcode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0
Empty file added tests/expected/runtime/variables/shadowing.stderr
View file
Open in desktop
Empty file.
6 changes: 6 additions & 0 deletions tests/expected/runtime/variables/shadowing.stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
100
50
100
10
20
20
Loading
Loading