Skip to content

Add parser performance benchmarks with CI comparison workflow - #102

Draft
Nef10 with Copilot wants to merge 5 commits into
mainfrom
copilot/add-parser-performance-tests
Draft

Add parser performance benchmarks with CI comparison workflow#102
Nef10 with Copilot wants to merge 5 commits into
mainfrom
copilot/add-parser-performance-tests

Conversation

Copilot AI commented Jan 14, 2026

Copy link
Copy Markdown
Contributor

Restores performance testing for the parser using the ordo-one/package-benchmark framework, previously removed in #100.

Changes

  • Benchmark implementation: Two benchmarks using the existing 6.5k line Big.beancount test file

    • Parse Big File: Measures file I/O + parsing via Parser.parse(contentOf:)
    • Parse Big File (String): Measures pure parsing via Parser.parse(string:) with pre-loaded content
  • CI workflow: Runs on PRs, benchmarks base vs head in same runner, posts markdown comparison to PR, fails on >10% regression

  • Dependencies: Added package-benchmark 1.29.7+, includes jemalloc for memory profiling

let benchmarks = {
    let config = Benchmark.Configuration(
        metrics: [.wallClock, .throughput],
        scalingFactor: .kilo,
        maxDuration: .seconds(10)
    )

    Benchmark("Parse Big File", configuration: config) { benchmark in
        let fileURL = loadBigFile()
        benchmark.startMeasurement()
        for _ in benchmark.scaledIterations {
            _ = try Parser.parse(contentOf: fileURL)
        }
    }
}

Workflow follows the official CI comparison guide for baseline comparison in the same runner environment.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • esm.ubuntu.com
    • Triggering command: /usr/lib/apt/methods/https /usr/lib/apt/methods/https -parse-as-library -incremental -c @/home/REDACTED/work/SwiftBeanCount/SwiftBeanCount/.build/x86_64-REDACTED-linux-gnu/debug/SwiftBeanCountParser.build/sources -I /home/REDACTED/work/SwiftBeanCount/SwiftBeanCount/.build/x86_64-REDACTED-linux-gnu/debug/Modules rk/Sources/Benchmark/ARCStats/ARCStats.swift x86_64-REDACTED-linux-gnu rk/Sources/Benchmark/ARCStats/ARCStatsProducer.swift -enable-batch-mode rk/Sources/Benchmark/Benchmark+ConvenienceInitializers.swift -e rk/Sources/Benchmark/Benchmark.swift git rk/Sources/Benchmark/BenchmarkClock.swift l rk/Sources/Benchmark/BenchmarkExecutor+Extensions.swift 917bb95775729d4f-emit-module rk/Sources/Benchmark/BenchmarkExecutor.swift (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

This section details on the original issue you should resolve

<issue_title>Parser performance tests</issue_title>
<issue_description>The parser had an attempt at performance tests, which were removed in #100
We should bring them back, using https://www.swift.org/blog/benchmarks/

for this we should follow https://swiftpackageindex.com/ordo-one/package-benchmark/1.29.7/documentation/benchmark/comparingbenchmarksci and probably use CI to run the base and head branch of a PR after each other in the same runner (but different action from the testing) to compare and fail at a certain threshold.</issue_description>

Comments on the Issue (you are @copilot in this section)


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 4 commits January 14, 2026 09:02
- Added ordo-one/package-benchmark dependency to Package.swift
- Created SwiftBeanCountParserBenchmarks executable target
- Implemented two parser benchmarks: file parsing and string parsing
- Added Big.beancount test resource for benchmarks
- Updated .gitignore to exclude benchmark artifacts

Co-authored-by: Nef10 <4551135+Nef10@users.noreply.github.com>
- Created benchmarks.yml workflow to run on PRs
- Workflow compares base vs PR branch performance
- Configured 10% threshold for performance regression detection
- Added jemalloc installation step for Linux
- Fixed SwiftLint violations in benchmark code
- Benchmarks now lint cleanly

Co-authored-by: Nef10 <4551135+Nef10@users.noreply.github.com>
- Removed unnecessary availability check for String(contentsOf:encoding:)
- Use explicit UTF-8 encoding consistently
- Simplified code by removing conditional logic

Co-authored-by: Nef10 <4551135+Nef10@users.noreply.github.com>
- Added comment to clarify file loading happens before measurement
- This ensures only parsing performance is benchmarked, not I/O

Co-authored-by: Nef10 <4551135+Nef10@users.noreply.github.com>
Copilot AI changed the title [WIP] Add parser performance tests for benchmarking Add parser performance benchmarks with CI comparison workflow Jan 14, 2026
Copilot AI requested a review from Nef10 January 14, 2026 09:20
@Nef10 Nef10 added maintenance Internal, non-code related changes tests Improvements to automated tests labels Feb 11, 2026
Nef10 pushed a commit that referenced this pull request Jul 2, 2026
synced local file(s) with
[Nef10/common-swift-package](https://github.com/Nef10/common-swift-package).





---

This PR was created automatically by the
[repo-file-sync-action](https://github.com/BetaHuhn/repo-file-sync-action)
workflow run
[#28079135573](https://github.com/Nef10/common-swift-package/actions/runs/28079135573)

Co-authored-by: file-sync-app[bot] <89714990+file-sync-app[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Internal, non-code related changes tests Improvements to automated tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parser performance tests

2 participants