j-util Performance Lab

This repository contains reproducible benchmarks and performance experiments for j-util libraries. It is a benchmark and demo workspace, not a reusable library.

Benchmark results depend on the hardware, operating system, JDK, JVM options, and runtime conditions under which they are collected. Results from this repository must not be treated as universal performance claims.

Requirements

• JDK 25 or newer

The Maven Wrapper downloads the project's Maven version automatically.

Modules

• benchmark-core contains the shared benchmark models, deterministic data generation, workload implementations, and correctness tests.
• benchmark-jmh contains the OpenJDK JMH benchmark classes and produces the executable benchmark JAR.

Build and test

./mvnw clean verify

Ordinary verification tests belong under benchmark-core/src/test/java and use JUnit 5.

Generate benchmark data

Dataset generation is a standalone step and is never performed as part of a measured benchmark operation. Build the self-contained JAR, then pass the desired row count to the generator:

./mvnw clean package
java -cp benchmark-jmh/target/benchmarks.jar io.github.jutil.performancelab.CsvDatasetGenerator 1000000

This example deterministically writes 1,000,000 data rows plus a header to target/benchmark-data/benchmark-rows-1000000.csv. Repeating the command with the same row count produces identical CSV data. To choose another destination, pass it as the second argument:

java -cp benchmark-jmh/target/benchmarks.jar io.github.jutil.performancelab.CsvDatasetGenerator 1000 /tmp/benchmark.csv

Generated files under target/ are build artifacts and must not be committed.

For a quick development run, generate the benchmark's default 10,000-row dataset:

java -cp benchmark-jmh/target/benchmarks.jar io.github.jutil.performancelab.CsvDatasetGenerator 10000

Generate the 1BRC-style processor dataset

The processor comparison uses a separate deterministic, headerless UTF-8 dataset with one ordinary Commons CSV record per line:

station-name;temperature

Station names repeat realistically and temperatures always have one decimal digit. The default is 10,000,000 measurements. Build with the configured Maven settings and run the generator without arguments to use that default:

./mvnw -s /Users/karenbarseghyan/.m2/settings-j-util.xml clean package
java -cp benchmark-jmh/target/benchmarks.jar \
  io.github.jutil.performancelab.OneBrcStyleDatasetGenerator

Pass a row count to generate another scale. The conventional output is target/benchmark-data/1brc-style-measurements-<row-count>.csv:

java -cp benchmark-jmh/target/benchmarks.jar \
  io.github.jutil.performancelab.OneBrcStyleDatasetGenerator 50000000

The generator reuses an existing file at that path when it has the requested line count. Pass a second argument for an explicit output path. Generated data stays under the ignored target/ tree and must not be committed.

Run the benchmarks

The benchmark methods are organized into classes so that each class contains only directly comparable operations. Separate ready-data classes add the narrow and wide average comparisons and the maximum-by-double comparison described below, and a focused iteration suite compares traversal APIs within Columnar Projection Store.

HardwoodMaterializationBenchmark

This benchmark reads one deterministic flat market-data Parquet dataset through Hardwood with an identical projection and a 4,096-row configured batch size in all four paths:

• hardwoodDirectProcessing checksums all eight fields directly from each decoded Hardwood batch without creating row objects or retaining the dataset. It is the no-secondary-storage baseline for one-pass work. It performs a different responsibility from materialization and is not an ArrayList replacement benchmark.
• hardwoodToColumnarBatch passes caller-configured ColumnReaders to the generated HardwoodMarketDataProjectionHardwoodLoader. The loader uses the generated common-range batch API, loads the complete pre-sized columnar store, and seals it.
• hardwoodToColumnarPerRow reads the same Hardwood arrays, advances one reusable projection view through each batch, calls store.add(view) once per row, and seals the same pre-sized generated store representation. It isolates generated batch append from per-row append.
• hardwoodToArrayList creates one immutable HardwoodMarketDataRow record per decoded row and appends it to a pre-sized ArrayList, representing conventional object materialization.

Every invocation opens a fresh consumable Parquet reader and closes both the caller-owned projected column readers and the Parquet reader. The timed boundary includes file opening, Parquet reading, Hardwood decoding, checksum work or materialization, resource closing, and seal() for columnar results. Trial setup generates the input file and trial teardown deletes it, so input generation and correctness validation are outside measurement. The fixture uses uncompressed Parquet with dictionary encoding disabled and deterministic low-cardinality strings and exactly representable numeric values.

JMH consumes the direct checksum or returned materialized representation. The GC profiler reports allocation rate and bytes per operation; it does not measure peak heap. This is specifically an end-to-end decoding and materialization benchmark, not a general claim that a columnar store is better than an ArrayList for unrelated storage or access patterns.

Run the ordinary small correctness test, which checks row counts, row order, all-field checksums, multiple batches, sealed stores, and representation equivalence:

./mvnw -pl benchmark-core \
  -Dtest=HardwoodMaterializationCasesTest test

Build and run a short 10,000-row smoke invocation of all four methods (smoke output is diagnostic, not a stable benchmark result):

./mvnw clean package
java -jar benchmark-jmh/target/benchmarks.jar \
  HardwoodMaterializationBenchmark \
  -p rowCount=10000 \
  -wi 0 \
  -i 1 \
  -f 1

Run a representative 1,000,000-row measurement with JMH GC allocation metrics:

java -jar benchmark-jmh/target/benchmarks.jar \
  HardwoodMaterializationBenchmark \
  -p rowCount=1000000 \
  -wi 2 \
  -i 3 \
  -f 1 \
  -prof gc

The default rowCount parameters are 1,000,000 and 10,000,000. Because the same generated file is read for every invocation within a trial, measurements after the first read generally benefit from the operating system's page cache.

OneBrcStyleProcessorBenchmark

This is a fair 1BRC-style processor benchmark, not an optimized or official 1BRC submission. All five variants parse the same two fields with the same immutable Apache Commons CSV format (semicolon delimiter, no header, no trimming or surrounding-space assumptions) and map each record to Item(String key, double value) using Double.parseDouble:

Benchmark	Execution strategy	Parallelism
filesLinesSequential	sequential Files.lines()	1
filesLinesParallelForkJoinPool	Files.lines().parallel() in a caller-created ForkJoinPool	2, 4, 8
inputStreamProcessorCore	sequential inputstream-processor-core	1
parallelRangeProcessorForkJoinPool	parallel-range-processor with a caller-created ForkJoinPool	1, 2, 4, 8
parallelRangeProcessorFixedThreadPool	parallel-range-processor with a caller-created fixed-thread-pool ExecutorService	1, 2, 4, 8

Every aggregation uses Storage with a HashMap<String, Counter>. Each Counter maintains minimum, maximum, sum, and count and calculates mean only as sum / count. Parallel stream partitions and range-processor parsers create local Storage instances; their partial results merge only after local work, without a shared ConcurrentHashMap.

The range-processor states construct the processor and their caller-owned executor once in JMH trial setup. The processor receives that executor through its public constructor, never owns it, and never shuts it down. Trial teardown shuts down and awaits the ForkJoinPool or fixed thread pool. The parallel Files.lines() state likewise creates one dedicated reusable ForkJoinPool, so it never relies on the common pool. The fixed thread pool is an additional caller-supplied executor comparison for parallel-range-processor; it is not a library default or an internally owned pool.

Measured execution includes file opening, processing, parsing, aggregation, and partial-result merging. Dataset generation, executor construction, processor construction, and correctness assertions are outside measurement. Each invocation starts with fresh aggregation storage, and no benchmark method prints results.

Commons CSV parser lifecycle necessarily follows each API's natural boundary. inputstream-processor-core uses one parser for the complete input, parallel-range-processor creates one independent parser per actual range and another for reconstructed boundary records when needed, and each already-framed line from Files.lines() is parsed by its own Commons CSV parser. Thus parsing logic and format are shared, while parser-instance counts are an unavoidable fairness difference. The range variants also include byte-range framing and partial aggregation merging; the stream variants include the JDK stream's own splitting and line-decoding behavior.

For a short smoke run after generating 1,000 rows:

java -cp benchmark-jmh/target/benchmarks.jar \
  io.github.jutil.performancelab.OneBrcStyleDatasetGenerator 1000
java -jar benchmark-jmh/target/benchmarks.jar \
  OneBrcStyleProcessorBenchmark \
  -p rowCount=1000 \
  -wi 1 \
  -i 1 \
  -f 1

Run the full default 10,000,000-row comparison with:

java -cp benchmark-jmh/target/benchmarks.jar \
  io.github.jutil.performancelab.OneBrcStyleDatasetGenerator 10000000
java -jar benchmark-jmh/target/benchmarks.jar \
  OneBrcStyleProcessorBenchmark \
  -p rowCount=10000000 \
  -wi 5 \
  -i 10 \
  -f 2

Run the 50,000,000-row scaling comparison without code changes with:

java -cp benchmark-jmh/target/benchmarks.jar \
  io.github.jutil.performancelab.OneBrcStyleDatasetGenerator 50000000
java -jar benchmark-jmh/target/benchmarks.jar \
  OneBrcStyleProcessorBenchmark \
  -p rowCount=50000000 \
  -wi 5 \
  -i 10 \
  -f 2

Because the same file is read repeatedly across warmup and measurement iterations, results primarily represent warm OS-page-cache processing rather than raw disk throughput.

CsvFullRowProcessingBenchmark

• streaming directly to the full-row consumer;
• ArrayList materialization with the expected row count as its initial capacity;
• ArrayList materialization starting with an initial capacity of 10;
• LinkedList materialization;
• ProjectionStore materialization with the expected row count as its initial capacity; and
• ProjectionStore materialization starting with an initial capacity of 10.

Each measured operation includes opening and reading the file, parsing CSV, materializing the selected representation where applicable, and running the same full-row checksum consumer. Dataset generation remains a separate, unmeasured step.

CsvFilteredPriceSumEndToEndBenchmark

Three end-to-end benchmarks produce the same aggregate from the same CSV input:

• arrayListFilteredPriceSumEndToEnd parses every BenchmarkRow into an expected-size ArrayList, then scans the list;
• columnarFilteredPriceSumEndToEnd parses every BenchmarkRow into an expected-size columnar ProjectionStore, then scans its quantity and price projections; and
• reductionStoreFilteredPriceSumEndToEnd feeds every BenchmarkRow to the generated reduction store, which incrementally applies FilteredPriceSum without retaining a row collection.

All three compute:

quantity >= 5
sum(priceCents) for matching rows

Each measured operation includes opening and reading the file, parsing the same BenchmarkRow objects through the same parser and InputStreamProcessor, and producing the final long sum. The architectural work intentionally differs: the ArrayList retains objects and performs a later traversal, the columnar store retains projections and performs a later columnar traversal, and the reduction store computes during ingestion without materializing retained rows.

ReadyPriceSumBenchmark and ReadyFilteredPriceSumBenchmark

The ready-data benchmarks compare reductions after data has already been materialized. ReadyPriceSumBenchmark contains the unfiltered selected-column sum for ArrayList, LinkedList, and columnar ProjectionStore:

sum(priceCents)

ReadyFilteredPriceSumBenchmark contains the filtered business operation for the same three representations:

quantity >= 5
sum(priceCents) for matching rows

The existing retained LinkedList scan benchmarks remain available as an additional representation. These benchmarks prepare their structures in JMH trial setup, so measured execution excludes CSV ingestion and materialization. Separate JMH states retain only the representation required by a benchmark. The retained ArrayList and ProjectionStore use the expected row count as their initial capacity.

The initial-capacity comparison is intentionally limited to the end-to-end benchmarks because it measures construction and growth cost. Once a structure is already loaded, its starting capacity is not part of the measured scan.

ReadyPriceAverageBenchmark

This benchmark asks: after deterministic input has already been materialized, how long do native convenient operations and identical naive arithmetic take to calculate the average price across the compared storage representations?

All actual collection and store comparators retain the complete logical PriceTick(long timestamp, double price) record even though the calculation reads only price. ArrayList<PriceTick>, FastUtil ObjectArrayList<PriceTick>, and Eclipse Collections FastList<PriceTick> store records as objects. Tablesaw stores the same logical records column-wise in a Table with timestamp and price columns, while Columnar Projection Store stores both timestamp and price projections.

The ordinary-addition comparator methods are:

• ArrayList: indexed traversal with ordinary addition, divided by list size;
• FastUtil ObjectArrayList: traversal of the public elements() backing array through its logical size with ordinary addition, divided by list size;
• Eclipse Collections FastList naive: explicit indexed traversal of the existing list with ordinary addition, divided by list size;
• Tablesaw naive: explicit indexed traversal of the existing price DoubleColumn with ordinary addition, divided by column size; and
• Columnar Projection Store: price summation through its public cursor API, divided by store size.

The native-operation methods coexist with those identical-naive-arithmetic comparisons:

• Eclipse Collections FastList: native sumOfDouble(PriceTick::price), divided by list size; and
• Tablesaw: the price column's native mean() operation on the complete table.

The separate double[] calculation baseline contains only prices. It traverses that array directly with ordinary addition and divides by the array length, providing the lowest-abstraction reference for the measured calculation. It is intentionally not a complete PriceTick representation. This documented exception must not be used for full-record retained-memory-footprint claims.

Construction, deterministic data generation, row-count checks, and correctness validation run in JMH trial setup and are excluded from measurement. Each JMH state retains only its own representation, and the native and naive methods for each framework reuse the same state. Ordinary-addition implementations traverse the same finite prices in the same encounter order and are checked for exact agreement. Eclipse Collections and Tablesaw retain their existing native operation semantics and are validated with a small floating-point tolerance because their arithmetic order can differ.

This ready-data suite repeatedly traverses each already-materialized representation and reports its average execution time in microseconds per operation. Each warmup and measurement iteration lasts one second; construction and data generation remain outside the measured operation.

Without a rowCount override, ReadyPriceAverageBenchmark runs all eight methods at each of its four default sizes: 1,000, 100,000, 1,000,000, and 10,000,000 rows. Passing -p rowCount=... overrides the source parameter list for that invocation, so only the requested size is run.

Run only the eight price-average methods with a chosen positive row count:

java -jar benchmark-jmh/target/benchmarks.jar \
  ReadyPriceAverageBenchmark \
  -p rowCount=10000

This in-memory benchmark generates its deterministic records directly by row index and does not read or generate a CSV dataset.

MaxByDoubleBenchmark

This serial ready-data suite is an Eclipse Collections MaxByDoubleTest JMH-derived workload. It uses project-owned deterministic domain and fixture code; it is not an official Eclipse Collections benchmark result.

Every method performs the same logical operation: scan all Position rows by the primitive double marketValue, calculated as quantity * product.price(), and return the original Position having the maximum value. A deterministic shared product population and row order are used. The fixture places one unique maximum at a non-terminal position for datasets of at least three rows, avoiding tie-semantics differences between implementations.

The five benchmark methods are:

• arrayListImperativeMaxByDouble: indexed imperative traversal of a JDK ArrayList<Position>;
• arrayListStreamMaxByDouble: a serial JDK stream using Comparator.comparingDouble;
• eclipseFastListMaxByDouble: Eclipse Collections FastList.maxBy;
• columnarProjectionStoreMaxByDouble: one Columnar Projection Store cursor over projected market values and retained original references; and
• manualHybridMaxByDouble: a manual lower-bound baseline pairing a complete Position[] reference array with a precomputed double[] market-value array.

This is intentionally a repeated-query comparison after construction. Generation, allocation, representation population, projection evaluation, sealing, and correctness validation occur in JMH trial setup and are excluded from the measured scan. Columnar Projection Store and the manual hybrid compute and retain marketValue during population. The object collections instead call Position.marketValue() during every measured scan. This asymmetry is part of the ready-data comparison and means the suite does not measure total end-to-end cost.

The only default rowCount is 3,000,000, matching the scale of the source workload. A command-line -p rowCount=... overrides that value. Run a quick smoke test with:

java -jar benchmark-jmh/target/benchmarks.jar \
  MaxByDoubleBenchmark \
  -p rowCount=1000 \
  -wi 1 \
  -i 1 \
  -f 1

For a publication-quality run at the default scale:

java -jar benchmark-jmh/target/benchmarks.jar \
  MaxByDoubleBenchmark \
  -p rowCount=3000000 \
  -wi 5 \
  -i 10 \
  -f 2

Results are not directly comparable with official Eclipse Collections runs: this suite has a different implementation set, deterministic project-owned fixtures, and different harness modes and configuration. No parallel variants are included because they would add a separate execution-model comparison.

ReadyMarketDataSnapshotAverageBenchmark

This separate wide-record suite performs the same ready-data average operation over a realistic MarketDataSnapshot containing capture time, symbol, last trade price and size, best bid and ask prices, and best bid and ask sizes. Each fixture row is the state immediately after a distinct completed trade, and the measured operation reads only lastTradePrice.

Unlike the narrow PriceTick(timestamp, price) suite, every complete comparator in this suite retains all eight snapshot fields. The storage representations fall into four categories:

• heap row objects: ArrayList<MarketDataSnapshot>, FastUtil ObjectArrayList<MarketDataSnapshot>, and Eclipse Collections FastList<MarketDataSnapshot>;
• off-heap row records: a raw JDK MemorySegment baseline with a fixed-width 64-byte row layout, and typed Chronicle Values flyweights over consecutive direct Chronicle Bytes records;
• on-heap columnar storage: Tablesaw Table, a DFLib DataFrame, Columnar Projection Store, and complete manually assembled HPPC primitive columns; and
• off-heap columnar storage: an Apache Arrow VectorSchemaRoot containing one vector for each snapshot field.

The MemorySegment representation is the raw off-heap row baseline. Its symbol field stores a one-byte UTF-8 length followed by up to seven UTF-8 bytes, and it uses explicit Arena ownership. Chronicle Values + Bytes provides the typed off-heap row representation with the same seven-byte UTF-8 symbol capacity and one reusable flyweight. Apache Arrow is the established off-heap columnar representation, using BigIntVector for capture time, VarCharVector for the symbol, and Float8Vector for all six double fields. All three retain the complete eight-field snapshot and release native resources at JMH trial teardown. The representations therefore compare the same complete logical records even though the calculation selects a single field.

The DFLib comparator is one complete in-memory columnar DataFrame containing one long series, one String series, and six double series. Its native method delegates average calculation to DFLib's public DoubleSeries.avg() operation, while its naive method indexes that same retained lastTradePrice series and performs ordinary encounter-order addition. The HPPC comparator is not a DataFrame: it is a complete manual column layout made from one LongArrayList, one ObjectArrayList<String>, and six DoubleArrayList instances. Its measured method indexes only the retained lastTradePrice list with ordinary addition. Both representations retain all eight snapshot fields, while the measured operation reads only lastTradePrice.

As in the narrow suite, the separate double[] baseline contains only the generated lastTradePrice values. It is a calculation-only reference and must not be included in complete-record retained-memory comparisons. Eclipse Collections' native sumOfDouble(), Tablesaw's native mean(), and DFLib's native avg() methods retain their framework-defined numerical semantics; the naive companion methods define ordinary encounter-order addition explicitly. Native results are therefore validated with the existing small floating-point tolerance instead of making their arithmetic implementation part of this benchmark's contract.

The suite measures hot sequential traversal of lastTradePrice; every ordinary implementation adds doubles in encounter order and divides by its logical row count. Setup, allocation, fixture generation, population, and validation are excluded from the measured operation. Off-heap storage primarily provides an explicit lifecycle and reduced heap and garbage-collection pressure; it is not assumed to be faster, and no performance claim is made before results are collected.

The suite uses the same JMH configuration and default row counts as ReadyPriceAverageBenchmark. Run all fourteen wide-record methods at one positive row count with:

java -jar benchmark-jmh/target/benchmarks.jar \
  ReadyMarketDataSnapshotAverageBenchmark \
  -p rowCount=10000

Construction, fixture generation, and validation remain outside measured time. No comparative performance or retained-memory conclusion is claimed before controlled results are collected on the intended hardware and row counts.

Chronicle Values performs runtime value-class generation and Apache Arrow's Netty allocator accesses direct-buffer internals. Maven tests configure the required module access. The packaged JMH methods also append the narrowly scoped fork arguments automatically: Chronicle opens java.lang, exports jdk.compiler/com.sun.tools.javac.file, and enables native access; Arrow opens java.nio. Both native-library forks allow the legacy sun.misc.Unsafe memory operations required when running on JDK 26. Users do not need to discover or add these flags when running the packaged benchmark normally.

ColumnarProjectionStoreIterationBenchmark

This focused ready-data suite compares the ergonomics and efficiency of the three public row-oriented traversal APIs on the same sealed Columnar Projection Store. The cursor exposes one reusable projection view whose contents advance with the cursor and therefore must not be retained. Indexed viewAt(index) provides explicit random access through stable, retainable views. forEach is the conventional OO traversal API and also supplies stable, retainable views.

The benchmark measures the cost of the stable-view convenience; it does not assume in advance that one traversal will be faster or allocate more at runtime. JIT escape analysis may eliminate some temporary stable-view allocations. Run with JMH's GC profiler to observe the allocation behavior that remains in the measured runtime:

java -jar benchmark-jmh/target/benchmarks.jar \
  ColumnarProjectionStoreIterationBenchmark \
  -p rowCount=1000000 \
  -prof gc

Each API runs both a narrow lastTradePrice sum and a checksum that reads all eight fields. Within a workload, all traversal mechanisms pass every row to the same resettable accumulator, visit rows in encounter order, and reuse the Consumer between invocations. Store construction, fixture generation, sealing, and validation happen in trial setup rather than measured code. The cursor itself is created inside each cursor operation, matching normal public usage.

For publication-quality measurements, increase warmup, measurement, and fork counts and use the largest configured data set:

java -jar benchmark-jmh/target/benchmarks.jar \
  ColumnarProjectionStoreIterationBenchmark \
  -p rowCount=10000000 \
  -wi 5 \
  -i 10 \
  -f 2 \
  -prof gc

These APIs deliberately have different contracts: the cursor prioritizes maximum traversal efficiency through a reusable view, viewAt provides stable views for explicit random access, and forEach provides conventional OO traversal through stable views. Interpret timing and allocation results in that ergonomics-versus-efficiency context rather than treating convenience as an inferior contract.

Run all methods with:

java -jar benchmark-jmh/target/benchmarks.jar \
  'CsvFullRowProcessingBenchmark|CsvFilteredPriceSumEndToEndBenchmark|ReadyPriceSumBenchmark|ReadyFilteredPriceSumBenchmark|ReadyPriceAverageBenchmark|MaxByDoubleBenchmark|ReadyMarketDataSnapshotAverageBenchmark|ColumnarProjectionStoreIterationBenchmark'

Override the rowCount JMH parameter with -p; the corresponding dataset must already exist for the CSV-backed benchmarks:

java -jar benchmark-jmh/target/benchmarks.jar \
  'CsvFullRowProcessingBenchmark|CsvFilteredPriceSumEndToEndBenchmark|ReadyPriceSumBenchmark|ReadyFilteredPriceSumBenchmark|ReadyPriceAverageBenchmark|MaxByDoubleBenchmark|ReadyMarketDataSnapshotAverageBenchmark|ColumnarProjectionStoreIterationBenchmark' \
  -p rowCount=100000

Add JMH's GC profiler to collect allocation and garbage-collection metrics:

java -jar benchmark-jmh/target/benchmarks.jar \
  'CsvFullRowProcessingBenchmark|CsvFilteredPriceSumEndToEndBenchmark|ReadyPriceSumBenchmark|ReadyFilteredPriceSumBenchmark|ReadyPriceAverageBenchmark|MaxByDoubleBenchmark|ReadyMarketDataSnapshotAverageBenchmark|ColumnarProjectionStoreIterationBenchmark' \
  -p rowCount=10000 \
  -prof gc

The GC profiler does not directly measure retained heap or peak heap usage.

Dataset generation is separate and unmeasured for all categories. JMH warmup means filesystem and operating-system page-cache effects may be present in the end-to-end comparisons. No performance conclusions should be drawn without running controlled experiments on the intended hardware and dataset sizes.

Run a benchmark manually on GitHub Actions

The Manual Benchmarks workflow is a manually dispatched, artifact-producing alternative to the Bencher workflow. Choose one suite, one supported row count, and one execution preset. Each workflow run maps the suite to exactly one existing benchmark class, generates a CSV dataset only when that class requires one, and uploads the JMH JSON results together with commit, input, command, Java, Maven, operating-system, and CPU metadata. The in-memory ready-price-average and max-by-double suites never use a CSV dataset.

The presets control JMH execution as follows:

• smoke: 1 warmup iteration, 1 measurement iteration, and 1 fork;
• default: 2 warmup iterations, 3 measurement iterations, and 1 fork;
• extended: 5 warmup iterations, 10 measurement iterations, and 2 forks.

One workflow run executes one comparable benchmark class. Results from a GitHub-hosted runner are suitable for comparing methods within that same controlled run. Separate workflow runs may be scheduled on different hardware, so their results should not be treated as directly comparable without accounting for the recorded environment metadata.