From d927c93f3474652caa47869c1ca05217b08f1eb5 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Jul 2026 01:14:22 +0000 Subject: [PATCH 1/3] Add Java implementation of the h2 histogram A pure-Java port of the iopsystems h2 histogram, producing byte-for-byte identical bucketing to the Rust histogram crate. Mirrors the code organization of the Go and Python implementations: - Config: bucketing parameters and value<->index conversions - Histogram: dense histogram with record/percentile/merge/subtract/downsample - SparseHistogram: columnar (index, count) form for storage - CumulativeHistogram: read-only cumulative form with O(log n) percentiles and a precomputed midpoint-estimated mean - Bucket / PercentileResult / BucketWithQuantiles value types Values and counts are u64 carried in Java longs with unsigned semantics, so the full u64 range is supported. The bucketing math is verified against the exact assertions from the Rust crate's own unit tests (src/config.rs). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_016Bb8BttJZHuy6zh957NMGT --- .github/workflows/ci.yml | 24 ++ .gitignore | 11 + README.md | 160 +++++++++ pom.xml | 87 +++++ .../com/iopsystems/h2histogram/Bucket.java | 34 ++ .../h2histogram/BucketWithQuantiles.java | 14 + .../com/iopsystems/h2histogram/Config.java | 210 +++++++++++ .../h2histogram/CumulativeHistogram.java | 320 +++++++++++++++++ .../com/iopsystems/h2histogram/Histogram.java | 330 ++++++++++++++++++ .../h2histogram/PercentileResult.java | 9 + .../h2histogram/SparseHistogram.java | 175 ++++++++++ .../java/com/iopsystems/h2histogram/U64.java | 44 +++ .../iopsystems/h2histogram/package-info.java | 29 ++ .../iopsystems/h2histogram/ConfigTest.java | 111 ++++++ .../h2histogram/CumulativeHistogramTest.java | 128 +++++++ .../iopsystems/h2histogram/ExampleTest.java | 41 +++ .../iopsystems/h2histogram/HistogramTest.java | 216 ++++++++++++ 17 files changed, 1943 insertions(+) create mode 100644 .github/workflows/ci.yml create mode 100644 README.md create mode 100644 pom.xml create mode 100644 src/main/java/com/iopsystems/h2histogram/Bucket.java create mode 100644 src/main/java/com/iopsystems/h2histogram/BucketWithQuantiles.java create mode 100644 src/main/java/com/iopsystems/h2histogram/Config.java create mode 100644 src/main/java/com/iopsystems/h2histogram/CumulativeHistogram.java create mode 100644 src/main/java/com/iopsystems/h2histogram/Histogram.java create mode 100644 src/main/java/com/iopsystems/h2histogram/PercentileResult.java create mode 100644 src/main/java/com/iopsystems/h2histogram/SparseHistogram.java create mode 100644 src/main/java/com/iopsystems/h2histogram/U64.java create mode 100644 src/main/java/com/iopsystems/h2histogram/package-info.java create mode 100644 src/test/java/com/iopsystems/h2histogram/ConfigTest.java create mode 100644 src/test/java/com/iopsystems/h2histogram/CumulativeHistogramTest.java create mode 100644 src/test/java/com/iopsystems/h2histogram/ExampleTest.java create mode 100644 src/test/java/com/iopsystems/h2histogram/HistogramTest.java diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..f54c384 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,24 @@ +name: CI + +on: + push: + branches: [main] + pull_request: + +jobs: + test: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + java-version: ["17", "21"] + steps: + - uses: actions/checkout@v4 + - name: Set up JDK ${{ matrix.java-version }} + uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: ${{ matrix.java-version }} + cache: maven + - name: Build and test + run: mvn -B --no-transfer-progress verify diff --git a/.gitignore b/.gitignore index 524f096..2855e70 100644 --- a/.gitignore +++ b/.gitignore @@ -22,3 +22,14 @@ # virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml hs_err_pid* replay_pid* + +# Maven +target/ + +# IDE +.idea/ +*.iml +.vscode/ +.classpath +.project +.settings/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..a2f0e64 --- /dev/null +++ b/README.md @@ -0,0 +1,160 @@ +# h2histogram-java + +[![CI](https://github.com/iopsystems/h2histogram-java/actions/workflows/ci.yml/badge.svg)](https://github.com/iopsystems/h2histogram-java/actions/workflows/ci.yml) +[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) + +A pure-Java implementation of the [iopsystems h2 histogram](https://github.com/iopsystems/histogram). + +`h2histogram` produces histograms with **byte-for-byte identical bucketing** to the +Rust `histogram` crate, so histograms recorded here can be consumed by +[Rezolus](https://github.com/iopsystems/rezolus) — and, conversely, you can read an +h2histogram produced by Rezolus (or the [Python](https://github.com/iopsystems/h2histogram-py) +and [Go](https://github.com/iopsystems/h2histogram-go) implementations) and analyze +it on the JVM. Values are carried in Java `long`s with unsigned semantics +(`Long.compareUnsigned` and friends), so the full `u64` value range is supported, +exactly like the Rust crate. + +## What is an h2 histogram? + +An h2 histogram quantizes values into buckets using two parameters: + +- **`groupingPower`** — the number of buckets spanning each power of two. It sets + the relative error to `2^-groupingPower` (e.g. `groupingPower=7` → ~0.78% error). +- **`maxValuePower`** — the largest representable value is `2^maxValuePower - 1`. + +Values below `2^(groupingPower+1)` are stored **exactly** (linear buckets of width 1); +larger values fall into logarithmic buckets. This gives HDR-histogram-like guarantees +with a simpler, faster bucket index computation. Rezolus records histograms with +`groupingPower=3` and `maxValuePower=64`. + +## Install + +Maven: + +```xml + + com.iopsystems + h2histogram + 0.1.0 + +``` + +Gradle: + +```kotlin +implementation("com.iopsystems:h2histogram:0.1.0") +``` + +The library requires Java 17 or later and has no runtime dependencies. + +## Quick start + +```java +import com.iopsystems.h2histogram.Bucket; +import com.iopsystems.h2histogram.Histogram; +import com.iopsystems.h2histogram.SparseHistogram; + +Histogram h = new Histogram(7, 64); // groupingPower, maxValuePower + +h.increment(42); +h.record(1000, 5); // value, count +h.recordMany(new long[] {12, 15, 900}); // bulk + +System.out.println(h.totalCount()); // 9 + +Bucket p99 = h.percentile(0.99).orElseThrow(); // Optional.empty() if empty +System.out.println(p99.start() + " " + p99.end() + " " + p99.midpoint()); + +// Combine / reduce +Histogram merged = h.merge(other); // element-wise sum +Histogram coarse = h.downsample(4); // fewer buckets, higher error, same total count +SparseHistogram sparse = h.toSparse(); // columnar (index, count) form for storage +``` + +### Fast repeated quantile queries + +For a snapshot you'll query many times, convert to a `CumulativeHistogram` +(the crate's `CumulativeROHistogram`). It stores non-zero buckets with +**cumulative** counts, so percentiles are answered with a binary search, and it +precomputes a midpoint-estimated mean: + +```java +import com.iopsystems.h2histogram.BucketWithQuantiles; +import com.iopsystems.h2histogram.CumulativeHistogram; + +CumulativeHistogram c = h.toCumulative(); // read-only; also SparseHistogram.toCumulative() +Bucket b = c.percentile(0.99).orElseThrow(); // O(log n) binary search (individual count) +double mean = c.mean().orElseThrow(); // midpoint-estimated mean, computed once +c.bucketQuantileRange(0); // quantile fractions of a stored bucket +for (BucketWithQuantiles bq : c.bucketsWithQuantiles()) { + // each non-zero bucket with its quantile span +} +``` + +## API overview + +| Type | Purpose | +|------|---------| +| `Config` | Bucketing parameters; `valueToIndex`, `indexToLowerBound`/`indexToUpperBound`, `totalBuckets`, `error` | +| `Histogram` | Dense histogram; `increment`, `record`, `recordMany`, `percentile(s)`, `merge`, `subtract`, `downsample`, `toSparse`, `toCumulative`, `fromBuckets` | +| `SparseHistogram` | Columnar `(index, count)` form; `fromHistogram`, `fromParts`, `toDense`, `toCumulative` | +| `CumulativeHistogram` | Read-only cumulative form (crate's `CumulativeROHistogram`); binary-search `percentile(s)`, `mean`, `bucketQuantileRange`, `bucketsWithQuantiles` | +| `Bucket` | A bucket's `count` and inclusive `[start, end]` range, plus `midpoint`/`width` | + +## Unsigned values + +All values and counts are unsigned 64-bit integers (`u64`) carried in Java +`long`s. A negative `long` represents a value above `Long.MAX_VALUE`; use +`Long.toUnsignedString`, `Long.parseUnsignedLong`, and `Long.compareUnsigned` +when working near the top of the range. `Bucket.midpoint()` and +`CumulativeHistogram.mean()` already account for this and return correct +`double` estimates across the full range. + +## Compatibility across implementations + +The same bucketing is implemented in: + +- [Rust](https://github.com/iopsystems/histogram) — the canonical implementation +- [Python](https://github.com/iopsystems/h2histogram-py) +- [Go](https://github.com/iopsystems/h2histogram-go) +- [JavaScript](https://github.com/iopsystems/h2histogram-js) (limited to `maxValuePower <= 53`, + since JS numbers are 64-bit floats) +- Java (this repository) — full `u64` range + +Because the bucket indices are identical, a `(bucket_indices, bucket_counts)` +pair produced by any of these can be loaded via `SparseHistogram.fromParts` / +`CumulativeHistogram.fromParts` and analyzed here. + +## Correctness + +The bucketing math is verified against the exact assertions from the Rust crate's +own unit tests (`src/config.rs`), so the bucketing is guaranteed bit-identical. +Run `mvn test` to see for yourself. + +## Building + +```bash +mvn verify +``` + +## Releasing + +Releases are published to Maven Central. To cut a release: + +1. **Land your changes on `main`** via a pull request. +2. **Set the release version** (drop the `-SNAPSHOT` suffix) in + [`pom.xml`](pom.xml), e.g. `0.1.0`, and merge that change. +3. **Tag and push** a `vX.Y.Z` tag on `main`: + + ```bash + git checkout main && git pull + git tag v0.1.0 + git push origin v0.1.0 + ``` + +4. **Deploy** with `mvn -Prelease deploy` using Sonatype (Maven Central) + credentials, then bump `pom.xml` back to the next `-SNAPSHOT` version. + +## License + +MIT — see [LICENSE](LICENSE). diff --git a/pom.xml b/pom.xml new file mode 100644 index 0000000..fb1dcf4 --- /dev/null +++ b/pom.xml @@ -0,0 +1,87 @@ + + + 4.0.0 + + com.iopsystems + h2histogram + 0.1.0-SNAPSHOT + jar + + h2histogram + + A pure-Java implementation of the iopsystems h2 histogram, producing + bucketing that is byte-for-byte identical to the Rust histogram crate. + + https://github.com/iopsystems/h2histogram-java + + + + MIT License + https://opensource.org/licenses/MIT + + + + + scm:git:https://github.com/iopsystems/h2histogram-java.git + scm:git:git@github.com:iopsystems/h2histogram-java.git + https://github.com/iopsystems/h2histogram-java + + + + 17 + UTF-8 + 5.10.2 + + + + + org.junit.jupiter + junit-jupiter + ${junit.version} + test + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.13.0 + + + org.apache.maven.plugins + maven-surefire-plugin + 3.2.5 + + + org.apache.maven.plugins + maven-javadoc-plugin + 3.6.3 + + + attach-javadocs + + jar + + + + + + org.apache.maven.plugins + maven-source-plugin + 3.3.0 + + + attach-sources + + jar-no-fork + + + + + + + diff --git a/src/main/java/com/iopsystems/h2histogram/Bucket.java b/src/main/java/com/iopsystems/h2histogram/Bucket.java new file mode 100644 index 0000000..d99217f --- /dev/null +++ b/src/main/java/com/iopsystems/h2histogram/Bucket.java @@ -0,0 +1,34 @@ +package com.iopsystems.h2histogram; + +/** + * A single histogram bucket: a count and an inclusive value range. + * + *

All three components are unsigned 64-bit values carried in Java + * {@code long}s; use {@link Long#toUnsignedString} and friends when a bound + * may exceed {@code Long.MAX_VALUE}. + * + * @param count the number of observations in the bucket (unsigned) + * @param start the inclusive lower bound of the bucket's value range (unsigned) + * @param end the inclusive upper bound of the bucket's value range (unsigned) + */ +public record Bucket(long count, long start, long end) { + /** + * Returns the arithmetic midpoint of the bucket range. It is a reasonable + * point estimate for values that fell into this bucket. + */ + public double midpoint() { + return (U64.toDouble(start) + U64.toDouble(end)) / 2.0; + } + + /** Returns the number of distinct integer values the bucket covers (unsigned). */ + public long width() { + return end - start + 1; + } + + @Override + public String toString() { + return "Bucket(count=" + Long.toUnsignedString(count) + + ", range=[" + Long.toUnsignedString(start) + + ", " + Long.toUnsignedString(end) + "])"; + } +} diff --git a/src/main/java/com/iopsystems/h2histogram/BucketWithQuantiles.java b/src/main/java/com/iopsystems/h2histogram/BucketWithQuantiles.java new file mode 100644 index 0000000..18b4acb --- /dev/null +++ b/src/main/java/com/iopsystems/h2histogram/BucketWithQuantiles.java @@ -0,0 +1,14 @@ +package com.iopsystems.h2histogram; + +/** + * Bundles a bucket with its quantile span. + * + *

{@code lowerQuantile} is the fraction of observations strictly before + * this bucket and {@code upperQuantile} the fraction at or before it, both in + * {@code [0.0, 1.0]}. + * + * @param bucket the bucket, with its individual (non-cumulative) count + * @param lowerQuantile the fraction of observations strictly before this bucket + * @param upperQuantile the fraction of observations at or before this bucket + */ +public record BucketWithQuantiles(Bucket bucket, double lowerQuantile, double upperQuantile) {} diff --git a/src/main/java/com/iopsystems/h2histogram/Config.java b/src/main/java/com/iopsystems/h2histogram/Config.java new file mode 100644 index 0000000..ef19272 --- /dev/null +++ b/src/main/java/com/iopsystems/h2histogram/Config.java @@ -0,0 +1,210 @@ +package com.iopsystems.h2histogram; + +/** + * An immutable bucketing configuration. + * + *

The bucketing strategy is fully determined by two parameters: + * + *

+ * + *

The layout has two regions: a linear region covering + * {@code 0 .. 2^(groupingPower+1)} where every bucket has width 1 (exact), and + * a logarithmic region above the cutoff subdivided into + * {@code 2^groupingPower} buckets per power of two. + * + *

This is a faithful port of the {@code Config} type from the Rust + * histogram crate and + * produces byte-for-byte identical bucketing. + * + *

Values are unsigned 64-bit integers carried in Java {@code long}s: a + * negative {@code long} represents a value above {@code Long.MAX_VALUE}, so + * the full {@code u64} range is supported. + */ +public final class Config { + private final int groupingPower; + private final int maxValuePower; + private final long max; // unsigned + private final int cutoffPower; + private final long cutoffValue; // unsigned + private final int lowerBinCount; + private final int upperBinDivisions; + private final int upperBinCount; + + /** + * Creates and validates a {@code Config}. + * + *

The constraints match the Rust crate: + * + *

+ * + * @throws IllegalArgumentException if the parameters are out of range + */ + public Config(int groupingPower, int maxValuePower) { + if (maxValuePower < 0 || maxValuePower > 64) { + throw new IllegalArgumentException( + "max_value_power must be <= 64, got " + maxValuePower); + } + if (groupingPower < 0 || groupingPower >= maxValuePower) { + throw new IllegalArgumentException( + "grouping_power (" + groupingPower + + ") must be non-negative and less than max_value_power (" + + maxValuePower + ")"); + } + + // The cutoff is the point at which the linear divisions and the + // logarithmic subdivisions have the same width: + // cutoffPower = groupingPower + 1. + this.groupingPower = groupingPower; + this.maxValuePower = maxValuePower; + this.cutoffPower = groupingPower + 1; + this.cutoffValue = 1L << cutoffPower; + this.upperBinDivisions = 1 << groupingPower; + this.max = maxValuePower == 64 ? -1L : (1L << maxValuePower) - 1; + this.lowerBinCount = (int) cutoffValue; + this.upperBinCount = (maxValuePower - cutoffPower) * upperBinDivisions; + } + + /** + * Infers a {@code Config} from a known bucket count and + * {@code maxValuePower}. + * + *

Rezolus/metriken parquet files store dense histograms as a bare list + * of bucket counts without recording {@code groupingPower}. Given the + * number of buckets and the (conventionally fixed) {@code maxValuePower} + * the grouping power can be recovered uniquely. + * + * @throws IllegalArgumentException if no grouping power produces + * {@code totalBuckets} + */ + public static Config fromTotalBuckets(int totalBuckets, int maxValuePower) { + for (int groupingPower = 0; groupingPower < maxValuePower; groupingPower++) { + Config candidate; + try { + candidate = new Config(groupingPower, maxValuePower); + } catch (IllegalArgumentException e) { + continue; + } + if (candidate.totalBuckets() == totalBuckets) { + return candidate; + } + } + throw new IllegalArgumentException( + "no grouping_power with max_value_power=" + maxValuePower + + " yields " + totalBuckets + " buckets"); + } + + /** Returns the grouping power used to create this configuration. */ + public int groupingPower() { + return groupingPower; + } + + /** Returns the max value power used to create this configuration. */ + public int maxValuePower() { + return maxValuePower; + } + + /** + * Returns the largest value representable by this configuration, + * i.e. {@code 2^maxValuePower - 1}, as an unsigned {@code long}. + */ + public long max() { + return max; + } + + /** Returns the total number of buckets for this configuration. */ + public int totalBuckets() { + return lowerBinCount + upperBinCount; + } + + /** + * Returns the relative error (as a percentage) of the logarithmic buckets. + * Linear buckets have width 1 and no error. If the config has no + * logarithmic buckets the error is zero. + */ + public double error() { + if (groupingPower == maxValuePower - 1) { + return 0.0; + } + return 100.0 / (double) (1L << groupingPower); + } + + /** + * Returns the bucket index that {@code value} (unsigned) falls into. + * + * @throws IllegalArgumentException if the value is greater than the + * configured maximum + */ + public int valueToIndex(long value) { + if (Long.compareUnsigned(value, cutoffValue) < 0) { + return (int) value; + } + if (Long.compareUnsigned(value, max) > 0) { + throw new IllegalArgumentException( + "value " + Long.toUnsignedString(value) + + " is out of range for max " + Long.toUnsignedString(max)); + } + + // power = floor(log2(value)); equivalent to 63 - leading_zeros for u64. + int power = 63 - Long.numberOfLeadingZeros(value); + int logBin = power - cutoffPower; + long offset = (value - (1L << power)) >>> (power - groupingPower); + + return lowerBinCount + logBin * upperBinDivisions + (int) offset; + } + + /** Returns the inclusive (unsigned) lower bound of the bucket at {@code index}. */ + public long indexToLowerBound(int index) { + long g = ((long) index) >>> groupingPower; + long h = ((long) index) - g * (1L << groupingPower); + if (g < 1) { + return h; + } + return (1L << (groupingPower + g - 1)) + (1L << (g - 1)) * h; + } + + /** Returns the inclusive (unsigned) upper bound of the bucket at {@code index}. */ + public long indexToUpperBound(int index) { + if (index == totalBuckets() - 1) { + return max; + } + long g = ((long) index) >>> groupingPower; + long h = ((long) index) - g * (1L << groupingPower) + 1; + if (g < 1) { + return h - 1; + } + return (1L << (groupingPower + g - 1)) + (1L << (g - 1)) * h - 1; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (!(obj instanceof Config other)) { + return false; + } + return groupingPower == other.groupingPower && maxValuePower == other.maxValuePower; + } + + @Override + public int hashCode() { + return 31 * groupingPower + maxValuePower; + } + + @Override + public String toString() { + return "Config(grouping_power=" + groupingPower + + ", max_value_power=" + maxValuePower + ")"; + } +} diff --git a/src/main/java/com/iopsystems/h2histogram/CumulativeHistogram.java b/src/main/java/com/iopsystems/h2histogram/CumulativeHistogram.java new file mode 100644 index 0000000..9835b38 --- /dev/null +++ b/src/main/java/com/iopsystems/h2histogram/CumulativeHistogram.java @@ -0,0 +1,320 @@ +package com.iopsystems.h2histogram; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Optional; +import java.util.OptionalDouble; + +/** + * A read-only histogram with cumulative counts for fast quantile queries. + * + *

It corresponds to {@code CumulativeROHistogram} in the Rust histogram + * crate. It is a variant of {@link SparseHistogram} that stores only non-zero + * buckets in columnar form, but with cumulative counts: {@code count[i]} is + * the running prefix sum of individual bucket counts, so the last element + * equals the total observation count. + * + *

Because the counts are cumulative, percentile queries are answered with + * a binary search (O(log n) in the number of non-zero buckets) rather than a + * linear scan. The histogram is read-only. A midpoint-estimated mean is + * computed once at construction. + */ +public final class CumulativeHistogram { + private final Config config; + private final int[] index; + private final long[] count; // cumulative (prefix-sum) counts + private final double mean; + private final boolean hasMean; + + private CumulativeHistogram(Config config, int[] index, long[] count) { + this.config = config; + this.index = index; + this.count = count; + double meanValue = 0.0; + boolean meanPresent = false; + if (count.length != 0 && count[count.length - 1] != 0) { + long total = count[count.length - 1]; + double weighted = 0.0; + for (int i = 0; i < index.length; i++) { + long individual = individualCount(i); + double midpoint = (U64.toDouble(config.indexToLowerBound(index[i])) + + U64.toDouble(config.indexToUpperBound(index[i]))) / 2.0; + weighted += midpoint * U64.toDouble(individual); + } + meanValue = weighted / U64.toDouble(total); + meanPresent = true; + } + this.mean = meanValue; + this.hasMean = meanPresent; + } + + /** + * Creates a {@code CumulativeHistogram} from raw parts. {@code count} must + * be cumulative (prefix sums). + * + * @throws IllegalArgumentException if the lengths differ, an index is out + * of range, the indices are not strictly ascending, the counts are not + * non-decreasing, or any count is zero + */ + public static CumulativeHistogram fromParts(Config config, int[] index, long[] count) { + if (index.length != count.length) { + throw new IllegalArgumentException( + "index and count must have the same length (" + + index.length + " != " + count.length + ")"); + } + int total = config.totalBuckets(); + int prev = -1; + for (int i : index) { + if (i < 0 || i >= total) { + throw new IllegalArgumentException("index " + i + " out of range for config"); + } + if (i <= prev) { + throw new IllegalArgumentException("indices must be strictly ascending"); + } + prev = i; + } + long prevCount = 0; + for (long c : count) { + if (c == 0) { + throw new IllegalArgumentException("cumulative counts must be non-zero"); + } + if (Long.compareUnsigned(c, prevCount) < 0) { + throw new IllegalArgumentException("cumulative counts must be non-decreasing"); + } + prevCount = c; + } + return new CumulativeHistogram(config, index.clone(), count.clone()); + } + + /** Builds a {@code CumulativeHistogram} from a dense {@link Histogram}. */ + public static CumulativeHistogram fromHistogram(Histogram histogram) { + long[] buckets = histogram.bucketsRef(); + int nonzero = 0; + for (long c : buckets) { + if (c != 0) { + nonzero++; + } + } + int[] index = new int[nonzero]; + long[] count = new long[nonzero]; + int k = 0; + long running = 0; + for (int i = 0; i < buckets.length; i++) { + if (buckets[i] != 0) { + running += buckets[i]; + index[k] = i; + count[k] = running; + k++; + } + } + return new CumulativeHistogram(histogram.config(), index, count); + } + + /** Builds a {@code CumulativeHistogram} from a {@link SparseHistogram}. */ + public static CumulativeHistogram fromSparse(SparseHistogram sparse) { + int[] index = sparse.indexRef().clone(); + long[] sparseCount = sparse.countRef(); + long[] cumulative = new long[sparseCount.length]; + long running = 0; + for (int i = 0; i < sparseCount.length; i++) { + running += sparseCount[i]; + cumulative[i] = running; + } + return new CumulativeHistogram(sparse.config(), index, cumulative); + } + + private long individualCount(int position) { + if (position == 0) { + return count[0]; + } + return count[position] - count[position - 1]; + } + + /** Returns the bucketing configuration. */ + public Config config() { + return config; + } + + /** Returns a copy of the non-zero bucket indices, ascending. */ + public int[] index() { + return index.clone(); + } + + /** Returns a copy of the cumulative (prefix-sum) counts aligned with {@link #index()}. */ + public long[] count() { + return count.clone(); + } + + /** Returns the number of stored (non-zero) buckets. */ + public int size() { + return index.length; + } + + /** Returns whether the histogram has no stored buckets. */ + public boolean isEmpty() { + return index.length == 0; + } + + /** Returns the total number of observations (unsigned). */ + public long totalCount() { + if (count.length == 0) { + return 0; + } + return count[count.length - 1]; + } + + /** + * Returns the midpoint-estimated mean of all observations, or empty if the + * histogram is empty. It is computed once at construction. + */ + public OptionalDouble mean() { + return hasMean ? OptionalDouble.of(mean) : OptionalDouble.empty(); + } + + /** Returns the first position where the cumulative count is {@code >= target}. */ + private int findQuantilePosition(long target) { + int lo = 0; + int hi = count.length; + while (lo < hi) { + int mid = (lo + hi) >>> 1; + if (Long.compareUnsigned(count[mid], target) < 0) { + lo = mid + 1; + } else { + hi = mid; + } + } + if (lo >= count.length) { + return count.length - 1; + } + return lo; + } + + private Bucket bucketAt(int position) { + return new Bucket(individualCount(position), + config.indexToLowerBound(index[position]), + config.indexToUpperBound(index[position])); + } + + /** + * Returns the bucket at {@code percentile} in {@code [0.0, 1.0]}, or empty + * if the histogram has no observations. The returned bucket carries the + * individual (non-cumulative) count. + * + * @throws IllegalArgumentException if the percentile is out of range + */ + public Optional percentile(double percentile) { + List results = percentiles(percentile); + if (results.isEmpty()) { + return Optional.empty(); + } + return Optional.of(results.get(0).bucket()); + } + + /** + * Returns a {@link PercentileResult} per requested percentile, in input + * order. Each percentile must be in {@code [0.0, 1.0]}. Returns an empty + * list if the histogram has no observations. + * + * @throws IllegalArgumentException if any percentile is out of range + */ + public List percentiles(double... percentiles) { + for (double p : percentiles) { + if (!(p >= 0.0 && p <= 1.0)) { + throw new IllegalArgumentException( + "percentiles must be in the range [0.0, 1.0], got " + p); + } + } + if (count.length == 0 || count[count.length - 1] == 0) { + return List.of(); + } + long total = count[count.length - 1]; + + List out = new ArrayList<>(percentiles.length); + for (double p : percentiles) { + long target = U64.ceilCount(p, total); + int pos = findQuantilePosition(target); + out.add(new PercentileResult(p, bucketAt(pos))); + } + return out; + } + + /** Alias for {@link #percentile(double)}. */ + public Optional quantile(double quantile) { + return percentile(quantile); + } + + /** + * Returns the quantile span for the {@code position}-th stored bucket, or + * empty if the histogram is empty or the position is out of range. The + * result's {@code lowerQuantile} is the fraction of observations strictly + * before this bucket and {@code upperQuantile} the fraction at or before + * it, both in {@code [0.0, 1.0]}. + */ + public Optional bucketQuantileRange(int position) { + if (position < 0 || position >= count.length || count[count.length - 1] == 0) { + return Optional.empty(); + } + double total = U64.toDouble(count[count.length - 1]); + double lower = position == 0 ? 0.0 : U64.toDouble(count[position - 1]) / total; + double upper = U64.toDouble(count[position]) / total; + return Optional.of(new BucketWithQuantiles(bucketAt(position), lower, upper)); + } + + /** Returns each stored bucket with its individual count. */ + public List buckets() { + List out = new ArrayList<>(index.length); + for (int i = 0; i < index.length; i++) { + out.add(bucketAt(i)); + } + return out; + } + + /** Returns each non-zero bucket with its (lowerQuantile, upperQuantile) span. */ + public List bucketsWithQuantiles() { + List out = new ArrayList<>(index.length); + double total = count.length == 0 ? 0.0 : U64.toDouble(count[count.length - 1]); + for (int i = 0; i < index.length; i++) { + double lower = i == 0 ? 0.0 : U64.toDouble(count[i - 1]) / total; + double upper = U64.toDouble(count[i]) / total; + out.add(new BucketWithQuantiles(bucketAt(i), lower, upper)); + } + return out; + } + + /** Reconstructs a dense {@link Histogram}. */ + public Histogram toDense() { + Histogram h = new Histogram(config); + long[] buckets = h.bucketsRef(); + for (int i = 0; i < index.length; i++) { + buckets[index[i]] = individualCount(i); + } + return h; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (!(obj instanceof CumulativeHistogram other)) { + return false; + } + return config.equals(other.config) + && Arrays.equals(index, other.index) + && Arrays.equals(count, other.count); + } + + @Override + public int hashCode() { + return 31 * (31 * config.hashCode() + Arrays.hashCode(index)) + Arrays.hashCode(count); + } + + @Override + public String toString() { + return "CumulativeHistogram(grouping_power=" + config.groupingPower() + + ", max_value_power=" + config.maxValuePower() + + ", nonzero_buckets=" + index.length + + ", total_count=" + Long.toUnsignedString(totalCount()) + ")"; + } +} diff --git a/src/main/java/com/iopsystems/h2histogram/Histogram.java b/src/main/java/com/iopsystems/h2histogram/Histogram.java new file mode 100644 index 0000000..8046bdd --- /dev/null +++ b/src/main/java/com/iopsystems/h2histogram/Histogram.java @@ -0,0 +1,330 @@ +package com.iopsystems.h2histogram; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.function.Consumer; + +/** + * A dense h2 histogram that stores a counter for every bucket. + * + *

Values are quantized into buckets according to a {@link Config} + * determined by {@code groupingPower} and {@code maxValuePower}. This is the + * Java analogue of the Rust {@code Histogram} type and produces byte-for-byte + * identical bucketing, so histograms recorded here can be consumed by + * Rezolus (and vice versa). + * + *

Values and counts are unsigned 64-bit integers carried in Java + * {@code long}s, so the full {@code u64} range is supported. + */ +public final class Histogram { + private final Config config; + private final long[] buckets; + + /** + * Creates an empty histogram with the given {@code groupingPower} and + * {@code maxValuePower}. + * + * @throws IllegalArgumentException if the parameters are invalid (see + * {@link Config#Config(int, int)}) + */ + public Histogram(int groupingPower, int maxValuePower) { + this(new Config(groupingPower, maxValuePower)); + } + + /** Creates an empty histogram from an existing {@link Config}. */ + public Histogram(Config config) { + this.config = config; + this.buckets = new long[config.totalBuckets()]; + } + + /** + * Creates a histogram from a full, dense array of bucket counts. The + * length of {@code buckets} must equal the config's + * {@link Config#totalBuckets()}. + * + * @throws IllegalArgumentException if the parameters are invalid or the + * length does not match + */ + public static Histogram fromBuckets(int groupingPower, int maxValuePower, long[] buckets) { + Config config = new Config(groupingPower, maxValuePower); + if (buckets.length != config.totalBuckets()) { + throw new IllegalArgumentException( + "expected " + config.totalBuckets() + " buckets, got " + buckets.length); + } + Histogram h = new Histogram(config); + System.arraycopy(buckets, 0, h.buckets, 0, buckets.length); + return h; + } + + /** Returns the bucketing configuration. */ + public Config config() { + return config; + } + + /** Returns a copy of the dense bucket counts (one entry per bucket). */ + public long[] bucketCounts() { + return buckets.clone(); + } + + /** Returns the internal bucket array without copying. */ + long[] bucketsRef() { + return buckets; + } + + /** Returns the number of buckets. */ + public int size() { + return buckets.length; + } + + /** Returns the total number of observations recorded (unsigned). */ + public long totalCount() { + long total = 0; + for (long c : buckets) { + total += c; + } + return total; + } + + /** + * Adds one observation of {@code value}. + * + * @throws IllegalArgumentException if value is out of range for the histogram + */ + public void increment(long value) { + record(value, 1); + } + + /** + * Adds {@code count} observations of {@code value}. + * + * @throws IllegalArgumentException if value is out of range for the histogram + */ + public void record(long value, long count) { + buckets[config.valueToIndex(value)] += count; + } + + /** + * Records each value in {@code values} once. + * + * @throws IllegalArgumentException (stopping) if any value is out of range + */ + public void recordMany(long[] values) { + for (long v : values) { + record(v, 1); + } + } + + /** + * Records each value with the corresponding weight from {@code counts}. + * + * @throws IllegalArgumentException if the arrays differ in length or any + * value is out of range + */ + public void recordMany(long[] values, long[] counts) { + if (values.length != counts.length) { + throw new IllegalArgumentException( + "values and counts must have the same length (" + + values.length + " != " + counts.length + ")"); + } + for (int i = 0; i < values.length; i++) { + record(values[i], counts[i]); + } + } + + // Bucket iteration ------------------------------------------------------ + + private Bucket bucketAt(int index) { + return new Bucket(buckets[index], + config.indexToLowerBound(index), config.indexToUpperBound(index)); + } + + /** Calls {@code action} for every bucket in ascending index order. */ + public void forEachBucket(Consumer action) { + for (int i = 0; i < buckets.length; i++) { + action.accept(bucketAt(i)); + } + } + + /** Returns every bucket with a non-zero count, in ascending index order. */ + public List nonzeroBuckets() { + List out = new ArrayList<>(); + for (int i = 0; i < buckets.length; i++) { + if (buckets[i] != 0) { + out.add(bucketAt(i)); + } + } + return out; + } + + // Combination ----------------------------------------------------------- + + private void checkCompatible(Histogram other) { + if (!config.equals(other.config)) { + throw new IllegalArgumentException("histograms have incompatible configurations"); + } + } + + /** + * Returns a new histogram that is the element-wise sum of {@code this} and + * {@code other}. Both histograms must share the same configuration. + * + * @throws IllegalArgumentException if the configurations differ + */ + public Histogram merge(Histogram other) { + checkCompatible(other); + Histogram result = new Histogram(config); + for (int i = 0; i < buckets.length; i++) { + result.buckets[i] = buckets[i] + other.buckets[i]; + } + return result; + } + + /** + * Returns a new histogram that is the element-wise difference of + * {@code this} and {@code other}. + * + * @throws IllegalArgumentException if any bucket would go negative or the + * configurations differ + */ + public Histogram subtract(Histogram other) { + checkCompatible(other); + Histogram result = new Histogram(config); + for (int i = 0; i < buckets.length; i++) { + if (Long.compareUnsigned(other.buckets[i], buckets[i]) > 0) { + throw new IllegalArgumentException( + "subtraction would produce a negative bucket count"); + } + result.buckets[i] = buckets[i] - other.buckets[i]; + } + return result; + } + + /** + * Returns a coarser histogram with a smaller {@code groupingPower}. Every + * step down approximately halves the number of buckets while doubling the + * relative error. The new grouping power must be strictly less than the + * current one. + * + * @throws IllegalArgumentException if {@code groupingPower} is not smaller + * than the current grouping power + */ + public Histogram downsample(int groupingPower) { + if (groupingPower >= config.groupingPower()) { + throw new IllegalArgumentException( + "target grouping_power must be less than the current grouping_power"); + } + Histogram result = new Histogram(groupingPower, config.maxValuePower()); + for (int i = 0; i < buckets.length; i++) { + if (buckets[i] != 0) { + result.record(config.indexToLowerBound(i), buckets[i]); + } + } + return result; + } + + // Quantiles / percentiles ------------------------------------------------- + + /** + * Returns the bucket at a single percentile in {@code [0.0, 1.0]}, or + * empty if the histogram has no observations. The percentile uses the same + * fractional convention as the Rust crate: 0.5 is the median. + * + * @throws IllegalArgumentException if the percentile is out of range + */ + public Optional percentile(double percentile) { + List results = percentiles(percentile); + if (results.isEmpty()) { + return Optional.empty(); + } + return Optional.of(results.get(0).bucket()); + } + + /** + * Returns a {@link PercentileResult} for each requested percentile, in the + * same order as the input. Each percentile must be in {@code [0.0, 1.0]}. + * Returns an empty list if the histogram has no observations. This mirrors + * the algorithm used by the Rust crate. + * + * @throws IllegalArgumentException if any percentile is out of range + */ + public List percentiles(double... percentiles) { + for (double p : percentiles) { + if (!(p >= 0.0 && p <= 1.0)) { + throw new IllegalArgumentException( + "percentiles must be in the range [0.0, 1.0], got " + p); + } + } + + long total = totalCount(); + if (total == 0) { + return List.of(); + } + + // Deduplicate and sort while remembering the original order for output. + double[] sortedUnique = Arrays.stream(percentiles).distinct().sorted().toArray(); + + Map resultsByP = new HashMap<>(sortedUnique.length * 2); + int bucketIdx = 0; + long partialSum = buckets[0]; + + for (double p : sortedUnique) { + long target = U64.ceilCount(p, total); + while (Long.compareUnsigned(partialSum, target) < 0 + && bucketIdx < buckets.length - 1) { + bucketIdx++; + partialSum += buckets[bucketIdx]; + } + resultsByP.put(p, bucketAt(bucketIdx)); + } + + List out = new ArrayList<>(percentiles.length); + for (double p : percentiles) { + out.add(new PercentileResult(p, resultsByP.get(p))); + } + return out; + } + + /** Alias for {@link #percentile(double)} (the crate uses "quantile"). */ + public Optional quantile(double quantile) { + return percentile(quantile); + } + + // Conversions ------------------------------------------------------------- + + /** Converts to the sparse (columnar) representation. */ + public SparseHistogram toSparse() { + return SparseHistogram.fromHistogram(this); + } + + /** Converts to a read-only cumulative histogram for fast quantiles. */ + public CumulativeHistogram toCumulative() { + return CumulativeHistogram.fromHistogram(this); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (!(obj instanceof Histogram other)) { + return false; + } + return config.equals(other.config) && Arrays.equals(buckets, other.buckets); + } + + @Override + public int hashCode() { + return 31 * config.hashCode() + Arrays.hashCode(buckets); + } + + @Override + public String toString() { + return "Histogram(grouping_power=" + config.groupingPower() + + ", max_value_power=" + config.maxValuePower() + + ", total_count=" + Long.toUnsignedString(totalCount()) + ")"; + } +} diff --git a/src/main/java/com/iopsystems/h2histogram/PercentileResult.java b/src/main/java/com/iopsystems/h2histogram/PercentileResult.java new file mode 100644 index 0000000..2d3693a --- /dev/null +++ b/src/main/java/com/iopsystems/h2histogram/PercentileResult.java @@ -0,0 +1,9 @@ +package com.iopsystems.h2histogram; + +/** + * Pairs a requested percentile with the {@link Bucket} it resolves to. + * + * @param percentile the requested percentile, in {@code [0.0, 1.0]} + * @param bucket the bucket the percentile falls into + */ +public record PercentileResult(double percentile, Bucket bucket) {} diff --git a/src/main/java/com/iopsystems/h2histogram/SparseHistogram.java b/src/main/java/com/iopsystems/h2histogram/SparseHistogram.java new file mode 100644 index 0000000..787bbbd --- /dev/null +++ b/src/main/java/com/iopsystems/h2histogram/SparseHistogram.java @@ -0,0 +1,175 @@ +package com.iopsystems.h2histogram; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Optional; + +/** + * A histogram stored as {@code (index, count)} pairs for non-zero buckets. + * + *

Only non-zero buckets are stored, as two parallel arrays {@code index} + * and {@code count} in ascending index order. This is the form Rezolus uses + * for its {@code :bucket_indices} / {@code :bucket_counts} parquet columns. + */ +public final class SparseHistogram { + private final Config config; + private final int[] index; + private final long[] count; + + private SparseHistogram(Config config, int[] index, long[] count) { + this.config = config; + this.index = index; + this.count = count; + } + + /** Builds a sparse histogram from a dense {@link Histogram}. */ + public static SparseHistogram fromHistogram(Histogram histogram) { + long[] buckets = histogram.bucketsRef(); + int nonzero = 0; + for (long c : buckets) { + if (c != 0) { + nonzero++; + } + } + int[] index = new int[nonzero]; + long[] count = new long[nonzero]; + int k = 0; + for (int i = 0; i < buckets.length; i++) { + if (buckets[i] != 0) { + index[k] = i; + count[k] = buckets[i]; + k++; + } + } + return new SparseHistogram(histogram.config(), index, count); + } + + /** + * Creates a sparse histogram from raw parts, validating invariants. + * + * @throws IllegalArgumentException if the lengths differ, an index is out + * of range, or the indices are not strictly ascending + */ + public static SparseHistogram fromParts(Config config, int[] index, long[] count) { + if (index.length != count.length) { + throw new IllegalArgumentException( + "index and count must have the same length (" + + index.length + " != " + count.length + ")"); + } + int total = config.totalBuckets(); + int prev = -1; + for (int i : index) { + if (i < 0 || i >= total) { + throw new IllegalArgumentException("index " + i + " out of range for config"); + } + if (i <= prev) { + throw new IllegalArgumentException("indices must be strictly ascending"); + } + prev = i; + } + return new SparseHistogram(config, index.clone(), count.clone()); + } + + /** Returns the bucketing configuration. */ + public Config config() { + return config; + } + + /** Returns a copy of the non-zero bucket indices, ascending. */ + public int[] index() { + return index.clone(); + } + + /** Returns a copy of the counts corresponding to {@link #index()}. */ + public long[] count() { + return count.clone(); + } + + /** Returns the number of stored (non-zero) buckets. */ + public int size() { + return index.length; + } + + /** Returns whether the histogram has no stored buckets. */ + public boolean isEmpty() { + return index.length == 0; + } + + /** Returns the total number of observations (unsigned). */ + public long totalCount() { + long total = 0; + for (long c : count) { + total += c; + } + return total; + } + + /** Returns each stored bucket with its individual count. */ + public List buckets() { + List out = new ArrayList<>(index.length); + for (int k = 0; k < index.length; k++) { + out.add(new Bucket(count[k], + config.indexToLowerBound(index[k]), config.indexToUpperBound(index[k]))); + } + return out; + } + + /** Converts to a dense {@link Histogram}. */ + public Histogram toDense() { + Histogram h = new Histogram(config); + long[] buckets = h.bucketsRef(); + for (int k = 0; k < index.length; k++) { + buckets[index[k]] = count[k]; + } + return h; + } + + /** Converts to a read-only {@link CumulativeHistogram}. */ + public CumulativeHistogram toCumulative() { + return CumulativeHistogram.fromSparse(this); + } + + /** Computes a percentile via the dense representation. */ + public Optional percentile(double percentile) { + return toDense().percentile(percentile); + } + + /** Computes percentiles via the dense representation. */ + public List percentiles(double... percentiles) { + return toDense().percentiles(percentiles); + } + + int[] indexRef() { + return index; + } + + long[] countRef() { + return count; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (!(obj instanceof SparseHistogram other)) { + return false; + } + return config.equals(other.config) + && Arrays.equals(index, other.index) + && Arrays.equals(count, other.count); + } + + @Override + public int hashCode() { + return 31 * (31 * config.hashCode() + Arrays.hashCode(index)) + Arrays.hashCode(count); + } + + @Override + public String toString() { + return "SparseHistogram(grouping_power=" + config.groupingPower() + + ", max_value_power=" + config.maxValuePower() + + ", nonzero_buckets=" + index.length + ")"; + } +} diff --git a/src/main/java/com/iopsystems/h2histogram/U64.java b/src/main/java/com/iopsystems/h2histogram/U64.java new file mode 100644 index 0000000..90b1f3a --- /dev/null +++ b/src/main/java/com/iopsystems/h2histogram/U64.java @@ -0,0 +1,44 @@ +package com.iopsystems.h2histogram; + +/** + * Helpers for treating Java {@code long} values as unsigned 64-bit integers + * ({@code u64}). + * + *

All values and counts in this library are {@code u64}, stored in Java + * {@code long}s with unsigned semantics ({@link Long#compareUnsigned}, + * {@link Long#toUnsignedString}, ...). This gives the full {@code u64} value + * range, exactly like the Rust crate. + */ +final class U64 { + private U64() {} + + private static final double TWO_POW_63 = 9.223372036854775808E18; + + /** Converts an unsigned 64-bit value to the nearest {@code double}. */ + static double toDouble(long value) { + if (value >= 0) { + return value; + } + return (double) (value >>> 1) * 2.0 + (value & 1L); + } + + /** Converts a non-negative {@code double} to an unsigned 64-bit value. */ + static long fromDouble(double value) { + if (value < TWO_POW_63) { + return (long) value; + } + return ((long) (value - TWO_POW_63)) + Long.MIN_VALUE; + } + + /** + * Computes {@code max(1, ceil(p * total))} with {@code total} unsigned, + * matching the Rust crate's {@code max(1, (q * total).ceil())}. + */ + static long ceilCount(double p, long total) { + double target = Math.ceil(p * toDouble(total)); + if (target < 1.0) { + return 1; + } + return fromDouble(target); + } +} diff --git a/src/main/java/com/iopsystems/h2histogram/package-info.java b/src/main/java/com/iopsystems/h2histogram/package-info.java new file mode 100644 index 0000000..0d9dc2e --- /dev/null +++ b/src/main/java/com/iopsystems/h2histogram/package-info.java @@ -0,0 +1,29 @@ +/** + * A pure-Java implementation of the iopsystems h2 histogram. + * + *

It produces histograms with byte-for-byte identical bucketing to the Rust + * histogram crate, so + * histograms recorded here can be consumed by (and interoperate with) Rezolus + * and the Python, Go, and JavaScript implementations. + * + *

The bucketing strategy is fully determined by two parameters: + * + *

+ * + *

The layout has two regions: a linear region covering + * {@code 0 .. 2^(groupingPower+1)} where every bucket has width 1 (exact), and + * a logarithmic region above the cutoff subdivided into + * {@code 2^groupingPower} buckets per power of two. + * + *

Values and counts are unsigned 64-bit integers ({@code u64}) carried in + * Java {@code long}s with unsigned semantics, so the full {@code u64} range is + * supported, exactly like the Rust crate. + */ +package com.iopsystems.h2histogram; diff --git a/src/test/java/com/iopsystems/h2histogram/ConfigTest.java b/src/test/java/com/iopsystems/h2histogram/ConfigTest.java new file mode 100644 index 0000000..271b603 --- /dev/null +++ b/src/test/java/com/iopsystems/h2histogram/ConfigTest.java @@ -0,0 +1,111 @@ +package com.iopsystems.h2histogram; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import org.junit.jupiter.api.Test; + +/** + * Expected values here are copied verbatim from src/config.rs in + * https://github.com/iopsystems/histogram so we are guaranteed bit-identical + * bucketing. + */ +class ConfigTest { + private static final long U64_MAX = -1L; // 2^64 - 1, unsigned + + @Test + void totalBuckets() { + assertEquals(252, new Config(2, 64).totalBuckets()); + assertEquals(7424, new Config(7, 64).totalBuckets()); + assertEquals(835_584, new Config(14, 64).totalBuckets()); + assertEquals(12, new Config(2, 4).totalBuckets()); + } + + @Test + void valueToIndex() { + Config c = new Config(7, 64); + assertEquals(0, c.valueToIndex(0)); + assertEquals(1, c.valueToIndex(1)); + assertEquals(256, c.valueToIndex(256)); + assertEquals(256, c.valueToIndex(257)); + assertEquals(257, c.valueToIndex(258)); + assertEquals(384, c.valueToIndex(512)); + assertEquals(384, c.valueToIndex(515)); + assertEquals(385, c.valueToIndex(516)); + assertEquals(512, c.valueToIndex(1024)); + assertEquals(512, c.valueToIndex(1031)); + assertEquals(513, c.valueToIndex(1032)); + assertEquals(7423, c.valueToIndex(U64_MAX - 1)); + assertEquals(7423, c.valueToIndex(U64_MAX)); + } + + @Test + void indexToLowerBound() { + Config c = new Config(7, 64); + assertEquals(0, c.indexToLowerBound(0)); + assertEquals(1, c.indexToLowerBound(1)); + assertEquals(256, c.indexToLowerBound(256)); + assertEquals(512, c.indexToLowerBound(384)); + assertEquals(1024, c.indexToLowerBound(512)); + assertEquals(Long.parseUnsignedLong("18374686479671623680"), + c.indexToLowerBound(7423)); + } + + @Test + void indexToUpperBound() { + Config c = new Config(7, 64); + assertEquals(0, c.indexToUpperBound(0)); + assertEquals(1, c.indexToUpperBound(1)); + assertEquals(257, c.indexToUpperBound(256)); + assertEquals(515, c.indexToUpperBound(384)); + assertEquals(1031, c.indexToUpperBound(512)); + assertEquals(U64_MAX, c.indexToUpperBound(7423)); + } + + @Test + void roundtripValueIndexRange() { + Config c = new Config(7, 64); + long[] values = {0, 1, 5, 127, 128, 255, 256, 257, 999, 1_000_000, (1L << 40) + 7}; + for (long v : values) { + int idx = c.valueToIndex(v); + long lo = c.indexToLowerBound(idx); + long hi = c.indexToUpperBound(idx); + assertTrue(Long.compareUnsigned(lo, v) <= 0 && Long.compareUnsigned(v, hi) <= 0, + "value " + v + " landed in bucket " + idx + " [" + lo + ", " + hi + "]"); + } + } + + @Test + void error() { + assertEquals(100.0 / 128, new Config(7, 64).error(), 1e-12); + // No logarithmic buckets -> zero error. + assertEquals(0.0, new Config(3, 4).error()); + } + + @Test + void invalidParams() { + assertThrows(IllegalArgumentException.class, () -> new Config(7, 65)); + assertThrows(IllegalArgumentException.class, () -> new Config(64, 64)); + assertThrows(IllegalArgumentException.class, () -> new Config(10, 5)); + assertThrows(IllegalArgumentException.class, () -> new Config(-1, 4)); + } + + @Test + void fromTotalBuckets() { + Config c = Config.fromTotalBuckets(7424, 64); + assertEquals(7, c.groupingPower()); + assertEquals(64, c.maxValuePower()); + // Rezolus default. + Config c2 = Config.fromTotalBuckets(496, 64); + assertEquals(3, c2.groupingPower()); + assertThrows(IllegalArgumentException.class, () -> Config.fromTotalBuckets(7425, 64)); + } + + @Test + void outOfRange() { + Config c = new Config(2, 4); // max = 15 + assertEquals(c.totalBuckets() - 1, c.valueToIndex(15)); + assertThrows(IllegalArgumentException.class, () -> c.valueToIndex(16)); + } +} diff --git a/src/test/java/com/iopsystems/h2histogram/CumulativeHistogramTest.java b/src/test/java/com/iopsystems/h2histogram/CumulativeHistogramTest.java new file mode 100644 index 0000000..3f1eff9 --- /dev/null +++ b/src/test/java/com/iopsystems/h2histogram/CumulativeHistogramTest.java @@ -0,0 +1,128 @@ +package com.iopsystems.h2histogram; + +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.Optional; +import java.util.OptionalDouble; +import org.junit.jupiter.api.Test; + +class CumulativeHistogramTest { + + @Test + void fromHistogram() { + Histogram h = new Histogram(7, 64); + h.record(1, 2); + h.record(500, 3); + h.record(1_000_000, 5); + + CumulativeHistogram c = h.toCumulative(); + assertEquals(10, c.totalCount()); + assertEquals(3, c.size()); + // Cumulative counts are prefix sums; last equals total. + assertArrayEquals(new long[] {2, 5, 10}, c.count()); + } + + @Test + void percentileMatchesDense() { + Histogram h = new Histogram(7, 64); + for (long i = 1; i <= 1000; i++) { + h.increment(i); + } + CumulativeHistogram c = h.toCumulative(); + for (double q : new double[] {0.0, 0.1, 0.5, 0.9, 0.99, 1.0}) { + Optional dense = h.percentile(q); + Optional cumulative = c.percentile(q); + assertTrue(dense.isPresent() && cumulative.isPresent(), "empty bucket at q=" + q); + assertEquals(dense.get().start(), cumulative.get().start(), "start at q=" + q); + assertEquals(dense.get().end(), cumulative.get().end(), "end at q=" + q); + } + } + + @Test + void empty() { + Histogram h = new Histogram(7, 64); + CumulativeHistogram c = h.toCumulative(); + assertTrue(c.isEmpty()); + assertTrue(c.percentile(0.5).isEmpty()); + assertTrue(c.mean().isEmpty()); + } + + @Test + void mean() { + Histogram h = new Histogram(7, 64); + // All in the linear (exact) region so the midpoint mean is exact. + h.record(10, 1); + h.record(20, 1); + h.record(30, 1); + OptionalDouble m = h.toCumulative().mean(); + assertTrue(m.isPresent()); + assertEquals(20.0, m.getAsDouble(), 1e-9); + } + + @Test + void fromParts() { + Config cfg = new Config(7, 64); + CumulativeHistogram c = CumulativeHistogram.fromParts( + cfg, new int[] {1, 256}, new long[] {3, 8}); + assertEquals(8, c.totalCount()); + // invalid: zero count + assertThrows(IllegalArgumentException.class, + () -> CumulativeHistogram.fromParts(cfg, new int[] {1}, new long[] {0})); + // invalid: decreasing + assertThrows(IllegalArgumentException.class, + () -> CumulativeHistogram.fromParts(cfg, new int[] {1, 2}, new long[] {5, 3})); + // invalid: not ascending + assertThrows(IllegalArgumentException.class, + () -> CumulativeHistogram.fromParts(cfg, new int[] {2, 1}, new long[] {1, 2})); + } + + @Test + void bucketQuantileRange() { + Histogram h = new Histogram(7, 64); + h.record(10, 2); + h.record(20, 2); + CumulativeHistogram c = h.toCumulative(); + + Optional first = c.bucketQuantileRange(0); + assertTrue(first.isPresent()); + assertEquals(0.0, first.get().lowerQuantile(), 1e-9); + assertEquals(0.5, first.get().upperQuantile(), 1e-9); + + Optional second = c.bucketQuantileRange(1); + assertTrue(second.isPresent()); + assertEquals(0.5, second.get().lowerQuantile(), 1e-9); + assertEquals(1.0, second.get().upperQuantile(), 1e-9); + + assertTrue(c.bucketQuantileRange(99).isEmpty()); + } + + @Test + void bucketsWithQuantiles() { + Histogram h = new Histogram(7, 64); + h.record(10, 2); + h.record(20, 2); + var withQuantiles = h.toCumulative().bucketsWithQuantiles(); + assertEquals(2, withQuantiles.size()); + assertEquals(0.0, withQuantiles.get(0).lowerQuantile(), 1e-9); + assertEquals(0.5, withQuantiles.get(0).upperQuantile(), 1e-9); + assertEquals(0.5, withQuantiles.get(1).lowerQuantile(), 1e-9); + assertEquals(1.0, withQuantiles.get(1).upperQuantile(), 1e-9); + assertEquals(2, withQuantiles.get(0).bucket().count()); + } + + @Test + void sparseToCumulative() { + Histogram h = new Histogram(7, 64); + h.record(1, 1); + h.record(500, 3); + SparseHistogram sparse = h.toSparse(); + CumulativeHistogram c = sparse.toCumulative(); + assertEquals(4, c.totalCount()); + assertEquals(h, c.toDense()); + assertFalse(c.isEmpty()); + } +} diff --git a/src/test/java/com/iopsystems/h2histogram/ExampleTest.java b/src/test/java/com/iopsystems/h2histogram/ExampleTest.java new file mode 100644 index 0000000..e44f185 --- /dev/null +++ b/src/test/java/com/iopsystems/h2histogram/ExampleTest.java @@ -0,0 +1,41 @@ +package com.iopsystems.h2histogram; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import org.junit.jupiter.api.Test; + +/** The README quick-start example, kept compiling and passing. */ +class ExampleTest { + + @Test + void quickStart() { + Histogram h = new Histogram(7, 64); // groupingPower, maxValuePower + + h.increment(42); + h.record(1000, 5); // value, count + h.recordMany(new long[] {12, 15, 900}); // bulk + + assertEquals(9, h.totalCount()); + + Bucket p99 = h.percentile(0.99).orElseThrow(); + assertTrue(p99.start() <= p99.end()); + assertTrue(p99.midpoint() > 0); + + // Combine / reduce + Histogram coarse = h.downsample(4); // fewer buckets, higher error, same total count + assertEquals(h.totalCount(), coarse.totalCount()); + SparseHistogram sparse = h.toSparse(); // columnar (index, count) form for storage + assertEquals(h.totalCount(), sparse.totalCount()); + + // Fast repeated quantile queries + CumulativeHistogram c = h.toCumulative(); // read-only; also SparseHistogram.toCumulative() + Bucket cumulativeP99 = c.percentile(0.99).orElseThrow(); // O(log n) binary search + assertEquals(p99, cumulativeP99); + assertTrue(c.mean().isPresent()); // midpoint-estimated mean, computed once + assertTrue(c.bucketQuantileRange(0).isPresent()); + for (BucketWithQuantiles bq : c.bucketsWithQuantiles()) { + assertTrue(bq.lowerQuantile() <= bq.upperQuantile()); + } + } +} diff --git a/src/test/java/com/iopsystems/h2histogram/HistogramTest.java b/src/test/java/com/iopsystems/h2histogram/HistogramTest.java new file mode 100644 index 0000000..618fce2 --- /dev/null +++ b/src/test/java/com/iopsystems/h2histogram/HistogramTest.java @@ -0,0 +1,216 @@ +package com.iopsystems.h2histogram; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.List; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicInteger; +import org.junit.jupiter.api.Test; + +class HistogramTest { + + @Test + void incrementAndTotal() { + Histogram h = new Histogram(7, 64); + for (long i = 0; i <= 100; i++) { + h.increment(i); + } + assertEquals(101, h.totalCount()); + } + + @Test + void recordWithCount() { + Histogram h = new Histogram(7, 64); + h.record(100, 5); + assertEquals(5, h.totalCount()); + int idx = h.config().valueToIndex(100); + assertEquals(5, h.bucketCounts()[idx]); + } + + @Test + void percentileExactLowRange() { + // In the linear region (values < cutoff) buckets have width 1, so + // percentiles are exact. + Histogram h = new Histogram(7, 64); + for (long i = 1; i <= 100; i++) { + h.increment(i); + } + assertBucketRange(h.percentile(0.5), 50, 50); + assertBucketRange(h.percentile(1.0), 100, 100); + assertBucketRange(h.percentile(0.0), 1, 1); + } + + private static void assertBucketRange(Optional bucket, long start, long end) { + assertTrue(bucket.isPresent()); + assertEquals(start, bucket.get().start()); + assertEquals(end, bucket.get().end()); + } + + @Test + void percentileEmpty() { + Histogram h = new Histogram(7, 64); + assertTrue(h.percentile(0.5).isEmpty()); + assertTrue(h.percentiles(0.5, 0.9).isEmpty()); + } + + @Test + void percentilesOrderPreserved() { + Histogram h = new Histogram(7, 64); + for (long i = 0; i < 1000; i++) { + h.increment(i); + } + List results = h.percentiles(0.9, 0.5, 0.99); + double[] want = {0.9, 0.5, 0.99}; + assertEquals(want.length, results.size()); + for (int i = 0; i < want.length; i++) { + assertEquals(want[i], results.get(i).percentile()); + } + } + + @Test + void percentileInvalid() { + Histogram h = new Histogram(7, 64); + h.increment(1); + assertThrows(IllegalArgumentException.class, () -> h.percentile(1.5)); + assertThrows(IllegalArgumentException.class, () -> h.percentile(-0.1)); + } + + @Test + void merge() { + Histogram a = new Histogram(7, 64); + Histogram b = new Histogram(7, 64); + a.record(10, 3); + b.record(10, 4); + b.record(2000, 1); + Histogram merged = a.merge(b); + assertEquals(8, merged.totalCount()); + int idx = merged.config().valueToIndex(10); + assertEquals(7, merged.bucketCounts()[idx]); + } + + @Test + void mergeIncompatible() { + Histogram a = new Histogram(7, 64); + Histogram b = new Histogram(6, 64); + assertThrows(IllegalArgumentException.class, () -> a.merge(b)); + } + + @Test + void subtract() { + Histogram a = new Histogram(7, 64); + Histogram b = new Histogram(7, 64); + a.record(10, 5); + b.record(10, 2); + Histogram diff = a.subtract(b); + assertEquals(3, diff.totalCount()); + assertThrows(IllegalArgumentException.class, () -> b.subtract(a)); + } + + @Test + void fromBucketsRoundtrip() { + Histogram h = new Histogram(3, 64); + h.record(5, 2); + h.record(1000, 7); + Histogram h2 = Histogram.fromBuckets(3, 64, h.bucketCounts()); + assertEquals(h, h2); + } + + @Test + void fromBucketsWrongLength() { + assertThrows(IllegalArgumentException.class, + () -> Histogram.fromBuckets(7, 64, new long[] {0, 0, 0})); + } + + @Test + void downsample() { + Histogram h = new Histogram(7, 64); + for (long i = 0; i < 10000; i++) { + h.increment(i); + } + Histogram coarse = h.downsample(3); + assertEquals(3, coarse.config().groupingPower()); + assertEquals(h.totalCount(), coarse.totalCount()); + assertThrows(IllegalArgumentException.class, () -> h.downsample(7)); + } + + @Test + void sparseRoundtrip() { + Histogram h = new Histogram(7, 64); + h.record(1, 1); + h.record(500, 3); + h.record(999999, 2); + SparseHistogram sparse = h.toSparse(); + assertEquals(h.totalCount(), sparse.totalCount()); + assertEquals(3, sparse.size()); + int prev = -1; + for (int i : sparse.index()) { + assertTrue(i > prev, "sparse indices not strictly ascending"); + prev = i; + } + assertEquals(h, sparse.toDense()); + } + + @Test + void sparseFromPartsValidation() { + Config c = new Config(7, 64); + assertThrows(IllegalArgumentException.class, + () -> SparseHistogram.fromParts(c, new int[] {1, 2}, new long[] {1})); + assertThrows(IllegalArgumentException.class, + () -> SparseHistogram.fromParts(c, new int[] {2, 1}, new long[] {1, 1})); + assertThrows(IllegalArgumentException.class, + () -> SparseHistogram.fromParts(c, new int[] {999999999}, new long[] {1})); + } + + @Test + void recordManyMatchesLoop() { + long[] base = {0, 1, 2, 300, 255, 256, 1024, 1_000_000, (1L << 50) + 3}; + long[] values = new long[base.length * 111]; + for (int i = 0; i < 111; i++) { + System.arraycopy(base, 0, values, i * base.length, base.length); + } + Histogram a = new Histogram(7, 64); + for (long v : values) { + a.increment(v); + } + Histogram b = new Histogram(7, 64); + b.recordMany(values); + assertEquals(a, b); + } + + @Test + void recordManyWithCounts() { + Histogram a = new Histogram(7, 64); + a.recordMany(new long[] {10, 20, 10}, new long[] {2, 3, 5}); + assertEquals(10, a.totalCount()); + int idx = a.config().valueToIndex(10); + assertEquals(7, a.bucketCounts()[idx]); + assertThrows(IllegalArgumentException.class, + () -> a.recordMany(new long[] {1, 2}, new long[] {1})); + } + + @Test + void iterBuckets() { + Histogram h = new Histogram(3, 6); + h.increment(0); + AtomicInteger count = new AtomicInteger(); + h.forEachBucket(b -> count.incrementAndGet()); + assertEquals(h.config().totalBuckets(), count.get()); + List nonzero = h.nonzeroBuckets(); + assertEquals(1, nonzero.size()); + assertEquals(1, nonzero.get(0).count()); + } + + @Test + void fullU64Range() { + // Values above Long.MAX_VALUE are handled with unsigned semantics. + Histogram h = new Histogram(7, 64); + h.increment(-1L); // u64::MAX + h.increment(Long.MIN_VALUE); // 2^63 + assertEquals(2, h.totalCount()); + Optional p100 = h.percentile(1.0); + assertTrue(p100.isPresent()); + assertEquals(-1L, p100.get().end()); // upper bound is u64::MAX + } +} From c8888a2cc7390dd578ebfd0ae1943b413dcffe87 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Jul 2026 02:02:48 +0000 Subject: [PATCH 2/3] Add JMH benchmarks comparing against HdrHistogram Single-threaded record-path benchmark at matched relative-error levels (h2 grouping powers 3/7/14 vs HDR 1/2/4 significant digits), with value streams touching from 64 buckets up to the full counter array to stress different cache levels. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_016Bb8BttJZHuy6zh957NMGT --- .gitignore | 4 + benchmarks/README.md | 45 +++++++ benchmarks/pom.xml | 85 ++++++++++++ .../h2histogram/bench/RecordBenchmark.java | 126 ++++++++++++++++++ 4 files changed, 260 insertions(+) create mode 100644 benchmarks/README.md create mode 100644 benchmarks/pom.xml create mode 100644 benchmarks/src/main/java/com/iopsystems/h2histogram/bench/RecordBenchmark.java diff --git a/.gitignore b/.gitignore index 2855e70..6895e49 100644 --- a/.gitignore +++ b/.gitignore @@ -33,3 +33,7 @@ target/ .classpath .project .settings/ + +# Benchmarks +benchmarks/dependency-reduced-pom.xml +benchmarks/results.json diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..ca8bb94 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,45 @@ +# Benchmarks + +JMH benchmarks comparing `h2histogram` against +[HdrHistogram](https://github.com/HdrHistogram/HdrHistogram) on the +single-threaded record path. + +## Method + +The two libraries are compared at matched relative-error levels. Because both +use a log-linear bucket layout, the matched settings also produce counter +arrays of almost identical size, so the cache behaviour is directly +comparable: + +| Precision | h2histogram | HdrHistogram | Counter array | +|-----------|-------------|--------------|---------------| +| `gp3` | `groupingPower=3` (12.5% error) | 1 significant digit (~10%) | ~4 KiB | +| `gp7` | `groupingPower=7` (0.78% error) | 2 significant digits (1%) | ~57 KiB | +| `gp14` | `groupingPower=14` (0.006% error) | 4 significant digits (0.01%) | ~6.3 MiB | + +The `spread` parameter controls how many distinct buckets the pre-generated +value stream touches, moving the hot working set through the cache hierarchy: + +- **`few`** — 64 buckets (512 B of counters; L1) +- **`quarter`** — a quarter of all buckets +- **`all`** — every bucket (up to ~6.3 MiB of counters at `gp14`; L3/DRAM) + +Bucket indices are sampled uniformly; sampling uniform *values* would +concentrate nearly all samples in the top power of two of a logarithmic +layout. Both histograms are fed the identical value stream, and use +`maxValuePower=63` / `highestTrackableValue=Long.MAX_VALUE` so every value is +in range for both. + +## Running + +```bash +# from the repository root: install the library, then build and run +mvn -B install -DskipTests +cd benchmarks +mvn -B package +java -jar target/benchmarks.jar +``` + +Useful options: `-p precision=gp7 -p spread=all` to run a single cell, +`-prof perfnorm` for per-op cache-miss counters (Linux), `-rf json` for +machine-readable output. diff --git a/benchmarks/pom.xml b/benchmarks/pom.xml new file mode 100644 index 0000000..2455856 --- /dev/null +++ b/benchmarks/pom.xml @@ -0,0 +1,85 @@ + + + 4.0.0 + + com.iopsystems + h2histogram-benchmarks + 0.1.0-SNAPSHOT + jar + + h2histogram-benchmarks + JMH benchmarks comparing h2histogram against HdrHistogram. + + + 17 + UTF-8 + 1.37 + + + + + com.iopsystems + h2histogram + 0.1.0-SNAPSHOT + + + org.hdrhistogram + HdrHistogram + 2.2.2 + + + org.openjdk.jmh + jmh-core + ${jmh.version} + + + org.openjdk.jmh + jmh-generator-annprocess + ${jmh.version} + provided + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.13.0 + + + org.apache.maven.plugins + maven-shade-plugin + 3.5.2 + + + package + + shade + + + benchmarks + + + org.openjdk.jmh.Main + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + + + + diff --git a/benchmarks/src/main/java/com/iopsystems/h2histogram/bench/RecordBenchmark.java b/benchmarks/src/main/java/com/iopsystems/h2histogram/bench/RecordBenchmark.java new file mode 100644 index 0000000..55570f5 --- /dev/null +++ b/benchmarks/src/main/java/com/iopsystems/h2histogram/bench/RecordBenchmark.java @@ -0,0 +1,126 @@ +package com.iopsystems.h2histogram.bench; + +import com.iopsystems.h2histogram.Config; +import com.iopsystems.h2histogram.Histogram; +import java.util.Random; +import java.util.concurrent.TimeUnit; +import org.openjdk.jmh.annotations.Benchmark; +import org.openjdk.jmh.annotations.BenchmarkMode; +import org.openjdk.jmh.annotations.Fork; +import org.openjdk.jmh.annotations.Measurement; +import org.openjdk.jmh.annotations.Mode; +import org.openjdk.jmh.annotations.OperationsPerInvocation; +import org.openjdk.jmh.annotations.OutputTimeUnit; +import org.openjdk.jmh.annotations.Param; +import org.openjdk.jmh.annotations.Scope; +import org.openjdk.jmh.annotations.Setup; +import org.openjdk.jmh.annotations.State; +import org.openjdk.jmh.annotations.Warmup; + +/** + * Single-threaded record-path comparison between h2histogram and HdrHistogram. + * + *

The two libraries are compared at matched relative-error levels; because + * both use a log-linear bucket layout, the matched settings also produce + * counter arrays of almost identical size, making the cache behaviour + * directly comparable: + * + *

+ * + *

The {@code spread} parameter controls how many distinct buckets the + * pre-generated value stream touches, moving the hot working set through the + * cache hierarchy: {@code few} (64 buckets, fits in L1 alongside the data), + * {@code quarter} (a quarter of all buckets), {@code all} (every bucket). + * Bucket indices are sampled uniformly — sampling uniform values + * would concentrate nearly all samples in the top power of two of a + * logarithmic layout. + * + *

Histograms use maxValuePower=63 so the identical value stream is in + * range for HdrHistogram, which only accepts values up to Long.MAX_VALUE. + */ +@State(Scope.Benchmark) +@BenchmarkMode(Mode.AverageTime) +@OutputTimeUnit(TimeUnit.NANOSECONDS) +@Warmup(iterations = 3, time = 1) +@Measurement(iterations = 5, time = 1) +@Fork(1) +public class RecordBenchmark { + + static final int N = 1 << 20; + + @Param({"gp3", "gp7", "gp14"}) + public String precision; + + @Param({"few", "quarter", "all"}) + public String spread; + + long[] values; + Histogram h2; + org.HdrHistogram.Histogram hdr; + + @Setup + public void setup() { + int groupingPower; + int hdrDigits; + switch (precision) { + case "gp3" -> { + groupingPower = 3; + hdrDigits = 1; + } + case "gp7" -> { + groupingPower = 7; + hdrDigits = 2; + } + case "gp14" -> { + groupingPower = 14; + hdrDigits = 4; + } + default -> throw new IllegalArgumentException(precision); + } + + Config config = new Config(groupingPower, 63); + h2 = new Histogram(config); + hdr = new org.HdrHistogram.Histogram(Long.MAX_VALUE, hdrDigits); + + int totalBuckets = config.totalBuckets(); + int targetBuckets = switch (spread) { + case "few" -> Math.min(64, totalBuckets); + case "quarter" -> Math.max(1, totalBuckets / 4); + case "all" -> totalBuckets; + default -> throw new IllegalArgumentException(spread); + }; + + Random rng = new Random(42); + values = new long[N]; + for (int i = 0; i < N; i++) { + values[i] = config.indexToLowerBound(rng.nextInt(targetBuckets)); + } + } + + @Benchmark + @OperationsPerInvocation(N) + public Histogram h2Record() { + Histogram h = h2; + for (long v : values) { + h.increment(v); + } + return h; + } + + @Benchmark + @OperationsPerInvocation(N) + public org.HdrHistogram.Histogram hdrRecord() { + org.HdrHistogram.Histogram h = hdr; + for (long v : values) { + h.recordValue(v); + } + return h; + } +} From f65124d33a360892d7bf83c7feb1ebf186e5482d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Jul 2026 02:04:01 +0000 Subject: [PATCH 3/3] Add single-threaded benchmark results vs HdrHistogram Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_016Bb8BttJZHuy6zh957NMGT --- benchmarks/README.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/benchmarks/README.md b/benchmarks/README.md index ca8bb94..3b481f0 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -43,3 +43,41 @@ java -jar target/benchmarks.jar Useful options: `-p precision=gp7 -p spread=all` to run a single cell, `-prof perfnorm` for per-op cache-miss counters (Linux), `-rf json` for machine-readable output. + +## Results + +Single-threaded, average time per recorded value (lower is better). Measured +on a 4-vCPU Intel Xeon @ 2.80 GHz cloud instance (32 KiB L1d / 1 MiB L2 per +core, 33 MiB shared L3), OpenJDK 21, JMH 1.37, 5×1 s iterations after 3×1 s +warmup, one fork. Treat the numbers as indicative — this is a shared cloud +machine. + +| Precision | Spread | h2histogram (ns/op) | HdrHistogram (ns/op) | +|-----------|--------|--------------------:|---------------------:| +| gp3 (~4 KiB) | few | 3.77 ± 0.37 | 2.90 ± 0.12 | +| gp3 | quarter | 2.70 ± 0.08 | 2.88 ± 0.12 | +| gp3 | all | 1.99 ± 0.07 | 2.86 ± 0.23 | +| gp7 (~57 KiB) | few | 0.76 ± 0.02 | 2.94 ± 0.27 | +| gp7 | quarter | 2.74 ± 0.09 | 2.72 ± 0.04 | +| gp7 | all | 2.03 ± 0.16 | 2.82 ± 0.12 | +| gp14 (~6.3 MiB) | few | 0.76 ± 0.03 | 3.01 ± 0.19 | +| gp14 | quarter | 3.47 ± 0.15 | 5.05 ± 0.23 | +| gp14 | all | 3.86 ± 0.88 | 6.82 ± 0.65 | + +Observations: + +- **Linear-region fast path.** When every value lands in the linear region + (`few` at gp7/gp14, where the first 64 buckets are all width-1), h2's + index computation collapses to a bounds check plus an array increment: + ~0.76 ns/op, roughly 4× faster than HDR on the same stream. +- **Large working sets favour h2.** At gp14 with the full ~6.3 MiB counter + array in play (past L2), h2 records ~1.8× faster than HDR at essentially + the same relative error, thanks to its cheaper index computation ahead of + the inevitable cache misses. +- **Mixed streams are a wash at moderate sizes.** At gp7/`quarter` the two + are within noise of each other; HDR is remarkably flat (~2.7–3.0 ns/op) + whenever its array fits in cache. +- The gp3/`few` cell is the one spread where h2 trails: at grouping power 3 + the linear region is only 16 buckets wide, so a 64-bucket stream mixes + linear and logarithmic paths and pays for branch misprediction, while at + `all` the branch becomes predictable-enough again and h2 pulls ahead.