Define deterministic floating-point grid-boundary semantics for RS_AsRaster

[jiayuasu]

Expected behavior

Rasterization should have a documented, deterministic rule for deciding whether a stored double-precision geometry coordinate represents a pixel grid boundary. The rule should behave consistently across pixel scales and world-coordinate magnitudes, while preserving coordinates that genuinely cross a boundary by a representable positive distance.

Actual behavior

Grid-boundary handling currently depends on whether the computed inverse affine coordinate happens to be exactly integral as a double:

double pixel = (coordinate - origin) / scale;

Common decimal and affine construction paths can leave a nonintegral residue for an intended boundary, causing an endpoint-only adjacent cell to be burned. Conversely, division can round a genuine adjacent-double crossing to an integral pixel coordinate, causing a cell with positive-length overlap to be omitted.

Examples:

• Decimal grid: (0.3 - 0.6) / -0.1 == 2.9999999999999996.
• Affine-generated value: with origin 0.1 and scale 0.1, 0.1 + 2 * 0.1 == 0.30000000000000004, and the inverse result is 2.0000000000000004.
• Large origin: (1000000.03 - 1000000.0) / 0.01 == 3.0000000027939677.
• Rounded quotient in the other direction: 0.030000000000000002 / 0.01 == 3.0, even though the coordinate is the next representable double above 0.03.

Steps to reproduce

Intended decimal boundary is classified as nonintegral

SELECT RS_BandAsArray(
  RS_AsRaster(
    ST_GeomFromWKT('LINESTRING (0.22 0.22, 0.3 0.3)'),
    RS_MakeEmptyRaster(1, 'B', 6, 6, 0.0, 0.6, 0.1, -0.1, 0.0, 0.0, 0),
    'B', true, 1.0, 0.0, false),
  1);

Under a policy that accepts the decimal relation as a boundary, the segment traverses only zero-based (row 3, column 2) and terminates at the pixel corner. The inverse-coordinate residue additionally burns endpoint-only (row 2, column 2). Under an exact stored-IEEE policy, that tiny extension would instead be considered real; choosing between those interpretations is the decision this issue requests.

Genuine representable crossing is classified as integral

SELECT RS_BandAsArray(
  RS_AsRaster(
    ST_GeomFromWKT('LINESTRING (0.025 0.045, 0.030000000000000002 0.045)'),
    RS_MakeEmptyRaster(1, 'B', 6, 6, 0.0, 0.06, 0.01, -0.01, 0.0, 0.0, 0),
    'B', true, 1.0, 0.0, false),
  1);

0.030000000000000002 is the next representable double above 0.03, so the line enters column 3 for a positive distance. Sedona burns only zero-based (row 1, column 2) because the inverse division rounds the endpoint to exactly 3.0; zero-based (row 1, columns 2 and 3) should be burned under exact stored-double geometry.

Why a tolerance is not a complete fix

The original source intent is no longer available after parsing or coordinate transformation. For origin 0.1, scale 0.1, and boundary index 2:

decimal boundary                         = 0.3
binary affine result (0.1 + 2 * 0.1)    = 0.30000000000000004
Math.nextUp(0.3)                        = 0.30000000000000004

The same stored bits can mean either an affine-generated boundary or a genuine one-double crossing beyond the decimal boundary. No epsilon can recover those two intents. A fixed or magnitude-scaled tolerance can also swallow real endpoint slivers, while unconditional exact-decimal arithmetic can add per-vertex allocation and CPU cost.

Decision required

Choose and document the boundary-equivalence policy before implementing it. Candidate policies include:

1. one canonical boundary encoding produced by the raster's actual forward affine transform;
2. an explicit finite set of boundary-equivalent encodings, such as forward-affine and canonical-decimal relations;
3. exact arithmetic over the stored IEEE-754 values, accepting that common decimal relations may not be boundaries.

This is a compatibility policy, not recovery of source intent. If multiple encodings are accepted, adjacent doubles may deliberately be boundary-equivalent and that exception must be documented.

Acceptance criteria

• Public documentation defines which stored coordinate encodings are treated as grid boundaries.
• Classification is axis-local, direction-independent, and consistent across unit, decimal, fractional, negative-scale, and large-origin grids.
• Values not in the accepted equivalence set preserve their exact mathematical side; quotient rounding alone is not an equality decision.
• Tests cover decimal literals, coordinates produced by the raster's actual affine operation, 1/3-style scales, large origins, bottom-up rasters, and adjacent Math.nextDown/Math.nextUp values.
• Tests explicitly document the unavoidable 0.3 versus 0.30000000000000004 policy choice.
• The proposed implementation includes benchmark results for long LineString and Polygon workloads against PR [GH-3120] Fix rasterization at grid-aligned endpoints #3251, with any accepted regression limit stated explicitly.
• Clipped endpoints carry explicit boundary/side provenance from the clipper rather than being reclassified solely from rounded coordinates. RS_AsRaster segment clipping drops crossings and loses grid-line sidedness #3255 should be implemented first.

Related work

• allTouched/line rasterization can burn a corner-touched pixel at grid-aligned vertices (differs from GDAL by one pixel) #3120 tracks the exact-grid-vertex case; [GH-3120] Fix rasterization at grid-aligned endpoints #3251 fixes it when the computed pixel coordinate is already exactly integral.
• RS_AsRaster segment clipping drops crossings and loses grid-line sidedness #3255 should land first so synthetic clipped endpoints do not lose their boundary provenance.

Settings

• Sedona version: PR [GH-3120] Fix rasterization at grid-aligned endpoints #3251 head 72d9cc26541 / 2.0.0-SNAPSHOT
• API type: SQL and Java
• JRE: 17
• Environment: local development checkout

I searched open and closed issues and did not find a duplicate.