Merge pull request #681 from apache/Fix_endian_issues

Made detection of BigEndian consistent throughout.

Author: leerho

NOTICE

@@ -3,7 +3,7 @@ Copyright 2025 The Apache Software Foundation
 
 Copyright 2015-2018 Yahoo Inc.
 Copyright 2019-2020 Verizon Media
-Copyright 2021-     Yahoo Inc.
+Copyright 2021-2025 Yahoo Inc.
 
 This product includes software developed at
 The Apache Software Foundation (http://www.apache.org/).

README.md

@@ -27,8 +27,9 @@ This is the core Java component of the DataSketches library.  It contains all of
 
 This component is also a dependency of other components of the library that create adaptors for target systems, such as the [Apache Pig adaptor](https://github.com/apache/datasketches-pig), the [Apache Hive adaptor](https://github.com/apache/datasketches-hive), and others.
 
-Note that we have a parallel core component for C++ and Python implementations of many of the same sketch algorithms, 
-[datasketches-cpp](https://github.com/apache/datasketches-cpp) and [datasketches-python](https://github.com/apache/datasketches-python)
+Note that we have a parallel core component for C++, Python and GO implementations of many of the same sketch algorithms, 
+[datasketches-cpp](https://github.com/apache/datasketches-cpp), [datasketches-python](https://github.com/apache/datasketches-python), and 
+[datasketches-go](https://github.com/apache/datasketches-go).
 
 Please visit the main [DataSketches website](https://datasketches.apache.org) for more information.
 
@@ -41,28 +42,19 @@ If you are interested in making contributions to this site please see our [Commu
 ### Installation Directory Path
 **NOTE:** This component accesses resource files for testing. As a result, the directory elements of the full absolute path of the target installation directory must qualify as Java identifiers. In other words, the directory elements must not have any space characters (or non-Java identifier characters) in any of the path elements. This is required by the Oracle Java Specification in order to ensure location-independent access to resources: [See Oracle Location-Independent Access to Resources](https://docs.oracle.com/javase/8/docs/technotes/guides/lang/resources.html)
 
-### OpenJDK Version 21
-An OpenJDK-compatible build of Java 21, provided by one of the Open-Source providers, such as Azul Systems, Red Hat, SAP, Eclipse Temurin, etc, is required.
+### OpenJDK Version 24
+An OpenJDK-compatible build of Java 24, provided by one of the Open-Source JVM providers, such as Azul Systems, Red Hat, SAP, Eclipse Temurin, etc, is required.
 All of the testing of this release has been performed with an Eclipse Temurin build.
 
-This release uses the new Java Foreign Function & Memory (FFM) features that are in "preview" in Java 21.
-As a result, the JVM flag <nobr>**--enable-preview**</nobr> must be set at compile and at runtime.
-
-**NOTE:** OpenJDK versions greater than 21 do not support running Java 21 class files (Class ID 65) with preview code. The runtime JVM version must be 21.
-
-**NOTE:** The Eclipse Compiler for Java (ECJ) version 21, used by default in both the Eclipse JDT IDE and the VScode IDE, will not allow compilation of preview code for Java version 21. Both Eclipse and VScode can be configured to use an OpenJDK compiler instead, but you will loose the incremental build capability of the ECJ.
-
-### DataSketches Memory 6.0.0
-
-This component depends on the [datasketches-memory-6.0.0](https://github.com/apache/datasketches-memory/tree/6.0.0) component, 
+This release uses the new Java Foreign Function & Memory (FFM) features that were made part of the Java Language in in Java 22.
 
 ## Compilation and Test using Maven
 This DataSketches component is structured as a Maven project and Maven is the recommended tool for compile and test.
 
 #### A Toolchain is required
 
-* You must have a JDK type toolchain defined in location *~/.m2/toolchains.xml* that specifies where to find a locally installed OpenJDK-compatible version 21.
-* Your default \$JAVA\_HOME compiler must be OpenJDK compatible, specified in the toolchain, and may be a version greater than 21. Note that if your \$JAVA\_HOME is set to a Java version greater than 21, Maven will automatically use the Java 21 version specified in the toolchain instead. The included pom.xml specifies the necessary JVM flags, so no further action should be required.
+* You must have a JDK type toolchain defined in location *~/.m2/toolchains.xml* that specifies where to find a locally installed OpenJDK-compatible version 24.
+* Your default \$JAVA\_HOME compiler must be OpenJDK compatible, specified in the toolchain, and may be a version greater than 24. Note that if your \$JAVA\_HOME is set to a Java version greater than 24, Maven will automatically use the Java 24 version specified in the toolchain instead. The included pom.xml specifies the necessary JVM flags, so no further action should be required.
 * Note that the paths specified in the toolchain must be fully qualified direct paths to the OpenJDK version locations. Using environment variables will not work.
 
 #### To run normal unit tests:

pom.xml

@@ -96,10 +96,9 @@ under the License.
     <!-- System-wide properties -->
     <maven.version>3.9.10</maven.version>
     <java.version>24</java.version>
-    <jvm-arguments>-Xmx10g -Duser.language=en -Duser.country=US -Dfile.encoding=UTF-8</jvm-arguments>
+    <jvm-arguments>-Xmx4g -Duser.language=en -Duser.country=US -Dfile.encoding=UTF-8</jvm-arguments>
     <maven.compiler.source>${java.version}</maven.compiler.source>
     <maven.compiler.target>${java.version}</maven.compiler.target>
-    <argLine>${jvm-arguments}</argLine>
     <charset.encoding>UTF-8</charset.encoding>
     <project.build.sourceEncoding>${charset.encoding}</project.build.sourceEncoding>
     <project.build.resourceEncoding>${charset.encoding}</project.build.resourceEncoding>
@@ -159,7 +158,7 @@ under the License.
           <version>${maven-compiler-plugin.version}</version>
           <configuration>
             <compilerArgs>
-              <arg>${jvm-arguments}</arg>
+              <arg></arg>
             </compilerArgs>
           </configuration>
         </plugin>

src/main/java/org/apache/datasketches/common/BigEndianNativeOrderNotSupportedException.java

@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.datasketches.common;
+
+/**
+ * The DataSketches Library is not supported on Big Endian machines.
+ */
+public class BigEndianNativeOrderNotSupportedException extends SketchesException {
+  private static final long serialVersionUID = 1L;
+
+  /**
+   * Constructs a new runtime exception with the message:
+   * "The DataSketches Library is not supported on Big Endian machines."
+   */
+  public BigEndianNativeOrderNotSupportedException() {
+    super("The DataSketches Library is not supported on Big Endian machines.");
+  }
+}

src/main/java/org/apache/datasketches/common/Family.java

@@ -19,6 +19,7 @@
 
 package org.apache.datasketches.common;
 
+import java.nio.ByteOrder;
 import java.util.HashMap;
 import java.util.Locale;
 import java.util.Map;
@@ -165,19 +166,22 @@ public enum Family {
 
   private static final Map<Integer, Family> lookupID = new HashMap<>();
   private static final Map<String, Family> lookupFamName = new HashMap<>();
-  private int id_;
-  private String famName_;
-  private int minPreLongs_;
-  private int maxPreLongs_;
+  private final int id_;
+  private final String famName_;
+  private final int minPreLongs_;
+  private final int maxPreLongs_;
 
   static {
     for (final Family f : values()) {
       lookupID.put(f.getID(), f);
       lookupFamName.put(f.getFamilyName().toUpperCase(Locale.US), f);
     }
+    if (ByteOrder.nativeOrder() != ByteOrder.LITTLE_ENDIAN) {
+      throw new BigEndianNativeOrderNotSupportedException();
+    }
   }
 
-  private Family(final int id, final String famName, final int minPreLongs, final int maxPreLongs) {
+  Family(final int id, final String famName, final int minPreLongs, final int maxPreLongs) {
     id_ = id;
     famName_ = famName.toUpperCase(Locale.US);
     minPreLongs_ = minPreLongs;

src/main/java/org/apache/datasketches/common/Util.java

@@ -28,6 +28,7 @@
 import static org.apache.datasketches.hash.MurmurHash3.hash;
 
 import java.lang.foreign.MemorySegment;
+import java.nio.ByteOrder;
 import java.util.Comparator;
 
 /**
@@ -38,6 +39,12 @@
 @SuppressWarnings("unchecked")
 public final class Util {
 
+  static {
+    if (ByteOrder.nativeOrder() != ByteOrder.LITTLE_ENDIAN) {
+      throw new BigEndianNativeOrderNotSupportedException();
+    }
+  }
+
   /**
    * The java line separator character as a String.
    */
@@ -812,7 +819,7 @@ public static <T> Object maxT(final Object item1, final Object item2, final Comp
   }
 
   /**
-   * Is item1 Less-Than item2
+   * Is item1 Less-Than item2?
    * @param <T> the type
    * @param item1 item one
    * @param item2 item two
@@ -824,7 +831,7 @@ public static <T> boolean lt(final Object item1, final Object item2, final Compa
   }
 
   /**
-   * Is item1 Less-Than-Or-Equal-To item2
+   * Is item1 Less-Than-Or-Equal-To item2?
    * @param <T> the type
    * @param item1 item one
    * @param item2 item two

src/main/java/org/apache/datasketches/cpc/PreambleUtil.java

@@ -31,7 +31,6 @@
 import static org.apache.datasketches.cpc.RuntimeAsserts.rtAssertEquals;
 
 import java.lang.foreign.MemorySegment;
-import java.nio.ByteOrder;
 import java.util.Objects;
 
 import org.apache.datasketches.common.Family;
@@ -142,12 +141,6 @@ final class PreambleUtil {
 
   private PreambleUtil() {}
 
-  static {
-    if (ByteOrder.nativeOrder() == ByteOrder.BIG_ENDIAN) {
-      throw new SketchesStateException("This sketch will not work on Big Endian CPUs.");
-    }
-  }
-
   private static final String fmt = "%10d%10x";
 
   /**
@@ -156,7 +149,7 @@ private PreambleUtil() {}
   static final byte SER_VER = 1;
 
   //Flag bit masks, Byte 5
-  static final int BIG_ENDIAN_FLAG_MASK     = 1; //Reserved.
+  static final int RESERVED_FLAG_MASK       = 1; //Reserved.
   static final int COMPRESSED_FLAG_MASK     = 2;
   static final int HIP_FLAG_MASK            = 4;
   static final int SUP_VAL_FLAG_MASK        = 8; //num Suprising Values > 0
@@ -584,7 +577,6 @@ static String toString(final MemorySegment seg, final boolean detail) {
 
     //Flags of the Flags byte
     final String flagsStr = zeroPad(Integer.toBinaryString(flags), 8) + ", " + (flags);
-    final boolean bigEndian = (flags & BIG_ENDIAN_FLAG_MASK) > 0;
     final boolean compressed = (flags & COMPRESSED_FLAG_MASK) > 0;
     final boolean hasHip = (flags & HIP_FLAG_MASK) > 0;
     final boolean hasSV = (flags & SUP_VAL_FLAG_MASK) > 0;
@@ -593,8 +585,6 @@ static String toString(final MemorySegment seg, final boolean detail) {
     final int formatOrdinal = (flags >>> 2) & 0x7;
     final Format format = Format.ordinalToFormat(formatOrdinal);
 
-    final String nativeOrderStr = ByteOrder.nativeOrder().toString();
-
     long numCoupons = 0;
     long numSv = 0;
     long winOffset = 0;
@@ -616,8 +606,6 @@ static String toString(final MemorySegment seg, final boolean detail) {
     sb.append("Byte 3: lgK                     : ").append(lgK).append(LS);
     sb.append("Byte 4: First Interesting Col   : ").append(fiCol).append(LS);
     sb.append("Byte 5: Flags                   : ").append(flagsStr).append(LS);
-    sb.append("  BIG_ENDIAN_STORAGE            : ").append(bigEndian).append(LS);
-    sb.append("  (Native Byte Order)           : ").append(nativeOrderStr).append(LS);
     sb.append("  Compressed                    : ").append(compressed).append(LS);
     sb.append("  Has HIP                       : ").append(hasHip).append(LS);
     sb.append("  Has Surprising Values         : ").append(hasSV).append(LS);

src/main/java/org/apache/datasketches/hll/PreambleUtil.java

@@ -32,7 +32,6 @@
 import static org.apache.datasketches.hll.HllUtil.RESIZE_NUMER;
 
 import java.lang.foreign.MemorySegment;
-import java.nio.ByteOrder;
 
 import org.apache.datasketches.common.Family;
 
@@ -133,7 +132,7 @@ private PreambleUtil() {}
   static int HLL_BYTE_ARR_START             = 40;
 
   //Flag bit masks
-  static final int BIG_ENDIAN_FLAG_MASK     = 1; //Set but not read. Reserved.
+  static final int RESERVED_FLAG_MASK       = 1; //Set to 0 but not read.
   static final int READ_ONLY_FLAG_MASK      = 2; //Set but not read. Reserved.
   static final int EMPTY_FLAG_MASK          = 4;
   static final int COMPACT_FLAG_MASK        = 8;
@@ -150,8 +149,6 @@ private PreambleUtil() {}
   static final int LIST_PREINTS             = 2;
   static final int HASH_SET_PREINTS         = 3;
   static final int HLL_PREINTS              = 10;
-  static final boolean NATIVE_ORDER_IS_BIG_ENDIAN  =
-      (ByteOrder.nativeOrder() == ByteOrder.BIG_ENDIAN);
 
   static String toString(final byte[] byteArr) {
     final MemorySegment seg = MemorySegment.ofArray(byteArr);
@@ -168,8 +165,6 @@ static String toString(final MemorySegment seg) {
     final int flags = seg.get(JAVA_BYTE, FLAGS_BYTE);
     //Flags
     final String flagsStr = zeroPad(Integer.toBinaryString(flags), 8) + ", " + (flags);
-    final boolean bigEndian = (flags & BIG_ENDIAN_FLAG_MASK) > 0;
-    final String nativeOrder = ByteOrder.nativeOrder().toString();
     final boolean compact = (flags & COMPACT_FLAG_MASK) > 0;
     final boolean oooFlag = (flags & OUT_OF_ORDER_FLAG_MASK) > 0;
     final boolean readOnly = (flags & READ_ONLY_FLAG_MASK) > 0;
@@ -219,8 +214,6 @@ else if (curMode == CurMode.HLL) {
     }
     //expand byte 5: Flags
     sb.append("Byte 5: Flags:                : ").append(flagsStr).append(LS);
-    sb.append("  BIG_ENDIAN_STORAGE          : ").append(bigEndian).append(LS);
-    sb.append("  (Native Byte Order)         : ").append(nativeOrder).append(LS);
     sb.append("  READ_ONLY                   : ").append(readOnly).append(LS);
     sb.append("  EMPTY                       : ").append(empty).append(LS);
     sb.append("  COMPACT                     : ").append(compact).append(LS);
@@ -286,8 +279,7 @@ static void insertLgK(final MemorySegment wseg, final int lgK) {
   }
 
   static int extractLgArr(final MemorySegment seg) {
-    final int lgArr = seg.get(JAVA_BYTE, LG_ARR_BYTE) & 0XFF;
-    return lgArr;
+    return seg.get(JAVA_BYTE, LG_ARR_BYTE) & 0XFF;
   }
 
   static void insertLgArr(final MemorySegment wseg, final int lgArr) {

src/main/java/org/apache/datasketches/quantiles/PreambleUtil.java

@@ -28,7 +28,6 @@
 import static org.apache.datasketches.quantiles.ClassicUtil.computeRetainedItems;
 
 import java.lang.foreign.MemorySegment;
-import java.nio.ByteOrder;
 
 //@formatter:off
 
@@ -92,15 +91,12 @@ private PreambleUtil() {}
   static final int COMBINED_BUFFER            = 32; //to 39 (Only for DoublesSketch)
 
   // flag bit masks
-  static final int BIG_ENDIAN_FLAG_MASK       = 1;
+  static final int RESERVED_FLAG_MASK         = 1;
   static final int READ_ONLY_FLAG_MASK        = 2;
   static final int EMPTY_FLAG_MASK            = 4;
   static final int COMPACT_FLAG_MASK          = 8;
   static final int ORDERED_FLAG_MASK          = 16;
 
-  static final boolean NATIVE_ORDER_IS_BIG_ENDIAN  =
-      (ByteOrder.nativeOrder() == ByteOrder.BIG_ENDIAN);
-
   /**
    * Default K for about 1.7% normalized rank accuracy
    */
@@ -143,8 +139,6 @@ private static String memorySegmentToString(final MemorySegment srcSeg, final bo
     final int familyID = extractFamilyID(srcSeg);
     final String famName = idToFamily(familyID).toString();
     final int flags = extractFlags(srcSeg);
-    final boolean bigEndian = (flags & BIG_ENDIAN_FLAG_MASK) > 0;
-    final String nativeOrder = ByteOrder.nativeOrder().toString();
     final boolean readOnly  = (flags & READ_ONLY_FLAG_MASK) > 0;
     final boolean empty = (flags & EMPTY_FLAG_MASK) > 0;
     final boolean compact = (flags & COMPACT_FLAG_MASK) > 0;
@@ -166,8 +160,7 @@ private static String memorySegmentToString(final MemorySegment srcSeg, final bo
     sb.append("Byte  1: Serialization Version: ").append(serVer).append(LS);
     sb.append("Byte  2: Family               : ").append(famName).append(LS);
     sb.append("Byte  3: Flags Field          : ").append(String.format("%02o", flags)).append(LS);
-    sb.append("  BIG ENDIAN                  : ").append(bigEndian).append(LS);
-    sb.append("  (Native Byte Order)         : ").append(nativeOrder).append(LS);
+    sb.append("  RESERVED                    : ").append(LS);
     sb.append("  READ ONLY                   : ").append(readOnly).append(LS);
     sb.append("  EMPTY                       : ").append(empty).append(LS);
     sb.append("  COMPACT                     : ").append(compact).append(LS);