Using OpenJDK JMH and JOL

[ShishkinDmitriy]

Hello.
After breaf discovering of the project and reading the article MethodologyRuntimeBenchmark I did found that benchmark uses System::currentTimeMillis and test will be evaluated 25 times (please, correct me if I did wrong conclusion). If I correct then I have a question:

• how you can guarantee that 25 iterations without "warm up" does not contains internal JVM processes like optimizations, working in interpretation mode and others?

Matrix benchmarking is reduced to benchmarking some java method invocation.
The only project for java benchmarking as I know is JMH. It is written by performance team of Oracle and aims to reduce any internal JVM influences on measurements. JMH uses well optimized code instrumentation and is used for internal measurements by performance team (including measurements of System::currentTimeMillis itself).
Additionally, for memory footprint there is another project JOL.
I think that benchmark that using JMH and JOL under the hood will produce more precise results of measurements.

JMH examples;
JOL examples;

I hope my question will be useful for further progress of you project.
Best regards,
Dmitriy

[lessthanoptimal]

Hi Dmitriy,

The documentation is a little bit out of date but it describes the general spirit. It actually uses nanoTime(). A single trial of the benchmark for one operation in one library works by increasing the number of iterations until a minimum amount of time has elapsed. For example, let's say the target time is 1 second, it might try 100 iteration and that only takes 0.1 seconds. It then tries 1000 iterations and it might get 0.6 seconds to complete. That guess and check process continues until it exceeds the target time. A warm up trial is baked into this process when it is a micro benchmark. Because the duration of a trial is guaranteed to have a minimum duration any imprecision of the timing mechanism is hopefully very small relatively. When the trial takes a long time (say 20 minutes) there is no warm up time.

Thanks for pointing out JMH and JOL. I'll take a look at them. This project has been semi-dormant for a bit. Just woke it up and testing/fixing everything after the major refactor to gradle + maven central.

[lessthanoptimal]

Since my last reply I've started using JMH in other projects. After becoming more familiar with JMH there is no plan to transition JMatBench over to using it. JMH would add complexity and give measurably less accurate results on very fast ops. JMH and this benchmark address many of the same issues. The primary difference is how the functions are called. JMH forces you to wrap every call in another function while JMatBench doesn't. See this page.

What JMatBench does is require you to call a for loop the ends after the specified number of iterations. This eliminates the overhead of a function call at the cost of being a little bit more difficult write. That overhead is measurable for very fast operations. The VM will sometimes inline function but doesn't always. On more resource constrained platforms it will often not.

The other issue I have with JMH is that more code is required to extract the results. JMH is great as a general purpose benchmark but the code contained in this benchmark was written specially for its use case.