build: add a meson build configuration

[oscarbenjamin]

This is incomplete but I am putting it here as a talking point:

Would there be any interest in flint using meson as a build system or as an alternative experimental build system?

For downstream usecases it would be nicer if Flint used something like meson rather than autotools et al. For python-flint for example if flint used either cmake or meson then it could be incorporated as a meson subproject after flintlib/python-flint#129 is merged.

From my limited testing it seems faster to build Flint with meson+ninja rather than configure+make. Currently though the meson configuration here does not build on all platforms and it is probably compiling things differently so any timing comparison is not meaningful at this stage.

Another potential advantage of using meson could be using its (experimental) SIMD module which facilitates using runtime detection for things like AVX:
https://mesonbuild.com/Simd-module.html
Runtime detection of CPU capabilities is what would be needed for python-flint to leverage AVX because it is not possible for Python wheel tags to be more fine-grained than e.g. x86_64.

The meson configuration code shown here is incomplete because it does not handle options and things and also it is deliberately written in the simplest possible way.

To test building this with meson first install meson and ninja (e.g. pip install meson ninja) and then:

./bootstrap.sh
./configure
meson setup build
meson compile -C build

You can then install with

meson install -C build

Arguments like --prefix should be passed to meson setup.

A usable implementation of building with meson would need to work without the bootstrap/configure steps. I cannot personally turn the changes in this PR into a complete replacement of configure etc but I could turn it into a usable experimental alternative.

Another possibility is that I could submit the same files to meson's wrapdb database so that other projects can build flint using meson without the meson build files being part of the flint source tree:
https://mesonbuild.com/Wrapdb-projects.html

[oscarbenjamin]

To be clear: the build here does not work. In Linux at least I get a completed build but some functions are missing. Probably not much is needed to get to a basic working state though.

[fredrik-johansson]

Maybe as a first step this can be brought up to parity with and replace CMake. I'd rather not have three parallel build systems.

To replace our autotools setup, we'd need to support all its capabilities and someone would have to commit to maintaining the build system in the medium term.

[albinahlback]

Would there be any interest in flint using meson as a build system or as an alternative experimental build system?

If we change build systems at all, I think they should be replaced. In other words, Meson would replace both CMake and Autotools. To answer your question, yes, under the right circumstances.

For downstream usecases it would be nicer if Flint used something like meson rather than autotools et al. For python-flint for example if flint used either cmake or meson then it could be incorporated as a meson subproject after flintlib/python-flint#129 is merged.

Apart from this example, why would it be more nice? I have gotten a lot of people seemingly being afraid of Autotools, but I have never understood why. So why?

From my limited testing it seems faster to build Flint with meson+ninja rather than configure+make.

So the most important things here: What was the difference, and what C flags where you using in each case?

Another potential advantage of using meson could be using its (experimental) SIMD module which facilitates using runtime detection for things like AVX: https://mesonbuild.com/Simd-module.html Runtime detection of CPU capabilities is what would be needed for python-flint to leverage AVX because it is not possible for Python wheel tags to be more fine-grained than e.g. x86_64.

I mean, we can establish run-time detection of CPU's. It is not hard. For x86, we can simply utilize the cpuid instruction which tells us everything we need to know. For ARM we can do something similar, but instead rely on the things in config/config.guess. The harder part is to nicely incorporating it into the runtime environment, I think. Please correct me if I'm wrong.

[albinahlback]

I could maintain a Meson build system if a PR was made that such both Autotools and CMake could be removed. Hence, the functionalities of Meson should include:

• Detecting the CPU during configuration and allow for cross-compilation,
• Setting the right assembly routines and parameters,
• Checking which CFLAGS are preferred for which CPU, and then checking which are supported, and setting them accordingly,
• A nice way to display all configure options for FLINT,
• Checking for GMP internals,
• Checking if fft_small is available,
• Checking for the correct assembly options, such as if symbols should be prefixed with an (additional) underscore or not.

[oscarbenjamin]

I could maintain a Meson build system if a PR was made that such both Autotools and CMake could be removed. Hence, the functionalities of Meson should include:

All these things will be doable somehow. I will check what looks like a good way though...

I'll see if I can actually get a working build first though :)

Apart from this example, why would it be more nice? I have gotten a lot of people seemingly being afraid of Autotools, but I have never understood why. So why?

From python-flint's perspective there are two primary advantages that would come from Flint using either meson or cmake rather than autotools. The same considerations would apply to many other downstream projects as well though:

1. Portability, particularly for Windows. The fact that flint uses autotools pretty much forces downstream build systems to operate in a unix shell with GNU tools which means using e.g. MinGW on Windows rather than the native tools and the MSVC compiler. I assume that the cmake configuration in Flint now was added to support building Flint with MSVC in conda-forge. Even if on unix it would be better to have something other than shell scripts to manage dependency building.

2. Integration with other build systems. Build systems like meson support building dependencies as subprojects which means that you can have a large stack of dependencies all tracked from git or from release versions. Then effectively make rebuilds your whole stack including all dependencies but keeping track of incremental rebuilds rather than starting from scratch. In the case of meson this works if the dependency uses either meson or cmake. There is some support for autotools-style dependencies but it is discouraged and after playing around with it I think it can only work in very simple cases and generally does not work well.

Projects that use autotools don't work properly as subprojects because they are opaque from the outside and cannot keep track automatically of when it is necessary to rerun configure or make clean etc. Relatedly I recently needed to bisect a bug in CPython but each time git checked out a new commit I couldn't just run make to get a clean rebuild because too many other things were changing. A usable subproject needs to be able to build cleanly after any changes like git checkout v3.0.1 etc but the only way to ensure a clean build with autotools is to wipe everything and start from scratch.

With meson a rebuild can keep track of when there are changes to the meson.build files or anything else or when built files should be removed, when dependencies have been updated to a new version etc. This is like if make could understand when the Makefile or configure scripts were changed and could know whether anything set or checked during configure changed but could know intelligently how to rebuild without just doing make clean or make -B.

Not all projects can use e.g. meson as their primary build system because build system tooling creates a bootstrapping problem. For example GMP and MPFR are dependencies of gcc itself. Likewise Python is a dependency of meson so it would be awkward for CPython to use meson as its own build system. A solution to that from python-flint's perspective is that we can add external wrapdb files for GMP and MPFR (other people want this as well mesonbuild/wrapdb#97).

In the case of Flint there is no bootstrapping issue though and the ideal solution for python-flint would be if Flint used either meson or cmake. If Flint used meson then it could also track GMP and MPFR as subprojects using the wrap file mechanism. In fact I think if Flint did that then it would not be necessary for python-flint or other downstream projects to do it explicitly because GMP and MPFR could just become recursive subprojects from python-flint's perspective (I haven't tested this).

[oscarbenjamin]

I'll see if I can actually get a working build first though :)

Currently I get:

    from .pyflint import *
  File "src/flint/pyflint.pyx", line 1, in init flint.pyflint
    """
ImportError: ./src/flint/types/fmpz.cpython-312-x86_64-linux-gnu.so:
undefined symbol: partitions_fmpz_fmpz

I guess some symbols are not being exported correctly.

[albinahlback]

Fair points. Let me know if you need help interpreting configure.ac and friends.

[oscarbenjamin]

I'll see if I can actually get a working build first though :)

I guess some symbols are not being exported correctly.

Actually this does give a working build I think at least on Linux. I was just linking python-flint against the wrong libflint.so.

The fix is that you have to actually install libflint.so after building it!

diff --git a/meson.build b/meson.build
index ae7cd95a1..1169a801b 100644
--- a/meson.build
+++ b/meson.build
@@ -28,6 +28,7 @@ install_headers(headers_all, subdir: 'flint')
 libflint = library('flint', c_files_all,
   dependencies: flint_deps,
   include_directories: include_directories('src'),
+  install: true
 )
 
 pkg_mod = import('pkgconfig')

[oscarbenjamin]

From my limited testing it seems faster to build Flint with meson+ninja rather than configure+make.

So the most important things here: What was the difference, and what C flags where you using in each case?

I'm not explicitly setting any flags so just the default ones are being used.

This is how the meson build compiles the C files by default:

cc -Ilibflint.so.p -I. -I.. -Isrc -I../src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O0 -g -fPIC -MD -MQ libflint.so.p/src_acb_dirichlet_turing_method_bound.c.o -MF libflint.so.p/src_acb_dirichlet_turing_method_bound.c.o.d -o libflint.so.p/src_acb_dirichlet_turing_method_bound.c.o -c ../src/acb_dirichlet/turing_method_bound.c

This is how make compiles them

cc -fPIC -DPIC -march=ivybridge -Wno-stringop-overflow -Wno-stringop-overread -Wall -Werror=implicit-function-declaration -std=c11 -pedantic -O3 -g -funroll-loops -I./src -DBUILDING_FLINT -DFLINT_NOSTDIO -DFLINT_NOSTDARG -c src/ulong_extras/is_square.c -o build/ulong_extras/is_square.lo -MMD -MP -MF build/ulong_extras/is_square.lo.d

The main difference for build time is -O0 vs -O3. Apparently meson's default is -O0 but we can add a different default. I've set -O3 as the default by making release the default for buildtype.

After fixing it to use -O3 I get similar times from both for a full rebuild. The only advantage of meson+ninja in that case is that you don't need to set -j N+1 because it is done automatically by ninja.

For a no-op build i.e. running make when there is nothing to do meson compile -C build takes about 1 second on this slow machine vs 5 seconds for make. If I touch src/gr_mpoly.h then both meson and it's 6 seconds vs 7 seconds to rebuild. For touching a single .c file its 3 seconds for meson vs 6 seconds for make. I'm not using ccache here which would affect these timings.

Probably basic build speed like this is not a significant factor here. What is more important is that it is a more reproducible build each time and that the reproducibility does not come at the cost of slower builds. You would get a big gain in speed in situations where you would otherwise need to use make clean for example because that shouldn't be needed with meson.

Also you can configure multiple build directories differently like:

meson setup release-dir -Dbuildtype=release
meson setup debug-dir -Dbuildtype=debug

Now you can have incremental rebuilds in each differently configured build directory without them clobbering each other:

meson compile -C release-dir
meson compile -C debug-dir

With autotools after ./configure --foo=bar you would have to rebuild everything even if you have configured with the exact same options as before and nothing has actually changed.

[albinahlback]

Yeah, I really like the rebuilding times you presented.

Could you just present the exact building times you get from Meson vs Make for building the whole library (not including the configuration process)? You can go with CFLAGS="-O0" for simplicity. I just want to make sure that building with Meson for one-time builds are not 30% slower or something.

[albinahlback]

And I just want to make it clear that I will not merge something that does not superset-ish what Autotools currently does. I know this is somewhat of a big task to take on, just so that you are aware of my requirements. Make sure that you exactly know what the Autotools stuff is doing.

And, once more, I am here to help you with the interpretation of the Autoconf stuff.

[oscarbenjamin]

These are timings from a fresh git clone using this PR as it stands. The timings are on a relatively slow Linux machine with 4 cores and build using Ubuntu's stock gcc in both cases. Both builds give a usable libflint but only because this machine does not have whatever is needed for fft_small. In both cases I am using the default compiler flags which means -O3. For the timings I am running them in separate directories for meson vs make and they are not sharing anything in terms of the build.

This is the timing for a full rebuild in a fresh clone with make:

$  git clone https://github.com/oscarbenjamin/flint.git
$  cd flint/
$  git checkout pr_meson
$  ./bootstrap.sh 
$  ./configure 
$  time make -j5 > make-output.txt
...
real	5m20.621s
user	18m48.493s
sys	2m8.120s

This is the full rebuild timing with meson+ninja:

$  git clone https://github.com/oscarbenjamin/flint.git
$  cd flint
$  git checkout pr_meson
$  ./bootstrap.sh 
$  ./configure 
$  meson setup build
$  time meson compile -C build
...
real	5m43.528s
user	17m48.237s
sys	2m24.565s

In this run meson is 7% slower than make but I've run these both a few times today and seen a lot of variability. Many times meson seemed faster but it all takes a bit too long for me to just run it many times and average. For the quoted timings I was careful not to use the computer while the build ran although the screen did go to sleep briefly during the meson build so I woke it up.

These are warm cache timings for a no-op build like running make after you have just run make:

$ time meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
ninja: no work to do.

real	0m1.329s
user	0m1.035s
sys	0m0.355s
$ time make
make: Nothing to be done for 'all'.

real	0m6.010s
user	0m5.764s
sys	0m0.242s

So in this case meson takes 1.3 seconds to see that nothing needs to be done and make takes 6 seconds.

These are timings after touch src/gr_mpoly.h:

$ touch src/gr_mpoly.h 
$ time make
  CC  gr/mpoly.c
  CC  gr_mpoly/add.c
  CC  gr_mpoly/combine_like_terms.c
  CC  gr_mpoly/equal.c
  CC  gr_mpoly/fit_bits.c
  CC  gr_mpoly/fit_length.c
  CC  gr_mpoly/fit_length_fit_bits.c
  CC  gr_mpoly/fit_length_reset_bits.c
  CC  gr_mpoly/gen.c
  CC  gr_mpoly/get_coeff_scalar_fmpz.c
  CC  gr_mpoly/get_coeff_scalar_ui.c
  CC  gr_mpoly/init.c
  CC  gr_mpoly/inlines.c
  CC  gr_mpoly/is_canonical.c
  CC  gr_mpoly/mul.c
  CC  gr_mpoly/mul_johnson.c
  CC  gr_mpoly/mul_monomial.c
  CC  gr_mpoly/mul_scalar.c
  CC  gr_mpoly/neg.c
  CC  gr_mpoly/push_term.c
  CC  gr_mpoly/randtest_bits.c
  CC  gr_mpoly/randtest_bound.c
  CC  gr_mpoly/set.c
  CC  gr_mpoly/set_coeff_scalar_fmpz.c
  CC  gr_mpoly/set_coeff_scalar_ui.c
  CC  gr_mpoly/set_scalar.c
  CC  gr_mpoly/sort_terms.c
  CC  gr_mpoly/sub.c
  CC  gr_mpoly/write.c
Building libflint.so.20.0.0

real	0m10.885s
user	0m9.677s
sys	0m1.055s
$ touch src/gr_mpoly.h 
$ time meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
[30/30] Linking target libflint.so

real	0m5.194s
user	0m8.241s
sys	0m1.769s

It looks like they both built 30 targets and meson did it in 5 seconds vs 10 seconds for make.

These are timings for touch src/fmpz/fmpz.c:

$ touch src/fmpz/fmpz.c 
$ time make
  CC  fmpz/fmpz.c
Building libflint.so.20.0.0

real	0m6.411s
user	0m5.942s
sys	0m0.467s
$ touch src/fmpz/fmpz.c
$ time meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
[2/2] Linking target libflint.so

real	0m3.550s
user	0m2.434s
sys	0m1.020s

Both built 2 targets (fmpz.o and libflint.so) and it took 3 seconds for meson vs 6 seconds for make.

[oscarbenjamin]

These are timings for a full build in a fresh clone but on a faster Mac laptop that has a 12 core M3 CPU. On this computer I have a few local changes meaning that the meson build can get to the end but it fails with a link error, due to some assembly preprocessor defines not being correct. I think that the timings are approximately comparable any way though because linking is only the last step.

This is the timing with make:

$  git clone https://github.com/oscarbenjamin/flint.git
$  cd flint/
$  git checkout pr_meson
$  ./bootstrap.sh 
$  ./configure 
$ time make -j13 > make-output.txt
...
make -j13 > make-output.txt  213.18s user 102.57s system 841% cpu 37.527 total

This is the timing with meson:

$  git clone https://github.com/oscarbenjamin/flint.git
$  cd flint/
$  git checkout pr_meson
$ git apply ../mac-fixes.diff  # some local fixes...
$  ./bootstrap.sh 
$  ./configure 
$ meson setup build
$ time meson compile -C build
...
meson compile -C build  189.25s user 86.31s system 710% cpu 38.780 total

In this case the meson build is about 11% faster than make but the final link step did not complete:

Undefined symbols for architecture arm64:
  "_flint_mpn_mul_10n", referenced from:
      _flint_mpn_mul_func_n_tab in src_mpn_extras_mul_basecase.c.o
  "_flint_mpn_mul_11n", referenced from:
      _flint_mpn_mul_func_n_tab in src_mpn_extras_mul_basecase.c.o
...

The problem is something to with this preprocessor define:

flint/src/mpn_extras/mul_basecase.c

Lines 275 to 291 in 940102e

	#elif FLINT_HAVE_ASSEMBLY_armv8
	mp_limb_t flint_mpn_mul_1n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_2n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_3n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_4n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_5n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_6n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_7n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_8n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_9n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_10n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_11n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_12n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_13n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_14n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	mp_limb_t flint_mpn_mul_15n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
	#else

The diff applied to make that get as far as it did was:

diff --git a/meson.build b/meson.build
index 6a73b0dbe..31734a767 100644
--- a/meson.build
+++ b/meson.build
@@ -1,7 +1,10 @@
 project('FLINT', 'c',
   version: '3.1.0',
   license: 'LGPL3+',
-  default_options: ['buildtype=release'],
+  default_options: [
+    'buildtype=release',
+    'c_std=c11',
+  ],
 )

 cc = meson.get_compiler('c')
@@ -12,6 +15,14 @@ mpfr_dep = dependency('mpfr', version: '>= 4.1.0')

 flint_deps = [gmp_dep, mpfr_dep, m_dep]

+add_project_arguments(
+  '-march=armv8-a',
+  '-DBUILDING_FLINT',
+  '-DFLINT_NOSTDIO',
+  '-DFLINT_NOSTDARG',
+  '-Werror=implicit-function-declaration',
+  language: 'c')
+
 subdir('src')

 headers_all = []
diff --git a/src/meson.build b/src/meson.build
index 66c0451ab..2a9d0d4fd 100644
--- a/src/meson.build
+++ b/src/meson.build
@@ -1947,15 +1947,15 @@ c_files_all_subdir = [
   'fft/mul_truncate_sqrt2.c',
   'fft/negmod_2expp1.c',
   'fft/normmod_2expp1.c',
-  #'fft_small/default_ctx.c',
-  #'fft_small/fmpz_poly_mul.c',
-  #'fft_small/mpn_helpers.c',
-  #'fft_small/mpn_mul.c',
-  #'fft_small/mulmod_statisfies_bounds.c',
-  #'fft_small/nmod_poly_mul.c',
-  #'fft_small/sd_fft.c',
-  #'fft_small/sd_fft_ctx.c',
-  #'fft_small/sd_ifft.c',
+  'fft_small/default_ctx.c',
+  'fft_small/fmpz_poly_mul.c',
+  'fft_small/mpn_helpers.c',
+  'fft_small/mpn_mul.c',
+  'fft_small/mulmod_statisfies_bounds.c',
+  'fft_small/nmod_poly_mul.c',
+  'fft_small/sd_fft.c',
+  'fft_small/sd_fft_ctx.c',
+  'fft_small/sd_ifft.c',
   'fft/split_bits.c',
   'fmpq/add.c',
   'fmpq/addmul.c',
@@ -5275,7 +5275,6 @@ c_files_all_subdir = [
   'qsieve/poly.c',
   'qsieve/primes_init.c',
   'qsieve/square_root.c',
-  'test/main.c',
   'thread_pool/clear.c',
   'thread_pool/distribute_work.c',
   'thread_pool/find_work.c',

[albinahlback]

Timings are looking very good. I think we can afford a small drop in a one-time build while we gain a lot in rebuilds! Thank you for looking into this.

[oscarbenjamin]

to help you with the interpretation of the Autoconf stuff.

What would be great is if you could just summarise from a high level what it is that configure is doing to modify the sources and also how that affects the generated Makefile. Right now I am (inadvisedly?) trying to make the meson build work after configure has run so that I don't need to replicate its effect on the sources yet. However it seems that configure has encoded some information into the Makefile in relation to fft_small that I don't immediately understand to be able to replicate in meson.

[albinahlback]

Regarding the wrong assembly linking: Is this with Autotools or is this with Meson?

[oscarbenjamin]

Regarding the wrong assembly linking: Is this with Autotools or is this with Meson?

With meson. It works fine with autotools.

[albinahlback]

Oh, I see. If it is with Meson then it should be expected (unless you have gotten that far already).

[oscarbenjamin]

unless you have gotten that far already

I definitely haven't :)

[albinahlback]

We (= me = Albin) use M4 to make writing assembly a little bit easier. One such things is making the assembler write the correct symbol. MacOS typically prefix their symbols with an underscore, which we test during the configuration.

[albinahlback]

So if you write int foo(void) in C, the generated symbol on some systems will actually be _foo and not foo.

[albinahlback]

There are a couple of other things that is needed to be able to write assembly, but I will try to write a list of what the whole Autotools process is doing.

[albinahlback]

Note that I will herein only describe the process for Autotools. Any local
functions used in configure.ac can be found in acinclude.m4. Remember that
stuff in configure.ac and acinclude.m4 is taken from GMP, so their copyright
claim should be preserved.

File structure

configure.ac

Contains the main script for the configuration.

acinclude.m4

Contains functions that configure.ac may use.

bootstrap.sh

Generates configure from configure.ac.

config/config.guess and config/configfsf.guess

In an Autotools project, one can push --host=aarch64-pc-linux-gnu to
configure to compile for 64-bit Arm architecture running Linux. In case you do
not specify what host is going to run the library, we run config/config.guess
to guess what host is going to run the library. In this script, we first run
config/configfsf.guess to get a guess, and then we get more precise in
config/config.guess to (try to) get the exact architecture.

config/config.sub and config/configfsf.sub

Validates system triplets.

Makefile.in

Basically Makefile but with some parts that has to be filled in during
configuration.

Layout in configure.ac

I have some titles in configure.ac which I will follow here.

preamble

We start by initializing the project, setting copyrights as well as version
number from the VERSION file. We keep a separate VERSION file to be able to
very easily change the version number, without bloating the Git-history on
configure.ac. We also set the SO-versions (manually?).

initialize libtool

We check if CFLAGS are set. Then we initialize libtool, for which we push the
option to disable static by default. If CFLAGS was not set, we remove any
CFLAGS that libtool may want to push.

build system

We do not allow specifying --target. We could get system triplets here, but
that is already invoked when we initialized libtool.

configure headers

We set which headers we are about to configure.

architecture specifics

We set the default CFLAGS. Then we check if we can recognize any architectures
we especially develop for, which is all x86 architectures supporting the ADX
instruction set as well as all Arm v8. We push -march options as well as what
assembly path and parameter path we should use for the host. Note that we do not
go for assembly when system is Windoze since Windoze for some reason chose to go
with their own ABI, which is incompatible with every other x86 OS.

Now, if we know by now that the processor supports AVX2 or NEON, we enable
fft_small module to reduce the time configuring. Otherwise we will test this
later.

features

Here we set the options available and the helpstrings. We check if we got an
okay value. Remember to preserve the defaults here.

packages

We set which packages one can include. Mandatory ones are GMP and MPFR. We allow
specifying include directory and library directory individually. We also check
that one does not specify both --with-PKG and --with-PKG-lib with
--with-PKG-include.

programs

We check what to use for mkdir -p, which is used in Make. Probably doesn't
have to be checked in Meson.

environment variables

We allow specifying different C flags to the test programs.

check programs and system

Check what keyword is used for defining inlined functions.

We check that the compiler supports -c and -o simultaneously (I think only
very old compilers did not allow this, so you can skip this).

We check for endianness.

We check which OS is being targeted, and set the filenames for the dynamic
library accordingly.

We set the static library name.

We also check how to push an option to the linker. This is usually -Wl. You
may or may not have to check for this, but it is required to test FLINT with the
correct library, so that it tests against the compiled library, and not some
other version that may be installed system-wide or whatever.

We check if the system is strongly-ordered. For now, it suffice to set all x86
systems to this. Some algorithms depend on this, which is why it is important.

check GMP

We check that the GMP version is greater or equal to 6.2.1. We check if we have
long long limbs, and set gmpcompat.h accordingly.

check MPFR

We check that MPFR >= 4.1.0.

check ABI

Some systems support multiple ABIs. Here, we check if ABI was specified. If it
was, then we set the ABI (duh). Else, we check how big GMP_LIMB_BITS is (can
only be 32 or 64).

check headers

We check for some headers. Some are optional, others are mandatory. Some are
mandatory only if some option was specified (such as --with-pthread requires
pthread.h).

check libraries

We save all CFLAGS as they may screw up with detecting if functions actually
exists or not. Then we make sure that the math library, GMP and MPFR can be
found. If user wants GMP internals (which is the default), we check for some GMP
internal functions. We always require some MPFR internals, so that has to be
checked. If user wants pthread, we check how to specify this. We also check
for BLAS and NTL if these are wanted.

Finally, we have an option for checking MPFR, which is maintainer level only. We
want to check for Nemo in a CI, but the MPFR that is built with Julia contains
some sanitiser settings, which gives an error when checking for such functions.
Hence, for this reason, we have this maintainer level option here. I do not know
if you will need it.

check settings and environment

The position independent flag that has to be pushed in order to generate a
shared library gets checked with libtool. We substitute this into the Makefile.

We check for cpu_set_t, which is used for threading stuff.

We check that alloca works as intended. You can skip this check if you want as
I've never seen any modern system not having it.

We check for aligned_alloc and _aligned_malloc. This is very important
downstream as old MacOS do not have either of these functions, although the
compiler state that it is C11 compatible.

CFLAGS

We check if compiler is Clang. If it is, we push an error flag for unknown
options so that it generates an error if we give some faulty option. Otherwise
the flags may be available, but still pushed to Make.

We check if we are going to generate code coverage. If so, we push adequate
flags.

If we are generating debugging information, which is the default, we check that
we actually can do it.

If ABI was set from user, we check that we can compile with that ABI.

We check if user wants AVX2 and AVX512, and set flags accordingly.

If the user did not specify CFLAGS, we check all flags we would like the
compiler to accepts. We also push -funroll-loops to some modules, so make sure
to test if this flag is available. If unrolling is not available, then we do not
push this to Make.

However, if the user specified the flags, make sure that all the flags are
accepted.

If the user specified TESTCFLAGS, we simply push these flags along with the
mandatory flags (such as -g for debugging) to Make. Otherwise, if the user did
not specify them, we use the same flags as for the rest of the repository.

We also set the C preprocessor flags to include src/ for the headers.

fft_small module

We check if the fft_small module is available. If x86, check for immintrin.h
and AVX2, for which the latter will be cached for most users. Else if Arm,
check for arm_neon.h and NEON instructions. Else it is not available. The
answer should be pushed to the Makefile.

Assembly

And now to the good parts!

If assembly is not wanted (then Albin is very disappointed), then we push that
we do not want assembly to the Makefile.

For the rest of this section, we assume that assembly is wanted.

First, we check that we have a path for the assembly code and that the ABI is
64-bit. If not, we send out a warning that we do not have any assembly code
available. Note that Windoze does not have a path for assembly since their ABI
is non-standard.

Now, we initialise the M4 configuration file used for the assembly routines,
called config.m4. We check that the M4 preprocessor is available, that nm
is available, and then we check what syntax to use. Some syntax is specific to
the architecture. Some syntax is architecture specific. We include the correct
M4 file(s) in config.m4 and set a C preprocessor constant according to what
assembly we are using (such as FLINT_HAVE_ASSEMBLY_armv8).

Finally, we push the assembler path to the Makefile.

parameters

We set the correct parameter path for the architecture.

substitutions and definitions

We basically just make substitutions, definitions and linking based on options
and found values.

epilog

We configure Makefile, flint.pc and flint.h according to the values we've
gotten.

[albinahlback]

Things to consider when replacing Makefile

1. We want to be able to compile with specific CFLAGS for certain modules.
   Unrolling of loops should be made for those listed at around line 100.

2. We want to be able to compile assembly. This includes generating the actual
   assembly file, which includes both PIC and non-PIC, as well as the actual
   object files.

3. As stated in the previous post, we want to be able to link test files with
   the generated library, not some installed library.

4. Source files should (preferably) be added automatically, as well as their
   dependencies. But I'm all ears if they should be added manually. This was a
   requirement from Bill Hart when I first started mixing with the build system,
   but I'm open to have it some other way if it reduces build times or whatever.
   Dependencies, however, should be automated.

5. Would be very good if one could list variable without altering the files. For
   instance, with make print-VAR we print variable VAR. This is very nice
   for debugging the Makefile.

6. I want nice printing of the building process. I do not want gcc -flag1 -flag2 ... -flag10000 -c foo.c -o foo.o as that bloats the screen. Although
   I think some information is nice to have, try to avoid long lines here.

7. I want it to state when it is building the actual library in a nice way, so
   that you know when it is about to be done. I suppose this is built into
   Meson, so I think you should not have to worry about this.

8. On some systems, the line width for the terminal is limited to some small
   number of characters (perhaps 1000 or something along that magnitude), which
   becomes a problem when FLINT has a lot of files. For this, we need to build
   merged object files. I do not know if you have to deal with this problem.
   This will become apparent in the CI.

9. Be able to create tests, tunes (?), profilers and example executables. Note
   that this includes NTL, which requires C++.

10. Being able to run tests.

11. Being able to run tests only for a set of modules (for the current module we
    do this via make check MOD="mpn_extras fmpz").

12. Being able to run tests for a set of modules with multiple threads. That is, if I want to test
    mpn_extras and fmpz with N threads, I want it to partition this test
    into at least N jobs running the tests in parallel (perhaps one job takes
    care of the test for fmpz_abs, fmpz_addmul, etc. while another job takes
    care of flint_mpn_mulhigh_n etc.). In other words, all tests should be
    executed in parallel and no test should be duplicated.

13. Saying when the test has completed running with a zero return value, so that
    one knows that they are completed (so that it did not stall, or did not
    abort or something).

14. Being able to compile profilers only for a set of modules.

15. Check examples. See make checkexamples.

16. Shortcut for running gdb on a test. Currently, this is done via make debug MOD=XXX ARGS=YYY.

17. Being able to run Valgrind, both on a set of modules, or on the whole
    library.

18. Being able to capture the code coverage.

19. Clean the build (opt-in for cleaning everything configurable).

20. Installing and uninstalling (equally important to installing).

21. Create a tarball. See make dist.

[albinahlback]

I don't think I missed anything, but of course I left some (more or less obvious) things unsaid. Let me know if there is something that needs to be explained in more detail.

[oscarbenjamin]

Things to consider when replacing Makefile

1. We want to be able to compile with specific CFLAGS for certain modules.
   Unrolling of loops should be made for those listed at around line 100.

In meson it is not possible to simply set different compiler flags on a per file basis but this kind of case where a few files use different flags can be handled by building them into an intermediate static library that is then linked into the end product:
mesonbuild/meson#1367

2. We want to be able to compile assembly. This includes generating the actual
   assembly file, which includes both PIC and non-PIC, as well as the actual
   object files.

This is all doable. Some kinds of assembly have explicit support in meson but in general you can use custom targets to run any commands that need to be run if needed.

3. As stated in the previous post, we want to be able to link test files with
   the generated library, not some installed library.

With meson this is taken care of automatically by the build configuration. Something like:

libflint = library('flint', c_files_all,
  dependencies: flint_deps,
  include_directories: include_directories('src'),
  install: true
)

test_exe = executable('test', 'test.c', link_with: libflint)

It doesn't use environment variables etc except when configuring.

4. Source files should (preferably) be added automatically, as well as their
   dependencies. But I'm all ears if they should be added manually. This was a
   requirement from Bill Hart when I first started mixing with the build system,
   but I'm open to have it some other way if it reduces build times or whatever.
   Dependencies, however, should be automated.

Adding source files automatically is disallowed in meson. There are workarounds but the intention is that files are listed explicitly and this is what enables it to be fast and reproducible. Dependencies among .c and .h files are determined automatically by ninja but the files themselves must be listed explicitly for this to work. We can easily make a Python script that could update this for all but one or two flint modules (I imagine fft_small or mpn_extras would be exceptions).

5. Would be very good if one could list variable without altering the files. For
   instance, with make print-VAR we print variable VAR. This is very nice
   for debugging the Makefile.

I won't show the full output but:

$ meson configure build-dir
Core properties:
  Source dir /Users/enojb/work/dev/flint/tmp/flint
  Build dir  /Users/enojb/work/dev/flint/tmp/flint/build

Main project options:

  Core options                             Current Value                              Possible Values                            Description
  --------------                           -------------                              ---------------                            -----------
  auto_features                            auto                                       [enabled, disabled, auto]                  Override value of all 'auto' features
  backend                                  ninja                                      [ninja, vs, vs2010, vs2012, vs2013,        Backend to use
                                                                                       vs2015, vs2017, vs2019, vs2022, xcode,
                                                                                       none]
  buildtype                                release                                    [plain, debug, debugoptimized, release,    Build type to use
                                                                                       minsize, custom]
...

Those are built-in options but Flint-specific custom options would get listed there as well.

6. I want nice printing of the building process. I do not want gcc -flag1 -flag2 ... -flag10000 -c foo.c -o foo.o as that bloats the screen. Although
   I think some information is nice to have, try to avoid long lines here.

By default it looks like what I showed above:

$ meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
[30/30] Linking target libflint.so

When you actually run that it shows a running counter like [10/30] compiling fmpz.c. If you pass -v then it will output the actual compiler commands one by one.

7. I want it to state when it is building the actual library in a nice way, so
   that you know when it is about to be done. I suppose this is built into
   Meson, so I think you should not have to worry about this.

It shows

[30/30] Linking target libflint.so

at the final link step. For a full build it's something like [5120/5120] Linking ....

8. On some systems, the line width for the terminal is limited to some small
   number of characters (perhaps 1000 or something along that magnitude), which
   becomes a problem when FLINT has a lot of files. For this, we need to build
   merged object files. I do not know if you have to deal with this problem.
   This will become apparent in the CI.

This was a meson bug that was fixed specifically because someone was using meson to build Flint:
mesonbuild/meson#7212

9. Be able to create tests, tunes (?), profilers and example executables. Note
   that this includes NTL, which requires C++.
10. Being able to run tests.
    ...

These things will all be possible somehow. It might be that some things that are handled in the Makefile would need to be a separate Python script or something.

19. Clean the build (opt-in for cleaning everything configurable).

The simple way to do this is just rm -r build-dir. You don't actually need to do that though because you can instead just make a new build directory and use that:

$ meson setup new-build-dir
$ meson configure new-build-dir -Dfoo=bar
$ meson compile -C new-build-dir

Note that meson does everything out-of-tree. It only creates/modifies files etc inside the build directory and leaves the source tree itself untouched.

20. Installing and uninstalling (equally important to installing).

This is just meson uninstall. You need to still have the build directory that was used for installing though and it does not restore any files that were overwritten during install.

[albinahlback]

In meson it is not possible to simply set different compiler flags on a per file basis but this kind of case where a few files use different flags can be handled by building them into an intermediate static library that is then linked into the end product: mesonbuild/meson#1367

I don't like this at all. Seems like a very basic thing that high performance libraries want to utilize. Hopefully we can do it gracefully.

Adding source files automatically is disallowed in meson. There are workarounds but the intention is that files are listed explicitly and this is what enables it to be fast and reproducible. Dependencies among .c and .h files are determined automatically by ninja but the files themselves must be listed explicitly for this to work. We can easily make a Python script that could update this for all but one or two flint modules (I imagine fft_small or mpn_extras would be exceptions).

Yeah, don't be too worried about this. I'm down for only doing it manually.

8. On some systems, the line width for the terminal is limited to some small
   number of characters (perhaps 1000 or something along that magnitude), which
   becomes a problem when FLINT has a lot of files. For this, we need to build
   merged object files. I do not know if you have to deal with this problem.
   This will become apparent in the CI.

This was a meson bug that was fixed specifically because someone was using meson to build Flint: mesonbuild/meson#7212

Haha, that's funny! Nice that is was being fixed.

9. Be able to create tests, tunes (?), profilers and example executables. Note
   that this includes NTL, which requires C++.
10. Being able to run tests.
    ...

These things will all be possible somehow. It might be that some things that are handled in the Makefile would need to be a separate Python script or something.

Alright. I just want to make it clear that I would like to avoid users having to call something else than meson in order to perform the tests. Additionally, I would really prefer to keep everything in the Meson ecosystem, just like we have contained (almost) everything into Makefile.

Although I have my worries, I think this all sounds good. I'm looking forward to see the progress for this.

[albinahlback]

And thank you for looking into this!

[oscarbenjamin]

• Being able to run tests.
• Being able to run tests only for a set of modules (for the current module we
  do this via make check MOD="mpn_extras fmpz").
• Being able to run tests for a set of modules with multiple threads. That is, if I want to test
  mpn_extras and fmpz with N threads, I want it to partition this test
  into at least N jobs running the tests in parallel (perhaps one job takes
  care of the test for fmpz_abs, fmpz_addmul, etc. while another job takes
  care of flint_mpn_mulhigh_n etc.). In other words, all tests should be
  executed in parallel and no test should be duplicated.
• Saying when the test has completed running with a zero return value, so that
  one knows that they are completed (so that it did not stall, or did not
  abort or something).

I think all of these things are handled natively by meson's test command:
https://mesonbuild.com/Unit-tests.html

• meson test runs the tests.
• By default tests are run in parallel but I think it is separate processes rather than separate threads.
• You can control how many processes with MESON_TESTTHREADS.
• Tests to run can be specified by test name or in groups using --suite.
• I'm sure it will set the return code correctly.
• Tests can be repeated and timeouts can be set.
• gdb, valgrind, coverage etc supported.

Testing won't work with the meson.build here yet though because I haven't added any configuration to build the tests.

[oscarbenjamin]

There is a meson bug about test executables being built by default:
mesonbuild/meson#2518

A workaround is to have a configure option for building tests:

meson setup build -Dbuild_tests=true
meson test

It should be possible to make it so that meson test errors if build_tests has not been configured.