mesonpy embeds random path into .o files for debug build

[bmwiedemann]

While working on reproducible builds for openSUSE (sponsored by the NLnet NGI0 fund), I found that various python packages (python-pandas, python-scikit-learn, python-scipy, python-contourpy, python-gobject) have variations in every build.

I am pretty certain this comes from

meson-python/mesonpy/__init__.py

Line 965 in 04e0066

build_dir = ctx.enter_context(tempfile.TemporaryDirectory(prefix='.mesonpy-', dir=source_dir))

but don't know how to best make this deterministic.

Is there a variant of tempfile.TemporaryDirectory that does not add randomness? I tried with pathlib.Path(os.path.join(source_dir, '.mesonpy')) but that lacks the automatic cleanup.

Here is a typical diff:

--- old /usr/lib/debug/.dwz/python-pandas-2.2.2-1.1.x86_64 (objdump)
+++ new /usr/lib/debug/.dwz/python-pandas-2.2.2-1.1.x86_64 (objdump)
@@ -14923,7 +14923,7 @@
  0010 01010101 081e3c64 777a3e00 2f757372  ......<dwz>./usr
  0020 2f737263 2f646562 75672f70 616e6461  /src/debug/panda
  0030 732d322e 322e322f 2e6d6573 6f6e7079  s-2.2.2/.mesonpy
- 0040 2d6e7777 6b673134 74002f75 73722f6c  -nwwkg14t./usr/l
+ 0040 2d73615f 6761376e 33002f75 73722f6c  -sa_ga7n3./usr/l
  0050 69623634 2f676363 2f783836 5f36342d  ib64/gcc/x86_64-
  0060 73757365 2d6c696e 75782f31 342f696e  suse-linux/14/in
  0070 636c7564 65002f75 73722f69 6e636c75  clude./usr/inclu
@@ -14965,7 +14965,7 @@
  02b0 2f6e756d 70792f64 61746574 696d6500  /numpy/datetime.
  02c0 2f757372 2f737263 2f646562 75672f70  /usr/src/debug/p
  02d0 616e6461 732d322e 322e322f 2e6d6573  andas-2.2.2/.mes
- 02e0 6f6e7079 2d757868 63736133 66002f75  onpy-uxhcsa3f./u
+ 02e0 6f6e7079 2d713166 36397136 61002f75  onpy-q1f69q6a./u
  02f0 73722f69 6e636c75 64652f70 7974686f  sr/include/pytho
  0300 6e332e31 31002f75 73722f69 6e636c75  n3.11./usr/inclu

[eli-schwartz]

You are "holding it wrong":

https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/distutils-r1.eclass#n1297

[thesamesam]

The line above is:

meson-python/mesonpy/__init__.py

Line 964 in 04e0066

if build_dir is None:

... so set build dir if you don't want one chosen for you.

[bmwiedemann]

So far our build-description just says %pyproject_wheel so not sure where to set builddir without patching every individual package. Maybe in our python-rpm-macros.

A more deterministic default would still be appreciated. e.g. scipy is listed on https://github.com/mesonbuild/meson-python/blob/main/docs/projects-using-meson-python.rst as "known to be adhering to best practices" but seems to also not specify a builddir.

Edit: docs suggest python -m build --wheel -Cbuild-dir=build

Now I got a reproducible build with

%pyproject_wheel --config-settings build-dir=mesonpybuild .

However, this is not so reassuring:
https://meson-python.readthedocs.io/en/stable/how-to-guides/build-directory.html

Warn that user-configured build directories are “use at your own risk”

[bnavigator]

The "how-to-guides" link leads to an older version.

https://mesonbuild.com/meson-python/reference/config-settings.html does not mention the at-own risk.

We have a clean build directory for every package (and python flavor) built by rpmbuild, so I think we can handle the risk.

[eli-schwartz]

So far our build-description just says %pyproject_wheel so not sure where to set builddir without patching every individual package. Maybe in our python-rpm-macros.

That sounds like an excellent question to ask the rpm or other OpenSUSE developers. I don't think meson-python has any opinion or indeed knowledge at all about what rpm-specific mechanisms rpm is using to execute meson-python or pass options to meson-python.

A more deterministic default would still be appreciated. e.g. scipy is listed on https://github.com/mesonbuild/meson-python/blob/main/docs/projects-using-meson-python.rst as "known to be adhering to best practices" but seems to also not specify a builddir.

It seems you are still confused about the option then? Why would scipy specify a builddir when it is the job of the build orchestration to decide whether a builddir makes sense? The default is carefully chosen -- the option is to "enable reproducibility and disable protections against clobbering manual user-configured build directories for people who don't use distro automation and just pip install .

However, this is not so reassuring: https://meson-python.readthedocs.io/en/stable/how-to-guides/build-directory.html

Warn that user-configured build directories are “use at your own risk”

Even for out of date documentation (which admittedly should be deleted) that is no longer referenced, it is marked as a TODO. It is vagueness upon vagueness. You shouldn't just take that with a grain of salt, you should take it with a whole bucket of salt.

[rgommers]

UX-wise there's a tradeoff here. With the extra tmpdir that meson-python creates:

1. The build always happens in a fresh directory and hence there's no risk of previous build settings being picked up from an earlier build.
2. Apparently there's non-reproducibility being introduced, which is annoying - repeated builds should yield identical results by default in the same build environment, ideally without extra build flags.

(1) seems more critical than (2) and has a much larger number of users, so our current choice does look right. However, the other question I'd have is what exactly gets embedded in those .o files - since if it's absolute paths, then that's anyway bad for build reproducibility (they'd be reproducible on the same machine or in the same CI config, but not otherwise). Could those be problems in pandas, SciPy & co?

I currently don't have much bandwidth, but if there's a fixable problem here I'd certainly want to prioritize fixing that for SciPy.

[rgommers]

I did a test and cannot actually reproduce the problem here with SciPy. Procedure:

1. export SOURCE_DATE_EPOCH=166887191
2. python -m build -wnx # in scipy repo, current main - build wheel without build isolation
3. mv dist/scipy-1.15.0.dev0-cp312-cp312-macosx_14_0_arm64.whl dist/scipy_bld1.whl
4. python -m build -wnx
5. mv dist/scipy-1.15.0.dev0-cp312-cp312-macosx_14_0_arm64.whl dist/scipy_bld2.whl
6. diffoscope dist/scipy_bld1.whl dist/scipy_bld2.whl

diffoscope says everything is identical.

[eli-schwartz]

It's the debug info in unstripped binaries.

[rgommers]

Ah that makes sense, thanks. It's not clear from the issue description or the links given there though. @bmwiedemann can you please confirm that all your regular package builds are fine, and it's only building debug packages that is showing non-reproducibility?

[bmwiedemann]

Yes, building without debuginfo is reproducible.
However, the default in openSUSE's Open Build Service is to always build with debuginfo and automatically strip it into a separate -debuginfo sub-packages at the end (maybe here). This leaves a hash of the debuginfo in the stripped .so file for gdb to find the right data, which is what makes the binary vary. In objdump diffs this shows up as a "shstrtab" section.

[rgommers]

This leaves a hash of the debuginfo in the stripped .so file for gdb to find the right data, which is what makes the binary vary.

The build ID hash depending on an absolute path to a build directory that doesn't exist at runtime is a design problem in debuginfo it looks like. While you can work around it by using --build-dir as pointed out above, this is an issue that ideally shouldn't occur to begin with. I think it's described in https://fedoraproject.org/wiki/Releases/FeatureBuildId:

The program /usr/lib/rpm/debugedit is used in rpm's separate debuginfo creation. It modifies DWARF data to replace the build-time absolute directory names with consistent names. This makes the installed debuginfo's source references usable, and it makes for reproducible rpm builds from identical constituents to produce identical binaries. The build ID computed by ld was affected by the name of the rpm build directory, so it will differ between two otherwise identical builds that used a different $RPM_BUILD_ROOT.

My patch adds the -i option to make debugedit recompute the build ID based on the contents of the file after the transformation. It also prints out the build ID bits in hex.

This is better than using --build-dir I think, since you don't want absolute paths to affect the hash and should apply independent of what build system or Python build backend you're dealing with.

[bmwiedemann]

Another way out could be the gcc -fdebug-prefix-map=old=new option.

[rgommers]

That looks like it's in the right direction too indeed. Not sure it can be made to work though, since you have to pass the name of the old directory, and if that's a tmpdir whose name you don't know beforehand (which will be the case for generated source files at least), what do you pass? Perhaps a good idea to use in combination with --build-dir though, if you end up going that way?

I'll include the description for fdebug-prefix-map=old=new from https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html for future readers:

When compiling files residing in directory old, record debugging information describing them as if the files resided in directory new instead. This can be used to replace a build-time path with an install-time path in the debug info. It can also be used to change an absolute path to a relative path by using . for new. This can give more reproducible builds, which are location independent, but may require an extra command to tell GDB where to find the source files. See also -ffile-prefix-map and -fcanon-prefix-map.

[bmwiedemann]

since you have to pass the name of the old directory, and if that's a tmpdir whose name you don't know beforehand (which will be the case for generated source files at least), what do you pass?

mesonpy knows the name of the tmpdir, so could it add this option to CFLAGS (and CXXFLAGS)?

[rgommers]

Inserting fairly niche and GCC-specific compiler flags into a build seems like a big no no. It's likely to break something else, confuse users, and bloat the build logs. Also, while we know what old is, we don't know what you want for new.

[eli-schwartz]

All this is already handled by the built-in reproducibility tooling for every entity working on reproducible builds, as long as you follow the existing directions in the mesonpy docs and pass the build-dir setting like some other distros already do.

[rgommers]

To be fair, there's no real docs for this yet, so I learned something here about how debuginfo is handled (and that we support SOURCE_DATE_EPOCH etc.). I'd like to add something more explicit to our docs, so that phrases like "reproducible builds" or "reproducibility" yield some info.

[eli-schwartz]

hmmm.... :)

#452 (review)

Mostly, SOURCE_DATE_EPOCH is supported by actual build tools e.g. gcc itself.

build paths are discussed a bit at https://reproducible-builds.org/docs/build-path/ and it's a generic problem which is generically solved by one of:

• always extracting the source tarball to the same fixed builder path
• using -fdebug-prefix-map computed based on the directory the source tarball is extracted to

(And using a stable build-dir option is also important, therefore.)

[rgommers]

Sure, it exists somewhere - all I meant is we need a page/section in https://mesonbuild.com/meson-python/ that mentions these things and then links to https://reproducible-builds.org and https://mesonbuild.com/Reproducible-builds.html.

build paths are discussed a bit at https://reproducible-builds.org/docs/build-path/

Ah yes, that page covers exactly all that we've talked about in this issue.

[bmwiedemann]

We solved this in openSUSE with https://build.opensuse.org/request/show/1204841 using a patch that sets a constant build_dir, if the SOURCE_DATE_EPOCH environment variable is set.

[jayaddison]

Hi folks - I'm a volunteer contributor to Reproducible Builds, and tend to focus on upstream and Debian patches; I began reading this while researching possible deterministic builds for Debian's contourpy.

Because I try to be a little circumspect about reading other projects' licensed code before writing my own patches (even in the context of mutual and compatible open source licenses), I haven't read the Gentoo or OpenSUSE approaches to fixing a builddir, though I gather from the discussion that that's the approach taken in each.

I 100% agree with the idea of isolating temporary build directories during default meson-python builds. Cached/existing on-disk artifacts simply do cause unexpected and weird problems otherwise.

But I'd also like to add some support for the fdebug-prefix-map suggestion. Folks involved in RB for much longer than I recognized that such an option would require multi-compiler support, and I can confirm that the same option is supported by LLVM/Clang.

As for what to set the new directory to with that option -- it's fine to simply replace the varying suffix with . - as mentioned it's not necessary at runtime for debug support other than to use to generate the build ID for matching debug symbols.

As to whether that complexity is worthwhile compared to adding a compiler option, adding noise in build logs, etc: I think that's a very fair question. The best response I can offer is that I personally think that software, by default, should build reproducibly. We can't always guarantee it -- people can always add code that will vary at build-time -- but deterministic-by-default (even with debugging enabled) I think would provide a lot of benefits and simplifications.

That said, I understand the caution about supporting this natively, and I think it's fairly reasonable to expect that build systems will handle this (some already are, and hopefully more will join). If I care about it enough, motivated self-interest should lead me to offer a pull request; I don't know whether I'll get around to that yet, but I'll consider it.

Thanks for reading if you made it this far, and no problem otherwise -- perhaps there could've been a more concise way for me to share my line of thinking.

[eli-schwartz]

Because I try to be a little circumspect about reading other projects' licensed code before writing my own patches (even in the context of mutual and compatible open source licenses), I haven't read the Gentoo or OpenSUSE approaches to fixing a builddir, though I gather from the discussion that that's the approach taken in each.

Gentoo utilizes https://mesonbuild.com/meson-python/reference/config-settings.html

OpenSUSE has patched the source code of meson-python to behave differently.

I recommend following the mesonpy documentation on passing args equivalent to GNU autotools dh_auto_configure --with-foo --enable-bar. You will anyways need it because mesonpy documents a setup-args key which can be used to pass options down to meson setup and there's quite a few of those that will be important (Gentoo uses --native-file, --pkg-config-path, -Dbuildtype, -Db_lto=$(is_lto_enabled), etc.). There is, generically speaking, no real reason standard debhelper arguments for meson shouldn't be respected here too.

And debhelper defines at minimum, buildtype=plain which you are going to want to pass to mesonpy as well for policy reasons.

But I'd also like to add some support for the fdebug-prefix-map suggestion. Folks involved in RB for much longer than I recognized that such an option would require multi-compiler support, and I can confirm that the same option is supported by LLVM/Clang.

But not MSVC! :) And maybe or maybe not ccrx, compcert, xc16, nvc, pgcc, icc, icx, lcc, emscripten, armclang, armltdclang, elbrus, metrowerks... quite literally, I genuinely have no clue! :)

Sure, some of those are derived from llvm. Are they new enough derivations to support fdebug-prefix-map, added practically yesterday as these things are reckoned -- in 2016?

So actually adding compile args by default would have to be done with care. Do we do it in mesonpy? How do we juggle setting this, guaranteeing it is set even when the user passed in their own CFLAGS or a native file, without totally overwriting the user-passed arguments? Most of all, how do we know what compiler is used? I mean, granted, I highly doubt mesonpy needs to care about Metrowerks even if meson itself does support it, but mesonpy is used on AIX. Meson might add support for xlC if the AIX scientific community submits patches for it. Do we instead add fdebug-prefix-map inside meson? But meson is more low-level and doesn't choose a mktemp style build-dir for you the way pip + mesonpy do.

How does one opt out of this, if they are positive they want the original paths? I mean, there's a reason the compiler vendors haven't simply unilaterally declared that prefixes will always be mapped without specifying an option. Which brings us to:

I would say that it is a moot point anyway because it cannot be added by default, only when it is specifically known that a reproducible build is desired...

As to whether that complexity is worthwhile compared to adding a compiler option, adding noise in build logs, etc: I think that's a very fair question. The best response I can offer is that I personally think that software, by default, should build reproducibly. We can't always guarantee it -- people can always add code that will vary at build-time -- but deterministic-by-default (even with debugging enabled) I think would provide a lot of benefits and simplifications.

... because this is unfortunately totally wrong. :( I mean, I do agree with my reproducible-builds hat on that many forms of reproducibility should simply be the default behavior. Stuff like __FILE__ should really be based on the project source root, not absolute filepaths. Stuff like __DATE__ and __TIME__ are absolute... &^$%.

But fdebug-prefix-map doesn't even affect __FILE__ at all. It affects debug info. And debug info has an extremely good reason to be accurate, because that is how you actually debug software! So we cannot set it to "dot" and we cannot in fact set it to anything whatsoever without knowing where the references should be redirected to. And the redirected references need to be to wherever debug data is actually stored on end user systems. If we don't do that, we break the debugger and people cannot step through code anymore.

This is why for reproducible builds we want to control the build-dir setting so we can ensure the debug info computes stable, reproducible paths, and we also want to inject fdebug-prefix-map outside of mesonpy, so that we can remap the extracted workdir to e.g. /usr/src/debug/${PACKAGE_NAME}/${PACKAGE_VERSION} which is where the package manager installs any source .c / .cpp files which are referenced in the debug info installed to /usr/lib/debug.

Thanks for reading if you made it this far, and no problem otherwise -- perhaps there could've been a more concise way for me to share my line of thinking.

No problem. It's a genuinely fascinating topic. :) Even if I have formed different conclusions than you have, I appreciate the thought that went into it as well as the opportunity to share the thoughts I have about it.

[eli-schwartz]

I realize that I am unsure whether Debian packages "installed-sources" for debug packages. I know dbgsym exists...

But fedora provides debuginfo and debugsource.

Arch packages sources as part of debug packages.

Gentoo provides FEATURES="splitdebug installsources".

[jayaddison]

Thanks - I definitely concede the many-other-compilers argument, and am not familiar enough (and probably shouldn't have implied that I was, previously!) with the way debug resources are loaded to find a way past that (other than, a naive thought perhaps, that some kind of content-addressing scheme might make sense to retrieve debug resources, because that'd be path-agnostic).

[jayaddison]

Thanks - I definitely concede the many-other-compilers argument, and am not familiar enough (and probably shouldn't have implied that I was, previously!) with the way debug resources are loaded to find a way past that (other than, a naive thought perhaps, that some kind of content-addressing scheme might make sense to retrieve debug resources, because that'd be path-agnostic).

Hmm... I think the debug Build-ID essentially is the content-addressing scheme, already; it's how a binary is linked to the associated debug resources (as retrieved from a filesystem directory, debuginfod, or whatever else).

What I don't get is the reasoning behind why some/any elements in the binary need to reference paths at all. For example, in the Debian reproducibility testing diffs at https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/contourpy.html the DWARF file contains a Directory Table entry in each comparative build for .mesonpy-xxxxxx -- but what is that, conceptually, and would there be a way to replace those labels/identifiers with something static?

[eli-schwartz]

Hmm... I think the debug Build-ID essentially is the content-addressing scheme, already; it's how a binary is linked to the associated debug resources (as retrieved from a filesystem directory, debuginfod, or whatever else).

Build-ID is that for the associative link between two objects that possess a build ID: the binary executable and the binary debug fields stripped out of an executable.

What I don't get is the reasoning behind why some/any elements in the binary need to reference paths at all. For example, in the Debian reproducibility testing diffs at https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/contourpy.html the DWARF file contains a Directory Table entry in each comparative build for .mesonpy-xxxxxx -- but what is that, conceptually, and would there be a way to replace those labels/identifiers with something static?

Because it's not necessarily about what goes into a Debian package. Like I said, Debian doesn't actually distribute all the data that other people use and rely on.

What is it conceptually? Well, I mentioned source code above.

If you build a program without installing it (for example to pip install --editable) the debug info is still inside the binary itself so it doesn't need a Build-ID at all in order to find that information. But it still needs to find the original source directory, or the debugger can only show symbol traces and cannot print out the lines of source code associated with those symbol traces. That's why the debug info contains those paths in the first place.

This is a fundamental difference between building software and running it immediately vs building software and packaging it in a package manager. In the latter case, and only in the latter case, a program is run on the packaged binaries to postprocess them, move their debug info into separate packages, and include the program source code into that separate package as well. This program also can rewrite the debug info to refer to the locations that the separate package will install, rather than referring to build directories, but gcc has an option to do the rewrite at compilation time and then all you have to do is move the sources.

(The program is appropriately called "debugedit".)

• In order to debug a program without installing it via a distro package manager, you MUST NOT mess with the default behavior of the compiler.
• In order to debug a program after installing it via a distro package manager you MUST mess with the default behavior of the compiler.
• if you don't care about stepping through sources, or your distro decides for you that you don't care, you MAY do whatever you want including deleting this information and replacing it with "." instead.

The build system can't know automatically which of the three is relevant, but CFLAGS can always know, or a specific option can be added. "Just automatically replace it with option 3" isn't on the table, since option 3 will break case 1 even if case 2 is superseded by a hashed content-addressable storage system. And the content-addressable storage system would have to be invented first and rolled out to existing toolchains by upgrading to new versions of gdb etc, I think.

[jayaddison]

To your question about Debian: yes, as you know dbgsyms packages are distributed; I found this blog post that describes a tool that retrieve the corresponding (signed-only, by default) sources.

Most of what you say I agree with, but I still think that, to take the case of contourpy specifically, something is unnecessarily nondeterministic here.

The first diff between two builds lists this difference in the output .so file (the built binary):

./.mesonpy-dkhufq4_/../src/contour_generator.cpp:42
./.mesonpy-klczfyfg/../src/contour_generator.cpp:42

So, first of all: the temporary build directory isn't actually relevant for debugging/source-stepping purposes. It provides build isolation, but the source files were outside of it anyway, in the ./src dir.

In the corresponding DWARF .debug file, we find the temporary path embedded again:

····<11>···DW_AT_comp_dir····:·(line_strp)·(offset:·0x17):·./.mesonpy-dkhufq4_
····<11>···DW_AT_comp_dir····:·(line_strp)·(offset:·0):·./.mesonpy-klczfyfg

There is -- separately! -- a reference to the src dir in the same .debug file. And it is the same for both builds:

··1»      (line_strp)»   (offset:·0x2b):·../src

So... it seems that the DW_AT_comp_dir is somewhat redundant here, unless I'm mistaken. It isn't going to be practically useful for locating any source code to read/step through, because that's not where the source files are -- and there's already another entry in the file (the Directory Table?) to list where the source files genuinely are.

Does that imply a bug with a different component? I don't know. To me it seems that the DWARF file shouldn't contain the .mesonpy-xxxxx temp dir here.

[bmwiedemann]

./.mesonpy-dkhufq4_/../src/contour_generator.cpp:42
./.mesonpy-klczfyfg/../src/contour_generator.cpp:42

maybe this can be solved by referencing source files differently as src/contour_generator.cpp here? Is there a place where the $tmpdir/.. could be dropped?

[eli-schwartz]

In the corresponding DWARF .debug file, we find the temporary path embedded again:

····<11>···DW_AT_comp_dir····:·(line_strp)·(offset:·0x17):·./.mesonpy-dkhufq4_
····<11>···DW_AT_comp_dir····:·(line_strp)·(offset:·0):·./.mesonpy-klczfyfg

There is -- separately! -- a reference to the src dir in the same .debug file. And it is the same for both builds:

··1»      (line_strp)»   (offset:·0x2b):·../src

So... it seems that the DW_AT_comp_dir is somewhat redundant here, unless I'm mistaken. It isn't going to be practically useful for locating any source code to read/step through, because that's not where the source files are -- and there's already another entry in the file (the Directory Table?) to list where the source files genuinely are.

Does that imply a bug with a different component? I don't know. To me it seems that the DWARF file shouldn't contain the .mesonpy-xxxxx temp dir here.

debug info when faced with relative filenames passed on the command line must store the compiler's working directory as well, specifically because the DW_AT_comp_dir is combined with the filename starting with "../" in order to compute the abspath().

It is an intentional part of the DWARF spec. I'm not familiar offhand with why this decision was chosen compared to e.g. performing canonicalization resolution and storing the computed realpath instead (maybe they didn't think it would matter? maybe they were concerned that a symlink destination would change and invalidate the computation? maybe they thought it did matter and actively wanted to be able to omit the comp_dir for display purposes?) but it is certainly the state of affairs as of today, so changing it would require a spec change conditional on consensus between various international stakeholders and is not a simple matter of declaring it a bug in the implementation.

[jayaddison]

Hmm.. well, regardless of canonicalization, the current-working-directory entry really needs to be there, according to the DWARF v5 spec, page 157 (hyperlink fragment is not a typo; page 157 of the document is page 175 of the PDF...):

Prior to DWARF Version 5, the current directory was not represented in the
directories field and a directory index of 0 implicitly referred to that directory as found
in the DW_AT_comp_dir attribute of the compilation unit debugging information
entry. In DWARF Version 5, the current directory is explicitly present in the
directories field. This is needed to support the common practice of stripping all but
the line number sections (.debug_line and .debug_line_str) from an executable.

Which I suppose is an argument for specifying a constant builddir during the build process.

I'll probably pause for a while, rather than being stubborn, but essentially what I'd like to achieve is: deterministic debug builds, with source-reference feasible at debugging-time (perhaps using path-based discovery of sources, or in rare cases, debugger config/command-line options.. though I'd prefer to avoid those), for both local development and on hosted/distro build infrastructure.

I think the 'three options' described earlier is an effective summary of the practical status-quo today.