build wheels without build isolation

[jameslamb]

Description

Contributes to rapidsai/build-planning#108

Proposes building wheels with --no-build-isolation, to improve the rate of sccache cache hits and therefore reduce CI times.

Notes for Reviewers

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

How I tested this

Ran a first build with these changes to populate the cache, then ran again... saw a 100% cache hit rate and the actual wheel-building part of a wheel-build-libcuspatial job take only 37 seconds.

Successfully built libcuspatial-cu11
Compile requests                     28
Compile requests executed            28
Cache hits                           28
Cache hits (c [gcc])                  2
Cache hits (c++ [gcc])                2
Cache hits (cuda [nvcc])             24

(build link)

[jameslamb]

Anticipating a review comment like "is this left over from debugging?"... I've intentionally left this in. I think it's generally useful to print the sccache stats in these CI logs, as they could provide a hint when you experience issues like CI timeouts or out-of-memory.

This only adds 27 lines to the logs, like this:

Compile requests                     28
Compile requests executed            28
Cache hits                           28
Cache hits (c [gcc])                  2
Cache hits (c++ [gcc])                2
Cache hits (cuda [nvcc])             24
Cache misses                          0
Cache timeouts                        0
Cache read errors                     0
Forced recaches                       0
Cache write errors                    0
Compilation failures                  0
Cache errors                          0
Non-cacheable compilations            0
Non-cacheable calls                   0
Non-compilation calls                 0
Unsupported compiler calls            0
Average cache write               0.000 s
Average compiler                  0.000 s
Average cache read hit            0.077 s
Failed distributed compilations       0
Cache location                  s3, name: rapids-sccache-east, prefix: /
Version (client)                0.7.7

[msarahan]

Yes, this is good IMO. I'm also going to add this kind of thing into the telemetry.

[vyasr]

I agree, I want us to start printing sccache in more jobs so that we can proactively identify these kinds of issues.

[bdice]

I merged in the upstream to get CI fixes from #1475.

[vyasr]

I opened rapidsai/dependency-file-generator#114 as a suggestion for simplifying these kinds of scripts, but feel free to proceed in whatever order you prefer. This LGTM now.

[vyasr]

Temporarily blocking merge pending offline discussion about whether we want to only turn off build isolation for C++ wheel builds so that Python builds can still test the build isolation code path via RBB and ensure that we get things mostly right.

[jameslamb]

I opened rapidsai/dependency-file-generator#114 as a suggestion for simplifying these kinds of scripts, but feel free to proceed in whatever order you prefer.

Ok, I'll try this right now. If it takes too long, I'll abandon it for now and proceed with 2 rapids-dependency-file-generator calls.

[jameslamb]

Suggested change

	git clone \
	--branch multiple-file-keys \
	https://github.com/jameslamb/dependency-file-generator.git \
	/tmp/rapids-dependency-file-generator
	pip uninstall --yes rapids-dependency-file-generator
	pip install /tmp/rapids-dependency-file-generator

TODO: remove this before merging

[jameslamb]

Ok I think this is working.

On the latest build on branch-24.12, you can see log lines about installing build dependencies and creating a temporary build directory.

Created temporary directory: /tmp/pip-wheel-ndbc3klu
Created temporary directory: /tmp/pip-ephem-wheel-cache-blfbvohw
Looking in indexes: https://pypi.org/simple, https://pypi.anaconda.org/rapidsai-wheels-nightly/simple, https://pypi.nvidia.com/
Processing /__w/cuspatial/cuspatial/python/libcuspatial
  Added file:///__w/cuspatial/cuspatial/python/libcuspatial to build tracker '/tmp/pip-build-tracker-vn6jabrn'
  Created temporary directory: /tmp/pip-build-env-64dc7d88
  Running command pip subprocess to install build dependencies
  Installing build dependencies: started

(build link)

On this PR, none of that.

Looking in indexes: https://pypi.org/simple, https://pypi.anaconda.org/rapidsai-wheels-nightly/simple, https://pypi.nvidia.com/
Processing /__w/cuspatial/cuspatial/python/libcuspatial
  Running command Preparing metadata (pyproject.toml)
  Preparing metadata (pyproject.toml): started

(build link)

[jameslamb]

I mention that because it looks to me like we're not getting sccache hits.

Successfully built libcuspatial-cu11
Compile requests                      28
Compile requests executed             28
Cache hits                             4
Cache hits (c [gcc])                   2

There hasn't been a new rmm commit merged since yesterday, and the last new cudf commit merged was an hour ago.

Gonna try another build to double-check this.

[jameslamb]

Yeah cool ok, I just did another build and saw all cache hits. Must have just gotten unlucky before with new cudf changes being committed and new nightlies appearing.

Compile requests                     28
Compile requests executed            28
Cache hits                           28
Cache hits (c [gcc])                  2
Cache hits (c++ [gcc])                2
Cache hits (cuda [nvcc])             24

(build link)