Apple Clang 16 build issue, requires ld_classic

[adamjstewart]

🐛 Describe the bug

When building from source with the new Apple Clang 16, I see the following error:

  ld: multiple errors: duplicate LC_RPATH '/Users/Adam/spack/opt/spack/darwin-sequoia-m2/apple-clang-16.0.0/cpuinfo-2023-11-04-4sylnhqsvndbdpgwvv3mydc6xhudikqh/lib' in '/Users/Adam/spack/opt/spack/darwin-sequoia-m2/apple-clang-16.0.0/py-torch-2.4.1-wsy5qo376pu3gqk6sazvaff3wf6uo6fl/lib/python3.11/site-packages/torch/lib/libc10.dylib'; duplicate LC_RPATH '/Users/Adam/spack/opt/spack/darwin-sequoia-m2/apple-clang-16.0.0/protobuf-3.13.0-sqetzfmrwohnuex4orfxr5loztdqj6rq/lib' in '/Users/Adam/spack/opt/spack/darwin-sequoia-m2/apple-clang-16.0.0/py-torch-2.4.1-wsy5qo376pu3gqk6sazvaff3wf6uo6fl/lib/python3.11/site-packages/torch/lib/libtorch.dylib'; duplicate LC_RPATH '/Users/Adam/spack/opt/spack/darwin-sequoia-m2/apple-clang-16.0.0/protobuf-3.13.0-sqetzfmrwohnuex4orfxr5loztdqj6rq/lib' in '/Users/Adam/spack/opt/spack/darwin-sequoia-m2/apple-clang-16.0.0/py-torch-2.4.1-wsy5qo376pu3gqk6sazvaff3wf6uo6fl/lib/python3.11/site-packages/torch/lib/libtorch_cpu.dylib'; duplicate LC_RPATH '/Users/Adam/spack/opt/spack/darwin-sequoia-m2/apple-clang-16.0.0/protobuf-3.13.0-sqetzfmrwohnuex4orfxr5loztdqj6rq/lib' in '/Users/Adam/spack/opt/spack/darwin-sequoia-m2/apple-clang-16.0.0/py-torch-2.4.1-wsy5qo376pu3gqk6sazvaff3wf6uo6fl/lib/python3.11/site-packages/torch/lib/libtorch_python.dylib'
  clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Current workaround is to inject -Wl,-ld_classic into the linker, but would be good to fix this as ld_classic is deprecated and will be removed in the future.

Versions

• torchvision: 0.19.1
• clang: 16.0.0

[adamjstewart]

We implemented a compiler wrapper in Spack that removes duplicate RPATHs, but we're still seeing the following issue:

ld: duplicate LC_RPATH '@loader_path' in '/Users/Adam/spack/opt/spack/darwin-sequoia-m2/apple-clang-16.0.0/py-torch-2.4.1-hlpsikrtvukd4ww7jyghncl63ux7f2pl/lib/python3.11/site-packages/torch/lib/libtorch_cpu.dylib'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/Users/Adam/spack/lib/spack/env/clang/clang++' failed with exit code 1

It's possible the bug is in PyTorch itself, but I'm not familiar enough with CMake to report it.

[NicolasHug]

Sorry @adamjstewart , this is kind of tricky to fix because we build torchvision on some subset of compilers, and that means not all may be properly supported. Especially in this case where the compiler/linker seems to be more pedantic than gcc (which tends to silently just ignore anything it doesn't expect).

[adamjstewart]

This is funny because Google (TF, JAX) have now dropped support for GCC and only support Clang. But I understand the pain of having to support too many compilers/platforms/etc. Ideally someone with more experience with Clang than both of us can submit a patch.

I'm still not 100% convinced that this isn't a bug in Spack's compiler wrappers, so if no one else reports this issue, I wouldn't prioritize it too much.