Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-43846: [Python][Packaging] Remove numpy dependency from pyarrow packaging #44148

Merged
merged 2 commits into from
Oct 10, 2024

Conversation

raulcd
Copy link
Member

@raulcd raulcd commented Sep 17, 2024

WIP

Creating the PR to validate if / what are the CI failures

Rationale for this change

Once the issue has been merged:

We do not require numpy as a pyarrow dependency.

What changes are included in this PR?

Remove numpy as a required dependency

Are these changes tested?

Via CI

Are there any user-facing changes?

Yes, pyarrow won't install numpy as a required dependency.

Copy link

⚠️ GitHub issue #43846 has been automatically assigned in GitHub to PR creator.

@raulcd
Copy link
Member Author

raulcd commented Sep 17, 2024

@github-actions crossbow submit -g python

Copy link

Revision: 2a52bce

Submitted crossbow builds: ursacomputing/crossbow @ actions-929cd4e6ac

Task Status
example-python-minimal-build-fedora-conda GitHub Actions
example-python-minimal-build-ubuntu-venv GitHub Actions
test-conda-python-3.10 GitHub Actions
test-conda-python-3.10-cython2 GitHub Actions
test-conda-python-3.10-hdfs-2.9.2 GitHub Actions
test-conda-python-3.10-hdfs-3.2.1 GitHub Actions
test-conda-python-3.10-pandas-latest-numpy-latest GitHub Actions
test-conda-python-3.10-substrait GitHub Actions
test-conda-python-3.11 GitHub Actions
test-conda-python-3.11-dask-latest GitHub Actions
test-conda-python-3.11-dask-upstream_devel GitHub Actions
test-conda-python-3.11-hypothesis GitHub Actions
test-conda-python-3.11-pandas-latest-numpy-1.26 GitHub Actions
test-conda-python-3.11-pandas-latest-numpy-latest GitHub Actions
test-conda-python-3.11-pandas-nightly-numpy-nightly GitHub Actions
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly GitHub Actions
test-conda-python-3.11-spark-master GitHub Actions
test-conda-python-3.12 GitHub Actions
test-conda-python-3.12-cpython-debug GitHub Actions
test-conda-python-3.9 GitHub Actions
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5 GitHub Actions
test-conda-python-emscripten GitHub Actions
test-cuda-python-ubuntu-22.04-cuda-11.7.1 GitHub Actions
test-debian-12-python-3-amd64 GitHub Actions
test-debian-12-python-3-i386 GitHub Actions
test-fedora-39-python-3 GitHub Actions
test-ubuntu-22.04-python-3 GitHub Actions
test-ubuntu-22.04-python-313-freethreading GitHub Actions
test-ubuntu-24.04-python-3 GitHub Actions

@raulcd
Copy link
Member Author

raulcd commented Sep 17, 2024

@github-actions crossbow submit -g wheel

@raulcd
Copy link
Member Author

raulcd commented Sep 17, 2024

@github-actions crossbow submit verify-python

Copy link

Revision: 2a52bce

Submitted crossbow builds: ursacomputing/crossbow @ actions-6a782cdfa6

Task Status
verify-rc-source-python-linux-almalinux-8-amd64 GitHub Actions
verify-rc-source-python-linux-conda-latest-amd64 GitHub Actions
verify-rc-source-python-linux-ubuntu-20.04-amd64 GitHub Actions
verify-rc-source-python-linux-ubuntu-22.04-amd64 GitHub Actions
verify-rc-source-python-macos-amd64 GitHub Actions
verify-rc-source-python-macos-arm64 GitHub Actions
verify-rc-source-python-macos-conda-amd64 GitHub Actions

Copy link

Revision: 2a52bce

Submitted crossbow builds: ursacomputing/crossbow @ actions-09c6d448ff

Task Status
python-sdist GitHub Actions
wheel-macos-monterey-cp310-amd64 GitHub Actions
wheel-macos-monterey-cp310-arm64 GitHub Actions
wheel-macos-monterey-cp311-amd64 GitHub Actions
wheel-macos-monterey-cp311-arm64 GitHub Actions
wheel-macos-monterey-cp312-amd64 GitHub Actions
wheel-macos-monterey-cp312-arm64 GitHub Actions
wheel-macos-monterey-cp313-amd64 GitHub Actions
wheel-macos-monterey-cp313-arm64 GitHub Actions
wheel-macos-monterey-cp39-amd64 GitHub Actions
wheel-macos-monterey-cp39-arm64 GitHub Actions
wheel-manylinux-2-28-cp310-amd64 GitHub Actions
wheel-manylinux-2-28-cp310-arm64 GitHub Actions
wheel-manylinux-2-28-cp311-amd64 GitHub Actions
wheel-manylinux-2-28-cp311-arm64 GitHub Actions
wheel-manylinux-2-28-cp312-amd64 GitHub Actions
wheel-manylinux-2-28-cp312-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-arm64 GitHub Actions
wheel-manylinux-2-28-cp39-amd64 GitHub Actions
wheel-manylinux-2-28-cp39-arm64 GitHub Actions
wheel-manylinux-2014-cp310-amd64 GitHub Actions
wheel-manylinux-2014-cp310-arm64 GitHub Actions
wheel-manylinux-2014-cp311-amd64 GitHub Actions
wheel-manylinux-2014-cp311-arm64 GitHub Actions
wheel-manylinux-2014-cp312-amd64 GitHub Actions
wheel-manylinux-2014-cp312-arm64 GitHub Actions
wheel-manylinux-2014-cp313-amd64 GitHub Actions
wheel-manylinux-2014-cp313-arm64 GitHub Actions
wheel-manylinux-2014-cp39-amd64 GitHub Actions
wheel-manylinux-2014-cp39-arm64 GitHub Actions
wheel-windows-cp310-amd64 GitHub Actions
wheel-windows-cp311-amd64 GitHub Actions
wheel-windows-cp312-amd64 GitHub Actions
wheel-windows-cp313-amd64 GitHub Actions
wheel-windows-cp39-amd64 GitHub Actions

@raulcd
Copy link
Member Author

raulcd commented Sep 19, 2024

@jorisvandenbossche is this something we want to do before 18.0.0?

@jorisvandenbossche
Copy link
Member

Should we bring that up on the mailing list?

My guess is that it won't be that disruptive because most people will have numpy installed via other dependencies anyway, but it might be a bit hard to predict.
If we want to do it at some point, I think we can as well do it for 18.0

@raulcd raulcd marked this pull request as ready for review October 2, 2024 15:13
@jorisvandenbossche
Copy link
Member

Looks good!

Can you update the documentation? https://arrow.apache.org/docs/dev/python/install.html#dependencies

@raulcd
Copy link
Member Author

raulcd commented Oct 3, 2024

Can you update the documentation? https://arrow.apache.org/docs/dev/python/install.html#dependencies

Thanks @jorisvandenbossche I've just updated the Numpy dependency from required to optional on the docs

@raulcd
Copy link
Member Author

raulcd commented Oct 9, 2024

@jorisvandenbossche if you are ok with this I'll merge for 18.0.0

@jorisvandenbossche
Copy link
Member

I am fine with that

@github-actions github-actions bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Oct 9, 2024
Copy link
Member Author

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@raulcd raulcd merged commit 8be5f9c into apache:main Oct 10, 2024
15 checks passed
@raulcd raulcd removed the awaiting merge Awaiting merge label Oct 10, 2024
@github-actions github-actions bot added the awaiting changes Awaiting changes label Oct 10, 2024
@raulcd raulcd deleted the GH-43846 branch October 10, 2024 10:02
@raulcd raulcd removed the awaiting changes Awaiting changes label Oct 10, 2024
raulcd added a commit that referenced this pull request Oct 10, 2024
…ckaging (#44148)

## WIP

Creating the PR to validate if / what are the CI failures 

### Rationale for this change
Once the issue has been merged:
- #25118

We do not require numpy as a pyarrow dependency.

### What changes are included in this PR?

Remove numpy as a required dependency

### Are these changes tested?

Via CI

### Are there any user-facing changes?

Yes, pyarrow won't install numpy as a required dependency.
* GitHub Issue: #43846

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
Copy link

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 8be5f9c.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

Fokko added a commit to Fokko/iceberg-python that referenced this pull request Oct 30, 2024
With Arrow 18.0.0 numpy is not a dependency anymore:
apache/arrow#44148

I think it would be good to also remove it from PyIceberg
Fokko added a commit to Fokko/iceberg-python that referenced this pull request Oct 30, 2024
With Arrow 18.0.0 numpy is not a dependency anymore:
apache/arrow#44148

I think it would be good to also remove it from PyIceberg
Fokko added a commit to apache/iceberg-python that referenced this pull request Oct 30, 2024
* Remove numpy as a hard dependency

With Arrow 18.0.0 numpy is not a dependency anymore:
apache/arrow#44148

I think it would be good to also remove it from PyIceberg

* Add link to issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants