Build platform-specific wheels containing libmagic #294

ddelange · 2023-09-06T12:59:42Z

This PR builds self-contained wheels as discussed in #233. For Windows users, this renders python-magic-bin from @julian-r obsolete.

pip install these wheels

pip can install from GitHub Release assets from my fork:

# comma separated list for --find-links
export PIP_FIND_LINKS=https://github.com/ddelange/python-magic/releases/expanded_assets/0.4.28.post7
pip install --force-reinstall python-magic

- python-magic-0.4.27.tar.gz
+ python-magic-0.4.28.tar.gz
- python_magic-0.4.27-py2.py3-none-any.whl
+ python_magic-0.4.28-py2.py3-none-macosx_10_9_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-macosx_11_0_arm64.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_aarch64.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_ppc64le.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_s390x.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-win32.whl
+ python_magic-0.4.28-py2.py3-none-win_amd64.whl

The wheels:

Come bundled with libmagic (.dylib on mac, .so on nix, .ddl on win) - no additional user action needed
- latest libmagic (5.45), except on windows (5.44, used the latest GitHub Release at https://github.com/julian-r/file-windows)
- building from source on linux and macos with maximum backwards compatibility
- skipping i686 linux wheels as they don't pass pytest ref ca4def3
Are forward compatible, they will install on cpython 2.7 and all current and future cpython 3.5+ distributions
Are built for the major cpu architectures (and also for alpine linux)
- ~~Linux architectures limited by availability: https://pkgs.org/search/?q=file-libs~~ now building from source on linux
- For windows I couldn't find compiled libmagic for ARM64, skipped it
Are uploaded as GitHub Release assets, as well as upload to PyPI, whenever a GitHub Release is (pre)released.
- 🚩 @ahupp for this, you still need to add wheels.yml as trusted publisher here

CI/CD

dists build with official cibuildwheel on GitHub Actions, and they build in parallel:

The PyPI publish step is failing here ^ because my fork is not a trusted publisher
For any system not covered by these wheels, pip will fall back to the source distribution, which will check for proper libmagic on the systen at install time.

fix #137, fix #288, fix #225, fix #276, fix #248, fix #87, fix #139, fix #233, fix #73, fix #60, fix #34, fix #293, fix #233, fix #278, fix #262, fix #248, fix #238, fix #145, fix #61, fix #12, fix #295, fix #311, fix #312, fix #313, fix #321, fix #332, fix #249, fix #333

apirogov · 2023-09-12T13:49:39Z

This is nice! Hope this will be merged soon!

Just ran into issues with my library being not usable by Mac and Windows users because I rely on python-magic. If there are wheels, I don't need to find a workaround or replace the library :)

python-magic-bin did not work for some of them, by the way.

stumpylog · 2023-09-12T14:40:45Z

.github/workflows/main.yml

+      with:
+        files: dist/*
+
+    - name: Upload to PyPI


A small improvement here might be to use the PyPa Action instead: https://github.com/pypa/gh-action-pypi-publish

The big advantage is trusted publishing, instead of storing a password or token as a secret

that's cool, thanks for sharing!

@ahupp shall I make that change and you set it up on PyPI side?

Sounds like trusted publishing is the way to go. I recently got this email:

Hi ddelange!
Earlier this year, we announced that PyPI would require all users to enable a form of two-factor authentication on their accounts by the end of 2023.

Keeping your PyPI account secure is important to all of us. We encourage you to enable two-factor authentication on your PyPI account as soon as possible.

What forms of 2FA can I use?
We currently offer two main forms of 2FA for your account:

Security device including modern browsers (preferred) (e.g. Yubikey, Google Titan)
Authentication app (e.g. Google Authenticator)
Once one of these secure forms is enabled on your account, you will also need to use either Trusted Publishers (preferred) or API tokens to upload to PyPI.

What do I do if I lose my 2FA device?
As part of 2FA enrollment, you will receive one-time use recovery codes. One of them must be used to confirm receipt before 2FA is fully active. Keep these recovery codes safe - they are equivalent to your 2FA device. Should you lose access > to your 2FA device, use a recovery code to log in and swap your 2FA to a new device.

Read more aboutrecovery codes.

Why is PyPI requiring 2FA?
Keeping all users of PyPI is a shared responsibility we take seriously. Strong passwords combined with 2FA is a recognized secure practice for over a decade.

We are requiring 2FA to protect your account and the packages you upload, and to protect PyPI itself from malicious actors. The most damaging attacks are account takeover and malicious package upload.

To see this and other security events for your account, visit your account security history.

Read more on this blog post.

If you run into problems, read the FAQ page. If the solutions there are unable to resolve the issue, contact us via support@pypi.org.

Thanks,
The PyPI Admins

@ahupp so the last thing for you to do is adding this repo as trusted publisher to https://pypi.org/manage/project/python-magic/settings/publishing/

.github/workflows/main.yml

ahupp · 2023-09-28T17:18:10Z

This is huge, thank you! Apology for the delay I thought I'd commented earlier but guess not. I'll look this over soon; I didn't quite understand how bad the binary dep situation was expecially on windows.

.github/workflows/main.yml

jean-humann · 2023-12-03T17:02:05Z

@ahupp @stumpylog could we have this one merged (and released) by the end of the year please ?

add_libmagic.sh

ferdnyc · 2024-05-28T17:44:30Z

magic/compat.py

+magic_file = None
+if not os.environ.get("MAGIC"):
+    # wheels package the mime database in this directory
+    # prefer it when no magic file is specified by the user
+    mime_db = os.path.join(os.path.dirname(__file__), 'magic.mgc')
+    if os.path.exists(mime_db):
+        magic_file = mime_db


Seems weird duplicating this logic between here and __init__.py, but I guess it's unavoidable since the __init__.py version is hidden inside Magic.__init__().

one way to avoid code duplication (and cyclic import) would be to add from magic.compat import magic_file on top of __init__.py, and use it as default argument:

Magic.__init__(magic_file=magic_file, ...)

option 2 would be to put the snippet in a new file and import it in both places (avoids cyclic import as well).

any preference here @ahupp?

add from magic.compat import magic_file on top of __init__.py,

I don't think that's a good idea, as it means that it'd be impossible to import any part of python-magic without also importing magic.compat. A compat module, specifically, should only be imported when needed.

The code could be moved to the top level of __init__.py, and imported into compat.py with from . import magic_file. I think that would be a better direction for things to flow, if going that route.

Oh, never mind. __init__.py already unconditionally imports magic.compat in its _add_compat() function. Which is always run on import.

Co-authored-by: Frank Dana <ferdnyc@gmail.com>

ddelange · 2024-05-29T16:31:26Z

Hi 👋 I'll be AFK until end of June. @ahupp feel free to take over my branch, or merge as is!

Privat33r-dev

Generally LGTM apart from some minor issues. Ideally, to have better control over dependencies, I would keep binaries inside of the repo, like pyexiv2 and some other libraries do.

.github/workflows/ci.yml

CHANGELOG

README.md

Privat33r-dev · 2024-06-25T15:22:16Z

README.md

+Depending on your system and CPU architecture, there might be no compatible wheel uploaded.
+However, precompiled libmagic might still be available for your system:
+
+```sh


It may be beneficial to add a library installation guide for SUSE as well

can you provide the relevant command?

fwiw, I think mostly all linux flavours will be covered by the wheels in the PR description, so those users won't be needing the install from source instructions provided here.

I guess that it would be
zypper install file-devel
it might as well be required to do (in vivo test is required though)
zypper install file-magic

Currently I don't have OpenSUSE at my disposal for tests and it's likely that it would be a default package. I don't promise anything, but I might find time soon-ish to test it.

Privat33r-dev · 2024-06-26T11:07:52Z

add_libmagic.sh

+        tar xvf "${tmpfile}" &&
+        cd "${version}" &&
+        ./configure &&
+        make &&


Dev dependencies to use "make" and similar commands are not shipped by default in some distros, especially in those that are usually used by Docker. I'd suggest to ensure that necessary dependencies are installed first.

I wished set -euxo pipefail also effects commands inside function definitions. then we could remove && from every line here (and the subshell parentheses here and the || on L21), and we could test which make on top of this function or similar.

now, it simply falls back to install_precompiled (L69) when curl or make or compilation fails (error will be in output).

if make is not available, in the current setup we will only have done an unnecessary curl. not the worst but could be avoided.

any ideas?

I wished set -euxo pipefail also effects commands inside function definitions. then we could remove && from every line here (and the subshell parentheses here and the || on L21), and we could test which make on top of this function or similar.

any ideas?

It appears to be doing so already. Try this (and also the same script, but remove the first line or replace exit 1 to exit 0):

set -euxo pipefail func(){ echo 1 (exit 1) echo 2 } func

You made me think about other topic when you mentioned that curl can be executed. There is no cleanup on any step, it might be logical to make a ZIP (or gzip, doesn't matter) file cleanup from external function and do it regardless of the outcome. Same for windows. Pseudocode:

install_stuff(){ real_install() rm potential_zip }

README.md

Privat33r-dev · 2024-06-26T11:45:19Z

add_libmagic.sh

+    (
+        version="file-5.45" &&
+        tmpfile="$(mktemp)" &&
+        curl -sSLo "${tmpfile}" "https://astron.com/pub/file/${version}.tar.gz" &&


Tried it on ubuntu's Docker, fails here :)
bash: curl: command not found

install_precompiled would work though

yes, the script is only intended for github actions default runner images and the cibuildwheel docker images (both have curl and make)

cclauss · 2024-06-26T12:42:21Z

There is wonderful work in this pull request and it has four positive reviews. Unfortunately, it has been open for ~9 months without landing. Perhaps it would be useful to break it into three separate PRs that are easier to review and land.

One PR that deals only with macOS and another that deals only with Windows might be easier to land. Once that is done then this PR could be rebased to deal only with Linux and friends. I know it is extra work but I sense that new momentum is needed.

estarfoo · 2024-06-26T15:37:48Z

Hi! Great PR, is there any real contention about it beyond the scope of OS/distro support in add_libmagic.sh and README.md?

Most users will only ever want the wheels from this repo. In particular, this looks like it will solidly cover usage in Docker images. Anyone who wants to use the source version and provide libmagic themselves, probably knows best how to do the latter in their environment. given info on where this package will look for the library. (Those who package python-magic for their Linux distro of choice will already have a preferred way of ensuring libmagic presence. This will probably not exactly match anything suggested in python-magic docs.)

Even for those particularly invested in having a range of setup instructions, the PR in its current state should look like a clear improvement on master, and further improvements in that are will come more easily as separate PRs (because they won’t be tied to CI scripts).

So: how about merging this without completing libmagic setup instructions for every possible platform? Seems like it already does what the PR title says.

Privat33r-dev · 2024-06-26T15:42:39Z

So: how about merging this without completing libmagic setup instructions for every possible platform? Seems like it already does what the PR title says.

Totally for it. My suggestions just show the way to improve it, but I would merge it "as is" since it already provides a huge value. "Perfect is the enemy of good".

@ahupp hopefully you can find some time to review this most discussed PR in the python-magic's history :)

ferdnyc · 2024-07-30T08:06:41Z

add_libmagic.sh

+        tmpfile="$(mktemp)" &&
+        curl -sSLo "${tmpfile}" "https://astron.com/pub/file/${version}.tar.gz" &&
+        tar xvf "${tmpfile}" &&


@estarfoo

Not a big deal, but if going with the piped curl output, personally I'd pair that with an explicit tar xvf -. Again, for the paranoid.

add_libmagic.sh

+import platform, sysconfig, io, zipfile, urllib.request
+assert platform.system() == "Windows"
+machine = "x86" if sysconfig.get_platform() == "win32" else "x64"
+url = f"https://github.com/julian-r/file-windows/releases/download/v5.44/file_5.44-build104-vs2022-{machine}.zip"


ddelange · 2024-06-29T07:20:12Z

@ahupp this also fixes failing CI on master (looks like the github actions linux runner image no longer ships libmagic by default)

ashtonpaul · 2024-07-31T19:32:08Z

@ahupp @ddelange Any update on this PR and the release? 👀 Is there anything that needs to be tested or are we waiting on anything other than a review?

ddelange · 2024-09-02T07:51:58Z

@ahupp please review, merge, add trusted publisher (see PR description) and release 🙏

wilhelmklopp · 2024-10-18T08:17:58Z

Also would love to see this merged, it's causing some issues for windows users of our package (which depends on mocket which depends on python-magic). Thank you! 😄

cclauss · 2024-10-18T09:25:34Z

@ashtonpaul @ddelange @wilhelmklopp @ferdnyc #294 (comment)

ferdnyc · 2024-10-21T03:08:06Z

@ahupp Speaking personally, this is a big, complex PR that I don't really feel qualified to approve given the scope and nature of all the changes here. I guess I'm in agreement with your #294 (comment)

Perhaps it would be useful to break it into three separate PRs that are easier to review and land.

One PR that deals only with macOS and another that deals only with Windows might be easier to land. Once that is done then this PR could be rebased to deal only with Linux and friends. I know it is extra work but I sense that new momentum is needed.

OTOH, the PR has received a number of approvals as-is, so there's ample indication that others are satisfied with it in its current form. I left some comments, which were addressed, so my input certainly shouldn't be viewed as disapproval by any means.

ddelange changed the title ~~Build ABI3 wheels containing libmagic~~ Build platform-specific wheels containing libmagic Sep 7, 2023

ddelange force-pushed the abi3-wheels branch 2 times, most recently from a98f13b to dc9c393 Compare September 7, 2023 07:26

ddelange mentioned this pull request Sep 7, 2023

Build platform-specific wheels containing libmagic ddelange/python-magic#1

Closed

ddelange force-pushed the abi3-wheels branch 12 times, most recently from d672b91 to 14f7dbb Compare September 7, 2023 10:03

Build platform-specific wheels containing libmagic

ec952d7

ddelange force-pushed the abi3-wheels branch from 14f7dbb to ec952d7 Compare September 7, 2023 10:09

jspraul mentioned this pull request Sep 9, 2023

Update file to 5.44 hey-red/Mime#49

Closed

apirogov mentioned this pull request Sep 12, 2023

python-magic has no wheels for Windows/Mac Materials-Data-Science-and-Informatics/metador-core#85

Open

stumpylog reviewed Sep 12, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

Move persmissions into job scope, remove ABI3 reference

a437409

ahupp mentioned this pull request Oct 10, 2023

Binary distribution for libmagic on Windows #293

Open

Switch to PyPI trusted publishing

4a715e2

ddelange mentioned this pull request Nov 23, 2023

bugfix: add path to load libmagic bin on Windows #249

Open

ddelange commented Nov 23, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

ferdnyc reviewed May 28, 2024

View reviewed changes

Apply suggestions from code review

da5b330

Co-authored-by: Frank Dana <ferdnyc@gmail.com>

ddelange force-pushed the abi3-wheels branch 5 times, most recently from 2822bca to 3d7698c Compare May 28, 2024 23:53

Revert partially: fix install on Windows

258efa4

ddelange force-pushed the abi3-wheels branch from 3d7698c to 258efa4 Compare May 29, 2024 07:43

Merge branch 'master' into abi3-wheels

3a55538

mikestaub mentioned this pull request Jun 19, 2024

support ARM chips on mac drforse/not-llama-fs#7

Closed

Privat33r-dev approved these changes Jun 25, 2024

View reviewed changes

Privat33r-dev reviewed Jun 26, 2024

View reviewed changes

ddelange commented Jun 26, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

Apply suggestions from code review

65fb61c

Privat33r-dev reviewed Jun 26, 2024

View reviewed changes

estarfoo approved these changes Jun 26, 2024

View reviewed changes

nitely mentioned this pull request Sep 16, 2024

django-spirit[files] python-magic-bin cannot be installed on mac silicon (m3) nitely/Spirit#331

Open

wilhelmklopp mentioned this pull request Oct 18, 2024

Dependency python-magic-bin seems to be unmaintained kolofordjango/kolo#95

Closed

ashtonpaul approved these changes Nov 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build platform-specific wheels containing libmagic #294

Build platform-specific wheels containing libmagic #294

ddelange commented Sep 6, 2023 •

edited

Loading

apirogov commented Sep 12, 2023

stumpylog Sep 12, 2023

ddelange Sep 12, 2023

ddelange Oct 26, 2023 •

edited

Loading

ddelange Nov 13, 2023

ddelange Nov 22, 2023

ahupp commented Sep 28, 2023

jean-humann commented Dec 3, 2023

ferdnyc May 28, 2024

ddelange May 28, 2024

ferdnyc May 29, 2024

ferdnyc May 29, 2024

ddelange commented May 29, 2024

Privat33r-dev left a comment

Privat33r-dev Jun 25, 2024

ddelange Jun 26, 2024

Privat33r-dev Jun 26, 2024 •

edited

Loading

Privat33r-dev Jun 26, 2024 •

edited

Loading

ddelange Jun 26, 2024

Privat33r-dev Jun 26, 2024 •

edited

Loading

Privat33r-dev Jun 26, 2024 •

edited

Loading

ddelange Jun 26, 2024

cclauss commented Jun 26, 2024 •

edited

Loading

estarfoo commented Jun 26, 2024

Privat33r-dev commented Jun 26, 2024

This comment was marked as resolved.

ferdnyc Jul 30, 2024

This comment was marked as resolved.

ddelange commented Jun 29, 2024

ashtonpaul commented Jul 31, 2024

ddelange commented Sep 2, 2024

wilhelmklopp commented Oct 18, 2024

cclauss commented Oct 18, 2024 •

edited

Loading

ferdnyc commented Oct 21, 2024

Build platform-specific wheels containing libmagic #294

Are you sure you want to change the base?

Build platform-specific wheels containing libmagic #294

Conversation

ddelange commented Sep 6, 2023 • edited Loading

pip install these wheels

CI/CD

apirogov commented Sep 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddelange Oct 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahupp commented Sep 28, 2023

jean-humann commented Dec 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddelange commented May 29, 2024

Privat33r-dev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Privat33r-dev Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

Privat33r-dev Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Privat33r-dev Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

Privat33r-dev Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cclauss commented Jun 26, 2024 • edited Loading

estarfoo commented Jun 26, 2024

Privat33r-dev commented Jun 26, 2024

This comment was marked as resolved.

Choose a reason for hiding this comment

This comment was marked as resolved.

ddelange commented Jun 29, 2024

ashtonpaul commented Jul 31, 2024

ddelange commented Sep 2, 2024

wilhelmklopp commented Oct 18, 2024

cclauss commented Oct 18, 2024 • edited Loading

ferdnyc commented Oct 21, 2024

ddelange commented Sep 6, 2023 •

edited

Loading

ddelange Oct 26, 2023 •

edited

Loading

Privat33r-dev Jun 26, 2024 •

edited

Loading

Privat33r-dev Jun 26, 2024 •

edited

Loading

Privat33r-dev Jun 26, 2024 •

edited

Loading

Privat33r-dev Jun 26, 2024 •

edited

Loading

cclauss commented Jun 26, 2024 •

edited

Loading

cclauss commented Oct 18, 2024 •

edited

Loading