Python 3.12 support blocked by aiohttp

[martin-traverse]

Is your feature request related to a problem? Please describe.

To use async IO with the azure-core library there is a dependency on aiohttp. The aiohttp project currently does not publish wheels for Python 3.12. Although the async transport is an optional dependency of azure-core, it is a hard dependency of the Azure fsspec implementation, adlfs. We would like to support Python 3.12 on Azure and cannot do so until this issue is resolved. This will also be impacting other projects that require async transport.

The lead maintainers of the aiohttp project are saying they need help to make the necessary updates, and that there are significant testing gaps which they are concerned about and wish to fix before a 3.12-compatible version can be released. To drive the point home, they also published this image in a recent thread on the subject:

Describe the solution you'd like

I think this is a broader issues than simply getting the 3.12 package for aiohttp out of the door. I can see two potential solutions:

• MS can contribute development effort to aiohttp, to resolve this specific issue and help maintain the package going forward as and when issues or feature requirements arise
• I see there is an alternate async transport in development using HTTPX, with sample code in the azure-core-experimental library and also in a separate module called "corehttp". If the HTTPX transport were completed and made available in azure-core that should remove the dependency on aiohttp for projects requiring async, including the Azure implementation of fsspec.

Describe alternatives you've considered

The Apache Arrow project is also working on an Azure implementation of their FileSystem interface, for projects using Apache Arrow these Python libraries may not be needed at all once that is available. This might help my particular project eventually, but probably doesn't help most people who want to use Python on Azure.

Additional context

Issue thread for aiohttp: aio-libs/multidict#887
Note in adlfs about requirement of aiohttp: fsspec/adlfs#452
Azure core docs on HTTP transport: CLIENT_LIBRARY_DEVELOPER#transport
Azure core experimental library: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/core/azure-core-experimental
Azure corehttp library: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/core/corehttp

[xiangyan99]

Thanks for reaching out.

It seems to me aiohttp has released 3.12 support. https://pypi.org/project/aiohttp/#files

Did I miss anything?

[github-actions]

Hi @martin-traverse. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[martin-traverse]

There is a thing called "multidict", a library which is part of the AIO project and sits near the bottom of the stack. The aiohttp package has a hard dependency on multidict, and multidict is not available for Python 3.12.

https://github.com/aio-libs/multidict
https://pypi.org/project/multidict/#files

Running pip install aiohttp, or pip install adlfs or even pip install azure-core[aio] on Python 3.12 will trigger a build for multidict, which fails if you don't have the C build chain installed and configured.

[kristapratico]

@kashifkhan

[xiangyan99]

There was a new multidict library released which supported 3.12 about half hours ago.
(I saw you were in the thread so maybe you knew it already).

[github-actions]

Hi @martin-traverse. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[martin-traverse]

The issue in aiohttp is now resolved which addresses the immediate problem, Python 3.12 now works with azure-core AIO so fsspec (adlfs) and anything else using async storage is unblocked (including my own project). I am happy if this issue is closed.

I do think it is worth discussing the broader point around the open source dependency tree. There are obviously big upsides to a shared ecosystem of common components but two major downsides are inconsistent support model and proliferation of dependency bloat. These issues can be mitigated in specific cases, either by contributing upstream or selecting dependencies with a well defined support model and small transitive dependency tree.

In this case both options are available, ether working with the upstream project or selecting alternate dependencies could help alleviate issues before they arise.

[github-actions]

Hi @martin-traverse, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.