Migrate to pyproject.toml and hatchling

[JoshMock]

I used Hatch to convert setup.py into pyproject.toml, and then did some cleanup to drop dev-requirements.txt, ensure nox scripts keep running, etc.

[pquentin]

Thanks, moving from setup.py to pyproject.toml is great! Since the CI setups of elasticsearch-py and elasticsearch-serverless-are based on what urllib3 was doing in 2021, I have opinions. 😬

The main thing I'm wondering about is inlining scripts inside pyproject.toml. While setup.py is bad because it requires executing arbitrary Python code and building a wheel to learn about simple facts such as the package name, I don't think everything should move to pyproject.toml:

• In particular, I think keeping scripts in a Python file where they can be highlighted, linted, formatted and expanded easily is a better option. (And I like using scripts, I find using nox much simpler than tox, which planned at some point to support scripts too, but apparently that never happened.)
• Additionally, moving to Hatch the project manager is harder to undo that just moving to Hatchling the build backend. I prefer using small tools that can do one thing well.

But then I am biased as I've been using nox for a long time now. And apparently Hatch is so easy to use that this is the second time I'm seeing a coworker moving to it :) elastic/rally-tracks#289

[JoshMock]

@pquentin Good points! And all things I was hoping to uncover by opening a draft PR.

I'm more than happy to keep things in Nox. I really like how Hatch manages envs and their dependencies, but it's more of a convenience than a necessity.

Part of the inspiration here was that elasticsearch-py has an open issue where Seth seconded the use of Hatch to help migrate away from setup.py. But your point about just using Hatchling for packaging strikes a nice balance. I'll see about shifting back that direction.

[JoshMock]

@pquentin cleaned things up a bit. what do you think?

[JoshMock]

I kept Hatch as a dependency for version bumping and build targets, but I assume I could drop tool.hatch.version and tool.hatch.build.targets.sdist from the config if you only wanted to use hatchling.

[JoshMock]

Actually I just talked myself out of tool.hatch.version. We won't really need to automate versioning of this package since it will not need to stay in sync with the ES stack version.

[pquentin]

Thanks! Is there any way to test the CI parts?

[pquentin]

Is this really needed since those dependencies are now in docs extra requirements?

[JoshMock]

nox -s docs was failing due to missing config values that sphinx-rtd-theme needed. I just looked into it more closely and needed to enforce versions on sphinx and the theme to work around it. Fixed in 16c972f.

[pquentin]

I wondered what was going and investigated too to be able to tell when we can drop the pins. As you've noticed, the root cause is that the two sphinx packages that we install currently require incompatible versions of Sphinx:

• sphinx-rtd-theme 1.2.2 (the last release) supports Sphinx >=1.6,<7: https://github.com/readthedocs/sphinx_rtd_theme/blob/9899ee4ee2f547f81e51297dc12317f018e62fdd/setup.cfg#L48
• sphinx-autodoc-typehints 1.24.0 (the last release) supports Sphinx>=7.0.1: https://github.com/tox-dev/sphinx-autodoc-typehints/blob/a41715889940c8f8c37b32bac59db9416f47c958/pyproject.toml#L41

Then pip considers that the plugin "at fault" is sphinx-rtd-theme. I'm not sure how the resolver works, maybe it's the only package that does not support Sphinx 7. Anyway, pip then tries to find a version that does in fact support Sphinx 7:

INFO: pip is looking at multiple versions of sphinx-rtd-theme to determine which version is compatible with other requirements. This could take a while.
Collecting sphinx-rtd-theme (from elasticsearch-serverless==0.1.0)
  Obtaining dependency information for sphinx-rtd-theme from https://files.pythonhosted.org/packages/c2/73/946cdd32f29c87843edaa059b9fc52ab9267d1137b3a76dee1d5
9d02a01a/sphinx_rtd_theme-1.2.1-py2.py3-none-any.whl.metadata                  
  Using cached sphinx_rtd_theme-1.2.1-py2.py3-none-any.whl.metadata (4.5 kB)
  Using cached sphinx_rtd_theme-1.2.0-py2.py3-none-any.whl (2.8 MB)
  Using cached sphinx_rtd_theme-1.1.1-py2.py3-none-any.whl (2.8 MB)
  Using cached sphinx_rtd_theme-1.1.0-py2.py3-none-any.whl (2.8 MB)
  Using cached sphinx_rtd_theme-1.0.0-py2.py3-none-any.whl (2.8 MB)
  Using cached sphinx_rtd_theme-0.5.2-py2.py3-none-any.whl (9.1 MB)
  Using cached sphinx_rtd_theme-0.5.1-py2.py3-none-any.whl (2.8 MB)

Since sphinx-rtd-theme 0.5.1 is the latest release without bounds, pip considers that it does support Sphinx 7. And then installs it. This is something we are likely to continue seeing in the future, given:

• Sphinx releasing new major version ~every 6 months
• sphinx-autodoc-typehints always requiring the latest Sphinx release (Sudden incompatibility for Sphinx<7 tox-dev/sphinx-autodoc-typehints#371)
• sphinx-rtd-theme being a bit behind on latest Sphinx support (1.3.0 is not released yet)

I can see two ways out of this issue:

1. We can use a different theme like which updates faster than sphinx-autodoc-typehints (but does not use a lower bound): https://github.com/pradyunsg/furo.

2. We can pin Sphinx ourselves, and periodically bump to the latest version that sphinx-rtd-theme supports:

   docs = [                                                                                                                                                       
       # sphinx-rtd-theme does not support Sphinx 7 yet                                                                                                           
       "sphinx<7",                                                                                                                                                
       "sphinx-rtd-theme",                                                                                                                                        
       "sphinx-autodoc-typehints",
   ]

I think I prefer the second approach. Although it does not have to block the pull request, we can do this later as this pull request is otherwise ready to be merged IMO.

[pquentin]

This looks great now! Thanks for iterating.

[JoshMock]

Of course! Thanks for all the useful feedback @pquentin. First time migrating from a setup.py. 😄