GH-44461: [Release][Packacing][Python] Set PARQUET_TEST_DATA on verify-release-candidate-wheels.bat #44462

raulcd · 2024-10-17T13:39:57Z

Rationale for this change

The Windows wheel verification fails due to missing PARQUET_TEST_DATA

What changes are included in this PR?

Add PARQUET_TEST_DATA to verify-release-candidate-wheels.bat which is only tested on the binary verification job.

Are these changes tested?

Via archery

Are there any user-facing changes?

No

GitHub Issue: [Packaging][Release][Python] The Windows wheel verification is failing due to missing PARQUET_TEST_DATA env variable #44461

github-actions · 2024-10-17T13:40:26Z

⚠️ GitHub issue #44461 has been automatically assigned in GitHub to PR creator.

raulcd · 2024-10-17T13:40:36Z

@github-actions crossbow submit verify-rc-binaries-wheels-windows --param release=18.0.0 --param rc=0

github-actions · 2024-10-17T13:42:54Z

Revision: 6ba7a5a

Submitted crossbow builds: ursacomputing/crossbow @ actions-f980a8883a

Task	Status
verify-rc-binaries-wheels-windows

raulcd · 2024-10-17T13:59:51Z

The error has been fixed but there seems to be a new error with ORC and Timezones :( :

 >       result = dataset_reader.to_table(dataset)
C:\tmp\arrow-verify-release-wheels\_verify-wheel-3.9\Lib\site-packages\pyarrow\tests\test_dataset.py:3248: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\tmp\arrow-verify-release-wheels\_verify-wheel-3.9\Lib\site-packages\pyarrow\tests\test_dataset.py:267: in to_table
    return dataset.to_table(**kwargs)
pyarrow\\_dataset.pyx:562: in pyarrow._dataset.Dataset.to_table
    ???
pyarrow\\_dataset.pyx:3841: in pyarrow._dataset.Scanner.to_table
    ???
pyarrow\\error.pxi:155: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>   ???
E   pyarrow.lib.ArrowException: Unknown error: Time zone file /usr/share/zoneinfo/UTC does not exist. Please install IANA time zone database and set TZDIR env.
pyarrow\\error.pxi:92: ArrowException

raulcd · 2024-10-17T14:03:53Z

@github-actions crossbow submit verify-rc-binaries-wheels-windows --param release=18.0.0 --param rc=0

github-actions · 2024-10-17T14:06:08Z

Revision: 95b3016

Submitted crossbow builds: ursacomputing/crossbow @ actions-5687ee222c

Task	Status
verify-rc-binaries-wheels-windows

raulcd · 2024-10-17T14:16:55Z

dev/release/verify-release-candidate-wheels.bat

+@rem Download IANA Timezone Database for ORC C++
+curl https://cygwin.osuosl.org/noarch/release/tzdata/tzdata-2024a-1.tar.xz --output tzdata.tar.xz || exit /B
+mkdir %USERPROFILE%\Downloads\test\tzdata
+arc unarchive tzdata.tar.xz %USERPROFILE%\Downloads\test\tzdata


arc does not seem to be present:

'arc' is not recognized as an internal or external command, operable program or batch file.

There's some more work to be done here :(

dev/release/verify-release-candidate-wheels.bat

kou · 2024-10-17T21:11:53Z

@github-actions crossbow submit verify-rc-binaries-wheels-windows --param release=18.0.0 --param rc=0

kou · 2024-10-17T22:13:06Z

@github-actions crossbow submit verify-rc-binaries-wheels-windows --param release=18.0.0 --param rc=0

kou · 2024-10-18T00:23:01Z

@github-actions crossbow submit verify-rc-binaries-wheels-windows --param release=18.0.0 --param rc=0

github-actions · 2024-10-18T00:25:15Z

Revision: d43d971

Submitted crossbow builds: ursacomputing/crossbow @ actions-2851110c41

Task	Status
verify-rc-binaries-wheels-windows

kou · 2024-10-18T01:42:55Z

@github-actions crossbow submit verify-rc-binaries-wheels-windows --param release=18.0.0 --param rc=0

github-actions · 2024-10-18T01:45:08Z

Revision: acf0493

Submitted crossbow builds: ursacomputing/crossbow @ actions-628dc0a01d

Task	Status
verify-rc-binaries-wheels-windows

kou · 2024-10-18T02:22:16Z

@github-actions crossbow submit verify-rc-binaries-wheels-windows --param release=18.0.0 --param rc=0

github-actions · 2024-10-18T02:24:36Z

Revision: 6c3e4a5

Submitted crossbow builds: ursacomputing/crossbow @ actions-c2ac566e6f

Task	Status
verify-rc-binaries-wheels-windows

kou · 2024-10-18T02:43:16Z

@github-actions crossbow submit verify-rc-binaries-wheels-windows --param release=18.0.0 --param rc=0

github-actions · 2024-10-18T02:45:28Z

Revision: a753b3b

Submitted crossbow builds: ursacomputing/crossbow @ actions-55cf78c3ae

Task	Status
verify-rc-binaries-wheels-windows

kou · 2024-10-18T03:38:12Z

@github-actions crossbow submit verify-rc-binaries-wheels-windows --param release=18.0.0 --param rc=0

kou · 2024-10-18T07:02:36Z

It seems that we need to update bundled datetime...

… verify-release-candidate-wheels.bat

…for nightly wheels verification

raulcd · 2024-10-24T15:26:22Z

@github-actions crossbow submit verify-rc-binaries-wheels-windows --param release=18.0.0 --param rc=0

github-actions · 2024-10-24T15:28:44Z

Revision: 7200add

Submitted crossbow builds: ursacomputing/crossbow @ actions-2790dff4a5

Task	Status
verify-rc-binaries-wheels-windows

raulcd · 2024-10-24T16:10:11Z

oh! yeah! I'm not sure why I run that, the change won't be there :)

kou · 2024-10-24T21:50:25Z

Yes. I did the same mistake.

GH-44482 isn't included in 18.0.0 RC0. So this is still failing. But the PARQUET_TEST_DATA problem is fixed by this. So I think that we can merge this.

raulcd

I am 👍 with merging this, just a small question

raulcd · 2024-10-25T10:41:43Z

dev/tasks/verify-rc/github.win.yml

@@ -29,6 +29,7 @@ jobs:
      {{ key }}: {{ value }}
    {% endfor %}
    {% endif %}
+    timeout-minutes: 60


I am not against setting the timeout, any specific reason to add it? Those jobs only run on verifying binaries at the moment and it would be unfortunate to start hitting the timeout there

Some jobs such as https://github.com/ursacomputing/crossbow/actions/runs/11393132955/job/31700762815 got stuck when I was working on this. I canceled them manually because we don't have the timeout.

I think that we should have a reasonable timeout to avoid wasting CI time (even when we don't pay money for it). If "60" isn't enough, we can use more long time such as "120". It's better than the default "360".

ok, I went back to historic runs and the longest seems to be 52 minutes:
https://github.com/ursacomputing/crossbow/actions/runs/8704336160/usage
Others:
https://github.com/ursacomputing/crossbow/actions/runs/7507682263/usage
https://github.com/ursacomputing/crossbow/actions/runs/9837997876/usage
Probably 60 is good enough

conbench-apache-arrow · 2024-10-25T22:21:50Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 1b40800.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.

github-actions bot added the awaiting committer review Awaiting committer review label Oct 17, 2024

raulcd commented Oct 17, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Oct 17, 2024

kou reviewed Oct 17, 2024

View reviewed changes

dev/release/verify-release-candidate-wheels.bat Outdated Show resolved Hide resolved

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Oct 17, 2024

This comment was marked as outdated.

Sign in to view

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 18, 2024

raulcd and others added 12 commits October 24, 2024 11:28

apacheGH-44461: [Release][Packacing][Python] Set PARQUET_TEST_DATA on…

bb9d643

… verify-release-candidate-wheels.bat

Replicate on Windows wheel release candidate verification what we do …

51289a4

…for nightly wheels verification

Use tar

803e356

Set timeout

9af43db

Use zoneinfo by MSYS2

f8ff823

Use the latest tzdata

0b2ac1b

Fix URL

fc0b803

Debug

84aceab

Debug

293bcd9

Use old tzdata again

fe77f00

Use TZDIR from conda

f7ee4ca

Clean up

39084c9

kou force-pushed the GH-44461 branch from 04c0982 to 39084c9 Compare October 24, 2024 02:30

Remove debug print

7200add

raulcd marked this pull request as ready for review October 25, 2024 10:39

raulcd requested review from assignUser and jonkeane as code owners October 25, 2024 10:39

raulcd commented Oct 25, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 25, 2024

raulcd merged commit 1b40800 into apache:main Oct 25, 2024
19 checks passed

raulcd removed the awaiting changes Awaiting changes label Oct 25, 2024

raulcd mentioned this pull request Oct 25, 2024

[Packaging][Release][Python] The Windows wheel verification is failing due to missing PARQUET_TEST_DATA env variable #44461

Closed

raulcd deleted the GH-44461 branch October 25, 2024 14:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-44461: [Release][Packacing][Python] Set PARQUET_TEST_DATA on verify-release-candidate-wheels.bat #44462

GH-44461: [Release][Packacing][Python] Set PARQUET_TEST_DATA on verify-release-candidate-wheels.bat #44462

raulcd commented Oct 17, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Oct 17, 2024

raulcd commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

raulcd commented Oct 17, 2024

raulcd commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

raulcd Oct 17, 2024

kou commented Oct 17, 2024

This comment was marked as outdated.

kou commented Oct 17, 2024

This comment was marked as outdated.

kou commented Oct 18, 2024

github-actions bot commented Oct 18, 2024

kou commented Oct 18, 2024

github-actions bot commented Oct 18, 2024

kou commented Oct 18, 2024

github-actions bot commented Oct 18, 2024

kou commented Oct 18, 2024

github-actions bot commented Oct 18, 2024

kou commented Oct 18, 2024

kou commented Oct 18, 2024

raulcd commented Oct 24, 2024

github-actions bot commented Oct 24, 2024

raulcd commented Oct 24, 2024

kou commented Oct 24, 2024

raulcd left a comment

raulcd Oct 25, 2024

kou Oct 25, 2024

raulcd Oct 25, 2024

conbench-apache-arrow bot commented Oct 25, 2024

GH-44461: [Release][Packacing][Python] Set PARQUET_TEST_DATA on verify-release-candidate-wheels.bat #44462

GH-44461: [Release][Packacing][Python] Set PARQUET_TEST_DATA on verify-release-candidate-wheels.bat #44462

Conversation

raulcd commented Oct 17, 2024 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Oct 17, 2024

raulcd commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

raulcd commented Oct 17, 2024

raulcd commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

raulcd Oct 17, 2024

Choose a reason for hiding this comment

kou commented Oct 17, 2024

This comment was marked as outdated.

kou commented Oct 17, 2024

This comment was marked as outdated.

kou commented Oct 18, 2024

github-actions bot commented Oct 18, 2024

kou commented Oct 18, 2024

github-actions bot commented Oct 18, 2024

kou commented Oct 18, 2024

github-actions bot commented Oct 18, 2024

kou commented Oct 18, 2024

github-actions bot commented Oct 18, 2024

kou commented Oct 18, 2024

kou commented Oct 18, 2024

raulcd commented Oct 24, 2024

github-actions bot commented Oct 24, 2024

raulcd commented Oct 24, 2024

kou commented Oct 24, 2024

raulcd left a comment

Choose a reason for hiding this comment

raulcd Oct 25, 2024

Choose a reason for hiding this comment

kou Oct 25, 2024

Choose a reason for hiding this comment

raulcd Oct 25, 2024

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Oct 25, 2024

raulcd commented Oct 17, 2024 •

edited by github-actions bot

Loading