[Checkpoint] Fix symlink issue where symlink file uploaded before checkpoint files upload #3376
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… into checkpoint_saver
bigning
changed the title
bigning
changed the title
b-chu
reviewed
Jun 21, 2024
eracah
suggested changes
Jun 21, 2024
There was a problem hiding this comment.
This PR introduces a lot of code debt by copying and pasting a lot of the logic from remote_uploader into checkpoint_saver with the only difference being that the files being uploaded are symlinks. Since remote symlinks are just text files we need to find a way to upload symlinks using the remote_uploader before we can merge this in.
eracah
reviewed
Jun 28, 2024
eracah
reviewed
Jun 28, 2024
mvpatel2000
approved these changes
Jul 8, 2024
mvpatel2000
pushed a commit
to mvpatel2000/composer
that referenced
this pull request
Jul 21, 2024
…ckpoint files upload (mosaicml#3376) * a * a * a * a * a * a * a * a * fix test * a * a * a * a * fix unit test * a * a * a * a * a * fix 2gpu unit test * a * a * a * a * fix doctest * a * fix test and lint * up * a * a * a * a * a * a * a * a * address comments * a * a * a * a * rerun test * add logging * remove debug comments * comments * a * cleanup * a * linter * lint * Update composer/callbacks/checkpoint_saver.py Co-authored-by: Evan Racah <evan.racah@databricks.com> * commenst * a * fix test * fix test * comments * a --------- Co-authored-by: Evan Racah <evan.racah@databricks.com>
mvpatel2000
pushed a commit
that referenced
this pull request
Jul 21, 2024
…ckpoint files upload (#3376) * a * a * a * a * a * a * a * a * fix test * a * a * a * a * fix unit test * a * a * a * a * a * fix 2gpu unit test * a * a * a * a * fix doctest * a * fix test and lint * up * a * a * a * a * a * a * a * a * address comments * a * a * a * a * rerun test * add logging * remove debug comments * comments * a * cleanup * a * linter * lint * Update composer/callbacks/checkpoint_saver.py Co-authored-by: Evan Racah <evan.racah@databricks.com> * commenst * a * fix test * fix test * comments * a --------- Co-authored-by: Evan Racah <evan.racah@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fix the symlink issues.
How?
[updated]: in the checkpoint saver, on rank-0 which saves the symlink, it all_gather the remote checkpoint file names, and start a new process to check if those remote files finish uploading by calling
object_store.get_object_size. It only upload symlink file once all the remote file finish uploading. This way:Unit test
Integration test
2-nodes OCI:
save: test-uploader-0Tkv9O
autoresume: test-uploader-yac88U
2-nodes mflow:
save: l38bi-full-sweep-train-bb-1-0e-6-5-VMY2Xo
load: l38bi-full-sweep-train-bb-1-0e-6-5-TyRcPi
Daily test:
https://github.com/mosaicml/composer/actions/runs/9700144963
composer regression test:
https://github.com/databricks-mosaic/regression-testing/actions/runs/9700161085
Perf test (100 batches with 9 batch save interval. The training time varies because of unstable uploading speed, but just want to make sure test didn't regress training time)
64 gpu test: 77b-bs1024-g512-res2-f60-37gSXK time: 26 minutes
64 GPU baseline: 77b-bs1024-g512-res2-f60-2VZPxB time: more than 40 minutes because rank 29 uploading delay
In case 1 rank upload fails, it won't hang:
test-uploader-MXcoEp