Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Added New doc page for Flyte Type system and Data Passing #5867

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

10sharmashivam
Copy link
Contributor

Tracking issue

Reference #3322

Why are the changes needed?

This documentation aims to provide users with a comprehensive understanding of Flyte’s type system and data passing mechanisms, ensuring they can effectively leverage these features in their workflows.

What changes were proposed in this pull request?

•	Created a new documentation page detailing the Flyte type system and data passing.
•	Included code examples for primitive types, collections, and custom types to illustrate the concepts clearly.

Additional Comments-

I intend to submit a follow-up PR in flyteorg/flytesnacks to include additional code examples under flyteorg/flytesnacks/examples/basics/basics. Once that PR is reviewed and merged, we can use the raw GitHub user content link for the examples and incorporate them into the documentation page, following the format used in other files in the documentation.

Check all the applicable boxes

  • [ X] I updated the documentation accordingly.
  • All new and existing tests passed.
  • [ X] All commits are signed-off.

Signed-off-by: 10sharmashivam <10sharmashivam@gmail.com>
Copy link
Member

@Future-Outlier Future-Outlier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, can you follow the Docs link section like this PR?

#5844

Copy link

codecov bot commented Oct 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 34.96%. Comparing base (bdaf79f) to head (0f636cf).
Report is 16 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5867      +/-   ##
==========================================
- Coverage   36.71%   34.96%   -1.76%     
==========================================
  Files        1304     1143     -161     
  Lines      130081   122102    -7979     
==========================================
- Hits        47764    42695    -5069     
+ Misses      78147    75718    -2429     
+ Partials     4170     3689     -481     
Flag Coverage Δ
unittests-datacatalog 51.58% <ø> (ø)
unittests-flyteadmin 54.03% <ø> (-0.39%) ⬇️
unittests-flytecopilot 11.73% <ø> (ø)
unittests-flytectl ?
unittests-flyteidl 6.92% <ø> (+0.03%) ⬆️
unittests-flyteplugins 53.57% <ø> (-0.05%) ⬇️
unittests-flytepropeller 43.00% <ø> (+0.15%) ⬆️
unittests-flytestdlib 55.41% <ø> (+0.62%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@10sharmashivam
Copy link
Contributor Author

10sharmashivam commented Oct 21, 2024

Hi, can you follow the Docs link section like this PR?

#5844

@Future-Outlier

Thank you for the feedback! I’ve added the Docs link as requested:
Docs Link

Once this PR gets approved, I intend to submit a follow-up PR in flyteorg/flytesnacks to include additional code examples under flyteorg/flytesnacks/examples/basics/basics. Once that PR is reviewed and merged, we can use the raw GitHub user content link for the examples and incorporate them into the documentation page, following the format used in other files in the documentation.

I’ve also added basic sample examples in this PR to give an initial idea of how they can be incorporated into the new documentation page.

Please let me know if there are any further adjustments needed!

:lines: 1
```

### When to Use the Type System
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this section is needed. Sounds like there are situations when you don't need to use Flyte's type system and that's not the case. This is a core construct in Flyte

1. Metadata: Information about the data, such as the structure of the data (e.g., data types, paths to data storage).
2. Raw Data: The actual data itself (e.g., a Pandas DataFrame, a file in cloud storage).

### How Data is Handled in Workflow Execution
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this section repeats some of what is already covered here

@davidmirror-ops
Copy link
Contributor

@10sharmashivam thanks for volunteering, this one is not a small feat! Would you be open to have a call where we can chat more about Flyte's type system, hear your questions and then we can eventually update this PR?

@10sharmashivam
Copy link
Contributor Author

@10sharmashivam thanks for volunteering, this one is not a small feat! Would you be open to have a call where we can chat more about Flyte's type system, hear your questions and then we can eventually update this PR?

@davidmirror-ops I’d be happy to set up a call to discuss the Flyte-type system further and make sure the PR is aligned with Flyte’s core principles. I appreciate the opportunity to dive deeper into this, and I look forward to your insights!

@davidmirror-ops
Copy link
Contributor

I think timezone difference will make it a bit hard (I'm on UTC-5). Tomorrow (Wednesday 23) 9:00 a.m. UTC-5 would work for you?

@10sharmashivam
Copy link
Contributor Author

Yes, that time works for me. Looking forward to the call.

@davidmirror-ops
Copy link
Contributor

@10sharmashivam already synced with @eapolinario on this

  1. Dynamic typing: if this is how we understand it (types that only exist at runtime) this is not accessible today in flytekit but you have to declare type hints upfront.
  2. I think this section would be even better as a blog post on flyte.org. Would you be open to porting the content here to this doc? There, we can iterate and add you as a co-author in the Flyte blog. wdyt?

@10sharmashivam
Copy link
Contributor Author

@davidmirror-ops Thank you for the opportunity! I would be thrilled to co-author the blog post for flyte.org

…gle meet

Signed-off-by: 10sharmashivam <10sharmashivam@gmail.com>
@10sharmashivam
Copy link
Contributor Author

10sharmashivam commented Oct 24, 2024

Hi @davidmirror-ops,

I have added content about dynamic typing & Static typing in flyte in the given doc file, starting from 3rd page.

I also applied feedbacks and suggestions giving earlier, and discussed in our google meet. Like reference to related already existing docs. Doc Link

And I also submitted PR, about coding examples, which can be used (when approved, and merged) as raw github user content link, serving as code example snippets in the Flyte type system documentation.

Thank you!

Signed-off-by: 10sharmashivam <10sharmashivam@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants