-
Notifications
You must be signed in to change notification settings - Fork 221
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] support reader doris using arrow flight driver #465
[Feature] support reader doris using arrow flight driver #465
Conversation
...ris-connector/src/main/java/org/apache/doris/flink/source/reader/DorisFlightValueReader.java
Outdated
Show resolved
Hide resolved
flink-doris-connector/src/main/java/org/apache/doris/flink/rest/SchemaUtils.java
Show resolved
Hide resolved
flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisConfigOptions.java
Outdated
Show resolved
Hide resolved
...ris-connector/src/main/java/org/apache/doris/flink/source/reader/DorisSourceSplitReader.java
Outdated
Show resolved
Hide resolved
partition.getTabletIds().stream() | ||
.map(Object::toString) | ||
.collect(Collectors.joining(",")); | ||
sql += " TABLET(" + tablet + ") "; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why does the tablet need to be spliced in here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In DorisSource.createEnumerator() method, readers split it by node. Currently, doris does not support specifying be node to read data for the time being, so you need to specify a tablet to query
protected AtomicBoolean eos = new AtomicBoolean(false); | ||
|
||
public DorisFlightValueReader( | ||
PartitionDefinition partition, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think arrow flight is a new reading method, should we no longer rely on the old reading method?
PartitionDefinition
This class was created using the old reading method. Should we give up using this class?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Obtaining the be node arrow readerInfo instance through ticket returns an exception, so the old reading form is temporarily used
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good job, Thanks @MaoMiMao, LGTM
Proposed changes
Support read doris using arrow flight driver, such as:
Checklist(Required)
Further comments
1.motivation by this issue motivation
2. split reader idea by this issue apache/doris#10170
Actual effect after completion
use clickbatch hits table