Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Proof-of-concept Parquet GEOMETRY logical type implementation #43977

Open
wants to merge 63 commits into
base: main
Choose a base branch
from
Open

Proof-of-concept Parquet GEOMETRY logical type implementation #43977

wants to merge 63 commits into from

Commits on Aug 12, 2024

  1. update thrift

    @paleolimbot
    paleolimbot committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    4a4ebc9 View commit details
    Browse the repository at this point in the history

  2. update thrift defs

    @paleolimbot
    paleolimbot committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    6f4e4b7 View commit details
    Browse the repository at this point in the history

  3. add stubs

    @paleolimbot
    paleolimbot committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    6acb2f2 View commit details
    Browse the repository at this point in the history

  4. split methods out of line

    @paleolimbot
    paleolimbot committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    6ff5855 View commit details
    Browse the repository at this point in the history

  5. maybe to/from thrift

    @paleolimbot
    paleolimbot committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    0ac5d84 View commit details
    Browse the repository at this point in the history

  6. a few more serializers

    @paleolimbot
    paleolimbot committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    8a80ac7 View commit details
    Browse the repository at this point in the history

  7. add basic test for serialization

    @paleolimbot
    paleolimbot committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    f330997 View commit details
    Browse the repository at this point in the history

  8. add sort order check

    @paleolimbot
    paleolimbot committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    b34b1c2 View commit details
    Browse the repository at this point in the history

  9. update thrift

    @paleolimbot
    paleolimbot committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    7e9f9b2 View commit details
    Browse the repository at this point in the history

  10. start geom utiles

    @paleolimbot
    paleolimbot committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    6a9ac3a View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. test roundtrip thrift cases

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    a22908f View commit details
    Browse the repository at this point in the history

  2. more geometry utils

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    dcc083a View commit details
    Browse the repository at this point in the history

  3. more

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    9acc840 View commit details
    Browse the repository at this point in the history

  4. bounder

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    e18412c View commit details
    Browse the repository at this point in the history

  5. add basic test

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    76b3f59 View commit details
    Browse the repository at this point in the history

  6. a few more strings

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    fc77ff2 View commit details
    Browse the repository at this point in the history

  7. test some bounding box things

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    21f11a1 View commit details
    Browse the repository at this point in the history

  8. more tests

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    6bbce5c View commit details
    Browse the repository at this point in the history

  9. fix test

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    bb41b06 View commit details
    Browse the repository at this point in the history

  10. with passing tests

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    88a42f5 View commit details
    Browse the repository at this point in the history

  11. add in WKT equiv

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    3798ef1 View commit details
    Browse the repository at this point in the history

  12. more tests

    @paleolimbot
    paleolimbot committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    ccbd616 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2024

  1. start on stats

    @paleolimbot
    paleolimbot committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    3752f83 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2024

  1. implement update/merge for geometry statistics

    @paleolimbot
    paleolimbot committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    2ff6078 View commit details
    Browse the repository at this point in the history

  2. more complete stats

    @paleolimbot
    paleolimbot committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    79e4715 View commit details
    Browse the repository at this point in the history

  3. start on factory methods

    @paleolimbot
    paleolimbot committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    4444757 View commit details
    Browse the repository at this point in the history

  4. more stats things

    @paleolimbot
    paleolimbot committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    176c997 View commit details
    Browse the repository at this point in the history

  5. maybe work with serde

    @paleolimbot
    paleolimbot committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    1de193e View commit details
    Browse the repository at this point in the history

  6. Update cpp/src/parquet/types.cc 

    Co-authored-by: Gang Wu <ustcwg@gmail.com>
    @paleolimbot @wgtmac
    paleolimbot and wgtmac authored Aug 19, 2024
    Configuration menu
    Copy the full SHA
    d2f8157 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Updated parquet.thrift and re-generated cpp sources

    @Kontinuation
    Kontinuation committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    b525e84 View commit details
    Browse the repository at this point in the history

  2. Geometry value writer could make use of the geometry statistics class to 

    populate geometry statistics
    @Kontinuation
    Kontinuation committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    6f0500e View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Geometry column writer now populates correct statistics

    @Kontinuation
    Kontinuation committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    c052ae0 View commit details
    Browse the repository at this point in the history

  2. format/tidy

    @Kontinuation
    Kontinuation committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    3c6b222 View commit details
    Browse the repository at this point in the history

  3. Run clang-tidy

    @Kontinuation
    Kontinuation committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    f6ae9ae View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Added a test that writes and reads a parquet file containing a geomet… 

    …ry column
    @Kontinuation
    Kontinuation committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    1c9523b View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Remove redundant include

    @Kontinuation
    Kontinuation committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    5a50790 View commit details
    Browse the repository at this point in the history

  2. Fix problems found by reviewers

    @Kontinuation
    Kontinuation committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    51e4ab8 View commit details
    Browse the repository at this point in the history

  3. Try to make it build properly on other platforms

    @Kontinuation
    Kontinuation committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    c40c04e View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into kontinuation-parquet-geometry

    @Kontinuation
    Kontinuation authored Sep 6, 2024
    Configuration menu
    Copy the full SHA
    e89962f View commit details
    Browse the repository at this point in the history

  5. Address review comments in apache#43196

    @Kontinuation
    Kontinuation committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    fb134d3 View commit details
    Browse the repository at this point in the history

  6. Resolve compile errors for MSVC

    @Kontinuation
    Kontinuation committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    2f4329e View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Expose getters in GeometryStatistics, Change geometry_types from 

    std::vector<uint32_t> to std::vector<int32_t>, several other minor fixes.
    @Kontinuation
    Kontinuation committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    1db855f View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Add test case for UpdateSpaced, don't generate min/max stats for geom… 

    …etry

columns.
    @Kontinuation
    Kontinuation committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    ad92bb6 View commit details
    Browse the repository at this point in the history

  2. Support covering

    @Kontinuation
    Kontinuation committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    f782e30 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. MakeStatistics and Statistics::Make should not be a breaking change

    @Kontinuation
    Kontinuation committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    9813f48 View commit details
    Browse the repository at this point in the history

  2. ColumnIndex, as well as some other fixes and refacturings

    @Kontinuation
    Kontinuation committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    c56133c View commit details
    Browse the repository at this point in the history

  3. Fix compiler warnings on AMD platforms as well as sanitizer warnings

    @Kontinuation
    Kontinuation committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    174e1e1 View commit details
    Browse the repository at this point in the history

  4. Remove all newly added include directives

    @Kontinuation
    Kontinuation committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    bd0e2ad View commit details
    Browse the repository at this point in the history

  5. include cmath for std::isnan

    @Kontinuation
    Kontinuation committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    1521bac View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. Test writing WKB encoded geometries using WriteArrow

    @Kontinuation
    Kontinuation committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    572e865 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Change the sort order of geometry from unknown to unsigned; resolved … 

    …several review comments
    @Kontinuation
    Kontinuation committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    6c322d5 View commit details
    Browse the repository at this point in the history

  2. Add generate_covering_ member to be explicit that' we'll generate the… 

    … coverings from bounding box when populating the encoded statistics
    @Kontinuation
    Kontinuation committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    cd43ba5 View commit details
    Browse the repository at this point in the history

  3. Refactor unscoped enums in geometry_util_internal to enum classes

    @Kontinuation
    Kontinuation committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    33803bc View commit details
    Browse the repository at this point in the history

  4. Revert more special case handling for unknown sort order

    @Kontinuation
    Kontinuation committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    31a70c5 View commit details
    Browse the repository at this point in the history

  5. Fix WKB covering test to take native endianness into consideration

    @Kontinuation
    Kontinuation committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    7dfbf4b View commit details
    Browse the repository at this point in the history

  6. min/max of geometry columns are the WKB representation of lower-left … 

    …and upper-right points
    @Kontinuation
    Kontinuation committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    5d8ab77 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Address latest review comments

    @Kontinuation
    Kontinuation committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ae5926f View commit details
    Browse the repository at this point in the history

  2. A better implementation of geometry min/max statistics

    @Kontinuation
    Kontinuation committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    672f19c View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Update the code to accomodate the latest changes of the standard: 

    1. Remove metadata property of geometry logical types
2. Remove covering from geometry statistics
    @Kontinuation
    Kontinuation committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    26ba162 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. Fix problem decoding WKB geometries with more than 32 coordinates

    @Kontinuation
    Kontinuation committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    fe8a3e5 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2024

  1. Re-implemented geometry statistics according to the updated spec: 

    1. geometry statistics moved out of statistics, it is now a field of column metadata
2. geometry statistics is removed from page index
    @Kontinuation
    Kontinuation committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    ba80f3e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into kontinuation-parquet-geometry

    @Kontinuation
    Kontinuation committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    9aca79d View commit details
    Browse the repository at this point in the history

  3. Revert some unnecessary changes

    @Kontinuation
    Kontinuation committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    da55a55 View commit details
    Browse the repository at this point in the history