FR: add Snowflake adapter

[kylelundstedt]

Hi @tconbeer! Just started playing with Harlquin; seems very cool.
In addition to DuckDB, we also use Snowflake. Any chance we'll see an adapter for it?

Thanks!

[tconbeer]

Yes, for sure. Any interest in contributing it? Otherwise I may get to it in a while, after a few other features and an odbc adapter. Happy to help you if you want to write it

[rupurt]

It would be awesome to implement this as an ADBC adapter. Snowflake has support and there is an Arrow Snowflake driver https://arrow.apache.org/adbc/0.5.1/driver/snowflake.html

[tconbeer]

Agreed that Arrow is the way to go. Harlequin's data table is backed by arrow, so results returned as arrow tables will load faster. Snowflake connector 2.2+ also lets you fetch results as arrow with fetch_arrow_all.

[rymurr]

Ive started working on one here: https://github.com/rymurr/harlequin-snowflake

Basics are workign but the catalog is very slow.

[TylerHillery]

@rupurt I have implemented an ADBC adapter here and I got it to work with my Snowflake account but it appears others are having issues. I can't seem to reproduce on my end so would love some more people to test it out.

I've tired adding to Harlequin pyproject.toml as an extra but the ADBC adapter requires python 3.9 or greater so it wouldn't let me. I also just haven't gotten around to contributing to the docs yet.

[tconbeer]

I've tired adding to Harlequin pyproject.toml as an extra but the ADBC adapter requires python 3.9 or greater so it wouldn't let me. I also just haven't gotten around to contributing to the docs yet.

@TylerHillery you can add a python version constraint to the extra dependency, like this:

harlequin-adbc = { version = "^0.1", python = ">=3.9.0", optional = true }

However, I tried that just now, and there is a dependency issue with PyArrow versions; harlequin-databricks basically requires pyarrow 14 (via databricks-sql-connector), and you require pyarrow 15:

    Because harlequin-databricks (0.1.1) depends on databricks-sql-connector (>=3.0.3,<4.0.0)
 and no versions of harlequin-databricks match >0.1,<0.1.1 || >0.1.1,<0.2, harlequin-databricks (>0.1,<0.2) requires databricks-sql-connector (>=3.0.3,<4.0.0).
(1) So, because harlequin-databricks (0.1.0) depends on databricks-sql-connector (>=3.0.3,<4.0.0), harlequin-databricks (>=0.1,<0.2) requires databricks-sql-connector (>=3.0.3,<4.0.0).

    Because no versions of databricks-sql-connector match >3.0.3,<3.1.0 || >3.1.0,<3.1.1 || >3.1.1,<4.0.0
 and databricks-sql-connector (3.1.0) depends on pyarrow (>=14.0.1,<15.0.0), databricks-sql-connector (>3.0.3,<3.1.1 || >3.1.1,<4.0.0) requires pyarrow (>=14.0.1,<15.0.0).
    And because databricks-sql-connector (3.1.1) depends on pyarrow (>=14.0.1,<15.0.0), databricks-sql-connector (>3.0.3,<4.0.0) requires pyarrow (>=14.0.1,<15.0.0).
    And because harlequin-databricks (>=0.1,<0.2) requires databricks-sql-connector (>=3.0.3,<4.0.0) (1), harlequin-databricks (>=0.1,<0.2) requires pyarrow (>=14.0.1,<15.0.0) or databricks-sql-connector (3.0.3)
    And because databricks-sql-connector (3.0.3) depends on pyarrow (>=14.0.1,<15.0.0), harlequin-databricks (>=0.1,<0.2) requires pyarrow (>=14.0.1,<15.0.0).
    And because harlequin-adbc (0.1.0) depends on pyarrow (>=15.0.0)
 and no versions of harlequin-adbc match >0.1,<0.2, harlequin-databricks (>=0.1,<0.2) is incompatible with harlequin-adbc (>=0.1,<0.2).
    So, because harlequin depends on both harlequin-databricks (^0.1) and harlequin-adbc (^0.1), version solving failed.

I think if you can loosen your dependency to pyarrow >= 14, then we should be good to go.

[TylerHillery]

I think if you can loosen your dependency to pyarrow >= 14, then we should be good to go.

Sounds good I should have some time next week to submit a PR

[tconbeer]

Just FYI, I'm cleaning up some dependabot stuff and merged a couple other changes, so please be sure to rebase onto the latest main commit before updating pyproject.toml and running poetry lock or you're in for a world of git pain.