Release 3.3.5

Overview

Added full complex type export to Parquet support and fixed some bugs.

Changes

• #522 Fixed an issue that caused a Spark Worker UI problem from conflicting dependencies.

  • Additionally, some extra variables were added to the docker-compose.yml to make the UI more accessible via #523.

• #515 Small change to ensure that Vertica container works on ARM based machines.

• #499 Added full support of complex type export to Parquet for the release of Vertica 12.0.2.

Release 3.3.4

Overview

Added a Jupyter Notebook Docker container plus truncation support when writing in overwrite mode.

• #488 Added a Jupyter Notebook container as an option for writing and executing new examples
• #507 Support table truncate when writing in overwrite mode

Release 3.3.3

Bug fixes

• #494 Correctly set JDBC connection client label so that data collection tables report it

Non-functional changes

• #470 Create a new video showing how to use the current Spark Connector (linked in the README)
• #471 Create a blog post that describes the current Spark Connector features, and how to use it (see here)

Release 3.3.2

Bug Fixes

This patch release contains the fix for #475.

Release 3.3.1

Overview

This patch release added support for Spark 3.3.0, allowing the query option to be used with dbschema, as well as other fixes.

Support Spark 3.3.0

The connector is now compatible with Spark 3.3.0 while maintaining backward compatibility with previous Spark 3.x

Make a query with dbschema option

With the option query and dbschema defined, the schema specified in dbschema will be appended to the table name of the query's FROM clause. This will also work when the dbschema is a string literals.

However, dbschema will not be appended if:

1. The FROM clause is a query.
2. The FROM clause already has a schema defined.

Bug fixes

• Fixes #443. The connector should now be able to handle empty element names.
• Fixes #463. Reading an empty table using Json format should no longer throw an exception.

Release 3.3.0

Overview

The connector can now read supported complex types like:

• Rows type.
• Complex arrays (any arrays that are not 1D arrays of primitives).

When reading complex types, all of the data will be exported as JSON files instead of Parquet files. The decision is done when complex type columns are detected, however the option json is available which will force the connector to always read using JSON exports.

For map type, the connector can only write to an external table but cannot read them. Vertica suggests using Array[Row(key, value)] instead for use in internal tables.

Requirements:
Reading complex data types require at least Vertica 11.1.1-0.

JSON Readers

For reading JSON files, we delegate to Spark's DataSource V2 to handle the reading. This includes partitioning the exported files and creating readers. On close, all exported files will be deleted using the same behaviours as that of reading Parquet files.

Release 3.2.2

Changes

• Added compatibility with Vertica 11.1:
  • An SQL method used by the connector was deprecated in Vertica 11.1.1, making it incompatible with previous connector releases. This release supports Vertica's new method as well as maintains backward compatibility.
• Fixed an issue with the connector creating many JDBC connections:
  • We now expect read/write operations to only create at most 2 JDBC connections to Vertica.

Release 3.2.1

Overview

This release includes support for maps and a fix for the connector with Amazon EMR.

Map Support

Support for writing map into Vertica is added. However, it is very limited. You can only write maps to external tables and maps cannot be queried. Thus we cannot support writing maps to internal tables or reading maps from external and internal tables.

These limitations are from Vertica. As of Vertica 11.x map type is not supported and exists like it it is only for interoperability with external data.

It is suggested by Vertica to use Array[Row(key, value)] to represent maps inside Vertica.

Bug fixes

• Fixed the issue where the connector failed to resolve Spark's version string when it contains non-numeric characters e.g. 3.2.1-amz-0
• Fixed the issue with Spark connector failing to detect complex type columns when used against Vertica 10.

Vertica Spark Connector V3.2.0 Release

Overview

This release contains support for writing Spark structs as Vertica rows, more user-friendly error messages, and bug fixes

Row Write Support

Spark structs can be written into Vertica as rows. Struct fields can be of primitive types or supported complex types. Note that row write is only supported for Vertica 11.x and newer.

Error Messages

More user-friendly errors message were also added. This include errors for Vertica incompatibility for when a complex type is used, as well as schema complex type schema errors.

Bug Fixes

• Fixed compatibility issue with Spark 3.2.1
• Fixed connector JDBC error when dbschema option is specified.

Vertica Spark Connector V3.1.0 Release

Overview

This release contains supports for array data type.

Array Support

Arrays support requires Vertica JDBC Connector 11 and is backwards compatible only with Vertica 10 with some restrictions. Refer to the readme for more information.

• Supports writing both native and complex arrays to Vertica.
• Support reading native arrays from Vertica.
• Added array_length option to specify a default array length value when writing to Vertica.

Fixes

• Fixes an issue with the Connector not properly deleting all files files written to staging area.