Flink CDC MySQL->Paimon 表中存在大写字母如何处理？

[beryllw]

Flink CDC MySQL->Paimon ，表中的字段是大写的，但是Paimon Hive catalog存在大小写敏感检查，会报错，如何处理？
我们能否在route中增加配置，支持字段默认转小写？

Caused by: java.lang.IllegalArgumentException: Field name [XXXX_ID, ...] cannot contain upper case in the catalog.
	at org.apache.paimon.utils.Preconditions.checkArgument(Preconditions.java:127) ~[flink-cdc-pipeline-connector-paimon.jar:3.2.0]
	at org.apache.paimon.catalog.AbstractCatalog.validateCaseInsensitive(AbstractCatalog.java:455) ~[flink-cdc-pipeline-connector-paimon.jar:3.2.0]
	at org.apache.paimon.catalog.AbstractCatalog.validateFieldNameCaseInsensitive(AbstractCatalog.java:482) ~[flink-cdc-pipeline-connector-paimon.jar:3.2.0]
	at org.apache.paimon.catalog.AbstractCatalog.createTable(AbstractCatalog.java:236) ~[flink-cdc-pipeline-connector-paimon.jar:3.2.0]

[Pluto201]

同样的问题。。

[yuxiqian]

Currently there's no other way than writing route rules to rename table names.

Option A: Add a route configuration indicating if table names should be converted to lowercase.
Option B: Add a Paimon connector option to do this job.

If Paimon connector is the only one that requires letter case conversion, Option B might be more appropriate. WDYT? @beryllw

[Pluto201]

Option B 要怎么做啊，我遇到的问题是字段名大写而不是表名，我在示例中没有看到将字段名改为小写的方法，我在paimon连接器pipeline的sink中添加了配置catalog.properties.allow-upper-case: True，还是报了相同的错误

[beryllw]

Option B 要怎么做啊，我遇到的问题是字段名大写而不是表名，我在示例中没有看到将字段名改为小写的方法，我在paimon连接器pipeline的sink中添加了配置catalog.properties.allow-upper-case: True，还是报了相同的错误

通过增加transform解决。（如果同时需要表路由，可以增加route，transform和route可以同时生效）
https://nightlies.apache.org/flink/flink-cdc-docs-release-3.2/docs/core-concept/transform/

transform:
  - source-table: mydb.web_order
    projection:ID as id, PRODUCT_NAME as product_name
    description: Convert table field names to lowercase

[Pluto201]

原表中主键字段名为大写，在transform中改为小写后报错，找不到原来的主键了。。。有啥方法能重新指定主键吗？

[beryllw]

可以指定主键，参考transform 文档 https://nightlies.apache.org/flink/flink-cdc-docs-release-3.2/docs/core-concept/transform/

[beryllw]

Option A: Add a route configuration indicating if table names should be converted to lowercase. Option B: Add a Paimon connector option to do this job.

The problem is that the Paimon table field requires letter case conversion, which I solved by adding a transform projection. Case sensitivity issues are not limited to Paimon but extend to many other sources(hive) as well. Should we add a transform configuration to conveniently convert table fields to lower case? @yuxiqian

[yuxiqian]

It's reasonable to add a "lowercased" option, considering it's a general issue affecting multiple sinks. Created FLINK-36475 to trace this.

[lvyanquan]

I've found the solution.

1. Update paimon dependency to 0.9.0
2. Add allow-upper-case option to paimon catalog, and set to true.
   refer to https://paimon.apache.org/docs/master/maintenance/configurations/#catalogoptions.

So, what we need to do is to update paimon dependency to 0.9.0 in FlinkCDC 3.3