-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PLUGIN-1813] CloudSQLMySQL Escape column names #514
[PLUGIN-1813] CloudSQLMySQL Escape column names #514
Conversation
58d2d12
to
307d311
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know this will fix one part of the problem but now this turns into a backward incompatible change because let's say if MySQL by default convert all identifiers to lowercase and if a cx has an existing pipeline which has any column name in uppercase, it will start failing.
The right thing to do would be as described in JIRA to introduce a new config named Field Mapping
which can map any CDAP field in the schema to a different name and will be passed as quoted.
Mysql Col names are not case sensitive. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please do the same changes for MySQL
sink as well.
Discussed offline, to address the larger problem in future. |
307d311
to
b8a9f71
Compare
b8a9f71
to
7be2c44
Compare
protected void insertOperation(PreparedStatement stmt) throws SQLException { | ||
for (int fieldIndex = 0; fieldIndex < columnTypes.size(); fieldIndex++) { | ||
ColumnType columnType = columnTypes.get(fieldIndex); | ||
Schema.Field field = record.getSchema().getField(columnType.getName(), true); | ||
writeToDB(stmt, field, fieldIndex); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why this change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ref:
database-plugins/oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSinkDBRecord.java
Lines 44 to 52 in acd47fa
@Override | |
protected void insertOperation(PreparedStatement stmt) throws SQLException { | |
for (int fieldIndex = 0; fieldIndex < columnTypes.size(); fieldIndex++) { | |
ColumnType columnType = columnTypes.get(fieldIndex); | |
// Get the field from the schema using the column name with ignoring case. | |
Schema.Field field = record.getSchema().getField(columnType.getName(), true); | |
writeToDB(stmt, field, fieldIndex); | |
} | |
} |
This is done to get the cols values from schema with ignoring case.
If cols in CDAP schema are in different case from the cols in the mysql DB, ignoring case is required else we will get null values.
CloudSQLMySQL Escape column names
Jira : PLUGIN-1813
Description
If column name is a MySQL reserved keyword say
insert
the query will fail due to incorrect sql syntax.This PR fixes this issue by escaping the column names using back ticks.
Similar column escapes is used in other plugins. Ref CloudSQLPostgreSQLSink
Code change
CloudSQLMySQLSink.java
MysqlSink.java
Unit Tests
CloudSQLMySQLSinkTest.java
MysqlSinkTest.java