无主键

同步前先将无主键的表过滤,或者加上主键

1
2
3
4
5
6
SELECT a.TABLE_SCHEMA,
a.TABLE_NAME,
a.index_name,
GROUP_CONCAT(column_name ORDER BY seq_in_index) AS `Columns`
FROM information_schema.statistics a WHERE a.index_name='PRIMARY' and a.table_schema=<数据库名称>
GROUP BY a.TABLE_SCHEMA,a.TABLE_NAME,a.index_name

主键为空字符

同步前先将对应的表过滤,或者加上主键

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
select concat('select ',a.COLUMN_NAME,' from ',a.TABLE_NAME,' where ',a.COLUMN_NAME,' = '''' ; ') from
(
select TABLE_NAME,COLUMN_NAME from information_schema.key_column_usage
where TABLE_SCHEMA = <数据库名称>
and constraint_name='primary') a
inner join
(
select TABLE_NAME,COLUMN_NAME from information_schema.columns
where TABLE_SCHEMA = <数据库名称> and DATA_TYPE in ('varchar')
) b
on a.TABLE_NAME=b.TABLE_NAME and a.COLUMN_NAME = b.COLUMN_NAME;

使用我维护的插件 https://github.com/lialzm/pipelinewise-tap-mysql.git 也可以解决该问题

时间类型为0000:00:00

修改了连接器 https://github.com/lialzm/pipelinewise-tap-mysql.git ,添加配置

1
2
3
metadata:
  '*':
    zero_date_time_behavior: convert_to_null

字段过大

如果表中有比较大的字段,会严重影响到同步速度,那可以判断下该字段是否必须,不是必须的话则考虑过滤