我的位置：首页 › 智能数据治理平台-睿治 › 睿治-问题求助 › 睿治如何抽取千万级别数据量

[数据交换] 睿治如何抽取千万级别数据量

复制链接 | 收藏本帖

3758

: AlexisP 小试身手Lv3

发表于2020-9-18 14:23

悬赏100

已解决

楼主

本帖最后由 AlexisP 于 2020-9-22 10:19 编辑

系统版本睿治V2.2.1

从mysql库中抽取800多万条数据报错，请问如何才能实现抽取800万数据？
目前最多有接近5000万数据，如何才能最高效率的分页进行抽取？
如果要分页的话，如何分页才能实现？

报错信息

2020-09-18 14:22:22 com.esen.exception.RuntimeException4I18N: 执行数据迁移时出现异常
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2115)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1989)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3400)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3112)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2341)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2736)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2486)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2444)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1381)
at com.esen.jdbc.pool.PooledStatement.getQureyResultSet(PooledStatement.java:54)
at com.esen.jdbc.pool.impl.mysql.MysqlPooledStatement.executeQuery(MysqlPooledStatement.java:21)
at com.esen.edataexchange.data.migrate.jdbc.JdbcReader.start(JdbcReader.java:95)
at com.esen.edataexchange.data.migrate.common.buffer.DataSenderRequestTask.call(DataSenderRequestTask.java:47)
at com.esen.edataexchange.data.migrate.common.buffer.DataSenderRequestTask.call(DataSenderRequestTask.java:24)
at com.esen.ethreadpool.req.RequestTaskFutureImpl.call(RequestTaskFutureImpl.java:100)
at com.esen.scheduling.LocaleTask.call(LocaleTask.java:56)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.esen.exception.RuntimeException4I18N: 发送数据时出现异常
... 23 more
Caused by: com.esen.exception.RuntimeException4I18N: 读取数据出现异常
... 23 more
Caused by: com.esen.jdbc.pool.impl.mysql.MysqlPooledSQLException: Query execution was interrupted, max_statement_time exceeded
... 23 more

2020-09-18 14:22:22 执行交换任务[xx_fee]出现异常：com.esen.exception.RuntimeException4I18N: 执行数据迁移时出现异常
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2115)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1989)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3400)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3112)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2341)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2736)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2486)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2444)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1381)
at com.esen.jdbc.pool.PooledStatement.getQureyResultSet(PooledStatement.java:54)
at com.esen.jdbc.pool.impl.mysql.MysqlPooledStatement.executeQuery(MysqlPooledStatement.java:21)
at com.esen.edataexchange.data.migrate.jdbc.JdbcReader.start(JdbcReader.java:95)
at com.esen.edataexchange.data.migrate.common.buffer.DataSenderRequestTask.call(DataSenderRequestTask.java:47)
at com.esen.edataexchange.data.migrate.common.buffer.DataSenderRequestTask.call(DataSenderRequestTask.java:24)
at com.esen.ethreadpool.req.RequestTaskFutureImpl.call(RequestTaskFutureImpl.java:100)
at com.esen.scheduling.LocaleTask.call(LocaleTask.java:56)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.esen.exception.RuntimeException4I18N: 发送数据时出现异常
... 23 more
Caused by: com.esen.exception.RuntimeException4I18N: 读取数据出现异常
... 23 more
Caused by: com.esen.jdbc.pool.impl.mysql.MysqlPooledSQLException: Query execution was interrupted, max_statement_time exceeded
... 23 more