同步操作将从 Gitee 极速下载/Addax 强制同步,此操作会覆盖自 Fork 仓库以来所做的任何修改,且无法恢复!!!
确定后同步将在后台操作,完成时将刷新页面,请耐心等待。
Addax is an open source universal ETL tool
Documentation detailed description of how to install and deploy and how to use each collection plugin
English | 简体中文
The project, originally from Ali's DataX, has been streamlined and adapted, as described below
Addax supports more than 20 SQL and NoSQL data sources. It can also be extended to support more.
docker pull wgzhao/addax:latest
docker run -ti --rm --name addax wgzhao/addax:latest /opt/addax/bin/addax.sh /opt/addax/job/job.json
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/wgzhao/Addax/master/install.sh)"
This script installs Addax to its preferred prefix (/usr/local for macOS Intel, /opt/addax for Apple Silicon and /opt/addax/ for Linux)
git clone https://github.com/wgzhao/addax.git addax
cd addax
mvn clean package
mvn package assembly:single
After successful compilation and packaging, a addax-<version>
folder will be created in the target/datax
directory of the project directory, where <version
indicates the version.
The job
subdirectory contains many sample jobs, of which job.json
can be used as a smoke-out test and executed as follows
bin/addax.sh job/job.json
The output of the above command is roughly as follows.
$ bin/addax.sh job/job.json
___ _ _
/ _ \ | | | |
/ /_\ \ __| | __| | __ ___ __
| _ |/ _` |/ _` |/ _` \ \/ /
| | | | (_| | (_| | (_| |> <
\_| |_/\__,_|\__,_|\__,_/_/\_\
:: Addax version :: (v4.0.3)
2021-09-16 11:03:20.328 [ main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2021-09-16 11:03:20.347 [ main] INFO Engine -
{
"content":[
{
"reader":{
"parameter":{
"column":[
{
"type":"string",
"value":"addax"
},
{
"type":"long",
"value":19890604
},
{
"type":"date",
"value":"1989-06-04 00:00:00"
},
{
"type":"bool",
"value":true
},
{
"type":"bytes",
"value":"test"
}
],
"sliceRecordCount":10
},
"name":"streamreader"
},
"writer":{
"parameter":{
"print":true,
"column":[
"col1"
],
"encoding":"UTF-8"
},
"name":"streamwriter"
}
}
],
"setting":{
"errorLimit":{
"record":0,
"percentage":0.02
},
"speed":{
"byte":-1,
"channel":1
}
}
}
2021-09-16 11:03:20.367 [ main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2021-09-16 11:03:20.367 [ main] INFO JobContainer - Addax jobContainer starts job.
2021-09-16 11:03:20.368 [ main] INFO JobContainer - Set jobId = 0
2021-09-16 11:03:20.382 [ job-0] INFO JobContainer - Addax Reader.Job [streamreader] do prepare work .
2021-09-16 11:03:20.382 [ job-0] INFO JobContainer - Addax Writer.Job [streamwriter] do prepare work .
2021-09-16 11:03:20.383 [ job-0] INFO JobContainer - Job set Channel-Number to 1 channels.
2021-09-16 11:03:20.383 [ job-0] INFO JobContainer - Addax Reader.Job [streamreader] splits to [1] tasks.
2021-09-16 11:03:20.383 [ job-0] INFO JobContainer - Addax Writer.Job [streamwriter] splits to [1] tasks.
2021-09-16 11:03:20.405 [ job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2021-09-16 11:03:20.412 [ taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2021-09-16 11:03:20.415 [ taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2021-09-16 11:03:20.415 [ taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
addax 19890604 1989-06-04 00:00:00 true test
addax 19890604 1989-06-04 00:00:00 true test
addax 19890604 1989-06-04 00:00:00 true test
addax 19890604 1989-06-04 00:00:00 true test
addax 19890604 1989-06-04 00:00:00 true test
addax 19890604 1989-06-04 00:00:00 true test
addax 19890604 1989-06-04 00:00:00 true test
addax 19890604 1989-06-04 00:00:00 true test
addax 19890604 1989-06-04 00:00:00 true test
addax 19890604 1989-06-04 00:00:00 true test
2021-09-16 11:03:23.428 [ job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2021-09-16 11:03:23.428 [ job-0] INFO JobContainer - Addax Writer.Job [streamwriter] do post work.
2021-09-16 11:03:23.428 [ job-0] INFO JobContainer - Addax Reader.Job [streamreader] do post work.
2021-09-16 11:03:23.430 [ job-0] INFO JobContainer - PerfTrace not enable!
2021-09-16 11:03:23.431 [ job-0] INFO StandAloneJobContainerCommunicator - Total 10 records, 260 bytes | Speed 86B/s, 3 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2021-09-16 11:03:23.432 [ job-0] INFO JobContainer -
任务启动时刻 : 2021-09-16 11:03:20
任务结束时刻 : 2021-09-16 11:03:23
任务总计耗时 : 3s
任务平均流量 : 86B/s
记录写入速度 : 3rec/s
读出记录总数 : 10
读写失败总数 : 0
Here and Here provides all kinds of job configuration examples
We recommend you use IntelliJ as your IDE. The code style template for the project can be found in the codestyle repository along with our general programming and Java guidelines. In addition to those you should also adhere to the following:
AddaxException(REQUIRE_VALUE, "lack of required item")
. This categorization lets you generate reports, so you can monitor the frequency of various failures.mvn license:format
.Formatter
class): format("Session property %s is invalid: %s", name, value)
(note that format()
should always be statically imported). Sometimes, if you only need to append something, consider using the +
operator.Assertions
class if there is one that covers your case rather than writing the assertion by hand. Over time, we may move over to more fluent assertions like AssertJ.This software is free to use under the Apache License Apache license.
Special thanks to JetBrains for his supports to this project.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。