gs_datachecker

JDK11+
Install kafka (start zookeeper and kafka service)

cd {path}/confluent-7.2.0

bin/zookeeper-server-start etc/kafka/zookeeper.properties
or
bin/zookeeper-server-start -daemon etc/kafka/zookeeper.properties

bin/kafka-server-start  etc/kafka/server.properties

bin/kafka-server-start -daemon etc/kafka/server.properties

# New connect configuration
vi etc/kafka/mysql-conect.properties

name=mysql-connect-all
connector.class=io.debezium.connector.mysql.MySqlConnector
database.hostname=
database.port=3306
database.user=root
database.password=test@123
database.server.id=1
database.server.name=mysql_debezium_connect-all
database.whitelist=test
database.history.kafka.bootstrap.servers=
database.history.kafka.topic=mysql_test_topic-all
include.schema.changes=true
transforms=Reroute
transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
transforms.Reroute.topic.regex=(.*)test(.*)
transforms.Reroute.topic.replacement=data_check_test_all

# Start the Kafka connect service 
bin/connect-standalone -daemon etc/kafka/connect-standalone.properties etc/kafka/mysql-conect.properties

Verify the service configuration and modify the application.yml file
	server.port is the verification service web port, which can not be modified by default
	logging.config is the absolute path of the extraction service log path config/log4j2.xml file
	bootstrap-servers is the working address of kafka, and the default installation can not be modified
	data.check.data-path is the output address of the verification result, and the default configuration can not be modified
	data.check.source-uri the source side service request address, and the default configuration can not be modified
	data.check.sink-uri is the service request address of the target end, and the default configuration can not be modified

Source side service configuration modification application-source.yml file
	server.port is the source side extraction service web port, which can not be modified by default
	logging.config is the absolute path of the extraction service log path config/log4j2source.xml file
	spring.check.server-uri is the verification service request address, and the default configuration can not be modified
	spring.extract.schema is the current validation data schema, and the name of the MySQL database
	bootstrap-servers is the working address of kafka, which can not be modified by default
	
	Data Source Configuration

Target side service configuration modification application-sink.yml file
	server.port  is the sink side extraction service web port, which can not be modified by default
	logging.config is the absolute path of the extraction service log path config/log4j2sink.xml file
	spring.check.server-uri is the verification service request address, and the default configuration can not be modified
	spring.extract.schema is the current validation data schema, and the name of the MySQL database
	bootstrap-servers is the working address of kafka, which can not be modified by default
	
	Data Source Configuration

use extract-endpoints shell command to start the source and sink service
sh extract-endpoints.sh start|stop|restart 
use check-endpoint shell command to start the check service
sh check-endpoint.sh start|stop|restart 

The extraction service must be started first, and then the verification service.

nohup java  -jar datachecker-extract-0.0.1.jar --source  >/dev/null 2>&1 &

nohup java  -jar datachecker-extract-0.0.1.jar --sink >/dev/null 2>&1 &

nohup java  -jar datachecker-check-0.0.1.jar >/dev/null 2>&1 &

1. Single instance verification uses sh script to start the verification service. If verification needs to be started in parallel, copy the current working directory file. After reconfiguration, use the java background startup command.

2. After the extraction service is started, it will automatically load the table related information of the database. If the data volume is large, the data loading will be time-consuming.

3. After the validation service is started, it will detect whether the table data information on the extraction end has been loaded. If the loading is not completed within a certain period of time, the validation service will automatically exit. At this time, you need to query the table information loading progress of the source and destination, and view the loading progress through the log information. Or restart the verification service directly.

4. The incremental verification service is started, and the source side configuration file  config  application source needs to be modified Debezium enable: true in yml and configure other debezium related configurations. Start the service to start the incremental verification service

Source side extraction service
-Dspring.config.additional-location=.\config\application-source.yml

Destination extraction service
-Dspring.config.additional-location=.\config\application-sink.yml

check service
-Dspring.config.additional-location=.\config\application.yml

JDK version requirements JDK11+
The current version only supports the verification of source MySQL and target openGauss data
The current version only supports data verification, not table object verification
MYSQL requires version 5.7+
The current version does not support the verification of geographic location geometry data