taosAdapter is a TDengine’s companion tool and is a bridge/adapter between TDengine cluster and application. It provides an easy-to-use and efficient way to ingest data from data collections agents(like Telegraf, StatsD, collectd) directly. It also provides InfluxDB/OpenTSDB compatible data ingestion interface to allow InfluxDB/OpenTSDB applications to immigrate to TDengine seamlessly.
taosAdapter provides the following functions.
taosAdapter is part of the TDengine server from TDengine v2.3.0.0. You don't need any additional steps to install taosAdapter if you already installed TDengine server. You can download TDengine server package (taosAdapter be included in v2.3.0.0 and above version) from the official website. If you want to deploy taosAdapter on another server, you need to install official TDengine server installation package. If you want to build taosAdapter from source code, you can refer to the How to build taosAdapter instruction.
taosAdapter service is managed by the systemd by default on the Linux system. It can be started by the command systemctl start taosadapter
and be stopped by the command systemctl stop taosadapter
.
The command rmtaos
will remove the TDengine server software including taosAdapter too.
taosAdapter only properly run with the same version of TDengine server. You need to upgrade TDengine server to upgrade taosAdapter. A separate deployed taosAdapter need to upgrade the corresponding version of TDengine server package too.
taosAdapter supports command line parameters, environment variables, and configuration files.
The default configuration file is /etc/taos/taosadapter.toml
.
Command-line parameters take precedence over environment variables take precedence over configuration files
The command line usage is arg=val such as taosadapter -p=30000 --debug=true
Usage of taosAdapter:
--collectd.db string collectd db name. Env "TAOS_ADAPTER_COLLECTD_DB" (default "collectd")
--collectd.enable enable collectd. Env "TAOS_ADAPTER_COLLECTD_ENABLE" (default true)
--collectd.password string collectd password. Env "TAOS_ADAPTER_COLLECTD_PASSWORD" (default "taosdata")
--collectd.port int collectd server port. Env "TAOS_ADAPTER_COLLECTD_PORT" (default 6045)
--collectd.user string collectd user. Env "TAOS_ADAPTER_COLLECTD_USER" (default "root")
--collectd.worker int collectd write worker. Env "TAOS_ADAPTER_COLLECTD_WORKER" (default 10)
-c, --config string config path default /etc/taos/taosadapter.toml
--cors.allowAllOrigins cors allow all origins. Env "TAOS_ADAPTER_CORS_ALLOW_ALL_ORIGINS" (default true)
--cors.allowCredentials cors allow credentials. Env "TAOS_ADAPTER_CORS_ALLOW_Credentials"
--cors.allowHeaders stringArray cors allow HEADERS. Env "TAOS_ADAPTER_ALLOW_HEADERS"
--cors.allowOrigins stringArray cors allow origins. Env "TAOS_ADAPTER_ALLOW_ORIGINS"
--cors.allowWebSockets cors allow WebSockets. Env "TAOS_ADAPTER_CORS_ALLOW_WebSockets"
--cors.exposeHeaders stringArray cors expose headers. Env "TAOS_ADAPTER_Expose_Headers"
--debug enable debug mode. Env "TAOS_ADAPTER_DEBUG"
--help Print this help message and exit
--influxdb.enable enable influxdb. Env "TAOS_ADAPTER_INFLUXDB_ENABLE" (default true)
--log.path string log path. Env "TAOS_ADAPTER_LOG_PATH" (default "/var/log/taos")
--log.rotationCount uint log rotation count. Env "TAOS_ADAPTER_LOG_ROTATION_COUNT" (default 30)
--log.rotationSize string log rotation size(KB MB GB), must be a positive integer. Env "TAOS_ADAPTER_LOG_ROTATION_SIZE" (default "1GB")
--log.rotationTime duration log rotation time. Env "TAOS_ADAPTER_LOG_ROTATION_TIME" (default 24h0m0s)
--logLevel string log level (panic fatal error warn warning info debug trace). Env "TAOS_ADAPTER_LOG_LEVEL" (default "info")
--monitor.collectDuration duration Set monitor duration. Env "TAOS_MONITOR_COLLECT_DURATION" (default 3s)
--monitor.identity string The identity of the current instance, or 'hostname:port' if it is empty. Env "TAOS_MONITOR_IDENTITY"
--monitor.incgroup Whether running in cgroup. Env "TAOS_MONITOR_INCGROUP"
--monitor.password string TDengine password. Env "TAOS_MONITOR_PASSWORD" (default "taosdata")
--monitor.pauseAllMemoryThreshold float Memory percentage threshold for pause all. Env "TAOS_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD" (default 80)
--monitor.pauseQueryMemoryThreshold float Memory percentage threshold for pause query. Env "TAOS_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD" (default 70)
--monitor.user string TDengine user. Env "TAOS_MONITOR_USER" (default "root")
--monitor.writeInterval duration Set write to TDengine interval. Env "TAOS_MONITOR_WRITE_INTERVAL" (default 30s)
--monitor.writeToTD Whether write metrics to TDengine. Env "TAOS_MONITOR_WRITE_TO_TD" (default true)
--node_exporter.caCertFile string node_exporter ca cert file path. Env "TAOS_ADAPTER_NODE_EXPORTER_CA_CERT_FILE"
--node_exporter.certFile string node_exporter cert file path. Env "TAOS_ADAPTER_NODE_EXPORTER_CERT_FILE"
--node_exporter.db string node_exporter db name. Env "TAOS_ADAPTER_NODE_EXPORTER_DB" (default "node_exporter")
--node_exporter.enable enable node_exporter. Env "TAOS_ADAPTER_NODE_EXPORTER_ENABLE"
--node_exporter.gatherDuration duration node_exporter gather duration. Env "TAOS_ADAPTER_NODE_EXPORTER_GATHER_DURATION" (default 5s)
--node_exporter.httpBearerTokenString string node_exporter http bearer token. Env "TAOS_ADAPTER_NODE_EXPORTER_HTTP_BEARER_TOKEN_STRING"
--node_exporter.httpPassword string node_exporter http password. Env "TAOS_ADAPTER_NODE_EXPORTER_HTTP_PASSWORD"
--node_exporter.httpUsername string node_exporter http username. Env "TAOS_ADAPTER_NODE_EXPORTER_HTTP_USERNAME"
--node_exporter.insecureSkipVerify node_exporter skip ssl check. Env "TAOS_ADAPTER_NODE_EXPORTER_INSECURE_SKIP_VERIFY" (default true)
--node_exporter.keyFile string node_exporter cert key file path. Env "TAOS_ADAPTER_NODE_EXPORTER_KEY_FILE"
--node_exporter.password string node_exporter password. Env "TAOS_ADAPTER_NODE_EXPORTER_PASSWORD" (default "taosdata")
--node_exporter.responseTimeout duration node_exporter response timeout. Env "TAOS_ADAPTER_NODE_EXPORTER_RESPONSE_TIMEOUT" (default 5s)
--node_exporter.urls strings node_exporter urls. Env "TAOS_ADAPTER_NODE_EXPORTER_URLS" (default [http://localhost:9100])
--node_exporter.user string node_exporter user. Env "TAOS_ADAPTER_NODE_EXPORTER_USER" (default "root")
--opentsdb.enable enable opentsdb. Env "TAOS_ADAPTER_OPENTSDB_ENABLE" (default true)
--opentsdb_telnet.dbs strings opentsdb_telnet db names. Env "TAOS_ADAPTER_OPENTSDB_TELNET_DBS" (default [opentsdb_telnet,collectd_tsdb,icinga2_tsdb,tcollector_tsdb])
--opentsdb_telnet.enable enable opentsdb telnet,warning: without auth info(default false). Env "TAOS_ADAPTER_OPENTSDB_TELNET_ENABLE"
--opentsdb_telnet.maxTCPConnections int max tcp connections. Env "TAOS_ADAPTER_OPENTSDB_TELNET_MAX_TCP_CONNECTIONS" (default 250)
--opentsdb_telnet.password string opentsdb_telnet password. Env "TAOS_ADAPTER_OPENTSDB_TELNET_PASSWORD" (default "taosdata")
--opentsdb_telnet.ports ints opentsdb telnet tcp port. Env "TAOS_ADAPTER_OPENTSDB_TELNET_PORTS" (default [6046,6047,6048,6049])
--opentsdb_telnet.tcpKeepAlive enable tcp keep alive. Env "TAOS_ADAPTER_OPENTSDB_TELNET_TCP_KEEP_ALIVE"
--opentsdb_telnet.user string opentsdb_telnet user. Env "TAOS_ADAPTER_OPENTSDB_TELNET_USER" (default "root")
--pool.idleTimeout duration Set idle connection timeout. Env "TAOS_ADAPTER_POOL_IDLE_TIMEOUT" (default 1h0m0s)
--pool.maxConnect int max connections to taosd. Env "TAOS_ADAPTER_POOL_MAX_CONNECT" (default 4000)
--pool.maxIdle int max idle connections to taosd. Env "TAOS_ADAPTER_POOL_MAX_IDLE" (default 4000)
-P, --port int http port. Env "TAOS_ADAPTER_PORT" (default 6041)
--prometheus.enable enable prometheus. Env "TAOS_ADAPTER_PROMETHEUS_ENABLE" (default true)
--restfulRowLimit int restful returns the maximum number of rows (-1 means no limit). Env "TAOS_ADAPTER_RESTFUL_ROW_LIMIT" (default -1)
--ssl.certFile string ssl cert file path. Env "TAOS_ADAPTER_SSL_CERT_FILE"
--ssl.enable enable ssl. Env "TAOS_ADAPTER_SSL_ENABLE"
--ssl.keyFile string ssl key file path. Env "TAOS_ADAPTER_SSL_KEY_FILE"
--statsd.allowPendingMessages int statsd allow pending messages. Env "TAOS_ADAPTER_STATSD_ALLOW_PENDING_MESSAGES" (default 50000)
--statsd.db string statsd db name. Env "TAOS_ADAPTER_STATSD_DB" (default "statsd")
--statsd.deleteCounters statsd delete counter cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_COUNTERS" (default true)
--statsd.deleteGauges statsd delete gauge cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_GAUGES" (default true)
--statsd.deleteSets statsd delete set cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_SETS" (default true)
--statsd.deleteTimings statsd delete timing cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_TIMINGS" (default true)
--statsd.enable enable statsd. Env "TAOS_ADAPTER_STATSD_ENABLE" (default true)
--statsd.gatherInterval duration statsd gather interval. Env "TAOS_ADAPTER_STATSD_GATHER_INTERVAL" (default 5s)
--statsd.maxTCPConnections int statsd max tcp connections. Env "TAOS_ADAPTER_STATSD_MAX_TCP_CONNECTIONS" (default 250)
--statsd.password string statsd password. Env "TAOS_ADAPTER_STATSD_PASSWORD" (default "taosdata")
--statsd.port int statsd server port. Env "TAOS_ADAPTER_STATSD_PORT" (default 6044)
--statsd.protocol string statsd protocol [tcp or udp]. Env "TAOS_ADAPTER_STATSD_PROTOCOL" (default "udp")
--statsd.tcpKeepAlive enable tcp keep alive. Env "TAOS_ADAPTER_STATSD_TCP_KEEP_ALIVE"
--statsd.user string statsd user. Env "TAOS_ADAPTER_STATSD_USER" (default "root")
--statsd.worker int statsd write worker. Env "TAOS_ADAPTER_STATSD_WORKER" (default 10)
--taosConfigDir string load taos client config path. Env "TAOS_ADAPTER_TAOS_CONFIG_FILE"
--version Print the version and exit
Note: If you support users using the web browser to access the interfaces, please configure the following CORS parameters according to your practical network setting:
AllowAllOrigins
AllowOrigins
AllowHeaders
ExposeHeaders
AllowCredentials
AllowWebSockets
If not, you don't need to configure them.
Please visit the webpage https://www.w3.org/wiki/CORS_Enabled or https://developer.mozilla.org/zh-CN/docs/Web/HTTP/CORS for the detailed CORS protocol.
For the default configuration file, see example/config/taosadapter.toml
You can use any http client to access the RESTful interface address http://<fqdn>:6041/<APIEndPoint>
to insert to or query from TDengine. Please refer to the official documentation for detail. The end point could be following:
/rest/sql
/rest/sqlt
/rest/sqlutc
You can use any http client to access the RESTful interface address http://<fqdn>:6041/<APIEndPoint>
to insert InfluxDB compatible protocol data to TDengine. The end point is:
/influxdb/v1/write
Support following InfluxDB query parameters:
db
Specify the necessary parameters for the databaseprecision
time precision non-essential parameteru
user non-essential parametersp
password Optional parameterNote: There is currently not supported token authentication in InfluxDB only supports Basic authentication and query parameter authentication.
You can use any http client to access the RESTful interface address http://<fqdn>:6041/<APIEndPoint>
to insert OpenTSDB compatible protocol data to TDengine. The end point is:
/opentsdb/v1/put/json/:db
/opentsdb/v1/put/telnet/:db
Modify the collectd configuration /etc/collectd/collectd.conf
. taosAdapter uses 6045 for collectd direct collection data write by default.
LoadPlugin network
<Plugin network>
Server "127.0.0.1" "6045"
</Plugin>
Modify the collectd configuration /etc/collectd/collectd.conf
. taosAdapter uses 6047 for collectd tsdb write by default.
LoadPlugin write_tsdb
<Plugin write_tsdb>
<Node>
Host "localhost"
Port "6047"
HostTags "status=production"
StoreRates false
AlwaysAppendDS false
</Node>
</Plugin>
modify the configuration file path_to_statsd/config.js
backends
add "./backends/repeater"
repeater
add { host:'host to taosAdapter', port: 6044}
An example configuration file as below:
{
port: 8125,
backends: ["./backends/repeater"],
repeater: [{ host: '127.0.0.1', port: 6044}]
}
Use icinga2 to collect check result metrics and performance data
opentsdb_telnet.enable
/etc/icinga2/features-enabled/opentsdb.conf
object OpenTsdbWriter "opentsdb" {
host = "host to taosAdapter"
port = 6048
}
TCollector is a client-side process that gathers data from local collectors and pushes the data to OpenTSDB. You run it on all your hosts, and it does the work of sending each host’s data to the TSD (OpenTSDB backend process).
opentsdb_telnet.enable
Prometheus exporter for hardware and OS metrics exposed by *NIX kernels
node_exporter.enable
Remote_read and remote_write are cluster schemes for Prometheus data read-write separation. Just use the REMOTE_READ and REMOTE_WRITE URL to point to the URL corresponding to Taosadapter to use Basic authentication.
http://host_to_taosadapter:port (default 6041) /prometheus/v1/remote_read/:db
http://host_to_taosadapter:port (default 6041) /Prometheus/v1/remote_write/:db
Basic verification:
Example Prometheus.yml is as follows:
remote_write:
- url: "http://localhost:6041/prometheus/v1/remote_write/prometheus_data"
basic_auth:
username: root
password: taosdata
remote_read:
- url: "http://localhost:6041/prometheus/v1/remote_read/prometheus_data"
basic_auth:
username: root
password: taosdata
remote_timeout: 10s
read_recent: true
taosAdapter will monitor itself memory usage during its running. You can adjust its thresholds via two parameters. The valid value is from -1 to 100. The unit is the percentage of physical memory on the system.
Query requests will be rejected when the pauseQueryMemoryThreshold is exceeded.
http return
All write and query requests will be rejected when the pauseAllMemoryThreshold is exceeded.
http return
The corresponding function will resume when the memory usage falls back below the threshold.
Status check interface http://<fqdn>:6041/-/ping
code 200
code 503
if memory exceeds pauseAllMemoryThresholdaction=query
returns code 503
if memory exceeds pauseQueryMemoryThreshold or pauseAllMemoryThresholdfor the corresponding configuration parameter
monitor.collectDuration duration Set monitor duration. Env "TAOS_MONITOR_COLLECT_DURATION" (default 3s)
monitor.incgroup Whether running in cgroup. Env "TAOS_MONITOR_INCGROUP"
monitor.pauseAllMemoryThreshold float Memory percentage threshold for pause query and insert. Env "TAOS_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD" (default 80)
monitor.pauseQueryMemoryThreshold float Memory percentage threshold for pause query. Env "TAOS_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD" (default 70)
You can adjust them according to the specific project scenarios and operation strategies, and it is recommended to use operation monitoring software to monitor system memory status in a timely manner too. You can configure the load balancer to check the interface for checking taosAdapter's running status too.
taosAdapter collects http related metrics, cpu percentage and memory percentage.
Provides an OpenMetrics interface.
http://<fqdn>:6041/metrics
taosAdapter supports writing metrics to TDengine.
Related configuration parameters
configuration | description | default values |
---|---|---|
monitor.collectDuration | cpu and memory collection interval | 3s |
monitor.identity | The identifier of the current taosadapter will be used if not set to 'hostname:port' | |
monitor.incgroup | whether running in a cgroup (set to true when running in a container) | false |
monitor.writeToTD | Whether to write to TDengine | true |
monitor.user | TDengine connection username | root |
monitor.password | TDengine connection password | taosdata |
monitor.writeInterval | write to TDengine interval | 30s |
taosAdapter controls the rows of result returned by the parameter restfulRowLimit
, -1 means no limit, default is no limit.
This parameter controls the following interface returns
http://<fqdn>:6041/rest/sql
http://<fqdn>:6041/rest/sqlt
http://<fqdn>:6041/rest/sqlutc
http://<fqdn>:6041/prometheus/v1/remote_read/:db
You can use systemctl status taosadapter
to check the running status of the taosAdapter.
Or you can set --logLevel
or the environment variable "TAOS_ADAPTER_LOG_LEVEL" to adjust the detail level how many log taosAdapter output. The valid values include panic, fatal, error, warn, warning, info, debug, and trace.
In the early version (2.2.x.x or earlier version), TDengine server provided an embedded http service by default. It will compulsorily running with the taosd
process. As mentioned early, taosAdapter need be manually run by 'systemctl start taosadapter' and has its own process. Some parameters and behaviors are different between the embedded httpd and taosAdapter. Please see below:
# | embedded httpd | taosAdapter | comment |
---|---|---|---|
1 | httpEnableRecordSql | --logLevel=debug | |
2 | httpMaxThreads | n/a | taosAdapter no need this paramter |
3 | telegrafUseFieldNum | please refer to taosAdapter telegraf configuration | |
4 | restfulRowLimit | restfulRowLimit | default value is 10240 in the embedded httpd. taosAdapter provides restfulRowLimit too but the default value is unlimited. User can set it according to the specific scenario |
5 | httpDebugFlag | not used | taosAdapter is immune to httpdDebugFlag
|
6 | httpDBNameMandatory | not used | taosAdapter requests the database name be specified in URL |
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。