Problem Description
When setting up an outbound Kafka connector from Aerospike the following error is observed in the logs:
May 31 2020 09:20:04 GMT: INFO (xdr): (xdr_ship.c:713) Connecting to 'kafka_connector'
May 31 2020 09:20:04 GMT: INFO (xdr): (xdr_dlog.c:106) dlog: free-pct 100 reclaimed 0 glst 1590912631088 (2020-05-31 08:10:31 GMT)
May 31 2020 09:20:04 GMT: INFO (xdr): (xdr.c:544) [kafka_connector]: dc-state CLUSTER_INACTIVE timelag-sec 0 lst 1590912636851 mlst 1590916804566 (2020-05-31 09:20:04 GMT) fnlst 0 (-) wslst 0 (-) shlat-ms 0 errcl 0 errsrv 0 rsas-ms 0.000 rsas-pct 0.0
May 31 2020 09:20:04 GMT: INFO (xdr): (xdr.c:586) summary: throughput 0 inflight 0 dlog-outstanding 0 dlog-delta-per-sec 0.0
May 31 2020 09:20:04 GMT: INFO (xdr): (xdr.c:596) detail: sh 0 ul 0 lg 14 rlg 0 rlgi 4 rlgo 3 lproc 14 rproc 1 lkdproc 0 errcl 0 errsrv 0 hkskip 8 hkf 1 flat 0
May 31 2020 09:20:04 GMT: WARNING (xdr-client): (as_cluster.c:236) Failed to connect to seed kafka-connector-customer-staging.domain.com 3000. AEROSPIKE_ERR_CLIENT Received invalid proto version: 72 Expected: 2, 10.0.0.12:3000
May 31 2020 09:20:04 GMT: WARNING (xdr-client): (as_cluster.c:236) Failed to connect to seed kafka-connector-customer-staging.domain.com 3000.
Explanation
This error will be displayed when the destination cluster specification on the source has not been done properly. In this instance the most likely culprit is that the Kafka destination has been specified in the same way as an Aerospike cluster, which is incorrect. The destination must be configured as a Change Notification DC.
Solution
To resolve the issue the destination cluster should be specified correctly for Change Notification. The error message above shows the connection is being made on port 3000 which is a smoking gun as this is normally the port an Aerospike cluster listens on. A Kafka connector would more likely listen on 8080. An example DC specification for Aerospike < 5.0 is shown below.
xdr {
enable-xdr true
enable-change-notification true
xdr-digestlog-path /etc/aerospike/digestlog.log1 100G
xdr-compression-threshold 1000
xdr-info-port 3004
datacenter kafka_connector {
dc-type http
http-version v1
http-url http://10.0.0.1:8080
http-url http://10.0.0.2:8080
}
}
Notes
- Full instructions for configuring Change Notification destinations can be found here 5.
- Change notification destinations are used by several Aerospike outbound connectors such as Kafka and JMS.
- Aerospike versions > 5.0 are simplified in their Change Notification configuration.
Applies To Earliest Version
Pre 4.9
Applies To Latest Version
4.9
