Context
When XDR is not able to ship a specific set successfully to a destination cluster due to an issue on the destination cluster, the set is retried continuously on the XDR side. Typically this happens when leveraging a connector where different sets can be shipped to different routes or backends.
In this case, XDR blocks all transactions behind the set being retried, and the XDR lag increases. You may want to skip the specific set causing the shipping failure to allow other sets to progress.
If the backend is Pulsar, use the Pulsar Outbound Connector configuration to specify which sets should be shipped to the Pulsar backend. This is done with a simple set-based inclusion filter, which lists only the desired sets to be shipped.
This can be also used in a scenario when only some specific sets of a namespace need to be shipped to the Pulsar backend.
Method
Make following changes in the Pulsar Outbound Connector Configuration :
- Check that there is no default routing set
- Check that there is no routing set at the ‘namespace’ level
- Check that the routing is set only at the ‘sets’ level for a given set
Sample /etc/aerospike-pulsar-outbound/aerospike-pulsar-outbound.yml changes
# one of json, flat-json, message-pack, or avro
format:
mode: json
# a list of transformations and mappings.
#bin-transforms:
# map:
# yellow: red
# transforms: # will be done in order
# - uppercase
#routing:
#mode: static
#destination: default
namespaces:
test:
sets:
premium:
routing:
mode: static
destination: premium
bin-transforms:
map:
gold: platinum
After these configuration changes and a restart of the Connector service, only the ‘premium’ set is published to the Pulsar backend. All other sets are skipped with a warning.