Detail
In Strong Consistency mode, it is possible for records to be in the Unreplicated state.a Re-replication can be triggered when the record is accessed by a client. Would a basic SI or PI query trigger such a re-replication?
Answer
A PI query (scan) or SI query will be able to read the un-replicated record but will not trigger the re-replication process.These records will only re-replicate when read or updated (ie: Simple gets calls, batch reads, batch writes , XDR recoveries, XDR write transactions)...
Example using asbackup:
1) Insert a single record:
2) Restart the entire cluster to clear replication state from all nodes:
3) Take a backup with asbackup.
An asbackup will issue a PI scan on the specified namespace and set.
root@e3de359ce85b:/# asbackup -n test -s demo -o backupfile.txt 2024-04-17 22:45:48 GMT [INF] [ 286] Starting backup of 127.0.0.1 (namespace: test, set: [demo], bins: [all], after: 1970-01-01 00:00:00 UTC, before: 1970-01-01 00:00:00 UTC, no ttl only: false, limit: 0) to backupfile.txt 2024-04-17 22:45:48 GMT [INF] [ 286] [src/main/aerospike/as_cluster.c:195][as_cluster_add_nodes_copy] Add node A1 127.0.0.1:3000 2024-04-17 22:45:48 GMT [INF] [ 286] [src/main/aerospike/as_cluster.c:195][as_cluster_add_nodes_copy] Add node A2 172.17.0.3:3000 2024-04-17 22:45:48 GMT [INF] [ 286] Processing 2 node(s) 2024-04-17 22:45:48 GMT [INF] [ 286] Node ID Objects Replication 2024-04-17 22:45:48 GMT [INF] [ 286] A1 1 2 2024-04-17 22:45:48 GMT [INF] [ 286] A2 1 2 2024-04-17 22:45:48 GMT [INF] [ 286] Set demo contains 1 record(s) 2024-04-17 22:45:48 GMT [INF] [ 324] No secondary indexes 2024-04-17 22:45:48 GMT [INF] [ 324] Backing up 0 UDF file(s) 2024-04-17 22:45:48 GMT [INF] [ 324] Starting backup for 4096 partitions from 0 to 4095 2024-04-17 22:45:48 GMT [INF] [ 324] Completed backup for 4096 partitions from 0 to 4095, records: 1, size: 0 (~0 B/rec) 2024-04-17 22:45:48 GMT [INF] [ 286] Backed up 1 record(s), 0 secondary index(es), 0 UDF file(s), 160 byte(s) in total (~160 B/rec)
4) Check the content of the backup file:
root@e3de359ce85b:/# cat /backupfile.txt Version 3.1 # namespace test # first-file + k S 4 key1 + n test + d 7JEZLUt/jONdXXjTS8ply6qqyWA= + s demo + g 1 + t 0 + b 3 - I foo 123 - S bar 3 abc - Z baz T
5) Confirm the record is still in the un-replicated state:
root@e3de359ce85b:/# asadm -e "show stat like unreplicated -flip"
Seed: [('127.0.0.1', 3000, None)]
Config_file: /root/.aerospike/astools.conf, /etc/aerospike/astools.conf
~test Namespace Statistics (2024-04-18 00:09:02 UTC)~
Node|unreplicated_records
172.17.0.3:3000 | 1
e3de359ce85b:3000| 1
Number of rows: 2
6) Trigger re-replication by reading the record:
aql> select * from test.demo where PK='key1' +--------+-----+-------+------+ | PK | foo | bar | baz | +--------+-----+-------+------+ | "key1" | 123 | "abc" | true | +--------+-----+-------+------+ 1 row in set (0.001 secs) OK
7) Check record re-replication state:
root@e3de359ce85b:/# asadm -e "show stat like unreplicated -flip"
Seed: [('127.0.0.1', 3000, None)]
Config_file: /root/.aerospike/astools.conf, /etc/aerospike/astools.conf
~test Namespace Statistics (2024-04-18 00:11:28 UTC)~
Node|unreplicated_records
172.17.0.3:3000 | 0
e3de359ce85b:3000| 0
Number of rows: 2
Notes
Write transactions in SC must be acknowledged by all replicas before a node acks back to the client.
In an SC namespace, records have 2 bits of the primary index reserved for the replication flag with the following possible states:
- OK
- Replicating,
- Re-replicating
- Unreplicated.
Failures of the ack from the replicas can lead to the record staying in the Unreplicated state.
Re-replication can be triggered using the following guide .
If a record is in Unreplicated state, XDR will trigger re-replication. The re-replication will be like a new write (with a new LUT) which could trigger XDR again, as a new write.