Problem Description
Client got "Invalid namespace" error exception even though the namespace is correctly specified.
Caused by: com.aerospike.client.AerospikeException: Invalid namespace: bar at com.aerospike.client.command.Command.getSequenceNode(Command.java:1052) at com.aerospike.client.command.Command.getNode(Command.java:1032) at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:60) at com.aerospike.client.AerospikeClient.get(AerospikeClient.java:783) ... at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:508) at com.ibm.ws.jaxrs20.server.LibertyJaxRsServerFactoryBean.performInvocation(LibertyJaxRsServerFactoryBean.java:651)
Explanation
The error means the namespace is not found in the partition map (keyed by the specified namespace). Namespaces are statically configured on the server-side and never change while the server is running. When the AerospikeClient instance is constructed, the namespaces are retrieved from the servers.Solution
There could be multiple solutions. We have identified the following:
-
Check Seed Nodes configured on client. The client could be pointing to the a different cluster with no such namespace.
-
It is also possible the access-address is not correctly configured.
-
One not common case that has been observed is one with an MTU mismatch in a docker environment.
One not common case that has been observed is one with an MTU mismatch in a docker environment.
After fixing the client MTU to match with the other instances the issue was resolved:
cat /etc/docker/daemon.json
{
"mtu" : 1400
}
In this case, the client also contained errors like these every second:
2018-10-30 19:02:54 WARN Node BB9AB1F9E565000 partition refresh failed: AEROSPIKE_ERR_TIMEOUT 2018-10-30 19:02:55 WARN Node BB9AB1F9E565000 partition refresh failed: AEROSPIKE_ERR_TIMEOUT 2018-10-30 19:02:56 WARN Node BB951619E565000 partition refresh failed: AEROSPIKE_ERR_TIMEOUT
The above is an indication of a network issue. This can also be reproduced by dropping packets using special firewall rules in a non-docker environment. When the client gets a timeout error, it will return an "invalid namespace" because the namespace is not found in the current partition map.