Global sequences allocate values in chunks of 10000 sequence numbers.
BDR uses two-level caching for the chunks. The first level is
used for immediate consumption by
nextval and has a
fixed size of 10 chunks. The second level cache, which is used to fill the
first level cache, is subject to voting. The size of the second level cache
defaults to 5 chunks. This means that BDR will keep somewhere between
100 and 150 thousand sequence numbers cached on each node. If this is not
enough (for example, the application is consuming sequences faster in peak
times), the voting cache size can be adjusted using the following command:
ALTER SEQUENCE sequence_name SET (cache_chunks = 10);
The above command will change the voting cache to 10 chunks (i.e. 100 thousand sequence numbers). Minimum value for cache_chunks is 5 and maximum is 100.
Note: BDR cannot currently currently automatically replicate the ALTER SEQUENCE ... SET (...) commands, so the above command will only affect the local node and it either needs to be executed on all nodes in the cluster manually or executed using the
bdr.bdr_replicate_ddl_command. This limitation is likely to be fixed in future versions of BDR.
Voting is done between the nodes to make sure that no chunk is ever allocated to more than one node. For this to work, enough nodes must be reachable to reach a binding vote. BDR requires that half the nodes, plus one, be able to communicate in order to vote successfully.
If more than half the nodes are down or are unreachable from a given node
then global sequence voting cannot achieve a quorum, so new chunks will not
be allocated in global sequences on that node. Inability to acquire new
global sequence chunks will eventually cause
calls on that node to fail with:
ERROR: could not find free sequence value for global sequence
until connectivity is restored and the next round of voting has completed.
It is possible for nodes that are up but isolated to be unable to get new global sequence chunks while the rest of the BDR group continues to function normally. If no individually connected group of nodes contains more than half the total node count, then no node will be able to allocate global sequence chunks.
Global sequences are not immediately ready for use. Attempting to use a global sequence immediately after its creation, but before the first round of voting has taken place, will result in an error like:
ERROR: global sequence public.myseq is not initialized yet
Simply insert a delay before using the sequence.
See also Global Sequence Limitations.