10.6. Global sequence voting

Global sequences allocate values in chunks of 10000 sequence numbers.

BDR uses two-level caching for the chunks. The first level is used for immediate consumption by nextval and has a fixed size of 10 chunks. The second level cache, which is used to fill the first level cache, is subject to voting. The size of the second level cache defaults to 5 chunks. This means that BDR will keep somewhere between 100 and 150 thousand sequence numbers cached on each node. If this is not enough (for example, the application is consuming sequences faster in peak times), the voting cache size can be adjusted using the following command:

    ALTER SEQUENCE sequence_name SET (cache_chunks = 10);
   

The above command will change the voting cache to 10 chunks (i.e. 100 thousand sequence numbers). Minimum value for cache_chunks is 5 and maximum is 100.

Note: BDR cannot currently currently automatically replicate the ALTER SEQUENCE ... SET (...) commands, so the above command will only affect the local node and it either needs to be executed on all nodes in the cluster manually or executed using the bdr.bdr_replicate_ddl_command. This limitation is likely to be fixed in future versions of BDR.

Voting is done between the nodes to make sure that no chunk is ever allocated to more than one node. For this to work, enough nodes must be reachable to reach a binding vote. BDR requires that half the nodes, plus one, be able to communicate in order to vote successfully.

If more than half the nodes are down or are unreachable from a given node then global sequence voting cannot achieve a quorum, so new chunks will not be allocated in global sequences on that node. Inability to acquire new global sequence chunks will eventually cause nextval calls on that node to fail with:

    ERROR: could not find free sequence value for global sequence
   

until connectivity is restored and the next round of voting has completed.

It is possible for nodes that are up but isolated to be unable to get new global sequence chunks while the rest of the BDR group continues to function normally. If no individually connected group of nodes contains more than half the total node count, then no node will be able to allocate global sequence chunks.

Global sequences are not immediately ready for use. Attempting to use a global sequence immediately after its creation, but before the first round of voting has taken place, will result in an error like:

    ERROR: global sequence public.myseq is not initialized yet
   

Simply insert a delay before using the sequence.

See also Global Sequence Limitations.