Blacklisting
Blacklisting can prevent situations in which individual messages cause servers to become overloaded and even to fail. For example, a message with a large payload could cause a server to fail. Blacklisting works by identifying and isolating messages that appear likely to cause a problem, or that were being processed when a server failure occurred.
Message Failover
In a cluster environment, if a server node fails, messages that were going to be processed by that node are reassigned (or failed over) to the remaining active nodes in the cluster. If there is only one cluster node, a bad message is reprocessed on the same node if the node is restarted.
In this situation, if a bad message is reassigned to another node, that node could fail as well. If that message is then failed over to a different active node, that node could also fail, as could all the remaining nodes, one after the other. In this way, one bad message can cause a whole cluster to fail. The cluster would then be restarted and fail again, possibly repeatedly. In the worst case, the whole cluster could be caught in an endless loop of restarts.
Blacklisting would prevent this endless cluster restart. If a node fails and a bad message cannot be processed successfully after two attempts, the message is assigned the non-delivered state (NDLV).
It may take some time before the node fails a second time. For this reason, there may be a delay before the bad message is set to non-delivered.
Message States
The graphic below provides an overview of the different processing states for messages.
Initially, a queued message has the state TBDL (To Be Delivered). While it is being delivered, it is assigned the state DLNG (Delivering). When the message is successfully delivered, it is assigned the state DLVD (Delivered).
If a server fails and is restarted, messages that are still in state DLNG are set to the initial state TBDL and returned to the queue.
Messages that are being delivered (state DLNG) at the time of the server failure are flagged as candidates for blacklisting. If a message has previously been flagged as a candidate for blacklisting, it is automatically assigned state NDLV. This prevents an endless series of failures.
When blacklisted messages are in state NDLV, an administrator must decide whether to allow the messages to be processed, or whether to set the message state to FAIL.