What you are looking at

Four producers publish into a queue that holds at most 300 messages; up to twelve consumers pull from its head, each handling about six messages a second. As long as consumers keep up, the queue stays nearly empty and lag — the age of the oldest waiting message — stays near zero. Push the production rate past what the consumers can drain and depth and lag climb together; that is the whole story of most queue incidents.

Backpressure or drop

A bounded queue has to do something when it fills. With backpressure on, producers are slowed once depth crosses the 70% high-water mark, in proportion to the headroom left — the pressure travels upstream, where a caller can wait, shed or return 429. Turn it off and nothing slows down: the queue fills to capacity and new messages are dropped on the floor. Neither is free, but only one of them is a decision you made.

Scaling on queue length

With KEDA on, the consumer count follows the queue itself: desired replicas = ⌈(waiting + in-flight) / 15⌉, up to the maximum you set. Scale-up is immediate but new consumers need a moment to start, so lag spikes before it recovers; scale-down waits for a 5-second cooldown so a brief lull does not cause flapping. Switch KEDA off, leave two consumers and hit Spike to see why scaling on CPU is the wrong signal for a worker that mostly waits on I/O.

Retries, poison and the dead-letter queue

Failures are retried at the tail with exponential backoff (0.25 s, 0.5 s, 1 s…). Transient failures — the flaky downstream — usually succeed on the second try. A poison message never will: here it hangs for 1.2 s and then fails, every time. With three retries it is dead-lettered after four attempts and a human can inspect it later; with infinite retries it circles forever, and six of them are enough to keep half the consumers busy doing nothing. The numbers are simulated and time is compressed.