The connection pool exhaustion that was not a connection pool problem

The alert said the connection pool was exhausted. The dashboard agreed: pool utilisation pinned at 100%, requests queuing, p99 latency through the roof. The obvious reading was that the database could not keep up.

It could keep up fine. Here is the trail.

The symptoms

A Node service in front of PostgreSQL, running a pool of 20 connections. Under normal load it sat at three or four in use. Roughly once a day, over about ninety seconds, it would climb to 20, stay there, and start rejecting work.

pool.total = 20   pool.idle = 0   pool.waiting = 143

pg_stat_activity was the first thing worth looking at, and the first thing that did not fit:

SELECT state, count(*), max(now() - state_change) AS longest
FROM pg_stat_activity
WHERE datname = 'app'
GROUP BY state;

    state    | count |     longest
-------------+-------+-----------------
 idle        |    20 | 00:01:47

Twenty connections, all idle. Not active, not idle in transaction — idle. The database was not executing anything. It had finished every query it had been given and was waiting for the application to either send more work or hand the connections back.

So the problem was not database throughput. Something was holding connections open without using them.

Finding the holder

The pool was checked out in a request handler that looked, in outline, like this:

async function handler(req, res) {
	const client = await pool.connect();
	try {
		const order = await client.query(ORDER_SQL, [req.params.id]);
		// enrich from an internal pricing service
		const pricing = await fetch(`${PRICING_URL}/quote/${order.rows[0].sku}`);
		const quote = await pricing.json();

		await client.query(AUDIT_SQL, [order.rows[0].id, quote.price]);
		res.json({ ...order.rows[0], price: quote.price });
	} finally {
		client.release();
	}
}

The connection is checked out, a query runs, and then the handler makes a network call to another service — while still holding the connection. The second query needs the same client, so the checkout has to span the fetch.

Under normal conditions the pricing service answers in about 15ms and nobody notices. The connection is held for maybe 20ms.

The actual cause

The fetch had no timeout.

Node’s global fetch has no default timeout. Neither does http.request. If the remote end accepts the TCP connection and then never responds, the promise stays pending — not for 30 seconds, not for 5 minutes. Indefinitely, until the socket is closed by something else.

The pricing service sat behind a load balancer that, during its own deploys, would accept connections before the upstream was ready. For about ninety seconds per deploy, requests were accepted and never answered.

Every request into our service during that window checked out a connection, issued its first query, called the pricing service, and stopped. Twenty requests was enough to take the entire pool. The 143 waiting requests were queued behind connections held by handlers waiting on a socket that was never going to reply.

The pool was not exhausted by database load. It was exhausted by another service’s deploy, transmitted through our code by a missing timeout.

The fixes

Three changes, in order of how much they mattered.

Do not hold a connection across a network call. This was the real defect. The pattern should be: talk to the database, release, do the remote work, reacquire.

async function handler(req, res) {
	const order = await pool.query(ORDER_SQL, [req.params.id]);

	// No connection held here.
	const quote = await getQuote(order.rows[0].sku);

	await pool.query(AUDIT_SQL, [order.rows[0].id, quote.price]);
	res.json({ ...order.rows[0], price: quote.price });
}

Using pool.query directly checks out a connection per statement and returns it immediately. The connection is now held for the duration of a query rather than the duration of a request. Where two statements genuinely must share a transaction, the remote call has to move outside it.

Give every outbound call a timeout. No exceptions, including calls to services you own — especially those, since you will deploy them.

const response = await fetch(url, {
	signal: AbortSignal.timeout(2000)
});

Fail fast on checkout. A bounded wait converts a pool starvation into fast errors on the affected requests rather than a queue that swallows everything:

new Pool({ max: 20, connectionTimeoutMillis: 5000 });

What to take from it

The metric that fired was pool utilisation, and pool utilisation is a symptom of concurrency, not of database performance. It rises when connections are held for longer, and holding is not the same as using. pg_stat_activity distinguishing idle from active was the whole diagnosis: the database was idle, so the delay was on our side of the socket.

The generalisation is worth internalising. A pool is a queue for a finite resource, and its depth is governed by holding time. Any unbounded wait inside a critical section will eventually consume the pool — and an HTTP call without a timeout is an unbounded wait, sitting in the middle of a critical section, waiting for a system you do not control.