2026-09-15

The proxy that erased the client's address

On our own platform, the attempt limit on the “forgotten password” page fired for every visitor at once: five requests from anywhere, and the feature closed for everyone. Every call reached the application with the same address, that of the tunnel between our two machines. The first diagnosis, on 18 August 2026, named the wrong culprit. The second, on 31 August, named the right one in three requests.

A diagnosis drawn from a single observation

On 18 August we read the X-Forwarded-For header as it arrived at the end of the chain: a single address, the tunnel's. We concluded that the edge server, the first to receive the request, was dropping the client's address, and that the fix belonged there. The note was written, the fix postponed. The reasoning was wrong, and it cost a second working session.

A marker planted at every hop

On 31 August we sent a recognisable X-Forwarded-For header to each hop of the chain, one after the other. Sent straight to the application, it survives. Sent to the intermediate service, it survives. Sent to the inner Caddy, the one that receives the tunnel's traffic, it is dropped and replaced with the peer's address. The culprit was named in three commands.

Caddy's behaviour, correct and little known

Since version 2.7, Caddy replaces the X-Forwarded-For header instead of appending to it, unless the immediate peer is on its list of trusted proxies. Our inner Caddy, version 2.11.4, trusted nobody: it therefore erased the chain the edge server had correctly built. That edge server was doing exactly the right thing, since its peer really is the public Internet. The fix is one line, trusted_proxies static followed by the tunnel's address, and nothing else.

The proof, before and after

We checked with the measurement that had failed two weeks earlier: from our own machine, the limit fires after five requests; at the same moment, a request from another public address goes through. The denial log now names the client's real address, where it used to write the tunnel's address for everyone. We then replayed the reverse attack: a client sending its own X-Forwarded-For through the edge server is still counted under its real address, because the edge overwrites whatever address the client claims.

The rule we keep

The first diagnosis had been drawn from a single observation, made at the end of the chain. When a header disappears between several proxies, we now plant a marker at every hop before reasoning: three requests name the culprit beyond dispute. One check remains in place: any request whose address chain contains no public address is written to the log. On a healthy platform that line never appears, and its absence is the proof that continues.

When a header disappears between several proxies, a marker planted at every hop names the culprit in three requests; reasoning from the end of the chain missed it for two weeks.