HTTP/2 became the default on the browser web long ago, and from there it gradually crept into service-to-service communication — usually unnoticed, riding along with a library or platform upgrade. Yesterday your client talked to the neighboring service over HTTP/1.1; today, after a version bump, it speaks HTTP/2. Nothing changed in your code, everything works… until the first idle period, traffic burst, or lost packet.
This post is about the set of pitfalls that HTTP/2 brings into internal traffic. The set is well known: the gRPC community worked through it about a decade ago and armored itself with protections, the Go community somewhat later and with loud issues. But if your stack got HTTP/2 "for free", those protections are most likely not there.
What HTTP/2 changes — and why every feature backfires
Compared to HTTP/1.1 there are three fundamental changes, and each has a flip side.
One connection instead of a pool. The client opens a single TCP socket and multiplexes all requests into it in parallel. It saves handshakes and ports — but now all requests share one fate: whatever happens to that socket takes down every request of the service at once. In HTTP/1.1 a stale pooled connection cost you one failed request; in HTTP/2 it can cost you the whole service.
Stream multiplexing. Requests travel interleaved, each with its own stream id. Convenient — but the session state (stream counters, limits, queues) becomes a shared resource that can leak or run out. An accounting bug in one request affects every request after it.
Session-level flow control. Besides the per-stream window there is a shared window for the whole connection: the receiver acknowledges consumed bytes (WINDOW_UPDATE), and the sender must not exceed the window. If the receiver stops reading, the window drains to zero and every stream on the connection freezes. This is by design: the RFC openly warns that careless flow control handling leads to deadlocks.
These three properties combine into four typical scenarios that look identical in production: "the service stopped responding, only a restart helped".
Pitfall 1: the half-open connection
Between two services in production there is almost always someone in the middle: a load balancer, a proxy, conntrack, NAT. Many of them have idle timeouts, and they kill connections differently — some with honest RSTs in both directions, some by silently dropping the entry from their table. Add the mundane loss of a FIN packet during close. The result is the same: the client holds an ESTABLISHED socket while nobody is left on the other end. Plain TCP will notice only on a write, after tens of minutes of retransmissions. With HTTP/1.1 such a connection would kill one request; with HTTP/2 every request multiplexes into the dead socket. One way to move that layer out of the application is a sidecarless mesh.
Pitfall 2: the flow control deadlock
A server under load (a GC pause, a heavy operation, an exhausted thread pool) stops reading request bodies. The shared session window hits zero, every sender stalls. If the server never gets back to reading, the connection is wedged forever — while looking perfectly healthy at the TCP level. The diagnostic sign on the server side: sockets in CLOSE_WAIT with unread bytes in Recv-Q — the clients gave up and left, the server never read or closed.
Pitfall 3: the pool that never evicts the dead
The most treacherous pitfall is on the client. You would expect: requests time out, so drop the connection and open a new one. But many implementations need a socket error to evict — and there is none: the socket is alive (pitfall 1) or formally alive (pitfall 2). Individual requests fail, the session stays "valid", the pool keeps handing it out. The textbook example is golang/go#39750: a dead HTTP/2 connection is cached in the pool forever, because new requests entering it mark it "busy" — so the idle-connection cleaner never touches it either.
Pitfall 4: contention for a single pipe
Even without failures: N workers writing large bodies in parallel through one session share one flow control window and one TCP buffer. A load burst creates the conditions for the deadlock from pitfall 2 — and then walks right into it. The profile "batch writer with ten threads" on HTTP/2 without window tuning is an incident waiting to happen.
How the grown-ups defused these pitfalls
All mature HTTP/2 ecosystems converged on the same set of protections:
| Protection | What it buys | Who made it a default |
|---|---|---|
| PING health checks | A half-open connection is detected in seconds, not half an hour | gRPC (keepalive), Go (ReadIdleTimeout) |
| Connection age limit | The connection is rotated before it has a chance to go stale | gRPC (MAX_CONNECTION_AGE, with jitter against reconnect storms) |
| Stalled-stream timeout (server) | A stream making no progress is reset and returns its window to the session | Jetty (streamIdleTimeout), gRPC servers |
| Sane window sizes | Parallel large bodies do not exhaust the session window | Tuned per workload; defaults are often small |
It is telling why these protections appeared: the gRPC documentation names the reasons directly — proxies killing idle connections, and lost FINs after which TCP stays silent for up to half an hour. Exactly pitfalls 1 and 3 from the list above, learned on other people's production years ago.
Our case: every pitfall in a single incident
A fresh illustration from our own practice. After a Solr 9 → 10 upgrade the SolrJ client library switched to HTTP/2 (cleartext h2c). The indexing service started hanging solid every few days — every request to Solr timing out, curable only by a restart. Its neighbor, writing the same content with the same client, ran without a single error.
The explanation matched the list point by point: the neighbor sends requests one at a time (a single POST cannot exhaust the session window), while the indexer's scheduler, after every pause, dumps its backlog into ten parallel workers pushing through one session (pitfall 4) exactly when Solr is busy committing and stops reading (pitfall 2). On the server — a collection of CLOSE_WAIT sockets with unread kilobytes; on the client we also found a half-open socket (pitfall 1). And the SolrJ client, young and without PING plumbing, never once evicted the wedged session from its pool (pitfall 3). The initial theory — "it's the network, conntrack kills idle connections" — did not survive the facts. The network turned out to be innocent, which is typical for this class of problems: the network gets blamed first.
The treatment also came from the standard set: a server-side streamIdleTimeout, a larger session window, a higher server idle timeout — and strategically, moving the batch writer to HTTP/1.1: for its workload profile, isolated pooled connections beat a shared pipe.
A checklist before adopting HTTP/2 between services
- Does your client send PINGs on an idle connection? If not — how will it learn about a half-open socket?
- Can the pool evict a session after a series of timeouts, not only after a socket error?
- Is the connection age limited? What happens on a server deploy or scale-out — will clients notice the new endpoints?
- Does the server reset streams that make no progress, or does a stalled stream live forever and hold the window?
- How many parallel large bodies does your workload push into one session? Is the window big enough?
- And an honest question to close: do you need HTTP/2 here at all? For "a dozen RPS between two pods" a pool of HTTP/1.1 connections is more boring — and noticeably more survivable.
Takeaway
HTTP/2 is a good protocol with an honest trade-off: connection economy in exchange for shared fate of requests. It expects mature plumbing around it — pings, rotation, stream timeouts. The browser web and gRPC have that plumbing out of the box. When HTTP/2 arrives in your internal traffic "for free", nobody ships the plumbing with it — and you get a failure class with a signature symptom: "everything hung, no errors, a restart helped". Now you know what it looks like and where to look: ss -tan on both sides plus Recv-Q/Send-Q will tell you more than any logs. And on when such an incident may be considered closed, there is a separate note on severity and the definition of recovered.