Eleven bytes can make an OpenSSL server set aside up to 131 KB of memory for a packet that never arrives, and that memory stays frozen until the process restarts. That’s the core of the OpenSSL HollowByte flaw disclosed by Okta’s Red Team on July 18, 2026.
Key Takeaways
- Older OpenSSL releases allocate a receive buffer based solely on the TLS header length, before any payload is verified.
- When the attacker sends a 11‑byte ClientHello with a fabricated length, the server reserves up to 131 KB per connection.
- glibc’s allocator keeps those chunks in user space, so the memory never returns to the kernel, causing heap fragmentation.
- Okta observed a 1 GB server OOM‑killed after 547 MB of memory was frozen; a 16 GB server lost about 25 % of RAM without hitting the connection limit.
- OpenSSL released patches on June 9 (4.0.1, 3.6.3, 3.5.7, 3.4.6, 3.0.21) without a CVE, advisory, or changelog entry, leaving users in the dark.
OpenSSL HollowByte flaw: How eleven bytes stall servers
What makes the issue so insidious is that the TLS handshake itself trusts the length declared in the three‑byte size field of the record header. That’s all it needs before it even looks at the first byte of the body. The older code grew the receive buffer to that declared size the moment the header landed, and then it waited for the rest of the payload that never showed up.
The technical root: premature buffer allocation
Every TLS handshake message starts with a four‑byte header, three of which indicate the body length. In the vulnerable OpenSSL versions, the library allocated a buffer equal to that length as soon as the header arrived. Because the allocation happens before any validation, an attacker can claim a maximum size—up to 131 KB for a ClientHello—so the server reserves that much memory instantly.
Then the worker thread blocks, waiting for the missing bytes. No authentication, no session, no key exchange ever happens. That’s a classic connection‑exhaustion pattern, but the twist is what follows when the client drops the connection.
Why glibc makes the issue linger
When the attacker aborts, OpenSSL frees the buffer, but glibc’s memory allocator keeps small and medium chunks around for reuse instead of returning them to the kernel. Okta’s tests varied the claimed size on each connection, which was enough to prevent the allocator from re‑using the freed chunks. The result is a fragmented heap that never shrinks, and the resident set size climbs and stays high long after the attack is over.
- On a 1 GB test server, the attack triggered an OOM kill after 547 MB of memory was frozen.
- On a 16 GB server, roughly 25 strong> of system RAM stayed locked without ever reaching the connection ceiling.
- Standard connection‑limiting defenses didn’t stop the attack, as Okta’s Red Team put it.
“standard connection-limiting defenses won’t stop it”
Impact on real‑world deployments
Okta ran the exploit against NGINX front‑ends backed by vulnerable OpenSSL builds. The servers didn’t need thousands of concurrent connections; a few hundred malformed ClientHello messages were enough to tip the memory usage over the edge. That’s why the Red Team says the flaw is more dangerous than a typical Slowloris‑style DoS.
Because the memory never returns, the attack can be sustained with minimal bandwidth. An attacker could keep a low‑volume flood going for hours, and the server would keep chewing RAM until it crashes or the administrator restarts the process.
Historical Context
OpenSSL has been the default TLS implementation for most Linux distributions for more than two decades. Over that span, the project has patched dozens of memory‑related bugs, many of which were disclosed with full CVE assignments and advisory notes. The community expects a similar level of transparency for every security‑relevant change.
Earlier this decade, a series of DoS vectors targeted the same allocation path but relied on deliberately malformed payloads that triggered a buffer overflow rather than a freeze. Those incidents prompted OpenSSL to introduce stricter length checks in newer branches. The HollowByte flaw shows that the older code path, still present in long‑term support releases, never received the same hardening.
In parallel, other TLS libraries such as BoringSSL and GnuTLS have implemented defensive programming patterns that allocate buffers only after the entire record is validated. Those patterns are now considered best practice, yet the vulnerable OpenSSL branches continued to allocate on header receipt.
When the Red Team uncovered HollowByte, the timing aligned with a broader industry focus on supply‑chain resilience. Organizations were already tightening their CI pipelines, but the lack of a CVE meant that many automated scanners simply missed the flaw.
That historical backdrop underlines why a seemingly tiny eleven‑byte packet can have outsized impact. The flaw lives at the intersection of legacy code, allocator behavior, and a disclosure model that assumes “hardening‑only” changes don’t need public notice.
What This Means For You
If you run an OpenSSL version older than the June 9 fixes, you should treat the HollowByte flaw as a high‑priority DoS risk. Restarting affected services after a patch is essential because the memory fragmentation only clears when the process exits.
For glibc‑based deployments, consider adding external connection‑rate limits or using a front‑end that enforces stricter TLS record size checks. Switching to a different allocator—such as jemalloc or tcmalloc—might mitigate the lingering fragmentation, but you’ll still want to patch the OpenSSL library itself.
Three concrete scenarios illustrate the immediate relevance:
- Micro‑service clusters. A Kubernetes pod running a Python Flask app behind an NGINX ingress can be taken down by a handful of malformed handshakes. The pod’s OOM killer will restart the container, causing a brief outage that could cascade through dependent services.
- Legacy payment gateways. Many point‑of‑sale terminals still rely on OpenSSL 1.1.x. A remote attacker who can reach the gateway’s public IP could trigger the memory freeze, forcing a reboot and potentially disrupting transaction processing during peak hours.
- Edge caching servers. CDNs that use OpenSSL for TLS termination often run on high‑memory VMs. Even though the VM has ample RAM, a sustained HollowByte attack can lock a quarter of that capacity, reducing cache hit rates and increasing latency for end users.
In each case, the mitigation steps are the same: patch the library, restart the process, and consider a front‑end that validates TLS record lengths before passing traffic to OpenSSL.
Beyond immediate fixes, teams should audit their monitoring stacks for unusually high RSS values that don’t correlate with traffic spikes. An unexplained rise could be the silent signature of a HollowByte‑style attack.
Competitive Landscape
OpenSSL isn’t the only TLS provider in the ecosystem. BoringSSL, maintained by a major technology company, has a more aggressive release cadence and typically publishes detailed security notes for every change. GnuTLS, another open‑source contender, follows a similar disclosure model, issuing CVEs for low‑severity bugs as well.
Because those libraries allocate buffers only after full record validation, they are not vulnerable to the exact same premature‑allocation trick. That difference makes them attractive alternatives for high‑risk environments, even though they may lack some of OpenSSL’s feature breadth.
Enterprises that already use OpenSSL for its extensive algorithm support can mitigate risk by deploying a reverse‑proxy that runs a different TLS stack. The proxy consumes the client‑side handshake, applies strict size checks, and then forwards a clean, verified record to the OpenSSL backend. This architecture adds a layer of defense without requiring an immediate library swap.
That approach mirrors a broader trend: decoupling TLS termination from application logic. When the termination point runs a hardened library, the downstream services inherit that protection automatically.
Key Questions Remaining
- Will OpenSSL adopt a policy that forces a CVE for any change that affects memory usage, even if classified as “hardening‑only”?
- How will downstream distributors like Red Hat revise their backporting practices to surface such fixes more clearly?
- Can allocator‑level mitigations be standardized across Linux distributions to reduce the impact of similar fragmentation bugs?
Answers to those questions will shape how quickly the community can respond to future low‑level flaws. Until then, visibility remains limited, and operators must rely on manual checks.
Looking ahead: Will OpenSSL change its disclosure policy?
OpenSSL’s decision to skip a CVE for a flaw that can silently kill production servers raises questions about transparency. Will future releases include a clearer severity label, or will the community push for a policy tweak that forces a CVE even for “hardening” fixes?
Only if the maintainers will adjust their triage process after this episode, but the pressure from downstream distributors and security‑focused users is already building.
Sources: The Hacker News, original report
