[ SECURITY ]·#0003·7 min read··status: resolved

My Monitoring Said the Server Was Too Slow. It Was Being Scanned.

Richard Carragher

My uptime monitor had been flapping my Nextcloud server for days. Timeouts — timeout of 48000ms exceeded, over and over. Availability had slid to around 92% over both 24 hours and 30 days. The graphs looked like a heartbeat.

The obvious diagnosis was staring at me: the box runs on an older, modest CPU, and Proxmox showed it pinned at 91% across four vCPUs during every flap. Clearly under-powered. The plan more or less wrote itself — migrate the VM to a beefier node and move on.

I’m glad I checked before I did, because the hardware was never the problem. The server was being scanned, and the way it was configured meant every scan cost me a PHP interpreter.

The symptom vs. the cause

Here’s the trap. Every observable signal pointed at CPU: the monitor timing out, Proxmox showing sustained 91% CPU with the “Full” pressure-stall metric sitting at 10–20%, the browser throwing “this site can’t be reached” during the worst windows. Memory was fine the whole time (~60% of 4 GiB). If you stopped at the dashboard, “needs a faster CPU” was a completely reasonable conclusion.

But the Apache error log told a different story. It was full of this:

[php:error] ... not found or unable to stat

Thousands of them. Requests for files like shadow-bot.php, zatura.php, wp-tot.php, t00l.php, and endless randomised *default.php variants — none of which existed on my server. Classic webshell reconnaissance: automated scanners spraying the names of known backdoors, hoping one of them is sitting on your box from a previous compromise.

They were hitting the origin at roughly 2.5 requests per second, per scanner, in long enumeration runs. And here’s the mechanism that turned an annoyance into an outage.

Why a scan for a file that doesn’t exist can take a server down

My Nextcloud runs on Apache with mpm_prefork and mod_php. That combination means every single request is served by a full OS process with a PHP interpreter embedded in it. There is no cheap path for a missing .php file. When a scanner asks for shadow-bot.php:

  1. Apache hands the request to mod_php.
  2. PHP spins up — roughly 300 milliseconds of process startup.
  3. PHP tries to stat the file, discovers it doesn’t exist, and errors out.
  4. That worker is now busy for the whole cycle.

A static “404, go away” would cost a few milliseconds. Instead each junk request cost ~300ms of a worker’s life. And the worker pool was deliberately small — 16 workers, sized so that 16 × ~70 MB fits comfortably in 4 GiB of RAM. That sizing is correct for the memory available. But 16 concurrent PHP forks will happily saturate four vCPUs, and once all 16 are tied up statting files that don’t exist, every legitimate request — including my monitor’s health check — queues up behind them until it blows past the 48-second timeout.

So the CPU graph wasn’t lying, exactly. The CPU really was pinned. It just wasn’t pinned because the hardware was too slow — it was pinned because it was doing 300ms of pointless work thousands of times, on demand, for anyone on the internet who asked.

How they found me in the first place

This is the part I found most instructive, because it punctures a comfortable assumption. I’d never published these hostnames anywhere. No links, nothing indexed. So how did the scanners know nextcloud.example.com existed to probe it?

Certificate Transparency. Every time you get a TLS certificate from a public authority like Let’s Encrypt, the hostname is written to public, append-only Certificate Transparency logs — by design, so that mis-issued certificates can be caught. That’s a genuinely good thing for the web. But it also means the moment you issue a certificate for a hostname, that hostname is public knowledge, searchable by anyone, including people building target lists for scanners.

There is no such thing as a secret hostname if it has a public certificate. Mine were sitting in the CT logs the day I first issued their certificates, and the scanners simply resolved the names, found they pointed straight at a residential connection with the web ports open, and started probing.

Four conditions had to line up for this to become an outage, and it’s worth naming them because breaking any one of them fixes it:

  1. The origin was discoverable — hostnames in CT logs, DNS pointing straight at the origin.
  2. The origin was reachable — the router forwarded the web ports directly to the box.
  3. Every probe cost an interpreter — mod_php handed missing files to PHP instead of cheaply denying them.
  4. The worker pool was small — correctly sized for RAM, but easily saturated.

The fix, in layers

I broke all four, not just one — defence in depth means not relying on any single control.

Stop feeding probes to PHP. The most direct fix: deny non-existent .php requests at Apache’s authorisation layer, before an interpreter is ever forked. A small drop-in config whitelists only the handful of real Nextcloud front-controller scripts (index.php, remote.php, status.php, and a few others) and denies every other .php request outright:

<Directory /var/www/nextcloud/>
    <FilesMatch "^(?!(index|remote|public|status|cron|v1|v2)\.php$).*\.php$">
        Require all denied
    </FilesMatch>
</Directory>

Now a probe for shadow-bot.php is refused by Apache in a few milliseconds with no PHP involved at all. The proof it’s working is in the logs: a denied request shows AH01630: client denied by server configuration with no accompanying php:error line. That absence is the whole point — PHP was never invoked. (One caveat: this also blocks Nextcloud’s web updater, which lives at a .php path, so you switch to the command-line updater. A worthwhile trade.)

Add a catch-all vhost. My Nextcloud vhost happened to be first in Apache’s load order, which quietly made it the default for any request that didn’t match a known hostname — so junk aimed at my bare IP or apex domain fell through to Nextcloud and got a PHP interpreter for its trouble. A tiny catch-all vhost that sorts first and denies everything now absorbs all of that unmatched noise into a cheap static rejection.

Take the origin off the internet entirely. The real structural fix. I removed the router’s port-forward completely and moved all ingress behind a Cloudflare Tunnel — an outbound-only connector running on the server that dials out to Cloudflare, so there is no inbound port to forward and nothing for a scanner to connect to directly. The origin’s own IP is no longer reachable from the internet at all; everything arrives via Cloudflare’s edge, where a WAF and rate limiting filter the obvious junk before it ever reaches home. My earlier post argued that “Zero Trust is not a product” — this is the same idea in practice: stop trusting the network position, and stop exposing an origin just because it’s “behind a firewall.”

Did it work?

Yes, and measurably. The clean signal is the application’s own error count. Here’s the daily error total across the weeks around the incident, versus the day I applied the fix:

  • 25 June: 430 errors
  • 13 July: 258
  • 3 July: 160
  • The day of remediation: 2

Two. The lowest daily count in the entire window, on a box running the same hardware, with the same 4 GiB of RAM I’d been about to declare insufficient. The CPU sits idle now. The monitor stopped flapping.

I also checked, carefully, that all this scanning hadn’t already succeeded before I noticed — no rogue .php files had been written under the web root, and Nextcloud’s own integrity check came back clean. This was reconnaissance that never landed, not a breach. But it’s a useful reminder that the reconnaissance itself was enough to take the service down, no successful exploit required.

What I’d take away from it

  • Your dashboard tells you what, not why. The CPU graph was accurate and completely misleading. “Server is slow” and “server is being made to do pointless work by the internet” produce identical CPU graphs. Read the actual logs before you spend money on the symptom.
  • There is no secret hostname. Certificate Transparency makes every certificated hostname public the moment it’s issued. Assume anything with a cert is known, and don’t rely on obscurity as a layer.
  • A directly-exposed origin is a standing liability, even a “hardened” one. The cheapest, most complete fix here wasn’t tuning Apache — it was making sure the origin simply couldn’t be reached directly at all. An outbound tunnel does that for free.
  • The most expensive request is the one that does real work for a stranger. If an unauthenticated probe can make your server fork a process, you’ve handed the internet a lever. Deny cheaply, and deny early.

I very nearly migrated a perfectly healthy server to fix a problem that had nothing to do with the server. The logs were right there the whole time. Read the logs.