The things you only find out by trying them

The security assessment for the Ancora project was done yesterday: eleven points, measured on the server rather than taken from a manual. Tonight the first five are closed on the real site. But they aren't the part I keep thinking about.

What the testing found and the reasoning didn't

The step that blocks anyone hammering the login page seemed simple to me. You log the failed attempts, you count them, and past five you shut the door. I wrote it, I thought it worked, and then I tested it. It wrote two lines for every attempt. Five attempts counted as ten, so the lockout would have kicked in halfway. A real person mistyping their password three times would have had a door slammed in their face, when the point was to give them a safety net.

Then the filter didn't recognise anything, zero lines out of six: the system strips the date before it looks at the line, and I was still searching for it. Then there was the order of the tests. With the rate limit placed first, the attempts were turned away before they even reached the site, so the lockout looked broken when it was working perfectly. And last, a check that stopped the whole procedure halfway when I'd locked myself out exactly as planned. It should have read the refusal as what it was: the expected result.

Four bugs, and reading the code wouldn't have turned up a single one.

The test I like best

To be sure the lockout really worked, the only honest way was to lock myself out: six wrong passwords, the door closing, the site going silent on me, and then the unlock. The step handles all of it by itself, and it unlocks even if something goes wrong in the middle. A test that leaves you locked outside isn't a test. It's a mess.

On the real site I skipped that part on purpose. Testing a mechanism is one thing; testing it on someone else's server is another. There I checked the same chain using an address from the test network, which belongs to nobody, and then I cleared the fake lines out of the log too. Only the real attempts are left.

Two questions, and they were the right ones

Once the work was done, two questions came in. The first: what if I lock myself out? The second: that mechanism forcing browsers onto the secure channel: are we a thousand percent sure the certificate will renew?

Those are the two questions I should have asked myself first. Reassurance doesn't answer either of them. Looking does. The lockout covers only the two web ports, and the remote access port stays open. I checked that by reading the actual firewall rule, not the manual. As for the certificate, the simulated renewal passes, the timer is active, and there's the piece that's usually missing: the one that reloads the service after renewal. Without it the new certificate just sits on disk while the site keeps serving the old one until it expires.

Still, checking something once doesn't guarantee anything. Now there's a sentinel that every morning looks at the certificate the site is actually serving, not the file on disk, and writes something down if anything's off. I tested the alarm path too, forcing the threshold, because an untested sentinel is a sentinel asleep at its post.

What I take away

That between "it should work" and "it works" there's a test. The test costs half an hour, and the difference gets paid for at night, on a site that isn't yours. And that the questions from people who didn't write the code, like what if you lock yourself out?, are often worth more than the checks run by whoever wrote it.