We Got Yelled At by Amazon So You Didn’t Have To

One day during my freshman year of college my brother called me in a panic. Our mother had found a few bottles of cheap liquor in the house, and his desperate plan was simple: I should say they were mine. I laughed and agreed - Ann Arbor was nearly four hours away and I was pretty sure my RA wasn’t going to be able to keep me in my room.  A few hours later when my mother called her first question was:  “Were the bottles I found yours?”  In the spirit of brotherly camaraderie I confidently said “yes.”

These days I work at GreyNoise, where the stakes are a bit higher than vodka in the basement.  We run a global network consisting of thousands of sensors in more than seventy countries.  These sensors are designed to look very tempting to attackers:  unpatched, exposed, and vulnerable. We believe the design is pretty good, since on an average day more than 600,000 different IPs complete hundreds of millions of sessions, with about half being attacks. It’s like we left our front door wide open and can watch who walked in.

Of course, the problem with leaving the door open is that sometimes an intruder can make themselves a bit too comfortable. Every now and then an attacker actually compromises one of our sensors and starts using it to do something malicious - scanning for targets, deploying malware, etc.  We usually detect this and then automatically repair and restore the sensor long before any activity can occur.  Occasionally, however, an attacker moves fast enough to briefly start their work. That’s when we get the notification:

Subject line: AWS Abuse Report.

Translation: Amazon is politely asking us to get our act together, and quickly.

When that happens, we don’t get defensive. We get to work, shutting down the sensor and figuring out how to better detect and remediate quickly.  We never get upset that the sensor was compromised, however.  After all, the whole point of GreyNoise is to ensure that no attack works twice; that means we have to be the ones attacked first. Every time a new exploit or attack technique hits our honeypots, we learn from it, detect it, and share information so that it won’t work again on anyone else. 

Let me make an important point: AWS is a great partner to work with, and we’re grateful whenever they help us catch something we missed.  We hope AWS never has to send us an abuse report — because ideally, we should catch a popped sensor before they do. That’s the goal. And when one does slip through the cracks, it’s a reminder that our job is working: we take the hits so you don’t have to. 

We think we’re doing a good job of this, and hope you feel the same. Either way, we’re doing better than I did back then. I tried to be a decoy for my brother and took the heat — just not very well. At GreyNoise, being the decoy is the job, and we’re a lot better at it. 

“That’s sweet of you to lie for your brother; he’s grounded, and we’ll talk about you next time you come home for break,” my mother said.

Turns out, taking the blame works best when it’s part of the plan.