Online Security Case Study: Donation Systems
Over the holiday weekend I got an email from Dianne, the Program Associate for our fiscal sponsorship program. Since I'm an evil slave driver of a boss, it's not entirely unusual for me to receive weekend emails from my staff. This one, however, warranted an unusual amount of attention. Dianne had noticed a barrage of suspicious donations made through our website. They all came within about an hour, they were all for either $5 or $10, and they were all made in behalf of various sponsored projects. Each of the lucky projects began with the letter 'A' or with a character like '"' that appears at the very top of alphabetized lists.
This was a situation I'd encountered a few times before, albeit on a smaller scale. I first discovered the phenomenon about 6 years ago, shortly after Fractured Atlas first started accepting donations through our website. Here it is, in a nutshell:
When someone (a waiter, a hacker, whoever) has stolen a bunch of credit cards he needs some way to find out which ones are legit and which ones have already been canceled or reported stolen. From the thief's perspective, the ideal testing mechanism:
- Is online and anonymous
- Is fast and easy (i.e. doesn't require an elaborate shopping cart system)
- Allows for posting transactions of any arbitrary dollar amount
Charities that offer online donation systems fit the bill perfectly. It's happened to us 2 or 3 times over the past 6 years. As far as I can tell, someone's got a list of credit card numbers and they're looking to run down the list as fast as possible. Based on this weekend's performance, it's possible to "test" about 50 stolen cards in 15 minutes using Fractured Atlas's online donation form. It would probably take an hour to do that on Amazon.com, which is why they use us instead.
On a practical level, this is mainly an irritation rather than a real problem. After all, none of our security systems have been breached and none of our customers' information has been stolen. We catch the issue quickly, refund all of the transactions, notify the relevant authorities, and apologize to the sponsored members for the inconvenience. Still, it's a pain in the butt and a distraction for my staff.
In the spirit of transparency and in the hopes that this might be useful for other folks having the same experience, I'll briefly describe our response and attempt to mitigate this exposure going forward. I'll also try to update this post if and when we find out whether our tactics have worked. (If you're not at least a little bit of a nerd, you may want to stop reading now.)
The very first course of action, of course, is exactly what I described above. Dianne refunded all of the transactions and notified all of the sponsees this morning. I notified our merchant account provider and our gateway processor to be on the lookout for any future transactions that match this pattern.
After that, I got my geek on. Our IT systems administrator and I started combing through server logs from the weekend to identify the IP addresses of the offending computers. Of the 250 bogus transactions, it's safe to say few fit the profile of the average Fractured Atlas donor:
~50% were from Jakarta
~30% were from China
~20% were from a small town in the U.K.
There's no way of knowing whether these were three different individuals or one culprit using various proxy servers. Either way, this is a slam dunk case for a more aggressive firewall policy. As of this afternoon we've blocked:
1) An entire ISP in Jakarta
2) An internet cafe in China
3) A small town in the U.K.
Is there some possibility that legitimate visitors to the site will get trapped in the net? Sure, but it's not especially likely. Eventually we'll probably scale back on the rules anyway; the important thing is to make this enough of a pain that the bad guys find some other website to use for their testing.
We've also tightened the address verification settings on our credit card processing system. The most interesting thing about this latest batch of transactions was that, in each and every case, the "donor" was able to provide an accurate CVC for the stolen card. It's illegal for merchants to store those, which means that the credit cards themselves must have been physically stolen (i.e. as opposed to stolen from some online merchant's database). Stricter address verification should help provide an additional line of defense. (Unfortunately, we've already seen a rise in legitimate transactions getting trapped in the filter because someone's address doesn't match perfectly, but this may be the price we have to pay, at least temporarily.)
And that brings me up to the moment. Hopefully this has been educational, or at least not mind-numbingly boring. Again, I'll try to to update this post if and when there are further developments.
To all of our sponsees who've been getting phony donation notices, I apologize for the inconvenience. I wish I could tell you that you've somehow become overnight fundraising superstars in Jakarta. Alas, for the moment, you should probably hold off on plans for any Indonesian bookings.
UPDATE (1/23/08): Zillions of members have had their legitimate billing information rejected by the new, tighter address verification scheme. I feared this might happen. It's really hard to verify billing addresses in a reliable way, since "St." and "Street" and "STREET" are all considered different and the credit card companies have no standardized format. We'll try tinkering with the verification settings and see if we can find the right balance between fraud-protection and usability.
UPDATE (2/21/08): Well, we managed to come up with a less aggressive address verification scheme that resulted in fewer failed membership dues rebills. For a few weeks there I thought we were home free. But then yesterday, the bastards were back with a vengeance...
Previously it appeared we were dealing with one (maybe 2-3 tops) individual credit card thieves, who were manually testing stolen cards with our donation processing system. Yesterday afternoon we were targeted by a far more sophisticated operation. Forensic analysis led to the inescapable conclusion that we were "attacked" by a botnet. This is much more difficult to defend against, since the zombie computers that were performing the test transactions were all over the world and coming from places that we couldn't just block (e.g. Verizon DSL customers in NJ). They were also ruthlessly efficient. Each "bot" would test roughly one new card every 2-3 seconds. They appeared to have a scarily good database of stolen information, too, since even with address verification and card security code checks, they were successfully charging perhaps 1 of every 10 credit cards. This resulted in perhaps 50 "donations" for $0.25 each within a 20 minute period.
My systems administrator and I immediately fought back. We began monitoring server traffic in real-time. Each time a suspicious computer would start posting rapid-fire donation attempts, we'd block the IP address. Within 2 minutes, however, another bot with a different IP address would take over and resume the assault. It quickly became clear that this was not a fight we could win, at least not this way.
Last night at around 6pm, I took down the online donation page on the Fractured Atlas website. This page has been up and running for 7 years now, so this was a disappointing development. However it was a necessary temporary move while we developed a more sophisticated defense. Fortunately, I was able to put something together that I think will be pretty effective, and the donation page went back up around 8:45pm. On the remote chance that one of the bad guys is reading this, I won't disclose all the gory details. Suffice it to say that our donation processing system now tracks the frequency of donation attempts in real-time and automatically blocks suspicious IP addresses.
This will almost certainly not be a 100% effective solution, but there's no such thing as perfect security. The goal is to make it annoying enough to mess with you that the bad guys decide to bother someone else (some easier target) instead. Hopefully we've done that.
Today we put up a CAPTCHA. Trust me, I find these as annoying as you do, so I really resisted taking this step. Of all the options available to us at this point, however, this is almost surely the least annoying and the least likely to deter or frustrate legitimate donors. As always, I'll keep you posted.