My run-in with unauthorised Litecoin mining on AWS(vertis.io)
vertis.io
My run-in with unauthorised Litecoin mining on AWS
http://vertis.io/2013/12/16/unauthorised-litecoin-mining.html
22 comments
The author suggested enabling billing alerts. For those running on Azure, billing alerts are currently in preview mode, and can be enabled via https://account.windowsazure.com/PreviewFeatures
I know there are a few code-quality bots on Github, but is there any service that you can install as a webhook which automatically checks for things like Amazon key pairs (which, IIRC, always start with "AKIA", at least the API keys anyway)?
I'm also curious if there's some utility or at least a list of regular expressions that can be used to scan for a number of credentials, not just AWS.
Amazon will refund you if you explain your situation.
sorry dude :(
Ugh, that sucks. Too late to help you now (but perhaps others) on your billing alerts points: check out http://cloudability.com -- alerts, analytics, prediction, suggestions, etc. Free for the most useful stuff.
Is this illegal? Could he somehow go to some authority?
EDIT: Why is it unlikely the FBI will successfully investigate?
EDIT: Why is it unlikely the FBI will successfully investigate?
Ask a simple question, get a simple answer: it is obviously and unambiguously illegal. You can certainly refer this to the FBI computer crimes folks and your local law enforcement. It is unlikely they will successfully investigate.
Yeah, I'll discuss this with Amazon when they get back to me, but I don't hold much hope for a positive outcome.
What specifically is illegal about that? Someone gave you credentials to an online service. Now you use those credentials. What law did you brake? You probably broke the ToS/EULA/whatever, but that by itself is not illegal.
(I'm not saying otherwise, I'm asking)
(I'm not saying otherwise, I'm asking)
Just because you have the means to do something does not mean you have the permission. If I drop my car key on the ground and you take it, it's still stealing. I gave you the means (accidentally), not the permission. Especially when using the ill-gotten item will cost me (gas, time, money).
Reminds me of the old phrase: What's the difference between hacking and penetration testing? Permission.
Reminds me of the old phrase: What's the difference between hacking and penetration testing? Permission.
Yes, I see that difference. I also know there are laws protecting computer systems and such. But was wondering if it could be applied to an online service.
For the US, pretty much anything is covered under https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act
statutory minimum is 5k. we work with federal law enforcement somewhat frequently at wepay to deter fraud, and it appears that the informal threshold to get their interest is about 100k of damages
Illegal? Yes. Convictable? Yes.
But will any law enforcement department or agency actually look into it? Highly unlikely. There's so much major scale white collar crime/cybercrime going on daily that it's generally too hard to dedicate resources to smaller cases like this.
Now, if they got evidence the perpetrator was doing this with dozens or hundreds of AWS accounts, that would be another story.
But will any law enforcement department or agency actually look into it? Highly unlikely. There's so much major scale white collar crime/cybercrime going on daily that it's generally too hard to dedicate resources to smaller cases like this.
Now, if they got evidence the perpetrator was doing this with dozens or hundreds of AWS accounts, that would be another story.
Possibly because it would cost the FBI orders of magnitude more than the damages to investigate.
The damages which amazon will likely refund.
The FBI does not target petty thieves.
The damages which amazon will likely refund.
The FBI does not target petty thieves.
Amusingly, I went to a meetup with some feds the other day.
Their statutory minimum to investigate is 5k, but they suggested it's worth getting in touch with them either way. If nothing else, it'll give them an excuse to say "cyber" a few more times.
Their statutory minimum to investigate is 5k, but they suggested it's worth getting in touch with them either way. If nothing else, it'll give them an excuse to say "cyber" a few more times.
This. I was defrauded to the tune of $3,500 (run of the mill stolen Paypal account bought a top of the line iMac from me on eBay) and after discovering to my horror that both eBay and Paypal sidestep all fraud for protection... I called the FBI, police, etc.
They listened politely until I said how much it was worth -- one of my favorite quotes was "If you added a zero to the end, we'd knock down their door at 6AM tomorrow. For $3,500 it's simply not worth our time."
They listened politely until I said how much it was worth -- one of my favorite quotes was "If you added a zero to the end, we'd knock down their door at 6AM tomorrow. For $3,500 it's simply not worth our time."
Oh, they certainly do, if powerful enough people complain.
http://en.wikipedia.org/wiki/Aaron_Swartz
The question is whether the poster is such a person.
http://en.wikipedia.org/wiki/Aaron_Swartz
The question is whether the poster is such a person.
The FBI wasn't involved in Swartz's prosecution, which was considerably easier for the US Attorney by dint of them actually knowing who was behind the incursions.
Now I'm curious, how many litecoins would it have generated in two days?
[deleted]
One cc2.8xlarge instance will give you 85 kH/s so 20 instances would give 1700 kH/s which only nets about $18 at current market prices / difficulty. So over 2 days he would have made off with a whopping $36.
Writing a script that generated $38 every time somebody accidentally commits a AWS key sounds like an amazing source of extra cash, depending on your current income.
If he would have used an altcoin that is limited by CPU like Primecoin he could have earned around 8000$ in 2days. Not sure if my calculation is legit through. It's based on the chains/day from the list at http://anty.info/primecoin-calculator/
Scary. I'm sure a lot of universities and servers will see an influx of hacks for coin-mining.
Scary. I'm sure a lot of universities and servers will see an influx of hacks for coin-mining.
Less than the cost of the ec2 instances, though that is still a wide window
Wish I knew.
You can (and should) set up an AWS CloudWatch alert on your account that will send you an email or SMS notification when your monthly bill exceeds a set threshold.
I was surprised how incredibly difficult that is to set up. Eventually I dead-ended following the instructions when CloudWatch told me there were 0 metrics to choose from for monitoring...
Hi,
There are a couple of things that are easy to overlook when using billing metrics.
1. All billing metrics are stored in us-east-1 even for usages in other regions.
2. If you are using consolidated billing, billing metrics will be published under the linked account, and will only be visible to that account.
Hope that helps.
There are a couple of things that are easy to overlook when using billing metrics.
1. All billing metrics are stored in us-east-1 even for usages in other regions.
2. If you are using consolidated billing, billing metrics will be published under the linked account, and will only be visible to that account.
Hope that helps.
> 1. All billing metrics are stored in us-east-1 even for usages in other regions.
...what? Is this true? If so, can someone explain the logic behind this?
...what? Is this true? If so, can someone explain the logic behind this?
Yeah, I've done that now, but it's 'after the horse has bolted'
I wonder if the author is going to be on the hook for the bill for this.
If he originally received this note from amazon, it makes me also wonder if amazon knew about the fraud while it was happening. I sense that they probably monitor the launch of many of the XXL servers more closely than others.
If he originally received this note from amazon, it makes me also wonder if amazon knew about the fraud while it was happening. I sense that they probably monitor the launch of many of the XXL servers more closely than others.
They are quite certainly not monitoring their servers closely, I once had a application running on a small instance costing me more than 30,000$ in just a few weeks because it was doing expensive API calls to AWS. When a growth from <10$ per month to 30,000$ in about 2 weeks isn't concerning them why would 3000$?
I have a support ticket open with Amazon about this. I'm hoping not, but I have to accept a certain amount of responsibility for leaking the key in the first place.
I highly doubt you will have to pay this large bill, they most likely will reverse the charges and you should be free and clear, especially if you detail what you found and that you also have an email from them that your key was exposed.
I for sure would go to your billing report on the AWS console and setup alerts. You can have AWS notify you automatically if your bill goes above any threshold you want to define.
I for sure would go to your billing report on the AWS console and setup alerts. You can have AWS notify you automatically if your bill goes above any threshold you want to define.
I had the exact same thing happen to me. 2 days of 20 instances running and then amazon called me. I shut it down and revoked permissions immediately.
They did refund the money.
They did refund the money.
I just checked my EC2 page and have found $45 worth of charges from instances that I shut down weeks ago. Would it have automatically re ordered some if the bidding price dropped below a threshold? When I looked at my management console there were no EC2 instances running. This has turned out to be a very expensive experiment in LTC mining.
Yeah, I'd say that anyone on Github that has somehow leaked keys is in for a nasty surprise (or has already received one)
> I wonder if the author is going to be on the hook for the bill for this.
Almost certainly not. I've seen AWS reverse charges even when it's unequivocally the user's fault rather than someone hacking in.
As an example, I launched two reserved instances outside of a VPC when I meant to launch inside a VPC. At the time, this couldn't be changed, and the docs were very clear on that. After filing a support ticket, they reversed the reservations... and gave me a $100 AWS credit. For my fuck-up.
Almost certainly not. I've seen AWS reverse charges even when it's unequivocally the user's fault rather than someone hacking in.
As an example, I launched two reserved instances outside of a VPC when I meant to launch inside a VPC. At the time, this couldn't be changed, and the docs were very clear on that. After filing a support ticket, they reversed the reservations... and gave me a $100 AWS credit. For my fuck-up.
I can't vouch for AWS, since I've never used it, but if the same principles carry across from their online store then one thing I've always found with Amazon is that they have the most amazing customer service.
I had an issue with a 9 month on SD card splitting, not expecting much I emailed them and they immediately sent out a replacement and told me to return the old one. I've had a couple of things go wrong with orders (sometimes just delayed in post) and they've always been very quick and helpful. Not all companies, online ot brick & mortar, understand this well enough.
I had an issue with a 9 month on SD card splitting, not expecting much I emailed them and they immediately sent out a replacement and told me to return the old one. I've had a couple of things go wrong with orders (sometimes just delayed in post) and they've always been very quick and helpful. Not all companies, online ot brick & mortar, understand this well enough.
Exactly same thing happened to me. 20 x xlarge instances raking up a total bill of $1800. I've opened a support case with them.
> Audit code before open sourcing
It's important to remember that open-sourcing is generally one-way: once it's out there, it's impossible to completely eliminate all traces. Always audit code, and if there's even a remote possibility that you'll regret it you should check again
It's important to remember that open-sourcing is generally one-way: once it's out there, it's impossible to completely eliminate all traces. Always audit code, and if there's even a remote possibility that you'll regret it you should check again
I'd recommend blowing away your repository's history before open sourcing as well.
If you decide to do this, using git replace so that you and other trusted people can see the history, but others can't is a particularly good idea.
CPU mining of scrypt-based cryptocurrency is highly inefficient. Let's do some math.
A cc2.8xlarge is reported to mine at 85 kh/s, so 20 of them would give you 1700 kh/s. That's roughly equivalent to a couple of high-end AMD GPUs (say a couple of overclocked 290x). This hashing power gives you a little over 0.5 LTC per day. It mined for two days, so it gained a little over 1 LTC. Let's call it $40.
That's right, the idiot behind this cost the OP $3000+ for $40 profit. A smarter criminal would have spawn GPU instances on EC2.
A cc2.8xlarge is reported to mine at 85 kh/s, so 20 of them would give you 1700 kh/s. That's roughly equivalent to a couple of high-end AMD GPUs (say a couple of overclocked 290x). This hashing power gives you a little over 0.5 LTC per day. It mined for two days, so it gained a little over 1 LTC. Let's call it $40.
That's right, the idiot behind this cost the OP $3000+ for $40 profit. A smarter criminal would have spawn GPU instances on EC2.
Criminals don't care how much money they're costing their victims, only how much money they'll make from their crime. This is no different than someone smashing your car's window ($100+) and destroying its dashboard ($1000+) to remove a stereo head unit that they'll flip for $50.
Correct, but for the same crime and effort, (so just as easy) the criminal could have made a bigger profit. They took the stereo and left behind the laptop. Criminals however are not always the brightest bulbs.
Someone broke into the office of a consulting shop I was working for, passing up dozens of macbook pros and who knows what else for an old-school translucent aqua imac and a $100 Walmart bike.
Many of these opportunistic thieves think about what they can move quickly so if they are caught they no longer have the goods on them. That imac and walmart bike they could have moved the same day for probably $100 profit, where as the macbooks they may have had to shop them around for longer, long enough for theft services to lock them down and track them once they are turned on... Not good news for the crook. They are smarter than you probably think.
They were probably hipsters... was it is a fixed gear bike? And Aqua iMacs are pretty culkin.
Notice any facial hair trimmers missing?
Please. No walmart "bike" has horizontal drop-outs.
Comment of the year. Hah!
To be fair, if they were tech savvy they would probably work as freelancers or at a computer store instead of stealing aqua imacs.
The thief probably wasn't bright and hadn't seen it before, thinking that iMac must be so new that he hasn't seen it in the news and therefore the most valuable (after all, Apple is making colored iPhones now).
Nah, just wait 15 years and sell the Aqua iMacs for thousands of dollars. Better than those mainstream macbook pros...
Sounds about right, those macbooks aren't worth anything.
You'd think somebody smart enough to glean a key from github and setup instances on aws would have thought this out more
Most scrypt mining software uses the CPU by default. It might have been too difficult for the thief to figure out how to get GPU mining setup.
No it doesn't. cgminer is the "default" that almost everyone uses and the current version doesn't even have CPU mining. The current version doesn't have script either, you gotta download a previous version but that's a different story. You gotta throw in a --script when starting it, otherwise it mines with SHA-256.
If you wanna CPU mine, you gotta download a specific CPU miner.
Source: I mine Litecoins.
If you wanna CPU mine, you gotta download a specific CPU miner.
Source: I mine Litecoins.
[deleted]
Makes you wonder why the criminal didn't launch g2.2xlarge instances which would get 185 kH/s per instance. In fact for awhile there litecoin mining was profitable at spot prices.
http://www.completefusion.com/profitable-litecoin-mining-on-...
http://www.completefusion.com/profitable-litecoin-mining-on-...
My guess would be ignorance. I don't think the criminal behind this knows what he is doing.
A smarter criminal would have opted for g2.2xlarge instances as well as mining for a currently more profitable coin. Granted he'd need to be careful not to leave a trail, he could essentially trade these coins for LTC and still obtain more litecoins.
A smarter criminal would have opted for g2.2xlarge instances as well as mining for a currently more profitable coin. Granted he'd need to be careful not to leave a trail, he could essentially trade these coins for LTC and still obtain more litecoins.
An even smarter criminal would have "mined" the Ripple give away using EC2, there are still people paying for those instances as we speak so surely they would be more profitable than litecoin mining (assuming you sell them right away).
https://www.computingforgood.org/
https://ripple.com/forum/viewtopic.php?f=18&t=4382
https://www.computingforgood.org/
https://ripple.com/forum/viewtopic.php?f=18&t=4382
Uh, nope. I "mined" Ripple through that little project for a couple of days (using the g2 instances) and got a couple of dollars worth of Ripple back for over $100 ec2 bill (460 hours of WCG runtime).
I'll boinc on my desktop for science and goodwill but will not waste any more time on Ripple's program or indeed on Ripple itself, as brilliant as the idea is. Ripple's founders are hanging on to their 60%+ outstanding Ripples so tightly that I suspect the project will never truly get off the ground.
As much as people criticize it, I think some variation of Bitcoin's seignorage mining is the only realistic way to bootstrap a successful virtual currency, at least at this point.
I'll boinc on my desktop for science and goodwill but will not waste any more time on Ripple's program or indeed on Ripple itself, as brilliant as the idea is. Ripple's founders are hanging on to their 60%+ outstanding Ripples so tightly that I suspect the project will never truly get off the ground.
As much as people criticize it, I think some variation of Bitcoin's seignorage mining is the only realistic way to bootstrap a successful virtual currency, at least at this point.
Premined strategies like Ripple chose certainly aren't the road to success.
I used it to mine dogecoin using those same instructions. I either had to use an older nvidia driver version or an older version of CUDAminer, though. So it wasn't that simple to set up once CUDAminer got the update that broke the newer NVidia software. This happened near the first of the month. http://doges.org/index.php?topic=43.0
Pardon my ignorance but why are you dogecoin mining?
My guess: the difficulty is low enough that it's profitable to mine dogecoin and flip it for Bitcoin or Litecoin. I know a lot of altcoin miners do this, and there are some pools (middlecoin) that are geared towards this.
I'm even more amazed that there is demand for dogecoin then. I see the value of LTC and others but what is the USP of dogecoin?
It was created as a joke altcoin, but some people actually desire it. Most likely to flip it, or just to say they own some.
The doge brand is the USP. Most LTC clones are already largely undifferentiated, so this is actually a strong contender given it's technically almost identical to the rest.
That sounds about on par with copper theft, it's often ripped out of installed equipment like AC units or transformers that require total replacement.
Not only that, LTC has been one of the lesser profitable scrypt-coins to mine for a while.
See https://www.multipool.us/ for reference. Although it fluctuates a lot, LTC has been around 4th-5th in profitability among scrypt coins. Often 5th. And wait for Doge to feature in these pools :-)
See https://www.multipool.us/ for reference. Although it fluctuates a lot, LTC has been around 4th-5th in profitability among scrypt coins. Often 5th. And wait for Doge to feature in these pools :-)
Actually, it's worse than that. Check http://coinwarz.com and http://coinchoose.com. Multipool is limited to the coins they mine. If you check out the more general list at the sites I mentioned, LTC is not even in the top 25.
I was trying to launch a GPU instance the other day for (scrypt-based) dogecoin mining, but I couldn't find it in the list. Perhaps they are sold out?
You need to make sure you have selected a compatible (cluster something, and green perhaps) AMI first.
Then the GPU instances will be available in the list. Ran into the same thing myself ;)
Then the GPU instances will be available in the list. Ran into the same thing myself ;)
How did Amazon detect your key in the wild? Or did they notice based on usage patterns/activity in your instances?
All AWS keys I've seen start with 'AKIA'. I am assuming that they have bots that search Github and other search engines for access keys. At that point it is easy for them to tie them back to an account and notify the user.
They must only have started doing that recently. This project has been out in the wild for at least a year.
Ha - no --- they do that when they see a spike in charges.
Or maybe it's just a coincidence they emailed immediately AFTER he racked up +3000$ in charges.
Or maybe it's just a coincidence they emailed immediately AFTER he racked up +3000$ in charges.
Well, kudos to them for doing that, at least. Of course it's awful that you could be out ~$3k, but imagine how bad it could have been if they hadn't been so proactive.
Yeah, it would have been another day at least before I checked amazon again.
Luke, drop me a note at werner [at] amazon with a link to the support ticket you created, and we'll see what we can do.
FYI all AWS keys start with AKIA - makes it easy to search for 'em.
Shoot, as someone who made the same mistake of leaving my AWS keys in an open source project, I think I narrowly dodged a bullet. I didn't realize this risk was so high. Thanks for this post!
I think smarter usage of IAM roles would have also helped here. Keys created strictly for S3 access should not have the ability to launch new instances and so on. Limiting keys to their specific purpose is a good security practice even for dev environments.
AWS IAM supports the ability to permit or prevent specific types of instances from being started by a given key; if folks are worried about a key being used to start G2 or CG1 or any other specific instance, take a look at the instructions here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-polic...
Role-based access control. :)
Though Amazon still has services that "don't support IAM" ... I'm looking squarely at Payments. That particular master key always makes me nervous.
Though Amazon still has services that "don't support IAM" ... I'm looking squarely at Payments. That particular master key always makes me nervous.
I suggest the OP check out Cloudability[1], which provides realtime cost management for AWS and other cloud providers. We help over 10,000 customers make sure this doesn't happen to them. (disclosure: I work there)
[1]: https://cloudability.com/
[1]: https://cloudability.com/
Another good habit to be in is never checking any kind of credentials into source control; even if it's some private personal project, just don't be tempted to check in your credentials to source control, because at some point you may find some portion of that that's useful that you import into a public project, accidentally preserving full history.
Sorry to the OP, hope that Amazon reverses those charges once you tell them what happened.
Sorry to the OP, hope that Amazon reverses those charges once you tell them what happened.
We know we shouldn't be committing password or personal key. But shit happens and it does happen very frequently, even top notched people do.
What we need is not to say "don't it", because no shit we shouldn't be doing that. instead we need defense mechanism. It would be helpful and interesting if git or hg has a plugin that detects when some credentials is leaking through and warn users "hey you better check this shit out" before doing a real commit.
The other thing is "don't commit key into a private repository". Don't chef and puppet users usually do that? How are people backing up their keys?
What we need is not to say "don't it", because no shit we shouldn't be doing that. instead we need defense mechanism. It would be helpful and interesting if git or hg has a plugin that detects when some credentials is leaking through and warn users "hey you better check this shit out" before doing a real commit.
The other thing is "don't commit key into a private repository". Don't chef and puppet users usually do that? How are people backing up their keys?
For rails, the Figaro gem is really helpful for managing environment variables, which is a good place to store credentials (the config file for Figaro is then added to .gitignore)
But how do you back up this set of environment variables. Yes, in practice I also use ignore file to prevent sensitive things leak into repository and I usually generate password dynamically on the fly or through some script.
Here's a handy GREP to find AccessKey/SecretKey pairs:
grep -RP '(?<![A-Z0-9])[A-Z0-9]{20}(?![A-Z0-9])' *
grep -RP '(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])' *
source: http://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-...
source: http://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-...
This would be neat for a pre-commit hook.
What's the best way to handle api keys without storing them in a repository? A separate configuration file?
I've saved api keys to a repository, but only ones with no payment information attached and read only access. Mainly because I couldn't think of a better way at the time.
I've saved api keys to a repository, but only ones with no payment information attached and read only access. Mainly because I couldn't think of a better way at the time.
Usally in environment variables rather than a config file.
for example https://devcenter.heroku.com/articles/config-vars
for example https://devcenter.heroku.com/articles/config-vars
This is very convenient on Heroku, but on my own servers, I have to use a post-deploy hook to copy a file that has all of my API keys to the server. That works great, except I now have a separate file to manage outside of source control, it's not under its own source control, and it's difficult to share amongst my team.
Anyone have any suggestions?
Anyone have any suggestions?
I've had some luck with having a separate repo. I'll have a "public" (even if it's not public) with all of the usual source code, and a "private" repo, with passwords, fabric files, run scripts, etc.
Works well on teams, where the subset of developers that don't need to do actual deployments don't need the private repo anyway.
Works well on teams, where the subset of developers that don't need to do actual deployments don't need the private repo anyway.
"Having a poke around confirmed what I had already guessed. The unauthorised user had been mining litecoin with the mining pool pool-x.eu."
Hmmm..you already guessed someone hacked your account to mine litecoin? Astroturfing much? That's the last thing I would have guessed. I would have thought someone was using it as some crazy web server or mail server to generate spam or phony websites for bogus ad clicks.
Hmmm..you already guessed someone hacked your account to mine litecoin? Astroturfing much? That's the last thing I would have guessed. I would have thought someone was using it as some crazy web server or mail server to generate spam or phony websites for bogus ad clicks.
shrug maybe I was wrong to make that kind of assumption.
It was probably the type and quantity of instance that tipped me off a little, having read about people trying to mine with EC2 again.
It was probably the type and quantity of instance that tipped me off a little, having read about people trying to mine with EC2 again.
So are you liable?
If you're hacked the most likely problem you'll get is a spike in data transfer costs. You can up the alarms to, for example, email you if the bandwidth usage goes above x (cost) over y time period.
I had a perl DOS bot get into a server, took about 2 hours to trigger the alarm. Shame I was fast asleep at the time, but the idea was there...
[0] https://console.aws.amazon.com/billing/home