Third cryptocurrency exchange becomes hacking victim, loses Bitcoin(zdnet.com)
zdnet.com
Third cryptocurrency exchange becomes hacking victim, loses Bitcoin
http://www.zdnet.com/third-cryptocurrency-exchange-becomes-hacking-victim-loses-bitcoin-7000027052/
55 comments
FTA:
Busoni said that a hacker took advantage of a processing flaw in the Bitcoin exchange post. When users submit a withdrawal request, the input is checked against your balance, deducted, and the new amount recorded within a database. However, it was discovered that placing several withdrawals all in practically the same instant meant each request was processed at more-or-less the same time, resulting in a negative balance but "valid insertions into the database, which then get picked up by the withdrawal daemon."
Isn't ACID the solution to these problems for like 30 years?
Isn't ACID the solution to these problems for like 30 years?
> Isn't ACID the solution to these problems for like 30 years?
Yes, but this is bitcoin, which is so totally unlike anything that existed before, that such outdated notions as ACID aren't valid anymore. We need to start with a clean slate and reinvent all the wheels all over again.
Yes, but this is bitcoin, which is so totally unlike anything that existed before, that such outdated notions as ACID aren't valid anymore. We need to start with a clean slate and reinvent all the wheels all over again.
Ok that totally made be laugh!
ACID is outdated? Shall we use mongodb for bitcoin?
I'm pretty sure they already do that.
Coinbase is way ahead of you on this one.
You can use an ACID compliant database and run into this problem, it's a classical race condition. The solution is to implement double bookkeeping and bank reconciliation, but it's tedious and complex to implement. I saw a pretty nice blog post with detailed information on the subject, but I can't remember where, didn't bookmark and my google-fu is weak at the moment. So either somebody remembers or you'll have to google yourself :)
> You can use an ACID compliant database and run into this problem, it's a classical race condition.
No it's not. It's just sloppy code and/or a misunderstanding of database transactions and isolation levels. If you use an ACID compliant database like PostgreSQL it's pretty straightforward to write proper code that would not have this issue.
The main concept to understand is that you need to lock all the rows that you'll be touching before you use them. In most case this method works fine with the default read-committed isolation level.
Code that queries your database in multiple round trips and then compares the values for validity on the app side will have this problem. Anything you read could have been modified by the time you "act on" the information you read. Heck even stored procedures running on the DB itself will have this problem if run in the default read-committed isolation level.
The high level solution is to use SELECT ... FOR UPDATE and lock everything that you'll be touching before you update anything.
> The solution is to implement double bookkeeping and bank reconciliation, but it's tedious and complex to implement.
This is a business requirement for any accounting/financial software but it's not a technical one. Having an audit trail, double entry accounting, and recon are business functions. You'd be crazy to design a system that didn't include them but it's not strictly required (again from a technical sense, not a business one!).
No it's not. It's just sloppy code and/or a misunderstanding of database transactions and isolation levels. If you use an ACID compliant database like PostgreSQL it's pretty straightforward to write proper code that would not have this issue.
The main concept to understand is that you need to lock all the rows that you'll be touching before you use them. In most case this method works fine with the default read-committed isolation level.
Code that queries your database in multiple round trips and then compares the values for validity on the app side will have this problem. Anything you read could have been modified by the time you "act on" the information you read. Heck even stored procedures running on the DB itself will have this problem if run in the default read-committed isolation level.
The high level solution is to use SELECT ... FOR UPDATE and lock everything that you'll be touching before you update anything.
> The solution is to implement double bookkeeping and bank reconciliation, but it's tedious and complex to implement.
This is a business requirement for any accounting/financial software but it's not a technical one. Having an audit trail, double entry accounting, and recon are business functions. You'd be crazy to design a system that didn't include them but it's not strictly required (again from a technical sense, not a business one!).
The problem appears to have been more straight forward than any of that. The actual transaction system was a different system from the order entry/webs system. It ran orders in serial after a delay using some kind of message passing. Obviously you can't have a SQL transaction that spans multiple applications, so SQL transactions wouldn't really work in this case. It was just a straight forward mistake: orders could be placed before previous orders were finished executing.
Transactions wouldn't actually work with bitcoin in this case anyway. eg, transaction 1 checks balance, deducts amount, does BTC transfer, and then commits. Transaction 2 does the same, but the commit would fail as transaction 1 has already modified it. Transaction 2 can't actually roll back though - the bitcoin transaction cannot be rolled back.
Transactions wouldn't actually work with bitcoin in this case anyway. eg, transaction 1 checks balance, deducts amount, does BTC transfer, and then commits. Transaction 2 does the same, but the commit would fail as transaction 1 has already modified it. Transaction 2 can't actually roll back though - the bitcoin transaction cannot be rolled back.
> The problem appears to have been more straight forward than any of that. The actual transaction system was a different system from the order entry/webs system. It ran orders in serial after a delay using some kind of message passing.
I'm thinking that they had both issues. If the system let you submit multiple withdraw requests (totalling more than your balance) then they definitely weren't checking at request submission before queuing them for later processing.
> Obviously you can't have a SQL transaction that spans multiple applications, so SQL transactions wouldn't really work in this case. It was just a straight forward mistake: orders could be placed before previous orders were finished executing.
XA transactions allow you to have transactions across multiple services. The most common use case is a database and a message queue. It's nowhere near as simple as dealing with a single transactional resource but it's not that uncommon in the finance world. I've used them quite a bit and they're really convenient. Getting it right from a dev-ops perspective is a bit pretty tricky though as you need to really understand how the transaction manager itself works and make sure it actually runs.
> Transactions wouldn't actually work with bitcoin in this case anyway. eg, transaction 1 checks balance, deducts amount, does BTC transfer, and then commits. Transaction 2 does the same, but the commit would fail as transaction 1 has already modified it. Transaction 2 can't actually roll back though - the bitcoin transaction cannot be rolled back.
Yes you can't have XA transactions with bitcoin as it doesn't include any concept of rollback. What you can do though is guarantee that a transaction is only run once and re-submit it if it failed to send. If you can uniquely identify a bitcoin transaction[1] then you can guarantee that you don't send it out more than once. Just add in a significant delay (couple hours? a day?) before any retries to ensure the transaction has been merged into the block chain.
[1]: And not by just using the transaction hash because we all know what can happen with that: https://en.bitcoin.it/wiki/Transaction_Malleability
I'm thinking that they had both issues. If the system let you submit multiple withdraw requests (totalling more than your balance) then they definitely weren't checking at request submission before queuing them for later processing.
> Obviously you can't have a SQL transaction that spans multiple applications, so SQL transactions wouldn't really work in this case. It was just a straight forward mistake: orders could be placed before previous orders were finished executing.
XA transactions allow you to have transactions across multiple services. The most common use case is a database and a message queue. It's nowhere near as simple as dealing with a single transactional resource but it's not that uncommon in the finance world. I've used them quite a bit and they're really convenient. Getting it right from a dev-ops perspective is a bit pretty tricky though as you need to really understand how the transaction manager itself works and make sure it actually runs.
> Transactions wouldn't actually work with bitcoin in this case anyway. eg, transaction 1 checks balance, deducts amount, does BTC transfer, and then commits. Transaction 2 does the same, but the commit would fail as transaction 1 has already modified it. Transaction 2 can't actually roll back though - the bitcoin transaction cannot be rolled back.
Yes you can't have XA transactions with bitcoin as it doesn't include any concept of rollback. What you can do though is guarantee that a transaction is only run once and re-submit it if it failed to send. If you can uniquely identify a bitcoin transaction[1] then you can guarantee that you don't send it out more than once. Just add in a significant delay (couple hours? a day?) before any retries to ensure the transaction has been merged into the block chain.
[1]: And not by just using the transaction hash because we all know what can happen with that: https://en.bitcoin.it/wiki/Transaction_Malleability
You wouldn't do the bitcoin transfer until the transaction authorizing it completed successfully. And you don't just use a single transaction that deducts the amount.
You'd use one transaction to indicate you were beginning a transfer, putting that amount in a hold state, attempt the bitcoin transfer, then a transaction to indicate the bitcoin transfer was successful or not.
If there is an error in the bitcoin transfer or after, and the second transaction is never run, the funds are still in hold whether transferred or not and can be manually corrected by checking the system logs.
You'd use one transaction to indicate you were beginning a transfer, putting that amount in a hold state, attempt the bitcoin transfer, then a transaction to indicate the bitcoin transfer was successful or not.
If there is an error in the bitcoin transfer or after, and the second transaction is never run, the funds are still in hold whether transferred or not and can be manually corrected by checking the system logs.
I might do something similar in the main.
Still, this doesn't inherently solve the problem. That is, you would still need to be aware of transaction isolation levels and ensure that you're locking rows properly as you conduct each transaction.
Still, this doesn't inherently solve the problem. That is, you would still need to be aware of transaction isolation levels and ensure that you're locking rows properly as you conduct each transaction.
The second part should be possible to solve using transfer accounts where the btc is stored temporarily while working out the transaction details. Everything would then be reversible until the transaction is finalized.
So, a btc escrow service? Sounds like an app!
Transactions wouldn't actually work with bitcoin in this case anyway. eg, transaction 1 checks balance, deducts amount, does BTC transfer, and then commits.
Continuing on the OMG, bitcoin makes proper bookkeeping obsolete(!)- bzz, NOT!!. Theme.
You certainly can use transactions for bitcoin orders, you just have to use the logic that many other business uses.
That is, a user has an account that they can place orders from. The only action they can take is turning their funds into an order. Correctly coded transactions assure that they can only purchase orders of up to the value in their account. Order processing is independent from this order entry system. Orders may succeed and become bitcoins or dollars and then be transferred back to the accounts or the order could fail and the dollars could be returned appropriately as well. This order processing is queued, could involve a series of messages, etc. I only have a degree in econ but I recall stock exchanges work basically this way (you place orders, money deducted immediately but you have to wait to see if your order actually worked, sorry).
Essentially, the final result of orders shouldn't be the point where the funds are debited from the users' accounts, the initially order placement should be where the debiting happens.
Continuing on the OMG, bitcoin makes proper bookkeeping obsolete(!)- bzz, NOT!!. Theme.
You certainly can use transactions for bitcoin orders, you just have to use the logic that many other business uses.
That is, a user has an account that they can place orders from. The only action they can take is turning their funds into an order. Correctly coded transactions assure that they can only purchase orders of up to the value in their account. Order processing is independent from this order entry system. Orders may succeed and become bitcoins or dollars and then be transferred back to the accounts or the order could fail and the dollars could be returned appropriately as well. This order processing is queued, could involve a series of messages, etc. I only have a degree in econ but I recall stock exchanges work basically this way (you place orders, money deducted immediately but you have to wait to see if your order actually worked, sorry).
Essentially, the final result of orders shouldn't be the point where the funds are debited from the users' accounts, the initially order placement should be where the debiting happens.
It's been a while, but I'm pretty sure I have worked on systems where the message queue endpoint and a database could both enlist in a distributed transaction.
Actually, I've worked on very large SOA systems that did share transactions and rollbacks across services. There are lots of standards for it, like WS-AtomicTransaction. So it's easily possible to do what the parent is asking, even when multiple services are involved.
FWIW, the A in ACID and the A in AtomicTransaction both stand for the same thing.
You can use an ACID compliant database and run into this problem, it's a classical race condition.
Isn't it a quick fix to add an ACID-friendly clause to the UPDATE statement, e.g. some pseuosql like:
UPDATE balances SET balance = (balance - amount) WHERE user_id = input_user_id AND (balance - amount > 0)
Isn't it a quick fix to add an ACID-friendly clause to the UPDATE statement, e.g. some pseuosql like:
UPDATE balances SET balance = (balance - amount) WHERE user_id = input_user_id AND (balance - amount > 0)
You shouldn't ever even have a "balance" that you update.
You should have a ledger.
"INSERT INTO Ledger (Account, Date, Amount) VALUES (PersonToDebit, Now, -withdrawl)"
And then someone's "balance" is the sum of their ledger.
If multiple withdrawls go through, the account could still go negative if you don't check at the time of the withdrawl, but worse case you still end up with negative balances rather than flat out losing track of the balance because you have a balance that you UPDATE.
You should have a ledger.
"INSERT INTO Ledger (Account, Date, Amount) VALUES (PersonToDebit, Now, -withdrawl)"
And then someone's "balance" is the sum of their ledger.
If multiple withdrawls go through, the account could still go negative if you don't check at the time of the withdrawl, but worse case you still end up with negative balances rather than flat out losing track of the balance because you have a balance that you UPDATE.
Well you'd probably want to have both, a negative balance is a problem with bitcoin if it can be arbitrarily negative. Say they issue 1000 withdraws in a millisecond. The balance will be "very" negative and there is nothing that can be done. And it might have to be done only a few times to bankrupt the exchange.
With the regular banking system or even with stock market, some trades and transactions can be reversed in case of errors and mistakes. So just having a ledger works, some bad updates can be reversed. Not so with bitcoin.
So when dealing with someone's account at a place like this an ACID compliant balance update is also needed.
Amateurs have jumped in and started developing banking software that deals with hundreds of millions of dollars -- so this is not surprising.
But at the same time it is kind of nice. It is sort of a Darwinian if you wish, the weakest and unlucky ones fail and get robbed the competent and/or lucky get to survive.
With the regular banking system or even with stock market, some trades and transactions can be reversed in case of errors and mistakes. So just having a ledger works, some bad updates can be reversed. Not so with bitcoin.
So when dealing with someone's account at a place like this an ACID compliant balance update is also needed.
Amateurs have jumped in and started developing banking software that deals with hundreds of millions of dollars -- so this is not surprising.
But at the same time it is kind of nice. It is sort of a Darwinian if you wish, the weakest and unlucky ones fail and get robbed the competent and/or lucky get to survive.
You shouldn't ever even have a "balance" that you update.
I was arguing against the grandparent's claim of ACID race conditions; my intention wasn't to advocate in favour of a "balance" column.
Edit: that said, I wouldn't mind hearing the pros or cons of maintaining a convenience (e.g. page display) "running balance" that is updated from the ledger after every transaction.
I was arguing against the grandparent's claim of ACID race conditions; my intention wasn't to advocate in favour of a "balance" column.
Edit: that said, I wouldn't mind hearing the pros or cons of maintaining a convenience (e.g. page display) "running balance" that is updated from the ledger after every transaction.
Well, I think the parent was saying that you shouldn't have a balance that is actually relied upon for record-keeping or transactional purposes.
If you want to keep one for optimization purposes (i.e. you are reading more than writing and don't want to constantly sum the ledger table), then that's probably outside the scope of the parent's point.
If you want to keep one for optimization purposes (i.e. you are reading more than writing and don't want to constantly sum the ledger table), then that's probably outside the scope of the parent's point.
This one?
http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on...
"What accounts for the consistency of bank data is not the consistency of its databases but the fact that everything is written down twice and sorted out later using a permanent and unalterable record that is reconciled later."
But isn't that talking about "traditional" financial institutions that (in my experience) have staggeringly complex (messy) computing environments just by the nature of how these businesses grow (e.g. lots of acquisitions).
Surely for a brand new service such a Bitcoin exchange there isn't that excuse?
http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on...
"What accounts for the consistency of bank data is not the consistency of its databases but the fact that everything is written down twice and sorted out later using a permanent and unalterable record that is reconciled later."
But isn't that talking about "traditional" financial institutions that (in my experience) have staggeringly complex (messy) computing environments just by the nature of how these businesses grow (e.g. lots of acquisitions).
Surely for a brand new service such a Bitcoin exchange there isn't that excuse?
Suddenly you don't have confidence in the balances due to a bug. How do you resolve it?
That is why you have ledgers. Being able to go back and "replay" step by step how you got where you are makes all the difference when you run into stupid bugs. And you will run into stupid bugs.
That you're dealing with a brand new service that has not been battle-tested for years is yet another reason why you want a ledger and proper reconciliation rather than relying on a running balance.
That is why you have ledgers. Being able to go back and "replay" step by step how you got where you are makes all the difference when you run into stupid bugs. And you will run into stupid bugs.
That you're dealing with a brand new service that has not been battle-tested for years is yet another reason why you want a ledger and proper reconciliation rather than relying on a running balance.
Yes, as I understand it, real banks don't use ACID databases because their software dates from before before the era of modern databases, they uses a lot of batch processing, because indeed they've grown by knitting together multiple institutions and etc.
To further clarify, an ACID database, used correctly, are a drop-in solution to problems like that of someone attempting multiple simultaneous account withdrawals.
And, of course, "you can still run into these problems using an ACID database" (lol), yes, all you have to do is not understand transactions and not use them.
To further clarify, an ACID database, used correctly, are a drop-in solution to problems like that of someone attempting multiple simultaneous account withdrawals.
And, of course, "you can still run into these problems using an ACID database" (lol), yes, all you have to do is not understand transactions and not use them.
It's been a few years but at least in theory ACID should avoid any race conditions. So you mean you'll face problems in a concrete implementation or due to performance optimizations? I can't image that ACID has a theoretical possibility for race-conditions - I'd like to learn if this is the case - Google did not turn up anything.
Check out the wikipedia article on Isolation (database systems) Anything below serializable has race conditions where the same query may return different results depending on what else the system is doing.
I think the solution is continuous monitoring and auditing. Keeping track of the balances is the simple part. Reconciling the balances is the complex part. Banking is inherently eventually consistent, so putting audits in place to verify integrity of the balances is critical.
Transaction isolation is not what you think it is. Most databases run at a liberal isolation level. For example, from the PostgreSQL docs[1] you can read that they use 'read committed' as an isolation level by default. In this example, it could mean that the following could happen:
* At T=1, transaction 1 will read the current balance * At T=2, transaction 2 will read the current balance * At T=3, transaction 1 will update the current balance * At T=4, transaction 2 is in a race condition situation, since the update of transaction 1 'leaks' into transaction 2, but the last read balance in transaction 2 still is the 'old' balance.
In other words: this is an area where a lot of people introduce race condition bugs without being aware of it.
[1] http://www.postgresql.org/docs/9.1/static/transaction-iso.ht...
* At T=1, transaction 1 will read the current balance * At T=2, transaction 2 will read the current balance * At T=3, transaction 1 will update the current balance * At T=4, transaction 2 is in a race condition situation, since the update of transaction 1 'leaks' into transaction 2, but the last read balance in transaction 2 still is the 'old' balance.
In other words: this is an area where a lot of people introduce race condition bugs without being aware of it.
[1] http://www.postgresql.org/docs/9.1/static/transaction-iso.ht...
That's the default, but you can, and should, use a serializable transaction isolation level where required by the problem domain.
Of course you should, the problem isn't that it isn't possible to serialize a transaction, the problem is that the default is optimistic. I can only imagine that there will be quite a hefty performance impact if serialized transactions were the default, so I understand their reasoning.
What looks like is needed is a serialization of user transactions.
And also risk-avoiding schemas (in the general provisions of the business, not only on the system running it)
And also risk-avoiding schemas (in the general provisions of the business, not only on the system running it)
And withdrawals are even the classic example used in schoolbooks too...
This is a repeat from yesterday: https://news.ycombinator.com/item?id=7340908
Thank you: I was starting to wonder… Three? Wasn't that yesterday's count already?
That's the problem with a decentralized count ;)
I'm pretty sure there's been more than three exchange heists.
Maybe… Still hate the secrecy around those.
Well, if someone could publish best practice as to how not get hacked, that would be great: it seems VCs and advocates are unhappy to see those scaring the seven billion people still to be convinced this is worth their attention. No points in having regulations, voluntary or state-enforced, if you have no idea what to check.
Well, if someone could publish best practice as to how not get hacked, that would be great: it seems VCs and advocates are unhappy to see those scaring the seven billion people still to be convinced this is worth their attention. No points in having regulations, voluntary or state-enforced, if you have no idea what to check.
It's easy: don't hold client funds.
This is why we need trustless exchanges:
http://www.reddit.com/r/Bitcoin/comments/1zgbza/i_am_buildin...
http://www.reddit.com/r/Bitcoin/comments/1zgbza/i_am_buildin...
How is this insulated against the future regulatory environment? For instance, the Feds have already come down on the KYC side of things. A trustless exchange is immune to such regulation. So, why wouldn't they outlaw participation in or operation of (as a gateway or other node) same?
These are earnest questions, not statements.
These are earnest questions, not statements.
The exchange itself would be impossible to shutdown or regulate. However regulatory agencies would be able to apply pressure at the gateway interface with the real world: the banks which accept USD wire transfers and issue "USDcoin" crypto tokens, as well as perform the reverse redemption.
It is entirely possible and I would argue acceptable that regulatory authorities require KYC compliance for these gateway entities, but that only affects people moving national currency in and out of the system. It's entirely possible that people will trade and do business in USDcoin without cashing out directly through the regulated banking gateway. This is a reality that regulators will have to get used to.
It is entirely possible and I would argue acceptable that regulatory authorities require KYC compliance for these gateway entities, but that only affects people moving national currency in and out of the system. It's entirely possible that people will trade and do business in USDcoin without cashing out directly through the regulated banking gateway. This is a reality that regulators will have to get used to.
>It's entirely possible that people will trade and do business in USDcoin...This is a reality that regulators will have to get used to.
I'm not sure about this one. Regulators have a tendency to "motivate" others to "get used to" things, not the other way around.
With E-gold years ago, the proprietors were criminally indicted for activity that took place on their network.
With your architecture, if the Feds deemed that their was laundering taking place or the exchange was otherwise being used to facilitate illicit activity, then I can easily envision them expanding the "responsible parties" definition beyond gateway operators to any node in the exchange. This, of course, would mean that anyone participating would be liable for activity that routed through them, thereby having a chilling effect on participating peers. So, if their goal is to shut down the exchange, they can effectively do so via this implicit threat of criminal prosecution.
If any of this sounds far-fetched, you really have to review the E-gold case [0] to see just how much the regulatory regime will contort itself to keep financial transactions under its constant watch.
[0] http://en.wikipedia.org/wiki/E-gold#Criminal_prosecution
I'm not sure about this one. Regulators have a tendency to "motivate" others to "get used to" things, not the other way around.
With E-gold years ago, the proprietors were criminally indicted for activity that took place on their network.
With your architecture, if the Feds deemed that their was laundering taking place or the exchange was otherwise being used to facilitate illicit activity, then I can easily envision them expanding the "responsible parties" definition beyond gateway operators to any node in the exchange. This, of course, would mean that anyone participating would be liable for activity that routed through them, thereby having a chilling effect on participating peers. So, if their goal is to shut down the exchange, they can effectively do so via this implicit threat of criminal prosecution.
If any of this sounds far-fetched, you really have to review the E-gold case [0] to see just how much the regulatory regime will contort itself to keep financial transactions under its constant watch.
[0] http://en.wikipedia.org/wiki/E-gold#Criminal_prosecution
It changes the dynamic of the regulatory environment from enforcing a few mega-exchanges organizations to having to enforce (possibly hundreds of) thousands of vendors selling BTC for currency and consumers buying it. It would be as difficult as enforcing drug dealing and we've seen how effective that has been.
If you were running an exchange why would you keep everyone's money in the same wallet? It seems like you would keep a wallet per user with their funds in it. Then any attempt to double transfer would fail on the bitcoin side because their wallet wouldn't have the money.
A quote: 'However, it was discovered that placing several withdrawals all in practically the same instant meant each request was processed at more-or-less the same time, resulting in a negative balance but "valid insertions into the database, which then get picked up by the withdrawal daemon.'
I can't believe programmers are alive today who don't know what a race condition is. In the future, when parallel processing is more the norm than the exception, race conditions will be the first thing we'll have to think about, not the last.
I can't believe programmers are alive today who don't know what a race condition is. In the future, when parallel processing is more the norm than the exception, race conditions will be the first thing we'll have to think about, not the last.
The owner of Poloniex posted details of the bug here: https://bitcointalk.org/index.php?topic=499580
Third?
I'm pretty sure it's been more than that in the last few years...
I'm pretty sure it's been more than that in the last few years...
"It's like all these bitcoin sites are running a bug bounty without even realizing it.."
https://twitter.com/jjarmoc/status/440927360821764097