With CDNs, there are definitely privacy (and GDPR) concerns, since often the complete traffic passes through the CDN. And in order to cache, rewrite and optimize responses, SSL connections are typically terminated in the CDN. And this implies that the CDN provider processes and potentially sees sensitive user data in plain text. Due to the distributed nature of CDNs with many edge locations, one can never be really sure what the exact legal circumstances are when user data is passed through the CDN nodes in different countries. And even though CDN providers won't leak user data on purpose, cloud bleed [1] has shown that this can happen by accident and at massive scale.
This is one of the reasons, why at Baqend we opted for a different approach: by using Service Workers, we can make sure that as a service provider we will only see public data. Personally identifiable information like cookies and sensitive session information does not leave the user's browser. And if you think about it that makes a lot of sense, since the public data really is what makes a site fast or slow. User-specific data (e.g., a profile, payment data, etc.) is hardly ever the cause of performance problems. Nonetheless, we do use CDNs for public data as they are an indispensable tool to achieve low latency.
"What if you could have a fully managed database service that's consistent, scales horizontally across data centers and speaks SQL?"
Looks like Google forgot to mention one central requirement: latency.
This is a hosted version of Spanner and F1. Since both systems are published, we know a lot about their trade-offs:
Spanner (see OSDI'12 and TODS'13 papers) evolved from the observation that Megastore guarantees - though useful - come at performance penalty that is prohibitive for some applications. Spanner is a multi-version database system that unlike Megastore (the system behind the Google Cloud Datastore) provides general-purpose transactions. The authors argue: We believe it is better to have application programmers deal with performance problems due to overuse of transactions as bottlenecks arise, rather than always coding around the lack of transactions. Spanner automatically groups data into partitions (tablets) that are synchronously replicated across sites via Paxos and stored in Colossus, the successor of the Google File System (GFS). Transactions in Spanner are based on two-phase locking (2PL) and two-phase commits (2PC) executed over the leaders for each partition involved in the transaction. In order for transactions to be serialized according to their global commit times, Spanner introduces TrueTime, an API for high precision timestamps with uncertainty bounds based on atomic clocks and GPS. Each transaction is assigned a commit timestamp from TrueTime and using the uncertainty bounds, the leader can wait until the transaction is guaranteed to be visible at all sites before releasing locks. This also enables efficient read-only transactions that can read a consistent snapshot for a certain timestamp across all data centers without any locking.
F1 (see VLDB'13 paper) builds on Spanner to support SQL-based access for Google's advertising business. To this end, F1 introduces a hierarchical schema based on Protobuf, a rich data encoding format similar to Avro and Thrift. To support both OLTP and OLAP queries, it uses Spanner's abstractions to provide consistent indexing. A lazy protocol for schema changes allows non-blocking schema evolution. Besides pessimistic Spanner transactions, F1 supports optimistic transactions. Each row bears a version timestamp that used at commit time to perform a short-lived pessimistic transaction to validate a transaction's read set. Optimistic transactions in F1 suffer from the abort rate problem of optimistic concurrency control, as the read phase is latency-bound and the commit requires slow, distributed Spanner transactions, increasing the vulnerability window for potential conflicts.
While Spanner and F1 are highly influential system designs, they do come at a cost Google does not tell in its marketing: high latency. Consistent geo-replication is expensive even for single operations. Both optimistic and pessimistic transactions even increase these latencies.
It will be very interesting to see first benchmarks. My guess is that operation latencies will be in the order of 80-120ms and therefore much slower than what can be achieved on database clusters distributed only over local replicas.
> So here are some facts and trivia that are not so well-known or published that I collected by talking to Parse engineers that now work at Facebook. As I am unsure about whether they were allowed to share this information, I will not mention them by name.
It's both lessons learned by Parse engineers and us, so I think the intended ambiguity is okay.
I do believe you, but can your share some more hard facts?
I've changed the passage to "one of the greatest". As in the disclaimer above the article, I'm mainly presenting assertions by Parse engineers, who were very sure about being the largest MongoDB user in terms of cluster size.
We also do have some benchmarks comparing real-time performance, which are under submission at a scientific conference, so we cannot share them yet.
Unfortunately, testing against Firebase using established benchmarks such as YCSB is not very useful, as it's unclear which limits are enforced on purpose and which limits are inherent to the database design. Would you be open to providing a Firebase setup for heavy load testing w.r.t. to throughput, latency and consistency?
The Google Datastore is built on Megastore. Megastore's data model is based on entity groups, that represent fine-grained, application-defined partitions (e.g. a user's message inbox). Transactions are supported per co-located entity group, each of which is mapped to a single row in BigTable that offers row-level atomicity. Transactions spanning multiple entity groups are not encouraged, as they require expensive two-phase commits. Megastore uses synchronous wide area replication. The replication protocol is based on Paxos consensus over positions in a shared write-ahead log.
The reason for the Datastore only allowing very limited queries is that they seek to target each query to an entity group in order to be efficient. Queries using the entity group are fast, auto-indexed and consistent. Global indexes, on the other hand, are explicitly defined and only eventually consistent (similar to DynamoDB). Any query on unindexed properties simply returns empty results and each query can only have one inequality condition [1].
Do you have a source for that? It is what the Parse engineers told me, but that kind of information is notoriously hard to corroborate. Any time I asked this question to MongoDB engineers they were evading concrete numbers for deployment sizes.
Author of the post here. If you have additional key insights about the Parse infrastructure, please post them here and I will directly add them to the article.
> Some problems are caused by DBMS to backend and backend-to-frontend network utilization, which is generally not something a specialized DBMS can solve.
I absolutely agree. RDBMSs are the kings of one-size-fits-most. In the NoSQL ecosystem, MongoDB starts growing into this role. Of course, many performance problems are not caused by the backend but rather the network (an average website makes 108 HTTP requests) or the frontend (Angular2 without AOT, I'm looking at you). We took an end-to-end approach to tackle this problem:
1. As objects, files and queries are fetched via a REST API, we can make them cachable at the HTTP level to transparently use browser caches, forward proxies, ISP interception proxies, CDNs and reverse proxies.
2. In order to guarantee cache coherence, we invalidate queries and objects in our CDNs and reverse proxy caches when the change using an Apache Storm-based real-time query matching pipeline.
3. To keep browser up-to-date we ship a Bloom filter to the client indicating any data that has a TTL that is non-expired but was recently written. This happens inside the SDK. Optionally client can subscribe to queries via websockets (as in Firebase or Meteor).
The above scheme is the core of Baqend and quite effective in optimizing end-to-end web performance. Coupled with HTTP/2 this gives us a performance edge of >10x compared to Firebase and others. The central lesson learned for us is that it's not enough to focus on optimizing one thing (e.g. database performance) when the actual user-perceived problems arise at a different level of the stack.
The core problem was that the sustained number of requests did not really cause any bottlenecks. Actual problems were:
- At the Parse side: expensive database queries on the shared cluster that could not be optimized since developers had no control over indexing and sharding.
- At the customer side: any load peaks (e.g. a front page story on hacker news) caused the Parse API to run into rate limits and drop your traffic.
There are definitely some similarities. Like in Gomix, the idea of Baqend is to let developers easily build websites, bots and APIs. However, Baqend has a stronger focus on how data is stored and queried, while Gomix is more focused on a rapid prototyping experience inside the browser.
In my opinion, the whole movement about BaaS is all about making things as smooth as possible for developers and shorten the time-to-market to its minimum. What some providers like Parse lost on the way, are the capabilities for small prototypes to grow to large systems. That requires being scalable at both the level of API servers, user-submitted backend code and the database system. And at some point, it also requires letting the user tune things like indices, shard keys and execution timeouts. This is the route we took at Baqend. We do not want to be the simplest solution out there, but we aim to provide the fastest (we use new web caching tricks) and most scalable one (we expose how the database is distributed).
The database was not the bottleneck for Parse, how they used it was. Actually that is the case for most databases systems, you have to choose a set of trade-offs your application can live with. We wrote about this in more detail here: https://medium.baqend.com/nosql-databases-a-survey-and-decis...
Of course, it's a little unfair to declare it failed. The exit definitely was impressive. However, the product still died. It's hard to tell without the real business metrics whether this was due to lack of sustainable revenue and growth or just missing alignment with FB's overall strategy.
From a standpoint of traction it was no failure. A lot of developers liked it for a good reason. However, the pricing model and the lack of performance guarantees lead Parse to being used for prototype stuff in the free-tier mainly. I think, if they had a little more trust in the capabilities of developers to know what they are doing, the service might still be alive.
That is definitely true for some of the older versions of Mongo. Still, a quorum write concern will yield far fewer consistency violations. An interesting trade-off to take: losing a little data is better than losing a little latency.
A little Parse trivia a gathered from talking to (very capable) Parse engineers that now work at FB:
- Still the world's largest MongoDB user
- Had 1M apps, largest one with 40M users
- Server was Rails at first (24 threads max. concurrency), later rewritten in Go
- >40 MongoDB Replica Sets with 3 nodes each. Storage Engine: RockDB (MongoRocks). No sharding (DB-to-replica-set-mapping). Only instance storage SSDs, no EBS.
- Write Concern 1 (!) - some people complained about lost data and stale reads (slave reads were allowed for performance reasons)
- Partial updates were problematic as small updates to large docs get "write amplification" when being written to oplog
- Experienced frequent (daily) master reelections on AWS EC2. Rollback files were discarded -> data loss
- Special flashback tool that recorded workloads that could be rerun for load and functional testing
- JS ran in forked V8 engine to enforce 15s execution limit
- No sharding automation: manual, error-prone process for largest customers
- Indexing not exposed: automatic rule-based generation from slow query logs. Did not work well for larger apps.
- Slow queries killed by cron job that polled Mongos currentOp and maintained a limit per API-key + query combination
- It was planned to migrate Parse to FB's infrastructure but the project was abandoned
- Clash of clans used Parse for push notifications and made up roughly half of all pushes
I find this extremely interesting, as we are building a BaaS, too, but have a very different approach (Baqend). Coming from a database background, our idea is that developers should know about details such as schemas and indexes (the Parse engineers strongly agreed in hindsight). Also we think that BaaS is not limited to mobile but very useful for the web.
Also I think that providers should be open about their infrastructure and trade-offs, which Parse only was after it had already failed.
We fixed that problem and deployed it. It turned out to be the iframe with the tutorial app. That sets a new hash with a random todo list id and that captures the back button. It was not intended to keep people on our site.
Thanks for pointing the issue out!
BTW, I agree, RDBMSs are a great choice if the hot data set fits in main memory and queries are well-optimized through appropriate indexes. Issues usually only start to occur if 1) the data set is too large 2) queries cannot be optimized by the query optimizer and involve full-table scans 3) interleaved transactions contend on few rows.
UnfilteredStackTrace Traceback (most recent call last) <ipython-input-2-0e20e3adf861> in <module>() 2 ----> 3 image = generate_image_from_text("alien life", seed=7) 4 display(image)
67 frames UnfilteredStackTrace: TypeError: lax.dynamic_update_slice requires arguments to have the same dtypes, got float16, float32.
The stack trace below excludes JAX-internal frames. The preceding is the original exception that occurred, unmodified.
--------------------
The above exception was the direct cause of the following exception:
TypeError Traceback (most recent call last) /content/min-dalle/min_dalle/models/dalle_bart_decoder_flax.py in __call__(self, decoder_state, keys_state, values_state, attention_mask, state_index) 38 keys_state, 39 self.k_proj(decoder_state).reshape(shape_split), ---> 40 state_index 41 ) 42 values_state = lax.dynamic_update_slice(
TypeError: lax.dynamic_update_slice requires arguments to have the same dtypes, got float16, float32.