With Salesforce specifically, getting a stable and reliable event-driven stream. That's why we ended up adopting a hybrid approach of subscribing to pub/sub events, but also periodically polling the REST API in case pub/sub misses anything.
Salesforce's scattered docs don't make best practices super clear, so designing the syncs took us some time.
Good question. Our ultimate goal is to be the go-to for any two-way data syncs.
We think the Salesforce syncing market by itself is 500 million to 1B. If you start including all of the tools we have our eyes on (Hubspot, Monday, ERPs), the market size comfortably gets into the billions.
Going after Heroku Connect makes sense as a starting point, but we've got our sights beyond that.
Gotcha, that makes sense. From our perspective, the one-way sync market (backups and ETL/rETL) was already saturated with heavily-capitalized players who were hitting each other hard with marketing dollars. We didn't feel that we had a unique insight with one-way syncs, but did feel like we'd come across something unique with two-way use cases.
So we decided to double down in this smaller, underserved area for now as we try to build something people love, even if it's a bit niche.
That being said, there probably is room in the backup space for challengers against the incumbents - it's a massive market.
Nice! Great point about the round trip - we do something similar for formulas and auto-generated fields like `Id`. That's awesome you built this in-house.
> Syncing data from Salesforce (to seed a database for example) is done via REST too. It works OK.
Have you thought about using the Bulk API for seeding? We started relying on that instead of REST, which helped us seed massive DBs much faster / more efficiently.
1. Yes, on our roadmap for Q1! Getting that request a lot
2. We don't currently sync metadata
3. Our footprint on your Salesforce API depends on whether you're using polling or streaming, and then it depends on the cadence of your syncs or frequency of changes. You can see some data on best/worst case scenarios here: https://docs.usebracket.com/connecting/salesforce_api
4. Yup, we priced based on the amount of data kept in sync. You can see more here: https://www.usebracket.com/pricing
Agreed on the trickiness! Our early users largely told us they preferred one source to take precedence in a conflict, and would rather set that general rule than review every conflict manually. But a handful have expressed interest in the latter approach, so it's on our roadmap to build.
> How specific is your solution to Postgres? Could it be ported to another db engine?
Our polling approach is relatively database-agnostic. We just need to handle each DB's quirks with our transformers (e.g. dealing with MySQL's lack of BOOL field types).
Streaming is currently Postgres-specific. We're planning on rolling out support for MySQL next, after we've finished our Hubspot integration. Do you have a specific DB in mind?
> (And, how are conflicts resolved? In a huge system with millions of records coming from everywhere it can fast become nightmarish?)
The primary source wins any merge conflicts that happen within a sync period. With polling, it's pretty straightforward: at every poll, we see how each side has changed, and for any record pairings for which there were edits on both sides, we prefer the primary source.
With streaming, we employ a hybrid method, where we only poll when events occur in either Salesforce or Postgres. If at that poll, the same record has been edited on both sides since the previous poll, we still prioritize the primary source (Salesforce). You can read the step-by-step flow here: https://docs.usebracket.com/streaming#the-streaming-sync-met...
Agreed! We’re big fans of consolidating in Postgres. We’re also hearing some downstream benefits to having things scaled in Postgres (makes reporting easier, allows the data team to use SQL rather than hitting an API, etc) from our users.
> Many clients I can think of this being most useful for would rather host it themselves, is that an option?
Right now you can self-host the associated datasets (like the Postgres event log table), but we're still working on allowing you to self-host the entire service. Stay tuned :)
> do you support subsetting a data source -- only sync records matching criteria?
In a one-way sync from Postgres to Salesforce, yes, you can apply filters using a SQL statement, but we’re working on adding this to the Salesforce side as well as two-way syncs. From your perspective, how would you want to set these filters? A SOQL query, or something else?
> Do you support to/from different instances of the same connector (e.g. Salesforce to Salesforce)?
Yup, our infra is data-source agnostic! But Salesforce <> Salesforce is not heavily tested, so there may be some funky behavior with oauth tokens if you’re trying to connect objects from two different instances during onboarding. Curious what use case you have in mind?
> Can you perform any transformations like map over the data, normalize or denormalize tables, etc?
Besides the one-way SQL filter I mention above, we try to make field mapping easy between the sources by automatically transforming when necessary (e.g., transforming a Salesforce picklist to a Postgres integer and vice versa). But we’re working on allowing users to create more detailed field-level transformations in the next few months.
Thanks for the feedback! Totally hear you on SSL, we’ll move that to every tier.
Agreed that many technical founders build on the pain points they've experienced themselves. I'm curious your take on why the managed services approach is hard to scale?
IMO, the reason many YC companies struggle with enterprise sales is because it's generally much harder than PLG, with long sales cycles and delayed feedback loops. It was definitely a learning curve for us!
We use a custom trigger to build an event log in Postgres. We actually found it more straightforward to use this approach than to use the WAL packages we considered.
Thanks! We've had to internalize the lesson "startups can do one thing well at a time" :)
At first, it was a matter of doing discovery with people who we thought would find Bracket useful. These weren't sales convos, but pure feedback/discovery. From that process, we got a good sense of which roles face these pain points, and exactly what those pain points are.
Since then we've split our time between 1) writing content/distributing it in places people would find it helpful and 2) reaching out directly to people who would use Bracket. In the latter case, we really don't like generic sales campaigns, so we spend a good bit of time crafting each message.
For sure, definitely has not been an easy problem to solve. The two-way syncing is one tough element (for the reasons mentioned above), and configuring our requests to the Salesforce API in a way that minimizes our footprint (while still keeping syncs fast) has been another tricky task.
Salesforce's scattered docs don't make best practices super clear, so designing the syncs took us some time.