Every time this topic comes up, it seems people invariably divide into one of two groups. In the first group are those who trust in the odds and play the probability game. In the other, those who demand absolute guarantees. Neither group seems fully capable of understanding the other's standpoint.
Let me add my own perspective: Distributed locks are a fallacy. They can be beneficial in decreasing contention, under the assumption that "most of the time, only one actor will be active". However, by themselves, they offer no solid guarantees. The blog post addresses this point by introducing the concept of fencing tokens. These tokens have the potential to provide concrete guarantees, but they require the cooperation of downstream systems for enforcement, which isn't always possible.
I was really surprised to see Antirez argue for the probability approach.
A use case that I wish authorization service providers would talk more about is support for "list" queries e.g. What resources of type foo can the user read?
In really simple cases you may model this as a one-shot check on a logical collection resource e.g. If I have an organization, and an organization owns many repositories, I may check for action:read on resource:/organizations/:id/repositories. It's very limiting though. What if I want to list all repositories across all organizations I have access to? What if I have multiple levels I want to cut across? Do I need to do all the sub-queries and aggregations myself? Do I need to do a one-shot check on every potential resource? What if the answer is "no" for every one, and I end up doing a table scan of the entire DB just to produce an empty result set? etc.
You provide a distributed lease, not a lock. A distributed lease by itself doesn’t provide mutual exclusion. Distributed leases are typically accompanied with a fencing token (which your service cannot provide out of band) or an optimistic lock on the underlying "exclusive" resources (which could be implemented by the consumer of your service). I think of distributed leases as an optimization e.g. they provide soft exclusion in the happy path, which may reduce thrashing on the underlying real mutual exclusion mechanism (like an optimistic lock) under general use.
Your docs and terminology guide users into building incorrect systems e.g “ensure only a single instance of a process runs at any given time” is simply not true.
edit: I had a quick look at your Python library, and while Python isn't my forte, it looks like it can be trivially induced to break the mutual exclusion mirage: the "requests" lib's default request timeout is None e.g infinite, so what happens when `client.try_heartbeat` goes out to lunch indefinitely? Looks like `locks._run_heartbeat_loop` will hang, the lease will never be renewed, and within 60 seconds a competing lease holder will claim it. Boom. So you fix the timeout issue and problem solved right? Still no dice... what happens when you hit a pathological hang in your non-real-time runtime or OS? Boom.
There's so much more to CI/CD than the build definitions (e.g. dashboarding, log viewing/retention, access control, manual interventions, secret management, test reporting, etc.) and while some of your points resonate very strongly with me (e.g. local builds), I can't help but wonder what the endgame is here?
You've raised $27m to what? Give away a free tool for engineers to define their builds in? While they continue to use and pay for another service to actually run the builds? I assume you intend to replace the CI service at some point and move up the stack to monetize?
Without more transparency its easy to imagine something like...
Step 1. Drive uptake of your tool by selling people on the pitfalls of "CI lock-in"
Step 2. Introduce your own CI solution, which people can now easily switch to
Step 3. Lock people in
I am yet to work on a team with a simple, reliable and repeatable build process.
Most build processes (from basic scripts or Makefiles all the way up to CI/CD product-specific pipeline configurations) are complex, brittle, opaque and poorly understood by most of the team. I've seen time and time again one or two people emerge as the "build" people because most team members want to run as far away as they can, and it's easier to let someone else gatekeep the mysterious black box that sometimes does something useful. These processes typically grow organically on an as-needed basis and are rarely designed under a unifying vision.
I think it's a combination of lack of experience on the engineers part, bad off the shelf tooling, lack of recognition of the cost and as a result under investment by the business.
Agreed - I’ve spent months working on a deep tech problem before I could even show a functional core, let alone putting it into production. Some problems take time, experience and foresight to solve and the solution only dovetails at the end.
Recognition and trust have never been a problem for me. Have you hired at this level before, or merely assuming? As always there is nuance, as other commenters have touched on.
“Someone that avoids talking about what they did, and focuses too much on "we"”
This is very interesting. I naturally use “we” when discussing past products and achievements. I do this in recognition of the fact that things are very rarely designed and built in true isolation. Even principal engineers socialize their designs and thinking with colleagues, making small tweaks here and there or gaining additional confidence to move forward.
My first experience with event sourcing was having a new hire at a previous company try to shill it. It was a smaller company and he had previously built a career consulting on ES. Every problem we had he would come up with an argument for why ES would solve it. I soon learned he had zero technical ability, but was great at sounding credible to management. I liked to imagine him as a "seagull ES" practitioner. He flies in, sh*ts ES on everything, and flies out without having to deal with the mess.
Ironically I went on to use a lot of ES prinicpals in building a distributed deduplication engine for my previous startup after I left that company. We used a WAL which we were able to rebuild databases/projections from, ship to replica nodes for read replicas, push to S3 for resilient storage etc. it was an extremely powerful architecture for us. It was also complex and required a special skill set to work on and reason about.
I have to imagine anyone that advocates for "moving from CRUD to Event Sourcing" has recently been sucked into drinking the coolaid, and has no idea about the world of hurt they've signed up for, completely necessarily.
Object tags instead of named resources look like a neat and flexible concept, and is something I hadn't considered before. It is quite jarring transitioning from the filesystem example to the introduction of tags though, as something quite important appears to be lost in the translation.
In the filesystem example I can visit any node in thee tree and set an ACL on it, optionally recursively. This gives me a lot of flexibility to slice and dice permissions in a complex potentially-multi-tenant resource model. Tags appear to lead to a flattened view of resources - all resources with tag foo are treated the same, regardless of their location in the model. It would be nice to see the author acknowledge this and expand on their examples to see how they might incorporate relational data models.
The beauty isn't in the UX of the API per se, it's in the reduction of historically complex file system APIs down to a simple set of primitives and behaviours that enable massive throughput and scale while still being broadly applicable to most modern application needs.
Deleting a bucket in an object store is surprisingly non-trivial. See Amazon’s UI - it works fine for small object counts but is very buggy for larger buckets.
I recommend using a static site generator and hosting on S3. It's the cheapest way to do it and will survive a hacker news storm. It's also easily managed in version control and zero maintenance. For extra points you could add a CDN in front of it but it's probably not worth it.
I'm a big golang user so I recommend Hugo for the site generator.
Sorry, I should have perhaps put a disclaimer in my original comment. I work for a company called StorReduce and built our replication feature* (an intelligent, continuous "sync" effectively). We currently have a patent pending for our method, so I'm not sure if I can offer any real insight unfortunately.
I haven't looked at your project, but based on what you've said I agree the way you're doing it is conceptually as fast as it can be (massively parallel and leveraging metadata) whilst being a general purpose tool that "just works" and has no external dependencies or constraints.
That depends on how you're storing the files. I was really just trying to highlight that for deduplication across files you need to deduplicate before you encrypt.