Lock contention is a real issue for any multi-threaded system, and while a RW mutex is useful when you have a longer executing critical section, for something very short lived there is still a cache coordination cost. In many of the HashiCorp applications, we work around this by using an immutable radix tree design instead [1].
Instead of a RW mutex, you have a single writer lock. Any writer acquires the lock, makes changes, and generates a new root pointer to the tree (any update operation generates a new root, because the tree is immutable). Then we do an atomic swap from the old root to the new root. Any readers do an atomic read of the current point in time root, and perform their read operations lock free. This is safe because the tree is immutable, so readers don't need to be concerned with another thread modifying the tree concurrently, any modifications will create a new tree. This is a pattern we've standardized with a library we call MemDB [2].
This has the advantage of making reads multi-core scalable with much lower lock contention. Given we typically use Raft for distributed consensus, you only have a single writer anyways (e.g. the FSM commit thread is the only writer).
We apply this pattern to Vault, Consul, and Nomad all of which are able to scale to many dozens of cores, with largely a linear speedup in read performance.
We use HCL broadly across the HashiCorp portfolio. It was popularized by Terraform, but also used in Vault, Consul, Nomad, and Packer.
Important to note, that HCL is designed to be completely interoperable with JSON. In practice, we find that HCL tends to be more human readable and writable and is significantly less verbose.
However, if you are doing any sort of machine generation, it can be convenient to generate JSON and feed that in.
We've continued to grow the Nomad team, and are working towards the big 1.0 release milestone later this month. The OSS usage continues to grow double digit every quarter and our commercial offering generates millions in revenue. Nomad is also the backbone of both HashiCorp Cloud Platform and Terraform Cloud. Suffice to say, we continue to support Nomad and depend on it!
We don't have a white paper on this yet, but we have a white board video that explains both how it works conceptually as well as at a more technical level of deployment architecture and data flow. https://www.youtube.com/watch?v=tUMe7EsXYBQ&feature=emb_titl...
Glad you found Consul easy to get going with. We definitely focus on enabling K8S as a first class platform. There is documentation on setting up Consul and operating with Kubernetes here: https://www.consul.io/docs/platform/k8s/index.html
HashiCorp builds DevOps tools for modern applications. We build open source tools including Vagrant, Packer, Terraform, Consul, Vault, and Nomad. We also have a set of cloud services (Vagrant Cloud, Terraform Registry, Terraform Cloud, etc) that are high scale SaaS services. The HashiCorp products are downloaded tens of millions of times per year and power tens of thousands of organizations. Join our growing team and help organizations deliver modern applications using DevOps practices and enable dynamic multi-cloud infrastructure.
At HashiCorp, we are committed to hiring and cultivating a diverse team. If you are on the fence about whether you meet our requirements, please apply anyway!
HashiCorp builds DevOps tools for modern applications. We build open source tools including Vagrant, Packer, Terraform, Consul, Vault, and Nomad. We also have a set of cloud services (Vagrant Cloud, Terraform Registry, Terraform Cloud, etc) that are high scale SaaS services. The HashiCorp products are downloaded tens of millions of times per year and power tens of thousands of organizations. Join our growing team and help organizations deliver modern applications using DevOps practices and enable dynamic multi-cloud infrastructure.
At HashiCorp, we are committed to hiring and cultivating a diverse team. If you are on the fence about whether you meet our requirements, please apply anyway!
Radix trees are one of my favorite data structures, and widely under used. For many "dictionary" type lookups, they can be faster and more efficient than hash tables. While hash tables are commonly described as being O(1) for lookup, this ignores the need to first hash the input, which is typically an O(K) operation, where K is the length of the input string. Radix trees do lookups in O(K), without needing to first hash and have much better cache locality. They also preserve ordering allowing you to do ordered scans, get min/max values, scan by shared prefix, and more.
If you combine them with an immutable approach (such as https://github.com/hashicorp/go-immutable-radix), you can support more advanced concurrency, such as lock free reads. This is important for highly concurrent systems to allow scalable reads against shared memory.
To keep it brief, we are more committed to Nomad today than before. The team has doubled in the last year, and we plan to grow further next year as well. Our goal has always been to build a simple, general purpose scheduler, that composes well with the rest of the HashiCorp ecosystem.
Kubernetes is an important ecosystem and a platform we tightly integrate with across our other products (Terraform, Consul, Vault). We've always believed that our tools would be "mixed and matched" with different technologies, and that pragmatically we should support the broadest range of integrations.
Nomad is an important piece of our ecosystem, and we have many open source users, enterprise customers, and our SaaS offerings are built on it. Rest assured, it's not going anywhere!
That is exactly right. At the bottom of the blog post we touch on this, but if you are using consul-template to provide secrets via a configuration file, it can either restart or reload (signal) the application to pickup the changes. Alternatively, an application could be Vault aware and use the SDK programmatically.
Exactly what you suggested would work! Having an AppRole that never expires would allow the trusted orchestrator to authenticate on each run, and then generate and inject ephemeral credentials.
Effectively, there are only two possible approaches. Platform specific integrations, such as AWS IAM, Kubernetes, Nomad, etc. Or Trusted Orchestrators, which help inject the initial "secret zero".
When Vault connects to the endpoint system to create a dynamic user, it presents a set of credentials only known to it. You have to authorize Vault to create dynamic users, so a malicious actor would need to somehow obtain a similar level of privilege.
Vault typically prefixes something to the username as well (e.g. "vault-...") and also audits the creation of dynamic users so you can either look for the prefix or cross check the audit logs.
We work with many Fortune 2000 customers, and having 500K secrets is on the extreme side and most certainly puts you in an infrastructure where you have 50K-100K+ machines under management.
In terms of the "compute cost", for an infrastructure of that size this is a negligible amount of overhead. For dynamic secrets that live 30 days, rotating 500K secrets works out to 1 secret every 5 seconds.
The advantage would be avoiding an incredible number of static credentials sprawled across a very large estate, plus having a unique audit trail that lets you identify points of compromise. Treating those credentials as dynamic will also reduce the human overhead of managing so many credentials, instead focusing on roles and high level intents.
I question if there is an non-disclosed bias given the anonymous user, created just in advance of the comment.
The goal of Vault was to be a modern Kerberos, but invert the integration model. I think that is the Achilles heel of Kerberos, since it has a complex API and only works if the endpoint systems are tightly integrated.
Vault operates in much the same way, and could be viewed as a "KDC". However, instead of requiring the Authentication Service (AS) to be Vault aware, Vault uses authentication plugins to do the integration in the other direction. Similarly, instead of network services being Vault aware, Vault uses secret plugins to do the integration with endpoint systems.
This lets Vault easily be extended to support new authentication systems and endpoint systems, without needing those systems to be modified. Otherwise, its conceptually very similar to Kerberos!
Edit: I'm a co-founder of HashiCorp, and one of the early authors on Vault.
I'm one of the co-founders of HashiCorp, and I wanted to just chime in on the release. We may have done a disservice in the blog post by highlighting the new features landing in the Enterprise versions, because it obscures all the new functionality and work that has gone into the open source. We call out some of these at the bottom of the post, but just for the sake of clarity:
* Identity / Group Management. This sounds like a minor bullet point, but an incredible amount of work went into supporting a unified identity system that could span multiple external identity brokers. Our goal was to allow Vault to act as a broker between many Identity Providers and apply a consistent access control and group management scheme to all of theme. This was something we originally landed in Enterprise with 0.8 so that we could bake it with a smaller number of customers and have since brought to open source as we feel more confident in the implementation and APIs.
* GCE integration. This is part of our ongoing partnership with Google and our goal of natively integrating Vault with all the common platforms our users want to use.
* K8S integration. Again, this is about the last mile integration into the platforms users want and is in the open source.
* Oracle database integration. The database backend provides dynamic secret functionality across a number of RDBMS and NoSQL systems, and we are continuing to broaden the supported systems.
* RSA support for transit encryption. The transit backend allows Vault to do key management and cryptographic offload, and we are continuing to extend the algorithms and operation types that are supported.
* Lazy loading of leases. Common feedback from our large scale users was that failover could take many minutes. We've improved the loading of leases and other metadata to be asynchronous resulting in a dramatically faster failovers for large scale deployments.
* Lots of other bug fixes and minor improvements.
This is only a small sample of what has been added in the Vault 0.8.3 and 0.9 releases as open source. In the last 6 months, we’ve added more to the OSS than ever before, because Vault Enterprise has allowed us to grow the team. By adding a small subset of features to our Pro and Premium Enterprise offerings we are able to continue developing Vault, and we take a similar approach for our other tools. I totally understand the frustration about our opaque pricing, and this is because we are still making pricing adjustments as we refine our fit and better understand the many different use cases customers have.
Mitchell has publically stated and I'll repeat, that our goal is to publish both our framework of determining what is open source versus commercial (largely a focus on compliance, governance, and collaboration) as well as our public pricing. Our goal is to be as transparent as possible, while still accommodating the need to understand product/market fit, refine pricing, and be a solvent business.
I'm curious if the author was aware of our implementation at HashiCorp (https://github.com/hashicorp/memberlist). We use that implementation in Serf (https://www.serf.io) to provide membership, failure detection, and an event system. Serf is in turn, used by both Consul (https://www.consul.io) and Nomad (https://www.nomadproject.io). The point being, our implementation has seen significant production hardening and is known to work on clusters with 10K+ machines.
We've made significant enhancements to SWIM to make it work at scale, but overall it's a simple algorithm to understand and works quite well in practice. Glad to see more implementations of it!
Instead of a RW mutex, you have a single writer lock. Any writer acquires the lock, makes changes, and generates a new root pointer to the tree (any update operation generates a new root, because the tree is immutable). Then we do an atomic swap from the old root to the new root. Any readers do an atomic read of the current point in time root, and perform their read operations lock free. This is safe because the tree is immutable, so readers don't need to be concerned with another thread modifying the tree concurrently, any modifications will create a new tree. This is a pattern we've standardized with a library we call MemDB [2].
This has the advantage of making reads multi-core scalable with much lower lock contention. Given we typically use Raft for distributed consensus, you only have a single writer anyways (e.g. the FSM commit thread is the only writer).
We apply this pattern to Vault, Consul, and Nomad all of which are able to scale to many dozens of cores, with largely a linear speedup in read performance.
[1] https://github.com/hashicorp/go-immutable-radix [2] https://github.com/hashicorp/go-memdb