This is a pretty hilarious and long-winded way to say "we have no idea how to lock someone out of a web service:"
> 1. While Ruby Central correctly removed access to shared credentials through its enterprise password manager prior to the incident, our staff did not consider the possibility that this credential may have been copied or exfiltrated to other password managers outside of Ruby Central’s visibility or control.
> 2. Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.
Those tools are certainly useful! In the recent production incident I mentioned, we we able to quickly use SLOWLOG to determine which calls were causing the latency spikes. Thanks deeply for providing all of these useful tools and docs.
> there is a percentage of users that will not read the doc and just deploy it
I wonder what this percentage is? I'd wager that it is higher than you might have anticipated, especially given the other comments on this thread.
As an engineer on a team ultimately responsible for the availability of a production service, it's my responsibility to ensure that the percentage of engineers that know the latency side effects of any Redis calls they make is near 100%. In the presence of such variable latency, any means of making that variability more obvious to all users of Redis would be a positive step towards happy users and operators.
This line jumped out to me as well. After a recent production incident determined to have been heavily influenced by several thousand O(N) commands being called against a list several orders of magnitude larger than normal, I'm thinking about this a lot.
I'm confident many other users of Redis are using O(N) operations in production against small datasets without knowledge of how much latency will be introduced by those operations when that dataset grows. This is exactly the kind of situation that makes me immediately skeptical when I find Redis in a emergent system design.
I'm considering what initially felt like a draconian means of remediation: using rename-command to rename all O(N) commands to BIG_O_N_$COMMAND to ensure everyone using them knows the possible impact and to allow for easy detection during code review and/or Redis latency spikes.
The more I think about it this approach, the more I feel that this should be the default mode of operation for Redis in production. SREs around the world would collectively save decades of time if the every engineer writing Redis queries to knew this fact by heart:
> Redis is very fast as long as you use O(1) and O(log_N) commands. You are free to use O(N) commands but be aware that it’s not the case we optimized for, be prepared for latency spikes.
I threw out that prototype soon after the talk. At the time, there weren't a lot of other engineers at the company doing Erlang, so maintenance was considered to be a long-term problem. I'm glad we made that call.
I threw out that prototype soon after the talk. At the time, there weren't a lot of other engineers at the company doing Erlang, so maintenance was considered to be a long-term problem. I'm glad we made that call.
Hi cdibona! I'm primary on-call at GitHub today, and I wanted to say thanks to everyone that worked on https://code.google.com/export-to-github/ and worked with us to load test it before this announcement. It's really cool to see so much work put into making it easy for folks to easily move their data.
First off, if you use a DNS provider that has support for a ALIAS records or something similar, pointing your apex domain to <username>.github.io will ensure your GitHub Pages site is served by our CDN without these redirects.
I wish we could provide better service for folks without a DNS provider that supports ALIAS domains in the face of the constant barrage of DDoS attacks we've seen against the IPs we've advertised for GitHub Pages over the years. We made the decision to keep DDoS mitigation enabled for apex domains after seeing GitHub Pages attacked and going down a handful of times in the same week. It's a bummer that this decision negatively impacts performance, but it does certainly improve the overall availability of the service.
FWIW, we considered pulling support for GitHub Pages on apex domains about a year ago because we knew it'd be slower than subdomains and would require DNS configuration that would be challenging and frustrating for a large number of our users. However, we ended up deciding not to go that route because of the number of existing users on apex domains.
has anyone out there actually used or evaluated clustrix? i haven't been able to find anything on the internet about this thing that hasn't come straight from the company.
You can't gain any more cycles, of course, but you can gain throughput by adding processes to an already cpu-bound app by giving faster incoming requests a place to go instead of sitting on the global queue while waiting for other requests to process.
> 1. While Ruby Central correctly removed access to shared credentials through its enterprise password manager prior to the incident, our staff did not consider the possibility that this credential may have been copied or exfiltrated to other password managers outside of Ruby Central’s visibility or control.
> 2. Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.