The call out on premature optimization is valid. However this article misses the mark on a couple fronts.
One, as others have called out, is the ability to control rollout (and rollback) without needing a deployment. Think mobile apps and the rollout friction. If something goes wrong, you need a way to turn off the offending functionality quickly without having to go through another deployment or a mobile app review.
Second, is to be able to understand the impact of the rollout. Feature flags can easily measure how the rollout of one feature can affect the rest of the system - whether it is usage, crash rates, engagement, or further down the funnel, revenue. It’s a cheat code for quickly turning every rollout into an experiment. And you don’t need a large sample size for catching some of these.
By having this power, you will find yourself doing more of it, which I believe is good.
This is a good list that includes a lot of things most people miss. I would also suggest:
1. Tight targeting of your users in an AB test. This can be through proper exposure logging, or aiming at users down-funnel if you’re actually running a down-funnel experiment. If your new iOS and Android feature is going to be launched separately, then separate the experiments.
2. Making sure your experiment runs in 7-day increments. Averaging out weekly seasonality can be important in reducing variance but also ensures your results accurately predict the effect of a full rollout.
Everything mentioned in this article, including stratified sampling and CUPED are available, out-of-the-box on Statsig. Disclaimer: I’m the founder, and this response was shared by our DS Lead.
As with anything, there was a long list of work that I never got around to mostly because life happened - kids, startups, etc. I would love to bring Small Basic to the browser where it's easy to share your programs and creations with your friends.
I continue to maintain the webservices needed for the app, but any substantial investment is at least a year out unfortunately.
Hey all, creator of Small Basic here. I made this back in 2008 as a side project when I used to work at Microsoft. My belief was/is that you can get more kids interested in programming when the IDE (language + runtime + libraries) encourages iterative coding with simple gratification loops built in.
You can see a bunch of decisions influenced by .Net which I was working with back then. I had been meaning to go back and revisit some of those decisions, but haven't had a chance yet. I have changed my stance around scoping and argument passing in functions, for instance.
(What is really neat is that 15 years later, my son is learning coding with Small Basic. And I hired an engineer for my current startup who had started coding with Small Basic as a kid in Brazil.)
Fast growing Series B startup, hitting strong traction with customers. Looking for generalists that can go up and down stack, and for Infra engineers that can build scalable systems - we process 7B+ events today, and doubling every few months.
Hey all, we're a small team of ex-Facebook engineers. We want to bring some of our favorite data/developer tools that we used at Facebook to everyone. Our goal is to help developers ship code faster using feature gates to decouple dependencies, realtime targeted config to reduce code churn, and experiments to validate features in production.
This is our first preview, and we have Feature Gates, Dynamic Config, and Event Logging, all accessible via our SDKs in JS, Node & Swift. We would love your feedback. Up next, we are building feature-launch based insights and metrics debugging, which we hope to ship in the next few weeks.
One, as others have called out, is the ability to control rollout (and rollback) without needing a deployment. Think mobile apps and the rollout friction. If something goes wrong, you need a way to turn off the offending functionality quickly without having to go through another deployment or a mobile app review.
Second, is to be able to understand the impact of the rollout. Feature flags can easily measure how the rollout of one feature can affect the rest of the system - whether it is usage, crash rates, engagement, or further down the funnel, revenue. It’s a cheat code for quickly turning every rollout into an experiment. And you don’t need a large sample size for catching some of these.
By having this power, you will find yourself doing more of it, which I believe is good.