Hadn't heard of PostHog before stumbling on this post. Just wanted to mention that "hybrid product analyst/product manager" caught my eye, and your other material on product management is great!
People have been putting their business into spreadsheets and then tweaking one number for so long they've lost the plot completely.
It's like if you sold cheeseburgers and thought "what if we charged the same price, and didn't include cheese?" and then when that worked because some people don't eat cheese, went "what if we charged the same price, and didn't include bread?" and that worked because some people don't eat bread. Eventually there is no lettuce, onion, or tomato and you're serving plain patties in a paper tray, condiments extra. Customers have dwindled, and the idiots left in charge have never made or sold a cheeseburger or did anything other than move numbers around in an already successful business can't think of a better idea than look at COGs on ground beef and suggest they try selling ketchup on a napkin.
I can see the misunderstanding, but I was not actually describing parimutuel betting (horseracing). In parimutuel betting the odds continue to change up until the race even once you've placed your bet ensuring that the payout is always the total amount wagered less a fee kept by the house.
Sportsbooks will open lines intelligently, but they absolutely do move the line in response to market forces in an attempt to balance money on both sides, because when the money is balanced, they are guaranteed profit.
It's true that when you make a sports wager, the house is paying you out of their wallet. It's also true that they employ a lot of energy and expertise in order to open the betting at accurate odds. However, no corporate, end user facing sportsbook is themselves fading action on one side of the match intentionally. They aggressively try to balance money on both sides so they can guarantee a profit.
Sportsbooks make money by taking bets on both sides of a game and offering odds that work in their favor. For example, even on an "even money" bet, you might have to bet $105 to win $100. The more one-sided a game seems, the bigger the gap between the odds on either side because the sportsbook is trying to manage its risk. As people place bets, they adjust the odds to balance the action. The sportsbook isn't banking on you being wrong—they want enough bets on both sides so they win no matter what. The difference between the odds is basically their "fee."
As a professional bettor, you're not really outsmarting the sportsbook—you’re trying to outsmart the public. The key is finding moments where the crowd is wrong enough that betting the other side makes sense, even with the sportsbook’s fees. That means you’ll often skip betting when the odds are pretty accurate.
Most sportsbooks will limit how much you can bet if you're too successful, but they usually won’t ban you outright.
I'm confused about the using the greyscale map tiles to estimate ping.
You don't want to have the users ping the servers themselves because those pings could be inaccurate or noisy, so you use historical average data for users in that region instead to get a nice simple number. But... how do you know where the user is? IP Geolocation? Can't that be wrong also?
Isn't it better to have a direct measurement which could be a little wrong than an average of a guess which could be really wrong?
Anecdote: eye exercises and subsequent vision therapy (went to an ophthalmologist out of pocket) were very beneficial for me (someone with mild myopia and no other diagnosed conditions), I'm thankful I did it, and would recommend to everyone, even those without myopia. I had a small reduction in my rx, and many other benefits.
That field didn't exist 30-40 years ago, and it is increasingly common now treating conditions (not myopia) that were previously thought to be non-reversible. But who knows what the future holds.
When doing this in the past, I settled on an sqlite database with one table that stores the compressed html (gzip or lzma) along with other columns (id/date/url/domain/status/etc.)
Also made it easy to alert on when something broke (query the table for count(*) where status=error) and rerun the parser for failures.
I'm a product manager. His post doesn't seem unreasonable.
Product managers are often hired to be a cat's paw for an unsustainable and/or ineffective way of getting technical work done. This is what many businesses want when they hire into product, because this is a more convenient explanation for why things aren't better than any alternative (such as you're doing too much stuff, you're building the wrong things, etc.).
This selects for people with the skill and temperament to thrive in this role. Being a bullshit artist is a great fit. Being a pushover and repeating everything your stakeholders say is easy. Asking difficult questions and being a skeptic is hard and doesn't make you popular.
The main reason that I think I make good decisions in the absence of data, is because of how much I've relied on whatever data is available to inform my decisions and learn going forward. This is a huge advantage to multivariate testing as a practice/culture. As a consequence, it's often very easy for me to pick out when readouts are giving a deceptive answer (i.e. oh, the scope of this uplift is too much, we need to double check if something happened to negatively impact the control).
I'm not sure I'd agree that people are often operating "close enough to optimal", but I would definitely agree that integrating experimentation is hard enough that sometimes the effort (or mistakes) you can introduce will cause more problems than you're helping. But I think this is more a function of how poor people are at the mechanics and the mindset of running experiments than that they're doing good enough pricing hot dogs. Experiments in many places are looked at for either CYA or boasting about quarterly results and not to truly learn/grow/improve.
The risk is not precisely and exactly quantified. The uncertainty isn't precisely quantified. This isn't an experiment where someone is asked "Would you like to have $2, or flip a coin and get $5 if its heads, or roll a die and get $16 if it's a six?"
I think it's more about how they are estimating risk in different areas, and ways in which one might commonly see people overvalue or undervalue risk, or overvalue or undervalue potential payoffs. It's not that they're trying to be more conservative, their incorrect calculation just makes one option seem like a bad (or worse idea) than another.
An alternative viewpoint is that maybe their risk calculation isn't actually wrong and they're not risk averse, they're just taking the strategy of only making decisions that don't make you look foolish and can be blamed elsewhere if you lose, but can take credit when you're lucky and they win.
Saying yes to 2$ over a coin flip to $5 is risk aversion. Saying no to $3 so you can coin flip for $5 is probably just foolish.
I am also team bayes for all the reasons you stated, but do want to argue a couple counterpoints:
* While you don't have to have a fixed sample size up front, you can still "cheat" in a bayesian analysis if you peek constantly and end early on promising results that you want to win, and let them run longer otherwise. So you want to do something to account for this (put some structure in place, approach with skepticism, laugh and put on sunglasses, whatever).
* It's very often useful in practice to have some idea of what kind of answer you're going to see in how long for planning reasons -- for example, rather than your boss saying "I need an answer tomorrow" they say "I need an answer as quick as you can". Bayesian methods give you the flexibility to be risky when you need to and accurately count for uncertainty, but sometimes you still need to predict and strategize around ideas like "We'll be about this certain in 2 days, and about this certain in 1 week, and about this certain in 4 weeks and it seems like planning on next Tuesday is the right call"
I've found understanding these frequentist methods to help inform my guesstimates of how experiments will play out with regards to sample size and impact as well as honestly evaluate the trade-offs in evaluating the tests where I wasn't running it -- AB testing is really widespread so I feel like it's important to understand frequentist tests well even if you intend to never use them if you can help it.
Right you are, but in defense of the author, understanding this concept while looking through readouts done by past colleagues has also made me feel depressed at times.
In my work experience so far, when it comes to A/B testing what I've observed is that:
* The better the tooling in general increases the proportion of people doing it _wrong_, because when it was harder, this selected better for people who wanted to do it right. Making it easier means more people do it right, but more people who would otherwise not do it at all can now do it, and do it wrong.
* Making aspects impossible by design will impress and amaze you with how humans and groups figure out how to not only prevail over what you tried to make impossible, but do it even harder than if you did nothing at all.
The best I think we can hope for is making things by default easier to catch, or easier to find later, or less deceptive.
Hard things are hard and that's ok. I think we should spend more time channeling our empathy into aspiring for ourselves and others to be better and do hard things, and making the ability to learn and do hard things accessible to everyone who wants it, as opposed to trying to pretend hard things are easy.
> What I don't understand is why power would be so relevant.
Doing the A/B test itself has a cost greater than just building the feature and releasing it (supporting two variants in production), and beyond that you also need to take into account the cost of acting on the results (i.e. if control wins what do you do? if it's a tie what do you do? best to budget for maximum possible effort, or the expected effort -- but expecting for the variant to win handily is budgeting for the minimum possible effort).
I've seen multiple businesses that always schedule around shipping an A/B test and context switching to the next project while the results stream in. Any result that isn't shipping the variant after x weeks is a huge inconvenience that throws off multiple teams, which means all those cognitive biases start to creep in and make it comfortable to declare loser variants as wins or ties.
While it's easy to write this behavior off as yet another way that groups make irrational decisions, I think the bit of truth in there is that sometimes, the cost of running the strictest, science-iest A/B test is simply too high. Power is a key part of how you reason that out up front, so you can make a rational decision not to test, or to modify your test to make the payoff worth it. For example:
* Let's set the goal metric for something higher up the funnel which is further from our true goal (more $) but happens much more often, so we can see the effect in 1 week instead of 2 months
* We really need to do this for [variety of business strategical decisions], so let's structure our experiment to make sure it won't cost us more than $X in a worst case scenario and find out in a few days rather than wait 2 months
> I now know that B is better than A.
You know that B outperformed A in the experiment. Checking statistical significance is like asking a trustworthy person "Are you sure?" and them saying "Yeah, I'm pretty sure". It's a percentage because it's sometimes wrong, and this doesn't account for the massive amount of real world factors that can still mean an experiment conducted with bulletproof math behind the analysis is still taking people down the wrong path.
This is really cool. I don't understand something about the architecture though -- you mention it relies on Postgres FDW. When you install Steampipe are you installing Postgres locally and using that, or are you using a remote postgres server?
Since there were a lot of people clamoring for more updates, subscription licensing, etc. I just wanted to add some volume to the opposing viewpoint.
I registered my copy of sublime text maybe 8 years ago. Before that I had used Notepad++ for years as well. I love that sublime is a thing that I paid for once and I can trust that it will always fundamentally work without problems for my basic use cases. I never have to think about "oh well I'm not doing so much text editing at the moment, maybe I'll turn it off for now..." like I do with some other subscriptions. I'm not worried it won't work or will degrade because it seems stable as a rock. I primarily used Windows when I bought it, and I've been Mac only for 4 years, and it's still with me.
Sublime text is special and I love what you guys have done! And I appreciate that for my $80 or whatever it was, I can use this thing the way I use it right now and never worry about it again.
Honestly, I think people are being reactionary from how common abandonware is in the open source community (obviously sublime is closed source, but people who use sublime use a lot of open source), as well as the long delay between ST2 and ST3. I remember plugins with docs that said "I have abandoned this project because Sublime Text is dead" when ST3 was still in beta. It is... probably not great that when I check the About window on my copy of SublimeText, it says Copyright 2019 and the changelog on your website says October 2019. It is useful for a customer facing product to not look dead, but this is completely possible without being forced to ship updates or features on a regular basis. Just be mindful of customer perceptions and what the touch points are and be patient and consistent within your bandwidth.