It can get complicated quickly if you're actually using it in a production system. At my prev enterprise saas company we had feature flags that could be turned on per customer / per environment (dev, staging, prod) with permission + logging model such that our support team could also toggle flags with history of who turned on what. We also had "per user" feature flags for certain test users at companies and had DSL rules to evaluate the features
Fish sauce is delicious but had to stop using it since it's high in histamine (gives me a stuffy nose) and potentially carcinogenic due to its high levels of nitrosamines
If you're interested in adversarial NLP, I also recommend reading this blog post on adversarial attacks on GPT2 with universal triggers (e.g. adding "nobody" as prefix for all inputs causes all entailments to be predicted as contradiction).
You could do something similar to how they trained a ML model to find antibiotics compounds: https://www.cell.com/action/showPdf?pii=S0092-8674%2820%2930.... First, train a deep learning model to learn a representation of molecules from their molecule structures. Then feed in the thousand or so known compounds that produce pleasant or unpleasant smells as training data with some score of "pleasantness". We can then use this model to quickly score millions of compounds and select candidates to test.
This is nothing like the sharpshooter fallacy. The analysis determined the average performance of the analysts with >100 stocks rated and 10 analysts out of 16 did better than the rest.
Outliers were removed to get a better measure of the "accuracy" of the price targets.
10 day windows were used to reduce the amount of volatility/noise in a time frame
Return horizons for 1 years was used because price targets are for one year.
Theres only 15 or so analysts I looked at.
I was doing this as an exploratory data analysis and didn't want to pull out my old stats textbook.
Cutoffs were chosen to reduce volatility of measurements since I was looking at percentages. A stock going from $1.5 to $2.0 is a 33% increase whereas the movement of $100 to $133 is significantly more impactful. Stock with lower market cap have more volatility. The minimum analyst rating was chosen to eliminate analysts with very small number of ratings as they would be unreliable.
I probably should have included the outliers when analyzing overall performance but if I recall correctly they did not have a significant effect.
The top analysts were determined by the average performance from one year after their ratings have been made. This isn't the top analysts out of 50,000 it's the top out of 50 or so analyst-rating pairs. There were only 16 or so analysts in total that I looked at. This isn't an instance of survivor bias as your example states. If I were to be more rigorous I could give a statistical test for this.
You're correct that I should have accounted for outliers when measuring performance of this strategy. If I recall correctly, even with the outliers, they did not significantly affect the average performance of analyst ratings. I would have to rerun the numbers though.
The analysis was more about measuring the performance of analysts which is why the price data for before and after the recommendation. For practical purposes of using this strategy, you are right that the price data from days after release would be better.
If the top 10 stocks you picked beat the market and have consistent earnings and dividends over a period of time, would this not be a repeatable strategy?