"Should" is the key word there. When Last Pass was breached in 2015, they were using only 5k iterations of PBKDF-2 to create the encryption key (but it was changed to ~100k in Feb 2018) [1].
Yes. Given how mass data leaks have shown just how bad people are at choosing passwords - I think it's a very safe assumption that a large proportion of Last Pass users have weak, easily guessable master passwords.
The ironic thing here is randomly generated passwords produced by a password manager are highly likely to be more secure than the password that protects the password vault itself.
Your arguments are sound in theory. But not in practice (for LastPass, 1Password has a better design). For example LastPass was deriving their encryption key with only 5k rounds of PBKDF-2 iteration (but used ~100k rounds to create their authentication hash)[1].
In LastPass's case, assuming the attacker has obtained a copy of the encrypted data, a smart attacker can ignore the authentication hash and just try encryption keys directly.
5k rounds of PBDKF-2 on a sub-$1,000 GPU is quite tractable to crack.
Was it a trivial thing for LastPass to fix? Of course. But that's not the point - if an attacker got a copy of the data before you fixed it - it's too late.
This is not an irrational fear given how Last Pass and many other password managers are designed. Last Pass (and others) derive encryption keys from their users passwords (via PBKDF-2).
Thus a (smart) attacker needn't guess the user's encryption key directly. They just need guess the user's password, "hash" it via PBDKF-2 with the proper params and see if it decrypts the data.
Using a sub $1,000 GPU and a table of common passwords obtained from popular website database leaks it's not that hard to crack the average joe's vault.
Attribution issues aside, there's two scarier potential issues I see, actually I see these with all posts on "here's our company's cool new A/B testing framework", that really scare me:
1. You're running a ton of tests, yet I see no mention of how you're adjusting your tests to account for multiple testing. The more tests you run the higher the chance you have of getting a false positive. Couple this with the majority of things you test probably won't be significantly better, and your chance of encountering a false positive is much much higher than you might think. You're running hundreds of tests and using a p-value of 0.05, but your chances of a false positive in your tests is much much much higher than 5%. Beware multiple tests and especially beware of the base rate fallacy.
2. Your post has no mention of statistical power or the size of the effect you're looking to detect. That makes me think you might not have considered this. If you don't know the effect size you're looking for or your statistical power, your A/B test results can't be trusted -- as you have no idea what your chances are of actually detecting a beneficial result if it exists.
[1] https://palant.de/2018/07/09/is-your-lastpass-data-really-sa...