Can you detail some of the rock solid solutions in 2008 for the problems addressed? For example, building multiple C++ packages from source. Google's Bazel wasn't open sourced until 2015, so solutions were still appearing long after 2008.
I feel that many of these legal arguments have not caught up with modern technology. If we had had the capabilities we have today 100 years ago, I am sure the lawyers would have pushed for maximum recording of all things. But face to face discussions are some how grandfathered in.
How is this any different than in-person conversations that are not recorded? Should companies be required to record all conversations that happen face to face?
I am thinking about the end application. All the more reason to ignore all of the benefits of a web solution and spin the highest performance version. Because performance is critical when viewing tons of cameras and lidar with near-realtime playback.
I have a serious question. Should this be a web visualization? Doesn't all the abstraction lead to performance hits? Tons of data needs to be pulled off the hard drive, network, RAM and shoved into the graphics card. Is putting a web browser in the middle of that pipeline the most efficient use of resources? I feel like it is a hammer situation.
That's not how that works and there are whole websites for figuring out if the plumber is useful. Angie's List, Yelp, etc... Plus plumbers normally have to be licensed by the state.
Instead this process is left to the individual corporations. So, you get mixed results in judging quality, because there is no standardization.
"Vogt couldn’t change lanes because of California state law around autonomous vehicles"
This statement is complete bullshit. The California DMV doesn't have any clear details defining operations of vehicles at this point in time. They only have rules about what is required to test on public roads. This includes 5 million dollar bond, applying for permit, etc...[1]
This statement alone (if correct from the reporter) makes me nervous about the amount of spin going on.
I was encouraging you to write down your mathematical evidence of better performing algorithms, because I know from my experience that when I try to write down proofs, my assumptions and reasoning become clearer. And often a step in the proof or logic that seems trivial becomes less trivial once you actually try to lay out the proof.
I believe that one of two things will arise when you go to write up a mathematical proof. One, you will be dependent on a statistical test that makes a normal distribution assumption implicitly or explicitly. Two, you will dependent on a bound that only applies in the limit. There is a third possibility, which is the most interesting to me, which is that you are dependent on a period of stationarity in your data, in other words, the distribution you are measuring must be stationary.
The proofs in the MAB papers are not intentionally obtuse or ignoring mainstream statistical ideas. They are written that way to say very explicit things under well defined assumptions. Locking down assumptions and being very precise is what writing down a mathematical proof is all about.
Ok, you are making a point about sampling. In periods of high traffic you have more samples that will bias the calculation. But, since the number of samples will be consistent between A and B, everything is fine. Fine.
I believe further discussion will not be productive. But, I suggest if you have a proof about A/B testing as it relates to the MAB problem or even if you have a proof about A/B testing in general that drops the normally distributed assumption or tells you exactly when to switch from exploring to exploiting, I suggest you write it up and publish it on arXv.
First, I was using Student t-test as a stand-in for whatever test or statistical measure you would like to use. I believe the popular one is Hoeffding's inequality in the bandit literature, hence the log term in the MAB algorithms. I agree this was a poor choice of example.
Second, I believe I am getting hung up on the fact that different arrival times are "weighted" differently. I think you are claiming that the Poisson assumption gives us equal numbers of A and B trials, so we can combined the statistics (counts) and avoid Simpson's paradox. This is fine, but why would you say "different times of arrival are not weighted the same". Does this mean you are somehow weighting periods of heavy traffic down and weighting low traffic up?
So, what happens when trial A becomes less favorable over time or is less favorable for brief periods? This means that the underlying random variable's mean is changing over time. Most statistical bounds cannot handle this situation.
I am not saying that A/B testing is not something we should do in general. I am saying that it is a good heuristic with very few provable properties compared with logarithmic regret MAB algorithms.
Also, I agree that MAB algorithms do not consider the case of adding bandit "arms" during the trial. A dynamic number of bandit arms problem may not be well studied, but is very interesting. This might be another reason for requiring a reweighing of the samples.
I don't know how we got onto arrival rates of our visitors. I would just like to state that Simpson's paradox is exactly why we shouldn't compare percent conversions. They are meaningless. However, many of the statistical tests like Student's t-test compensate for this paradox by including the number of samples in the tests. See : http://en.wikipedia.org/wiki/Student%27s_t-test#Unequal_samp...
I think you said one thing that are at the heart of the issue. We assume that the E[conversion of B] > E[conversion of A] for the entire period sampled.
I think all of the details about Poisson processes are not required if you just assume that each person is drawn IID from the population.
I just don't think you are answering the right questions here.
Let's assume that each person arrives IID from the infinite population. Then, we have a Bernoulli process for each A or B query. A "conversion" results in a 1, a failure results in a 0. Since, these people are arriving IID, we can select a sub-sample which is also IID. We would now like to estimate the parameters for each process and/or compare the two processes. We can do this using t-test. This will give us the statistical significance that the one group had a higher "conversion rate" than the other group. Note: rate does not factor into this problem at all because we assume the participants were IID, so the t-test (used correctly accounting for different number of samples) will tell us which test is larger.
My question is now what happens when the parameters of your queries for A or B change over time. Still under the assumption that E[B] > E[A], it now matters greatly in which order you use your samples.
I think the only reason you brought the Poisson model into the discussion is to weight the more recent samples higher and down weight the earlier samples in your basket of samples. This is a heuristic for considering a fixed interval in which the samples are stationary. It effectively considers a window that slides with the time of arrival.
https://youtu.be/zm_rlLyelQo