Thank you for reading the forum trying to better understand my POV.
re: #1. Yes, my stance with Anthony is that he doesn't think there is a bug in the software because he thinks this is a typical one phase competition. Flight Quest was more complicated than the typical Kaggle competition.
re: #2. I have done a poor job explaining the checkbox. I understand the confusion, as my statements are contradictory. In a single phase competition, the checkbox is important because over the course of the competition, numerous submissions are scored and you ultimately need to pick the one you want to apply as your final submission. In the case you do not make a final selection, the system selects the most recent submission.
The same UI is used for the two phase competition, including the same text that Anthony posted in his response to me, however the submissions work slightly differently as I mentioned.
> "As I have been saying over email repeatedly over the past 8 days, and yesterday on the phone with you and the Sorkin, there is no checkbox to check!"
I said this because I don't believe there should have been a checkbox on the screen. That is what I meant by saying there is no checkbox to check.
That is a good summary, but it is a little bit more complex than what you have stated. But your comment is helpful because this is exactly the understanding that Goldbloom has and it is incorrect in an important way.
The 23 entries that were submitted are for the public leaderboard set. The purpose of this set is so over the duration of the competition, competitors can see how they are performing relative to each other. These 23 submissions have no impact on the final prize winners.
A different data set, called the final data set is what actually determines the final prize winners. I submitted this file exactly one time. It was processed and scored by Kaggle and I followed the directions exactly as the competition admin stated. I did not check a checkbox because I did not believe it applied - I only submitted a single file and I recieved the 'all zeros' response from Kaggle's system as it said it would.
Also, the text Anthony quoted is not applicable to the Flight Quest competition either. This is the text that is used for the single phase competitions, not a two phase competition like Flight Quest.
If you haven't competed before I know the difference might not be clear. But if you read the response from the other competitors in the forum, you can see there is general agreement that the submission page is faulty for a two phase competition.
>'... declared winners are simply a matter of opinion, not fact'
I certainly agree with that part of your statement.
I have a hard time thinking about moving on to a new competition, this one has left such a bad taste in my mouth. Maybe time will change my mind, but certainly not happening any time soon.
I will do some research on how I can find some representation. Any tips, or experience with who I might get in contact with? Am I looking for some sort of technology or competition specialist?
I get the instant replay analogy, but I have a problem with a bug in their software impacting the results of the competition. I work in software, so I get living with bugs and understanding they are always going to be there. But I am unhappy that Kaggle/GE announced the winners via press release at the same the rest of the competition field was notified. It makes it impossible for a competitor to dispute the final results.
From the response I have received from Kaggle/GE I get the impression, under no circumstances what so ever would they go back and change the published results of this competition.
Ironically, that has been the best part about this ordeal - a number of the top competitors have shown support through the forum as well as private emails. I compete alone, so this issue has given me the opportunity to get to know some of the other folks.
I am not certain I want to work on anymore Kaggle competitions. I put (just a guess) somewhere between 100 and 200 hours into FlightQuest. It is hard for me to rationalize putting this amount of effort into another Kaggle hosted competition after going through the experiences I went through for this competition.
I have not consulted a lawyer yet, the reason being the competition rules state the following:
"By participating in the Competition, each Entrant agrees to release, indemnify and hold harmless GE, Kaggle Inc., and their respective affiliates, subsidiaries, advertising and promotions agencies, as applicable, and each of their respective agents, representatives, officers, directors, shareholders, and employees from and against any injuries, losses, damages, claims, actions and any liability of any kind resulting from or arising out of your participation in or association with the Competition. GE is not responsible for any miscommunications such as technical failures related to computer, telephone, cable, and unavailable network or server connections, related technical failures, or other failures related to hardware, software or virus, or incomplete, late or misdirected Entries. GE reserves the right to cancel, modify or suspend the Competition should any computer virus, bug or other technical difficulty or other causes beyond the control of GE corrupt the administration, security or proper play of the Competition, and to determine winners from among Entries not affected by the corruption, if any, in its sole discretion."
I interpret that as saying: GE can pick the winners using its sole discretion. So I don't believe this is a legal issue I can win. I have a lawyer in the family, I will run it by her and see what her opinion is.
I am a 'Kaggler', I have done well (prize winner) in some past competitions and I could not agree more.
In the last week of a competition, you go into kitchen sink mode trying anything and everything to squeeze the last few points out of your models. The objective of the competition from the competitors standpoint is to win, not create a solution that could be deployed into the sponsor's production environment. On going maintenance of the model is not a consideration.
As far as the netflix competition goes though, the final solutions did help publicize the potential of ensembling and RBMs - I am grateful for that. My personal approach to the competitions is to use them as an opportunity to try out new modeling techniques (i.e. RMBs, deep beliefs etc...) on real world data. It has the added bonus of potentially paying off with prize money (as long as you don't get screwed[1]).
I would like to hear from past Kaggle sponsors and see what they have done with the winning models.
re: #1. Yes, my stance with Anthony is that he doesn't think there is a bug in the software because he thinks this is a typical one phase competition. Flight Quest was more complicated than the typical Kaggle competition.
re: #2. I have done a poor job explaining the checkbox. I understand the confusion, as my statements are contradictory. In a single phase competition, the checkbox is important because over the course of the competition, numerous submissions are scored and you ultimately need to pick the one you want to apply as your final submission. In the case you do not make a final selection, the system selects the most recent submission.
The same UI is used for the two phase competition, including the same text that Anthony posted in his response to me, however the submissions work slightly differently as I mentioned.
> "As I have been saying over email repeatedly over the past 8 days, and yesterday on the phone with you and the Sorkin, there is no checkbox to check!"
I said this because I don't believe there should have been a checkbox on the screen. That is what I meant by saying there is no checkbox to check.