Salesforce hackathon comes under attack from devs(venturebeat.com)
venturebeat.com
Salesforce hackathon comes under attack from devs
http://venturebeat.com/2013/11/22/salesforce-hackathon-comes-under-attack-from-devs/
15 comments
Paying $500 for a 0% chance to win; that doesn't sound like the best deal to me. You can get a lot more value from a regular hackathon that you can actually get some attention and feedback from and that isn't going to charge you a crazy amount of money. I don't understand how you can justify the competition when your app wasn't even viewed.
Edit: Apparently the ticket price was $100 and then reimbursed. That makes it better, but I would still argue you get more benefit from a standard hackathon.
Edit: Apparently the ticket price was $100 and then reimbursed. That makes it better, but I would still argue you get more benefit from a standard hackathon.
Tickets for the Hackathon were $99 a head but they quickly started giving them out for free. I also heard stories that people got reimbursed fro the ticket price if they had payed.
Our team (2 people), paid $99 each for Hackerpass tickets, which Salesforce then reimbursed after they started giving them out for free.
Launch Hackathon had the best process of the dozen-or-so large hackathons I've been a part of. They split the judging into two rounds: a 4 minute face-to-face interview with 2 mentors (2-minute presentation, 2-minutes for questions), and if you were the top team in your group (10 teams per interview group) then you made it on stage for the grand final pitch to the audience and a judging panel of 6.
Things they did well: * Each team received a score and brief feedback afterwards. (Ours was "7/10, it looks like you've solved a real problem for that industry" - so not extensive but our effort was validate, which is psychologically important for the end of a weekend hackathon - especially when you don't make the final round : ) * 20 mentors for preliminary round meant that everyone had their pitch and the process went smoothly and ran on time (seemingly a rare occurrence for these events). * 3 wild-cards. (13 teams eventually pitched) * Face-to-face interviews. A nice change to the theater of pitching an audience. * Grand final was short instead of a 2-hour fiasco where everybody loses interest.
Things they did well: * Each team received a score and brief feedback afterwards. (Ours was "7/10, it looks like you've solved a real problem for that industry" - so not extensive but our effort was validate, which is psychologically important for the end of a weekend hackathon - especially when you don't make the final round : ) * 20 mentors for preliminary round meant that everyone had their pitch and the process went smoothly and ran on time (seemingly a rare occurrence for these events). * 3 wild-cards. (13 teams eventually pitched) * Face-to-face interviews. A nice change to the theater of pitching an audience. * Grand final was short instead of a 2-hour fiasco where everybody loses interest.
I don't know anyone involved in the judging or the winning team, but if what is alleged is true (that a team won that violated the terms of service) then my guess is that it was the team actively breaking the rules, and a judging panel selecting what they saw was the best product and being unaware of the violation. Honestly, judging hackathons is tough and screening each team ahead of time is close to impossible - it's not like you're going to code review everybody.
Cheaters always get caught.
Cheaters always get caught.
This article doesn't mention the allegation that the judges didn't even look at some of the other applicants: http://salesforce1million.challengepost.com/forum_topics/264...
I work at salesforce but I am not a spokesperson for salesforce nor do I have any authority to speak for the company. What I write here, I write as a member of HNN only.
I did not have any part in the hackathon process, but I had and still do have quite a bit of visibility on how the process was run. I can assure you that an enormous amount of effort went into the judging and that it was entirely unbiased. There were literally dozens of the most senior executives and engineers within the company that worked on the earlier judging rounds behind the scenes. I actually declined to be part of that process because it was too much of a commitment for me to make. I know, for example, that the next-to-final round judges were up until the early AM reviewing the submissions and that each judge looked at on the order of 20 different submissions per rounds.
Sadly, whenever something like $1M is on the line, there is always going to be someone that is unhappy with the outcome. People will speculate and make claims that are outright false. But please keep an open mind as you read more about this event. There will soon be some official communication on this topic from salesforce. I suspect that the more actual facts you learn about the process, the more reasonable it will seem.
I did not have any part in the hackathon process, but I had and still do have quite a bit of visibility on how the process was run. I can assure you that an enormous amount of effort went into the judging and that it was entirely unbiased. There were literally dozens of the most senior executives and engineers within the company that worked on the earlier judging rounds behind the scenes. I actually declined to be part of that process because it was too much of a commitment for me to make. I know, for example, that the next-to-final round judges were up until the early AM reviewing the submissions and that each judge looked at on the order of 20 different submissions per rounds.
Sadly, whenever something like $1M is on the line, there is always going to be someone that is unhappy with the outcome. People will speculate and make claims that are outright false. But please keep an open mind as you read more about this event. There will soon be some official communication on this topic from salesforce. I suspect that the more actual facts you learn about the process, the more reasonable it will seem.
Are you saying that the dozens of people claiming that no one tested their apps are all lying?
Like others have said previously, Salesforce must have forgotten about analytics.
Like others have said previously, Salesforce must have forgotten about analytics.
I am saying that there is a misunderstanding. The earliest rounds of judging looked at the videos of the apps that contestants produced as well as a code review as a sanity check. This was done most likely so that each app could be viewed in the best possible light as well as to help the judges scale to the number of entries.
So, no, the people complaining are not lying but they and anyone else would be incorrect to conclude that their app was not evaluated. Every app was evaluated. All by videos. Some by code review. And a few through every reasonable means.
So, no, the people complaining are not lying but they and anyone else would be incorrect to conclude that their app was not evaluated. Every app was evaluated. All by videos. Some by code review. And a few through every reasonable means.
What about the people who checked the analytics on their videos and determined that they hadn't been viewed?
I honestly haven't seen a single complaint like that. Can you point me to one? FWIW, I am pretty sure that each entry was reviewed at least twice.
This hn comment links to a few places where people complained: https://news.ycombinator.com/item?id=6779036
The link that specifically shows people complaining that nobody viewed their app or looked at their video is this: http://salesforce1million.challengepost.com/forum_topics/264...
The link that specifically shows people complaining that nobody viewed their app or looked at their video is this: http://salesforce1million.challengepost.com/forum_topics/264...
Your first link doesn't contain a single complaint claiming that people's videos were not viewed. It does, however, link to the second link, and the other two links it contains make no references to this sort of complaint.
The second link has exactly one person clearly stating that they've received zero launches and zero video plays. There is a second person that states "Yeah, same here. We had a workflow setup to send us notifications if any new used the app and received none." Notice that he/she doesn't mention video plays.
So, in total, we have exactly one person that says that they've received zero launches and zero video plays. If I've unfairly characterized the data behind your links, please let me know. But that's all that I can find.
In trying to find out if they have a valid complaint, I've discovered that the person behind that complaint is responsible for a significant portion of all of the discussions and complaints at the challenge site.
One thing is clear: their submission is in the same area as the winning entry. In their shoes, I might be disappointed for that reason alone. But for the sake of transparency, here are the two entries:
The winning entry (which is admittedly being evaluated to see if they conformed to all rules):
The second link has exactly one person clearly stating that they've received zero launches and zero video plays. There is a second person that states "Yeah, same here. We had a workflow setup to send us notifications if any new used the app and received none." Notice that he/she doesn't mention video plays.
So, in total, we have exactly one person that says that they've received zero launches and zero video plays. If I've unfairly characterized the data behind your links, please let me know. But that's all that I can find.
In trying to find out if they have a valid complaint, I've discovered that the person behind that complaint is responsible for a significant portion of all of the discussions and complaints at the challenge site.
One thing is clear: their submission is in the same area as the winning entry. In their shoes, I might be disappointed for that reason alone. But for the sake of transparency, here are the two entries:
The winning entry (which is admittedly being evaluated to see if they conformed to all rules):
http://salesforce1million.challengepost.com/submissions/18552-healthcare-love
http://vimeo.com/79921465
unfortunately, that was the cut of the video that i made prior to seeing the entry form. the actual video that was submitted to the office guideline is here:
https://vimeo.com/79931458
the only views were me. i included the other video after i spoke with a salesforce person about the issue. and they had directed me how to enter the links. that full one is the one that's getting hits because i like it more. it was obviously labeled as not the official video with directions to the official video in case it was accidentally seen.
here's the big problem though. this contest was supposed to validate real applications. healthcare.love is nice, but it's a hollywood set. it doesn't wire into that much. it makes the erroneous mistake of targeting the new healthcare marketplace with an application for expensive iphones? in a demo state that is already covered by covered california? not the government exchange that it's adorable name harkens? i actually wrote simulated systems to pull member plans off the db and fake authorize them and submit them back into the system so that notifications could travel back to the app. oh, and instead of text i included twilio for VOIP and data coordination between the client, not assuming the target individuals would be text savvy youngsters. oh, that, and having to pull of the 85,000 healthcare plans that actually come from the healthcare.gov open marketplace. not just a single json list. so in the end, yah, i guess maybe i should have just focused on the icons? didn't think that was the point.
i held back until now, but looks like the makers of that app are also employees of a salesforce invested company. so they are DQ'd. this is all the kind of thing an open judging with actual use or connection with the entrants could have simply identified. but it all happens in the dark.
https://vimeo.com/79931458
the only views were me. i included the other video after i spoke with a salesforce person about the issue. and they had directed me how to enter the links. that full one is the one that's getting hits because i like it more. it was obviously labeled as not the official video with directions to the official video in case it was accidentally seen.
here's the big problem though. this contest was supposed to validate real applications. healthcare.love is nice, but it's a hollywood set. it doesn't wire into that much. it makes the erroneous mistake of targeting the new healthcare marketplace with an application for expensive iphones? in a demo state that is already covered by covered california? not the government exchange that it's adorable name harkens? i actually wrote simulated systems to pull member plans off the db and fake authorize them and submit them back into the system so that notifications could travel back to the app. oh, and instead of text i included twilio for VOIP and data coordination between the client, not assuming the target individuals would be text savvy youngsters. oh, that, and having to pull of the 85,000 healthcare plans that actually come from the healthcare.gov open marketplace. not just a single json list. so in the end, yah, i guess maybe i should have just focused on the icons? didn't think that was the point.
i held back until now, but looks like the makers of that app are also employees of a salesforce invested company. so they are DQ'd. this is all the kind of thing an open judging with actual use or connection with the entrants could have simply identified. but it all happens in the dark.
Thanks for giving more feedback. Would you be kind to respond to the other child of your OP pointing out that the winner had been working and actually demoed their app way before the starting period set out in the rules? Thanks!
No problem. Will do in a second...
gwf. if you, and salesforce, really believe it was an improbable series of poor decisions and not malicious intent, tell your bosses it is time to build the redemption story. and we can speak about how to reboot this together, the company and the contestants.
And btw, the system won't operate now if anyone actually tries to test it. Too many dependent systems doing things that I spun down.
Tell Salesforce to hold an open judging and I'll spin them back up. And EVERYONE should get a chance to show off their applications.
Tell Salesforce to hold an open judging and I'll spin them back up. And EVERYONE should get a chance to show off their applications.
actually, you could do us all a favor. ask salesforce to open up the entry gallery. we've been trying to construct that ourselves. but some people are scattered to the wind. and others have relationships with salesforce and don't want them jeopardized by being an active voice.
people did hard work. and i'm not just advocating for myself. it is appauling that the work went into the circular file. and what they would have been judged on is the marketing brochure.
people did hard work. and i'm not just advocating for myself. it is appauling that the work went into the circular file. and what they would have been judged on is the marketing brochure.
so here's the thing. when you lure people into such a large development period with this statement:
But it’s not going to be easy—$1 million is going to bring out the best of the best. So don’t wait until Dreamforce, you’re going to want to get started now.
and you only have 150 entries, you owe everyone a run. video review becomes inexcusable. people flew in with weeks of work and teams of people. and you reviewed their video? was this supposed reward video craftsmanship? the last thing on many people's minds when you intentionally led them to believe it was the work that counted?
But it’s not going to be easy—$1 million is going to bring out the best of the best. So don’t wait until Dreamforce, you’re going to want to get started now.
and you only have 150 entries, you owe everyone a run. video review becomes inexcusable. people flew in with weeks of work and teams of people. and you reviewed their video? was this supposed reward video craftsmanship? the last thing on many people's minds when you intentionally led them to believe it was the work that counted?
What about the allegations that the winning team demoed their app weeks before the hackathon even started?
I honestly can't speak to the specific allegations because I am not in a position to know the facts one way or another. But I can tell you three things for certain.
First, salesforce is taking this very seriously and there is an ongoing discussion at the highest ranks to evaluate the complaints. Literally, the entire executive team is watching this.
Second, there were close to a hundred people that were internally part of the judging process. Each of these judges looked at a dozen or two entries. I could see little opportunity or motivation for collusion.
Finally, salesforce is one of the most self-critical companies that I've ever been a part of when it comes to evaluating ourselves in terms of trust and integrity. Nearly every internal communication begins and ends with a statement around how trust is our most important value. And while some our doubting our credibility in this moment, there's a reason why so many large and small companies trust us to hold their data; it's because we never compromise on any issue of integrity and if we make a mistake we investigate it and own up to it.
First, salesforce is taking this very seriously and there is an ongoing discussion at the highest ranks to evaluate the complaints. Literally, the entire executive team is watching this.
Second, there were close to a hundred people that were internally part of the judging process. Each of these judges looked at a dozen or two entries. I could see little opportunity or motivation for collusion.
Finally, salesforce is one of the most self-critical companies that I've ever been a part of when it comes to evaluating ourselves in terms of trust and integrity. Nearly every internal communication begins and ends with a statement around how trust is our most important value. And while some our doubting our credibility in this moment, there's a reason why so many large and small companies trust us to hold their data; it's because we never compromise on any issue of integrity and if we make a mistake we investigate it and own up to it.
2 of the five finalists now require disqualification. people are looking into the other 3. how thorough could that process really have been?
[deleted]
Hah Salesforce - you were better off having no hackathon at all. This is disgusting. Preselecting winners possibly?, not looking at everyone's app / video, selecting a winner that started way before hackathon was announced. yep total mess.
If developers are incentives by the possibility of winning $1m, they would start working months in advance. That's why it's not ok to offer $1m as a hackathon prize, as it's not in the spirit of hacking.
The prospect of a $1M prize wasn't announced until Oct 25th: http://thenextweb.com/insider/2013/10/25/salesforce-offering...
That's kind of nitpicky. My point was that if there is a hackathon with a highly desirable prize, and you know about it in advance, there is a direct incentive for you to spend time between the time you learn of the hack and the time when the hack officially begins working on ideas, designs, business strategy, and even code, and that that effect is the reason that it's not a good idea, or in the spirit of hacking, to offer high stakes prizes in hackathon style situations.
Apologies for the very long sentence, I love commas and am the arch-nemesis of conciseness.
Apologies for the very long sentence, I love commas and am the arch-nemesis of conciseness.
I think you could offer 1million but it has to be a competition where the teams that enter solve a single problem. That way accurate scoring metrics can be developed and used.
It doesn't make sense to say "Hackathon, 1 million dollars, Go!".
edit: To add. The problem should be announced the day of!
It doesn't make sense to say "Hackathon, 1 million dollars, Go!".
edit: To add. The problem should be announced the day of!
I guess that it's better to allow much more time for bigger prizes. Like the Netflix prize. That was a great mix of marketing and feeding the recommender system.
This is a general problem for hackathons, the idea of limiting something like development is impossible.
Even if you don't code, you could plan the whole concept, user your super familiar framework and there is a huge difference to the "spirit" of just developing the concept on the spot.
I don't want to say I dislike hackathons in general, but not for huge prices. Just meeting interesting people and trying to build something together is the really valuable part.
Even if you don't code, you could plan the whole concept, user your super familiar framework and there is a huge difference to the "spirit" of just developing the concept on the spot.
I don't want to say I dislike hackathons in general, but not for huge prices. Just meeting interesting people and trying to build something together is the really valuable part.
There are a number of ways to combat this.
- Have a theme, a dataset, or some other component that is not announced until the day it starts (and to be eligible an entry must demonstrate meaningful incorporation of that component).
- Have a manageable organizer-to-participant ratio so they can do a decent job of verifying that teams aren't using a developed concept.
- Don't allow premade teams, have individuals pitch concepts and form teams.
Each of those can be gamed to some degree, but they can go a long way, and to game them requires serious intent to deceive.
They didn't even try, at all. The winner was openly a fully formed startup that demoed their app at a Salesforce meetup weeks before the hackathon. That's just, mind-blowing.
- Have a theme, a dataset, or some other component that is not announced until the day it starts (and to be eligible an entry must demonstrate meaningful incorporation of that component).
- Have a manageable organizer-to-participant ratio so they can do a decent job of verifying that teams aren't using a developed concept.
- Don't allow premade teams, have individuals pitch concepts and form teams.
Each of those can be gamed to some degree, but they can go a long way, and to game them requires serious intent to deceive.
They didn't even try, at all. The winner was openly a fully formed startup that demoed their app at a Salesforce meetup weeks before the hackathon. That's just, mind-blowing.
There's another good way to eliminate these concerns: don't have big prizes.
When I hosted a hackathon last summer, I put no limits on projects but the max value of the prizes was $25. With prizes so small, there's no incentive to game the system nor is it grossly unfair to let people work on something they'd already started on.
The goal was purely to work on something cool for a day with like-minded people. One guy brought on a voice-controlled Nest lamp he'd been working on, another built an Arduino step-counter, a third took a crack at writing a basic chess AI.
We provided pizza and beer and just let everyone loose. It went great and there were no hard feelings.
When I hosted a hackathon last summer, I put no limits on projects but the max value of the prizes was $25. With prizes so small, there's no incentive to game the system nor is it grossly unfair to let people work on something they'd already started on.
The goal was purely to work on something cool for a day with like-minded people. One guy brought on a voice-controlled Nest lamp he'd been working on, another built an Arduino step-counter, a third took a crack at writing a basic chess AI.
We provided pizza and beer and just let everyone loose. It went great and there were no hard feelings.
I participated also, and enjoyed the hackathon itself. I wasn't expecting to win. I expected the winner to be someone that had a suitable product that was ready to launch around the same time as the hackathon was announced, and decided to seize the moment for some PR. This is more or less what happened. I think what is leaving a bad taste is that I didn't expect that it would done so brazenly.
The problems in my mind aren't about unfair judging (all judging is biased), but rather that:
* The winning company had clearly been started long before the permitted start date.
* It took less than an hour for this to be 'proved'; Salesforce obviously can't check compliance on every team, but not double-checking the five finalists seems foolish.
* The smoking gun was a demo Upshot gave at a Salesforce meetup. One of the two company founders had worked at Salesforce for 9 years until earlier this year. There's no way any sensible verification could have missed that they'd been working on this for a while. Asking a few of his former colleagues, for example, would likely have revealed the truth.
* Salesforce haven't yet acted to remedy this; which makes them appear complicit in it. I attribute their behavior to incompetence rather than malice. If it was a total fix from the start, there's no reason to exclude projects started before the hackathon in the rules.
The Upshot product itself looks fairly good, so I feel sorry for them in that they could do even better financially if they hadn't won. But not that sorry, given they have a million dollars cash-in-hand :-)
To be fair to Salesforce, their million dollar prize seems to have backfired rather spectacularly with the very audience they were trying to win over. They are probably more annoyed than anyone, and are probably having to tread very carefully for legal reasons. Even if they reclaimed the prize and gave it to #2, it seems that many of the top 5 may also not be entirely above suspicion, and they would want to be doubly-sure of those entries. They'd also presumably have to figure out who was #6 and give them the (new) 5th place prize.
All in all, a major mess.
The problems in my mind aren't about unfair judging (all judging is biased), but rather that:
* The winning company had clearly been started long before the permitted start date.
* It took less than an hour for this to be 'proved'; Salesforce obviously can't check compliance on every team, but not double-checking the five finalists seems foolish.
* The smoking gun was a demo Upshot gave at a Salesforce meetup. One of the two company founders had worked at Salesforce for 9 years until earlier this year. There's no way any sensible verification could have missed that they'd been working on this for a while. Asking a few of his former colleagues, for example, would likely have revealed the truth.
* Salesforce haven't yet acted to remedy this; which makes them appear complicit in it. I attribute their behavior to incompetence rather than malice. If it was a total fix from the start, there's no reason to exclude projects started before the hackathon in the rules.
The Upshot product itself looks fairly good, so I feel sorry for them in that they could do even better financially if they hadn't won. But not that sorry, given they have a million dollars cash-in-hand :-)
To be fair to Salesforce, their million dollar prize seems to have backfired rather spectacularly with the very audience they were trying to win over. They are probably more annoyed than anyone, and are probably having to tread very carefully for legal reasons. Even if they reclaimed the prize and gave it to #2, it seems that many of the top 5 may also not be entirely above suspicion, and they would want to be doubly-sure of those entries. They'd also presumably have to figure out who was #6 and give them the (new) 5th place prize.
All in all, a major mess.
hackathons should go back to what they used to be: great events to try out new technology, and meet up with fellow engineers while having fun
Where are the winners? I am sure they are on HN. Why are they so silent on this matter?
Uh... Not everyone is on HN. It's presumptuous to assume that because someone hasn't posted here, they're being "silent" on the matter.
I will be fair and yes not everyone. But they are the winners and we are poking at this matter for a day and I would be surprised they haven't checked HN (I am sure their friends are telling them about what readers think of them and Salesforce).
... That's still presumptuous of you. For example, I don't check Reddit. Ever. Is it so hard to believe that some other people who no doubt have something better to do with their time don't check HN? [1]
[1] Rhetorical question.
[1] Rhetorical question.
Sorry, I beg to differ in this story. Are you saying these two engineers have no idea what the hell is going on and that they have not been told about what is being discussed here? These guys are software engineers and they sure have friends and co-workers who are HN readers. And they being the main party involved you think they haven't visited the site is one big fat red presumptuous of you in your own word.
If I accuse you being a cheater you sure won't check HN. I am pretty sure.
If I accuse you being a cheater you sure won't check HN. I am pretty sure.
This is why I don't attend hackathons anymore. At the bigger events, there's always bound to be controversy, cheaters, questionable criteria, etc.
Perhaps you are attending hackathons for the wrong reason? I didn't personally attend, but I'm sure anyone who went in order to build an interesting indea with smart hackers probably had a decent time.
Allegations of cheating are serious, but they shouldn't affect your enjoyment of the event unless your main purpose was the win the prize.
Allegations of cheating are serious, but they shouldn't affect your enjoyment of the event unless your main purpose was the win the prize.
For the wrong reason? What's with all these smartie pants on hackernews? A big hackathon always has some sort of incentive, and aiming for that is wrong?
I don't know about you, but I enjoy winning, and I've won hackathons before. If I just wanted to build something with my hacker friends, I can do it any weekend I want.
I don't know about you, but I enjoy winning, and I've won hackathons before. If I just wanted to build something with my hacker friends, I can do it any weekend I want.
My first reaction to this was "it's too good to be true". Now I won't wonder if it wasn't true
the article is stuffed with salesforce.com backlinks, including a nice salesforce profile at the bottom.
any publicity is good publicity.
any publicity is good publicity.
The summary of the company is the best part.
With more than 100,000 customers, salesforce.com is the enterprise cloud computing company that is leading the shift to the social enterprise. Social enterprises leverage social, mobile and open cloud technologies to put customers at the heart of their business.
I don't know what this means.
With more than 100,000 customers, salesforce.com is the enterprise cloud computing company that is leading the shift to the social enterprise. Social enterprises leverage social, mobile and open cloud technologies to put customers at the heart of their business.
I don't know what this means.
It means some obnoxious twit telling you to start using Chatter instead of email, and spend half your day using one of the worst UI's known to man instead of getting work done.
Marc Benioff has no credibility
[deleted]
It was a good hackathon.
I am happy for the winners.
The truth of the matter is, there were so many entries, that it was a complete lottery who ended up winning. I personally thought our app was great, and had a decent shot at winning, but I went to bed assuming 99% odds we wouldn't get picked. With that many entries the odds were not that great.
As lame as this sounds, I've learned the way to really win at Hackathons is to build something you wanted to build anyway. That way, if you win, it's a nice bonus, but you'll come out ahead no matter what.
The judging processes in these things is always suboptimal. I knew it was going to be a nightmare to try and narrow down that many entries to 5 in just a few hours. They were very optimistic in thinking they could pull that off. I think ChallengePost has a great opportunity to improve here by innovating so that if you enter a Hackathon, you get direct feedback from the judges (either quantitative numerical feedback or qualitative comments), so you get some acknowledgement for the hard work put in. The judging was indeed very light--as another data point I don't think we got a single hit on our app-- but I thought they tried to do something unprecedented and did a decent enough job and can learn going forward.
It was a very cool event, I'm happy for the winners, and hopefully some of these Hackathon software tools like ChallengePost and HackerLeague can continue to get better and help alleviate some of these transparency concerns for future Hackathons.
I would definitely recommend participating in future Salesforce Hackathons.