True, but also need to know the basics well of what constitutes good code and how it should scale vs just working code. Too many people relying on LLMs to produce stuff which just about works but give users a terrible experience as it bearly works.
Nice concept. I've definitely seen this play out in practice.
A lot of sites aren't impossible to scrape, but they're steadily getting more expensive. We're having to lean more on residential proxies, headless browsers etc just to get the same data that used to be straightforward...
Instead of focusing solely on the "market boom" and technological advancements, we should be having a serious conversation about the societal implications and the long-term sustainability of this scraping free-for-all.
The "higher stakes" mentioned aren't just about cost and effort; they're about the potential erosion of online privacy, the destabilization of websites reliant on ad revenue, and the ethical responsibility of data practitioners.
It would be interesting to explore ways of profit sharing to cover loss of ad revenue, better caching to reduce the cost on websites themselves, and simple ways to opt in or opt out of personal information being used online!
Looks great! I've had the same issue as you with many projects - so many people leave the marketing till the end so don't beat yourself up about it now. (Hopefully you will have learned a lesson haha!)
Lets look at what you could do:
- Start in one local area - ideally one near you or one that you know. Make it work there and then expand.
- See if you can engage any local groups which might engage with the idea - I could see local city/town facebook groups liking this. Or maybe on nextdoor.com.
- Check out the book "The cold start problem" by Andrew Chen. He goes into many examples of lots of companies that had this issue and how they over came the issue and got their first customers - http://andrewchen.com/wp-content/uploads/2022/01/ColdStartPr...
Meetup really could be a 10x better platform.
As someone who's run a group on it having to pay to then put in the work to run a group on top of that is just crazy.
They could have monetised the platform in so many other different ways which could have helped the community instead of just going for the easy pickings of charging the event organisers.
Why not charge a percentage on top of any payments take?
Create/integrate with a room booking system in cities to help organisers book? (and take a fee on this)
The only thing Meetup does well is discovery - the domain name is amazing for the type of site it is and it is easy enough to search for groups in your city and find what you want. Everything else is pretty terrible.
I don't know how you could justify having more than 3 dev's working on this site the way it is at the moment. It just screams lazy...
Travel between countries in Europe is improving every year too. Next week I'll be traveling from Bordeaux to Berlin (over 1,600km) - it's faster than the car (16 hours by car vs 12 hours by train), and cheaper than flying, in the summer at least (150 euro by train, vs 300 euro by plane - booking 6 weeks before).
That will improve next year too with the direct Paris Berlin train that should only take 7 hours.
I doubt many website owners feel they have an ethical responsibility to provide public APIs to their data so people wouldn't scrape their content. It is up to them what they want to do with it.
I just feel like the whole thing is a chicken and egg problem - both sides could be nicer to the other but no one wants to go first!
I do like the idea of identifying web scrapers in the user agent but I'm not sure how many websites would use it...
What does one get for actually being nice and being an "ethical web scraper"?
I don't see many people bothering to be honest, developers who are out there scraping are mostly just doing a job, they want to be done as quick as possible and most don't care about things like this unless there are repercussions
My only thought on their refunding process is that maybe they don't have a simple way to just click one button to refund everyone for that specific campaign?
We should probably not assume bad intent on their part!
HN also has moderators who can step in when things look like they're getting out of control which really helps. Or in some cases just point people to the rules and tell them to stop causing flamewars or get out.
It really makes a difference to the overall discourse. Something that is missing on most other places on the web!
I think it's interesting how they justify giving 1 million of the donation proceeds to the Organisers but then decide that the rest just doesn't make the cut.
It would be nice if they provided some evidence showing the organisers having anything to do with - "promotion of violence and harassment".
With any of these large scale protests there will be some bad eggs who appear to cause havoc...
Madness that they haven't gotten back to your access request in 6 months!
Why even bother having the API there - so much value can be added by people building on top of YouTube and other large sites, its a shame that most of these large sites do nothing to provide API access and people have to go out of their way to scrape them them...
That looks like a great resource! How often is the data set "updated"?
I'd imagine most people's use cases need data which can change from day to day or week to week but I do think that this is fantastic if I was to have a project which was looking at data across a longer timeframe.