There is actually a thriving industry in data collection and these range from companies that provide tools for you to do your own data collection and those who will collect the data for you (or provide pre-made datasets). Full disclosure, I work for one such: Scrapinghub where we provide tools, platform, and data services.
I just ran across https://www.opensecrets.org/ and found it quite useful and comprehensive in tracking contributions to candidates.
I live in the US and am privileged with the level of transparency that exists, but it's still not necessarily enough. Similar issues are present with the clunky nature of government websites and databases and so I think we're in agreement that it's not even close to the potential of what it could be.
So for Scrapy and Portia, they are both free as in beer, specifically because we believe in the power of open source. Scrapy actually precedes Scrapinghub and was certainly not developed as a marketing tool.
Charges come with large scale crawls (above certain limits on our platform), additional products like Crawlera (our smart downloader that routes requests from a crawl through a pool of IP addresses to avoid bans), datasets, and for us to handle complex crawls for companies outsourcing to us.
Our model is that there is something for everyone whether you are looking to dip your toes into web scraping (free), use it occasionally (usually journalists) or dependent on web crawling for your business.
Full disclosure, I work for Scrapinghub. Our tools are Scrapy and Portia, both open source and both free as in beer. Scrapy is for those who want fine-tuned manual control and who have a background in Python. Portia is the visual web scraper for those who are non-technical to technical but don't want to bother with code.
Web scraping is everywhere, even if it's not necessarily spoken openly about or acknowledged. The publicized perception of web scraping is fairly negative, but doesn't take into account the benefits of data used in machine-learning or democratized data extraction (as in the case of this article or for building public service apps like transportation notifications), or the simple realities of competitive pricing and monitoring the activities of resellers.
Researchers, academics, data scientists, marketers, the list goes on for those who use web scraping daily.
Glad you enjoyed the article! I'm hoping that more examples of ethical data extraction will start to turn the tide of public perception.
Full disclosure, I work for Scrapinghub and the web UI you speak of is Portia - our open source visual web scraper. It's for those who range from non-technical to technical but want a quick way to scrape data. I think it's extremely important to develop tools to democratize the acquisition of data regardless of technical background and skill. Glad you find the article and tool interesting!
By overcharging, I mean that we compared prices to a baseline day in January (the 13th, to be exact) and then showed those prices against what was on offer from other retailers. Consumers do have access to this information and their buying decisions are their own. And that's true, this study reinforces the free market system we have. There is something to be said for the shadiness of certain retailers elevating their prices on days that are being touted as "sale" days.
I think the unethical aspect of arbitrary pricing that goes beyond daily fluctuations really came out in this project. I actually was not aware of the situation with JC Penny, but really fascinating glimpse into their consumer mindset. Probably comparable to Sam's Club, since even I expected a ton of lower prices from them based on membership. I like your cultural analysis!
Very much agreed. And this was very much the case on specific days. Prices do fluctuate, but there were definitely products that were continuously inflated over the course of multiple days.
The fascinating thing about this study is that in certain cases, prices went beyond traditional fluctuations to the point of being inflated for multiple days in a row irrespective of time of day and demand.
Thanks for the support! I agree especially that it's not just random price fluctuation, specifically with the Apple TV example listed in the article. Target dropped the price and then systematically kept it at an inflated rate before dropping it partially again. This was not a case of hourly fluctuations according to demand.
I like your extremely accurate title pitch. And no, we're not entitled to low or consistent prices, but the fact that there is inflation on "sale" days is noteworthy because of the air of false advertising. You're absolutely correct though, shoppers should buy when they feel comfortable because buyer's remorse is meaningless when you're happy with the purchase.
and since this is all open source, here's a link to GitHub: https://github.com/scrapinghub/portia2code