News.YC in Polar Coordinates(labs.diffle.com)
labs.diffle.com
News.YC in Polar Coordinates
http://labs.diffle.com/ycpolar/
10 comments
Nice hack nostro. What process did you use to scrape? Been working on a newsyc scraper
- extract & parse with python + beautifulSoup, display with YUI
- parsing the RSS feed for user, story points, story title and date posted
- then grabbing user details via homepage for karma, inception date
- exporting to xml
- skinning with YUI
Will display when ready, heres a screenshot ~ http://flickr.com/photos/bootload/1400863977/
- extract & parse with python + beautifulSoup, display with YUI
- parsing the RSS feed for user, story points, story title and date posted
- then grabbing user details via homepage for karma, inception date
- exporting to xml
- skinning with YUI
Will display when ready, heres a screenshot ~ http://flickr.com/photos/bootload/1400863977/
Regexps. John tried a beautifulSoup parser, but it turned out to be more trouble than it was worth. Insert joke about "now you have two problems", but it works in a pinch, and it really only had to last till we showed off our projects at 8:00 PM ;-).
I don't mind sharing the code if PG doesn't mind the potential flood of screen scrapers. It just handles the front page and comment page for individual articles, and it's pretty compact - only 60 lines including doc comments.
I don't mind sharing the code if PG doesn't mind the potential flood of screen scrapers. It just handles the front page and comment page for individual articles, and it's pretty compact - only 60 lines including doc comments.
"... Regexps. John tried a beautifulSoup parser, but it turned out to be more trouble than it was worth ...'
Took me a bit of mucking around to get it working. My trick was just utilising the FOX HTML Validator to find the page structure, then CUT+PASTE the text into IDLE and call up BS in IDLE, then write the BS expression to oarse the string till I had the right data.
"... I don't mind sharing the code if PG doesn't mind the potential flood of screen scrapers. ..."
A better suggestion might be supply the raw data you collected as a service and let others do what they want with it. Most just want the data and are not particularly interested in the HOW. If they are interested in the HOW then it's better to point them to some scrapers to play with.
Took me a bit of mucking around to get it working. My trick was just utilising the FOX HTML Validator to find the page structure, then CUT+PASTE the text into IDLE and call up BS in IDLE, then write the BS expression to oarse the string till I had the right data.
"... I don't mind sharing the code if PG doesn't mind the potential flood of screen scrapers. ..."
A better suggestion might be supply the raw data you collected as a service and let others do what they want with it. Most just want the data and are not particularly interested in the HOW. If they are interested in the HOW then it's better to point them to some scrapers to play with.
Similar commenters mostly appears to be Paul. Why did you choose that criterion?
Depends where you're looking. pg seems to be common to the south. However, up at the north there are a bunch that are all gwenhyfaer, while to the west is joshwa and Readmore. Which makes sense, since the plot tries to cluster by common commenters along the angular direction.
I chose this criterion mostly so you could see who's participating in other threads similar to the articles you like.
I chose this criterion mostly so you could see who's participating in other threads similar to the articles you like.
This is awesome! You may be able to improve it further by considering other things besides common comment posters to determine the angle measure for a story. I haven't seen this good of a web information graphic since the treemap used at "newsmap" found here: http://www.marumushi.com/apps/newsmap/newsmap.cfm
This is a hot new view.
A lot of the features (animation, selection, title/domain swapping) are basically gratuitous - I wanted to get more practice with JQuery since I need that functionality in my startup. Hell, the whole thing is basically gratuitous. But it was pretty fun to write.
John was working on a scraper for Reddit...that may be a bit more interesting, since it updates more frequently and the larger audience means more posts in common.