Show HN: Hacker News Comments(hncomments.nathancahill.com)
hncomments.nathancahill.com
Show HN: Hacker News Comments
http://hncomments.nathancahill.com/
72 comments
If that ever changes and HN is overrun (insert dramatic and sad Hans Zimmer score) what about starting a new HN just for programmers? I wouldn't be able to get in at the moment, but someday I'd hope to be able to.[1]
[1] I'm assuming you'd have some kind of rotating or individualized Project Euler style problems that you have to solve to get in, and that they would be pretty tough. It occurs to me that you could also have temporary student accounts for people who are just getting into the software world as well though.
[1] I'm assuming you'd have some kind of rotating or individualized Project Euler style problems that you have to solve to get in, and that they would be pretty tough. It occurs to me that you could also have temporary student accounts for people who are just getting into the software world as well though.
Passing a programming test is no guarantee for having anything interesting to contribute to an online community, just look at your company's internal IT mailing lists.
Totally! That's why I'm not suggesting anyone do it now. There's a place for kludgy solutions, but not until you've tried everything else.
An invitation based system, while not perfect, would probably be the most effective method. Plus, the owner of the site could restrict or encourage growth by the amount of invitations allocated... allowing for easy management of unwanted lulls or spikes in user activity.
Invitation based communities are also very self-policing. If you invite someone and they happen to be an idiot then there is egg on your face as well.
Invitation based communities are also very self-policing. If you invite someone and they happen to be an idiot then there is egg on your face as well.
Ideas were floating around for this: http://i.imgur.com/BuF7d.png
Clever. It's like CAPTCHA for programmers. It wouldn't be much more difficult than the current registration process.
By "get in" do you mean to comment or to simply read the comments? I'm all for a intelligence test (I personally would like a "actually RTFA test"), but I'm pretty sure that making that test required to even read the comments is a terrible idea.
I meant to comment. Though of the three strategies mentioned here (obscurity, invitation-only, and skill tested) I definitely prefer the first two.
could try what metafilter does. It seems like a nice community.
Iirc there's a certain site that requires you to solve a differential equation as captcha. The name escapes my mind.
One of the comics I read requires you to solve a simple calculus problem before you can comment, as a pseudo-captcha.
http://spikedmath.com/
http://spikedmath.com/
That's cool. By the way, I got mine as well. It's the Quantum bit service.
Link - http://random.irb.hr/signup.php
Link - http://random.irb.hr/signup.php
This has already happened. Someone should start something new, ideally a smaller, more focused crowd.
Scraping comments on HN hits the robots.txt wall rather quickly.
At a maximum 2 scrapes per minute, your cache will be 60 minutes old when you're watching 120 (single-page) comment threads. That might be tolerable. Now try 1200 or 12,000 comment threads.
You can do smart cache refreshes, but you will miss bursty activity. You can ignore articles older than X days, but you'll miss the "dupe, comments here:" resurgences.
HN gets about 150 posts per day that hit the top 60, at least briefly, as a point of reference.
This is a neat thing, but if it gets popular you will have a problem. On the plus side, it's self-limiting. As the usage gets higher, the experience will degrade and usage will decline. No intervention required by you. :)
I've wished for a policy-compliant way to cache comments in a timely manner, but there's just no way to stay current under the existing rules.
At a maximum 2 scrapes per minute, your cache will be 60 minutes old when you're watching 120 (single-page) comment threads. That might be tolerable. Now try 1200 or 12,000 comment threads.
You can do smart cache refreshes, but you will miss bursty activity. You can ignore articles older than X days, but you'll miss the "dupe, comments here:" resurgences.
HN gets about 150 posts per day that hit the top 60, at least briefly, as a point of reference.
This is a neat thing, but if it gets popular you will have a problem. On the plus side, it's self-limiting. As the usage gets higher, the experience will degrade and usage will decline. No intervention required by you. :)
I've wished for a policy-compliant way to cache comments in a timely manner, but there's just no way to stay current under the existing rules.
What if you're scraping with ajax? Wouldn't each individual user's IP take the hit and not the domain's IP?
You can't scrape with AJAX because of cross domain security restrictions.
One potential solution to obey robots.txt might be to spawn multiple small EC2 instances with different IP's and have them coordinate with each other to share the crawling without individually running over the limits. (This is also useful for scraping from sites that have rate limits)
One potential solution to obey robots.txt might be to spawn multiple small EC2 instances with different IP's and have them coordinate with each other to share the crawling without individually running over the limits. (This is also useful for scraping from sites that have rate limits)
robots.txt doesn't enforce itself so there is no IP limitation; this is still a violation and no better than simply lowering the delay on a single scraper.
The ajax request looks to be getting proxied through this guy's server. You have to do something like that because of cross domain ajax issues
Perhaps an iframe would work best. With some injected styles and javascript (some that would make the iframe expand automatically à la Disqus), I think this would work nicely.
I'd definitely use this on my blog (http://sachagreif.com) if it was officially supported (or at least tolerated) by HN somehow.
And the fact that people can only comment if they have an HN account makes it even better!
And the fact that people can only comment if they have an HN account makes it even better!
> Although many websites and blogs have a comment system, the quality of the discussion on Hacker News tends to be much better.
Wouldn't this risk changing that?
Wouldn't this risk changing that?
I would see that eventuating if this allowed commenting via the embedding blog, thankfully it doesn't. I don't have the same issue with this that others do, but I remember TheNextWeb getting a bit of slack from commenters for embedding HN comments (although that was mixed amongst other comment platforms, and is a little different)
API-wise, http://api.ihackernews.com/ , http://hndroidapi.appspot.com/ or http://www.hnsearch.com/api didn't suit? [edit] what am I thinking, an API doesn't need an API...
Oh, and simplicity of this seems pretty nice!
API-wise, http://api.ihackernews.com/ , http://hndroidapi.appspot.com/ or http://www.hnsearch.com/api didn't suit? [edit] what am I thinking, an API doesn't need an API...
Oh, and simplicity of this seems pretty nice!
Probably not, unless your blog has an organic audience that is bigger (or at least comparable) than that of HN homepage...
Blogs this big are unlikely to implement a third-party (and unsupported) discussion feed.
In my preferred model of Internet content, stuff like this would be available for every element without scraping.
Good job.
Good job.
The word for that is http://en.wikipedia.org/wiki/Transclusion , and it's interesting that it was a core component of Xanadu (see the Wikipedia link there).
Yeah. I know.
We also need a micropayment processor that can do sub-penny transactions.
EDIT: Actually, I would be fine with penny transactions considering that the current state of things with say, paypal; is 30+ cents per transaction.
We also need a micropayment processor that can do sub-penny transactions.
EDIT: Actually, I would be fine with penny transactions considering that the current state of things with say, paypal; is 30+ cents per transaction.
How hard would it be to have your script look for the submission on HN by url instead of having to manually put in the id? Maybe it could use the one with the highest score if there's more than one?
Yes, that's the plan! A new API endpoint will use the referring page to capture the URL, then link it with the appropriate discussion on HN.
Great contribution Nathan, looking forward to using this to embed comments after my next blog post. Can you let me know how the comment system works? Are you scraping HN?
Thanks! I'm respectfully scraping. I've rate limited all calls to YC's servers and am caching all comments on my side.
Seems respectable. Out of curiosity, why did none of the existing 3rd party API's suit your purpose?
Probably the same reason no one should you use this to power their comment section there is no revenue, and no real reason to keep it up. Things like this come and go.
This would make more sense if it was entirely client-side code towards an official JSONP endpoint, such as the official hacker news API.
Not saying this is necessarily true for Hacker News, but unfortunately a lot of sites don't have the time/resources to develop and then maintain a public API for the majority of their content. And mostly see an RSS or XML feed of their news/articles/stories as good enough.
Do you think people are going to wait for someone to submit each post to HN? I think this will encourage writers to submit every single story they write to HN, just to get a comments page.
Even if someone put every single story they write on HN, that's not a guarantee that it will get comments, since it might not get up-voted.
I think if I were to implement this on my own page, I would probably have two tabs that expand if clicked. Something like this:
Show [your site's name] comments | Show HN comments
And then for the HN comment tab, you would use the OP's awesome submission. Thanks, btw!
I think if I were to implement this on my own page, I would probably have two tabs that expand if clicked. Something like this:
Show [your site's name] comments | Show HN comments
And then for the HN comment tab, you would use the OP's awesome submission. Thanks, btw!
That's a good question, but it's probably a very small percentage. And they can always add it later.
Here's a question, is it possible to put a Hacker News upvote button on your blog post? Or is that against the HN rules, or something that's frowned upon in the community?
I created hnlike.com about a year ago that helps you to setup an upvote button for your blog. I have disabled the upvote feature as the community seemed worried. See the HN discussion on my story for details.
I'm not supporting upvoting or posting comments. I like HN how it is now (or more so a year ago).
I hope this doesn't further impact the quality of discussion on hacker news. It seems that every day I notice more and more people posting completely non-contributing crap in the comments that you'd expect from Reddit or something (one-liners, stupid jokes, etc). I go to Reddit for that, I come here for insight (which, granted, you can find a lot on Reddit also, but HN is supposed to be a bastion of intellectual thought and discussion).
This idea is kind of alluring. But you can't really include discussions on your page that might be happening all over the Internet. Reddit etc.
This is where the browser should really be smart enough to do a search engine lookup, and propose smart links alongside the page you're reading. Or even summarise and merge discussions found elsewhere.
This is where the browser should really be smart enough to do a search engine lookup, and propose smart links alongside the page you're reading. Or even summarise and merge discussions found elsewhere.
I'm going off topic a little but...
An element I really like from random blog posts I've seen off of HN is using the sidebars for additional, short commentary/details. Your comment reminded me of that, but sadly this might not work well if an article were all chopped up with tons of commentary (I'm thinking of used books where 'important' parts are highlighted and comments put beside them by a random reader, which I dislike). Well, then again, perhaps for people who like to dive into subjects and really digest an article, there could be a straight text version and a feature to see it marked up w/ smart commentary.
This also reminds me of a small 'movement' a year or two ago for citing in-article links at the bottom instead of linking to them throughout the article itself.
An element I really like from random blog posts I've seen off of HN is using the sidebars for additional, short commentary/details. Your comment reminded me of that, but sadly this might not work well if an article were all chopped up with tons of commentary (I'm thinking of used books where 'important' parts are highlighted and comments put beside them by a random reader, which I dislike). Well, then again, perhaps for people who like to dive into subjects and really digest an article, there could be a straight text version and a feature to see it marked up w/ smart commentary.
This also reminds me of a small 'movement' a year or two ago for citing in-article links at the bottom instead of linking to them throughout the article itself.
Does it really matter where the links are listed - in the article or at the end?
Opera has a side panel, one of which is the Links panel. That pulls links out of the page and lists them alongside. But this alone isn't that much use. The browser however could do something more interesting with them.
I think a lot of pages seem dreadfully wasteful by placing in the same old accessory content on their pages. Buttonitis etc. I don't think this necessarily should live 'inside' the 'page' at all. The content should remain dumb. The inbound links, and chat around the content augments it. And this could be summised somehow by browser tools / other services.
I understand the author's want for embedding HN comments on a page. There's a simple and elegant solution to the problem. Simply open up another browser window/tab and point it to the relevant HN comment page. To do so, the author could include a simple link in the page, leave it to the browser to infer, or just let the user discover it for themselves (pointers do help though.)
Why the obsession of stuffing our pages with content?
It also feels a little snooty doing something like this (embedding HN comments on your page that is.)
Opera has a side panel, one of which is the Links panel. That pulls links out of the page and lists them alongside. But this alone isn't that much use. The browser however could do something more interesting with them.
I think a lot of pages seem dreadfully wasteful by placing in the same old accessory content on their pages. Buttonitis etc. I don't think this necessarily should live 'inside' the 'page' at all. The content should remain dumb. The inbound links, and chat around the content augments it. And this could be summised somehow by browser tools / other services.
I understand the author's want for embedding HN comments on a page. There's a simple and elegant solution to the problem. Simply open up another browser window/tab and point it to the relevant HN comment page. To do so, the author could include a simple link in the page, leave it to the browser to infer, or just let the user discover it for themselves (pointers do help though.)
Why the obsession of stuffing our pages with content?
It also feels a little snooty doing something like this (embedding HN comments on your page that is.)
> but sadly this might not work well if an article were all chopped up with tons of commentary
I think this should be left to the domain of the browser. Create a tool for filtering and showing certain aspects.
I think this should be left to the domain of the browser. Create a tool for filtering and showing certain aspects.
Why isn't this open source?
Why would you want open source for this? Do you want to spend the time to deploy your own instance just to have the exact same result?
To learn from it; so that more than one person can work on improving it; just in case I don't trust the submitter to serve code to my visitors.
Yeah, that makes sense. What would give you the confidence necessary to have someone else/Nathan Cahill host the server?
yes i would. to implement the whole thing, instead of only this little jquery snippet shouldn't take much longer. but then i'm not dependent on someone elses server being up/fast.
If everyone could implement their own scrapper they would be able to follow robots.txt guidelines/limits, but HN servers might take a big blow it it gets popular.
That makes sense. Maybe it will go open source in the future. It doesn't look like Nathan has a revenue model for this so he may be open to that.
to counter myself... i've recently seen a couple of blogs move to github pages, where something like this would be quite handy
You'd hope it was optimally done client side, and then hosting would be moot.
Wow, I really like this, great work!
I've previously thought that comment threads on blogs tend to die down fairly quickly, whereas discussions on HN tend to last a bit longer, so this is doubly cool.
I guess the only downside is the near-impossibility of letting people comment directly from the page without having to log in again.
I've previously thought that comment threads on blogs tend to die down fairly quickly, whereas discussions on HN tend to last a bit longer, so this is doubly cool.
I guess the only downside is the near-impossibility of letting people comment directly from the page without having to log in again.
Not a downside. I'm not allowing posting comments or upvotes through this system to maintain the integrity of the discussion on HN.
Just a heads up, I tried to test out your inline "Give it a try below!" but it's not working right now. In the browser console: "GET http://hncomments.nathancahill.com/comments/4285019/html/ 500 (Internal Server Error)"
Thanks, there were a couple hiccups handling the large amount of traffic. Should be in working order now.
Nice work, this is pretty cool! I've been working on a similar approach but using Github (mainly because I host my blog there using Jekyll). I'll try and get it released tonight :)
Very, very interesting! So you already calculated that the cost of this will be insignificant for you even with millions of pageviews? Blogs have a long tail of visits... ;)
How do people opt-in to allow their comments to be shared on sites other than HN? Or is this strictly opt-out and, if so, what is the opt-out mechanism?
[deleted]
This is very interesting! Great work.
Probably because your random internet stranger has never heard of HN.
It's a shame, for instance, that I was ever allowed to find this place.