Please fix: Unknown or expired link.
19 comments
I'll try to respond to everything in one comment.
kogir and I talked recently about getting rid of "unknown or expired link". It's on our list. There are a lot of things on the list; we don't know yet when we'll get to it.
There's work ongoing to make HN more usable on mobile. This will likely roll out sooner than the above. We intend to roll it out incrementally, starting with the front page. We'll give people lots of notice so anyone running scrapers that are affected (like me!) will have time to change.
The look and feel of HN is unlikely to change much. We can argue about that one till the cows come home. I'm tempted to—my opinions are as strong as many of yours—but that doesn't seem like the best idea right now.
Feature requests should go to https://news.ycombinator.com/item?id=363. Software bugs should go to https://github.com/HackerNews/HN/issues. That would have been the appropriate destination for the OP, and indeed this issue has been posted about there.
Threads like the current one get lots of upvotes and comments, but they've never been on topic for HN. I'm going to bury this one now.
kogir and I talked recently about getting rid of "unknown or expired link". It's on our list. There are a lot of things on the list; we don't know yet when we'll get to it.
There's work ongoing to make HN more usable on mobile. This will likely roll out sooner than the above. We intend to roll it out incrementally, starting with the front page. We'll give people lots of notice so anyone running scrapers that are affected (like me!) will have time to change.
The look and feel of HN is unlikely to change much. We can argue about that one till the cows come home. I'm tempted to—my opinions are as strong as many of yours—but that doesn't seem like the best idea right now.
Feature requests should go to https://news.ycombinator.com/item?id=363. Software bugs should go to https://github.com/HackerNews/HN/issues. That would have been the appropriate destination for the OP, and indeed this issue has been posted about there.
Threads like the current one get lots of upvotes and comments, but they've never been on topic for HN. I'm going to bury this one now.
How is it not on topic for HN?
It would be on topic to discuss the implementation of any other site.
It would be on topic to discuss the implementation of any other site.
What's on topic for HN, as the guidelines say, are stories that gratify intellectual curiosity. Meta posts gratify several things, but that is not one of them. They generate masses of comments like fast-growing weeds, because we're all full of opinions about HN, but the discussions never go very deep, or really anywhere. This has been the case for many years—pretty much since the beginning of the site—and so has the policy on it.
You don't think a community of programmers and hackers and UX nerds would or should be intellectually curious about why a forum written in a Lisp dialect works and why it doesn't, and how it solves, or doesn't solve, various problems?
I can understand why it's annoying for the staff to have to encounter these threads so often - it's annoying for users to keep running into dead links. But I don't think it's entirely true that this topic has no value to the community at large. Having opinions and discussing things is kind of what forums are supposed to be about.
I can understand why it's annoying for the staff to have to encounter these threads so often - it's annoying for users to keep running into dead links. But I don't think it's entirely true that this topic has no value to the community at large. Having opinions and discussing things is kind of what forums are supposed to be about.
But look at the actual discussion here. It's mediocre.
I guess I can't argue with that, even though I personally find the subject more interesting than politics or talk about sexism in tech or bitcoin... everyone's mileage varies.
Though I would disagree with the premise that meta-discussion, necessarily, is a bad thing. At the very least given the nature of the site and its userbase, it's going to be inevitable.
Though I would disagree with the premise that meta-discussion, necessarily, is a bad thing. At the very least given the nature of the site and its userbase, it's going to be inevitable.
This has bearing on my absolutely favourite part of the Web Content Accessibility Guidelines (WCAG) 2.0:
Section 2.2.1, Timing Adjustable (http://www.w3.org/TR/WCAG20/#time-limits-required-behaviors):
----
2.2.1 Timing Adjustable: For each time limit that is set by the content, at least one of the following is true: (Level A)
- Turn off: The user is allowed to turn off the time limit before encountering it; or
- Adjust: The user is allowed to adjust the time limit before encountering it over a wide range that is at least ten times the length of the default setting; or
- Extend: The user is warned before time expires and given at least 20 seconds to extend the time limit with a simple action (for example, "press the space bar"), and the user is allowed to extend the time limit at least ten times; or
- Real-time Exception: The time limit is a required part of a real-time event (for example, an auction), and no alternative to the time limit is possible; or
- Essential Exception: The time limit is essential and extending it would invalidate the activity; or
- 20 Hour Exception: The time limit is longer than 20 hours.
Note: This success criterion helps ensure that users can complete tasks without unexpected changes in content or context that are a result of a time limit. This success criterion should be considered in conjunction with Success Criterion 3.2.1, which puts limits on changes of content or context as a result of user action.
----
Yea, verily! HN isn’t even Level A (the most basic level that "everyone" should comply with) compliant. Through this, HN fails Accessibility 101. (Its text colour choices are also often dubious as far as contrast goes—that’s covered in WCAG 2.0 as well, of course.)
I simply don’t care about the technical reasons. The choice of coroutines to back all of these things has broken the basic accessibility of all the content.
Section 2.2.1, Timing Adjustable (http://www.w3.org/TR/WCAG20/#time-limits-required-behaviors):
----
2.2.1 Timing Adjustable: For each time limit that is set by the content, at least one of the following is true: (Level A)
- Turn off: The user is allowed to turn off the time limit before encountering it; or
- Adjust: The user is allowed to adjust the time limit before encountering it over a wide range that is at least ten times the length of the default setting; or
- Extend: The user is warned before time expires and given at least 20 seconds to extend the time limit with a simple action (for example, "press the space bar"), and the user is allowed to extend the time limit at least ten times; or
- Real-time Exception: The time limit is a required part of a real-time event (for example, an auction), and no alternative to the time limit is possible; or
- Essential Exception: The time limit is essential and extending it would invalidate the activity; or
- 20 Hour Exception: The time limit is longer than 20 hours.
Note: This success criterion helps ensure that users can complete tasks without unexpected changes in content or context that are a result of a time limit. This success criterion should be considered in conjunction with Success Criterion 3.2.1, which puts limits on changes of content or context as a result of user action.
----
Yea, verily! HN isn’t even Level A (the most basic level that "everyone" should comply with) compliant. Through this, HN fails Accessibility 101. (Its text colour choices are also often dubious as far as contrast goes—that’s covered in WCAG 2.0 as well, of course.)
I simply don’t care about the technical reasons. The choice of coroutines to back all of these things has broken the basic accessibility of all the content.
> The choice of coroutines to back all of these things has broken the basic accessibility of all the content.
Exactly. This is actually a good example of More is Less.
Exactly. This is actually a good example of More is Less.
The choice of coroutines is not the cause of the problem, it's the decision to store session state on the server and clearing that state before the user is done.
This same problem can happen in a PHP application, and it can be avoided in a Racket web framework application that uses continuations.
This same problem can happen in a PHP application, and it can be avoided in a Racket web framework application that uses continuations.
Also, can someone please make the site render with a larger font on mobile? I am spending a fortune on new glasses and contact lenses, as well as monocles and loupes just to read HN when not on my computer. So many people on here have asked for this over the years. Perhaps this is a subtle attempt by the YC to make sure all the people in our industry have to use eyewear so we can recognize each other at the secret meetings. Maybe this has something to do with Gunnar Optiks marketing eyewear for geeks? Maybe YC is about to fund an optometrist practice. Or maybe someone here knows that Apple is about to release a 20 inch iPad?
Please, for the love of everything that is good and pure and true, make the font larger!
Sincerely, Blind among blind
Edit: please don't suggest one of the other HN wrappers. I know they exist. I don't want to hand over my HN password to them. My HN identity is important to me, so unless I can run my own wrapper on my own servers I don't want it.
Edit edit: I believe if HN mods/operators announced a contest for best mobile design they'd get a ton of submissions. Sadly I am not a designer and could not participate, but I am sure the amount of submissions would be huge.
Please, for the love of everything that is good and pure and true, make the font larger!
Sincerely, Blind among blind
Edit: please don't suggest one of the other HN wrappers. I know they exist. I don't want to hand over my HN password to them. My HN identity is important to me, so unless I can run my own wrapper on my own servers I don't want it.
Edit edit: I believe if HN mods/operators announced a contest for best mobile design they'd get a ton of submissions. Sadly I am not a designer and could not participate, but I am sure the amount of submissions would be huge.
There is a new layout that is supposed to be more mobile friendly in progress. dang has (briefly) posted about it here:
https://news.ycombinator.com/item?id=7606298
https://news.ycombinator.com/item?id=7606298
My solution to this for my Android phone (Galaxy Nexus) was to switch to the Opera browser. Opera is one of few browsers that still supports text reflow, meaning that I can easily zoom in and have the content nicely reformatted to fit the screen. Works like a charm on HN.
[deleted]
> Edit: please don't suggest one of the other HN wrappers. I know they exist.
What about one of the many apps? Have you tried news:yc?
What about one of the many apps? Have you tried news:yc?
In the meantime you can use this http://ihackernews.com
Can't you control the minimum font size on your mobile browser or something?
I believe on iOS you can increase the font size for the entire UI, not just the browser. Not something I would want to do. No idea about Android.
This happens with the next page button as well if I've been on the page for more than 3 minutes or so.
I see this almost every day. I have always assumed that Hacker News is too cool to fix/support navigation after the frontpage (or any other page) has changed.
It's not "cool", it's that the implementation is a quick hack, not intended to be a quality product. It just needs to be good enough to make users grudgingly return and see the PR for YC startups.
See [1] on why this is happening. Relevant part:
>Internally, the web server associates a token called the fnid with each instance of a link or form, and records the continuation function for each token. Periodically, old fnids are deleted.
[1]: http://arclanguage.github.io/ref/srv.html
>Internally, the web server associates a token called the fnid with each instance of a link or form, and records the continuation function for each token. Periodically, old fnids are deleted.
[1]: http://arclanguage.github.io/ref/srv.html
I'm not sure anybody particularly cares why it is happening, just that it is happening.
Thanks for the reference.
Any idea where the fnids are stored and how they are garbage-collected? Could it be that they're just dumped into a memcached instance and older fnids get pushed out whenever the memory limit is reached?
Any idea where the fnids are stored and how they are garbage-collected? Could it be that they're just dumped into a memcached instance and older fnids get pushed out whenever the memory limit is reached?
It's not the fnid that gets garbage collected, it's the coroutine itself.
I have a modest proposal for addressing the terrible dearth of fnids on this site. Since fnids are in such high demand, and yet are expensive to keep in memory, one small change might free up lots of them - remove the use of fnids for more pages and use a static url instead
so the more link on:
https://news.ycombinator.com/news
would point to:
https://news.ycombinator.com/news/2
and so on.... This would free up many fnids for use elsewhere, and also make the secondary list pages more easily cacheable and reduce load on the server.
The same could be done for other lists which currently use fnids without any appreciable change in the user experience or probably much change in the server code.
so the more link on:
https://news.ycombinator.com/news
would point to:
https://news.ycombinator.com/news/2
and so on.... This would free up many fnids for use elsewhere, and also make the secondary list pages more easily cacheable and reduce load on the server.
The same could be done for other lists which currently use fnids without any appreciable change in the user experience or probably much change in the server code.
Public Service Announcement:
If you're on Chrome use HackerNews Enhancement Suite.
https://chrome.google.com/webstore/detail/hacker-news-enhanc...
No more ass-ugly hackernews font choices and colors.
If you're on Chrome use HackerNews Enhancement Suite.
https://chrome.google.com/webstore/detail/hacker-news-enhanc...
No more ass-ugly hackernews font choices and colors.
Neat! Worth it for the option to collapse threads alone. Recently some extraordinarily lengthy, tedious threads have dominated the discussion on otherwise-interesting topics, and simply finding the end of the dreck was a problem.
One positive side effect of the architecture is proper isolation in pagination. When I fetch Page 1, the link to Page 2 will always return the same set, regardless of timing (up until it expires). You will never see the last entry on one page show up as the first entry on the next page.
Ya, for a community that is supposed to value great design and technical acumen, this site looks like somebody's pet project that was abandoned.
I don't really mind the design. I have a feeling they purposely keep the site design simple and, what some might consider "unappealing" to the average user. Helps weed out the normies.
It also weeds out the adults who think about what they read and don't have child eyes.
You can always zoom in or enable high DPI mode if your browser supports it
Or one can always make a mother*ucking web site [^1], which works perfectly in every browser. Zero issues with font size, layout and scrolling. Just-works.
[1]: http://motherfuckingwebsite.com/
[1]: http://motherfuckingwebsite.com/
So ... is this a bug or a feature?
Is the internal activity causing this unpopular behavior supposed to do this, in the first place? If the site is indeed behaving exactly as its designers want it to, then that's a rather separate debate as opposed to simply ways and means.
I'm not saying I agree with the behavior; I'm just trying to see what should be debated to start with.
Is the internal activity causing this unpopular behavior supposed to do this, in the first place? If the site is indeed behaving exactly as its designers want it to, then that's a rather separate debate as opposed to simply ways and means.
I'm not saying I agree with the behavior; I'm just trying to see what should be debated to start with.
Yes, the order changes over time, so the second page will often contain items from the old first page.
I agree with you: HN time-outs and failures (which also occur on reading) are an unnecessary pain. This is one reason you should never write into a web text box (a rule I also fail to adhere to). But really even "modern application grade" web stuff is so flakey if you value your text you write it in a local editor and then copy/paste later.
With this rule in mind, I wonder if a textures / notes area would be a good feature to include in the web browser itself (including mobile browsers) ...
> I'm pretty sure that the HN community likes to read thoughtful, thoroughly
> researched comments that are somewhat longer than one-liners.
[^1]: http://www.nicemice.net/par/
[^2]: http://jblevins.org/projects/markdown-mode/
[^3]: I just wish markdown-mode would renumber the footnotes if I go back and insert a new footnote. As it is I have to pipe the output through pandoc
It's not a priority for HN development. See Graham's 2009 essay for an explanation of what matters:
http://paulgraham.com/hackernews.html
http://paulgraham.com/hackernews.html
Can someone explain why this expiration thing is happening like I'm 5? Even the most basic PHP tutorial app doesn't have this problem so I don't understand people saying it's inevitable.
Yeah, it's really annoying to have to edit the URL on mobile when this happens..
I really do believe that HN is poorly designed for end users. The UI is not forward with intent, or pleasurable to use. It's been literally years and the expired link issue hasn't been addressed. ---No, I'll stop you right there. It's been acknowledged as an implementation bug, but not addressed. PG's lack of user empathy and stubbornness make HN worse.
The only reason people come here is the content.
The only reason people come here is the content.
The only reason people come here is the content.
I consider this a good thing. It attracts the right kind of people.
I consider this a good thing. It attracts the right kind of people.
Are you insinuating that the key to dissuading the wrong kind of people is to make Hacker News difficult to use? The notion that good users will "power through" a bad design is dubious at best.
I hope that you're not involved in product design. This thought process is not good for anyone.
I hope that you're not involved in product design. This thought process is not good for anyone.
I make no apologies for my opinion, however much you dislike it.
I've maintained for years that the key to a high quality online forum is for it to be limited in some way. It can be limited by bugs. Limited by nobody advertising that it exists. Limited by having a very focused topic of discussion. I'm willing to believe that it can be limited by moderation, but I self-select out of those forums so don't know.
But without some sort of limiting you get what happened to Usenet, Slashdot, kuro5hin, the main page of Reddit, and so on. Which is that they became the place that everyone knew that they should go. And then the noise went up faster than the signal, resulting in a poor signal to noise ratio. And then the best people found new forums to go to, and slowly disappeared.
Of course if you make the limiting TOO effective, eventually nobody will be left. There is a balance to be had.
And this only applies to high quality content of the kind that I want to engage with. Which is admittedly not to everyone's taste. It is not a way to make something popular.
I've maintained for years that the key to a high quality online forum is for it to be limited in some way. It can be limited by bugs. Limited by nobody advertising that it exists. Limited by having a very focused topic of discussion. I'm willing to believe that it can be limited by moderation, but I self-select out of those forums so don't know.
But without some sort of limiting you get what happened to Usenet, Slashdot, kuro5hin, the main page of Reddit, and so on. Which is that they became the place that everyone knew that they should go. And then the noise went up faster than the signal, resulting in a poor signal to noise ratio. And then the best people found new forums to go to, and slowly disappeared.
Of course if you make the limiting TOO effective, eventually nobody will be left. There is a balance to be had.
And this only applies to high quality content of the kind that I want to engage with. Which is admittedly not to everyone's taste. It is not a way to make something popular.
For any new readers: this is a "bug by design", it has been extensively complained about in the past, and it's marked "will not fix, stop complaining" by management.
This has been a public service announcement.
This has been a public service announcement.
Not true. It's an open bug. It's just really hard to fix.
https://github.com/HackerNews/HN/issues/11
https://github.com/HackerNews/HN/issues/11
With all due respect, if a framework makes storing sessions for more than 30 minutes "hard", there's something fundamentally wrong with the framework. This has been a solved problem for longer than I have worked in the industry.
Yes, it's been frustrating to me as well: https://news.ycombinator.com/item?id=5032739. Just an unfortunate consequence of starting out as a prototype.
@akkartik's JS redirect suggestion sounds like it will be trivial to implement. No need to fix the legacy arc code, just add a stopgap measure that will improve the UX for a lot of people :)
This 'bug' has been around so long that at this point I suspect that it's really just a long-running troll.
There's a reason things are set up as they are:
* Although HTTP connections are normally stateless, HN has a login system for account holders, which means there's a state, a connection, associated with each logged-in user.
* Because of the state information associated with logins, A Web server has a limited number of login connections it can handle.
* To serve the maximum number of visitors, a server needs to recycle connections that are no longer in use.
* There's no reliable heartbeat method between browser and server that a server can use to determine whether a given connection is still in use. Also, heartbeats would represent an extra traffic load on the server. And now that tabs have become popular, there's no way to be sure that a heartbeat response isn't from an unused, forgotten tab among dozens.
* Therefore, after a connection has not seen packet requests for a set time interval, the server expires the connection to make it available to other visitors.
* The shorter the timeout, the more connections the server can handle.
> I would suggest a sensible default of 86400 seconds.
Let's say the server can handle 2048 simultaneous login sessions (I don't know the actual number). According to your suggestion, after the first 2048 visitors log in, the server would refuse to accept any new logins for 24 hours. This isn't realistic.
Here's how I handle the problem -- if I decide a reply is going to take more than a few seconds, I transfer my reply to a text editor and let the connection expire. When my long-winded reply is ready, I refresh the original page, paste my reply in, and click "Add Comment".
I hope this helps.
* Although HTTP connections are normally stateless, HN has a login system for account holders, which means there's a state, a connection, associated with each logged-in user.
* Because of the state information associated with logins, A Web server has a limited number of login connections it can handle.
* To serve the maximum number of visitors, a server needs to recycle connections that are no longer in use.
* There's no reliable heartbeat method between browser and server that a server can use to determine whether a given connection is still in use. Also, heartbeats would represent an extra traffic load on the server. And now that tabs have become popular, there's no way to be sure that a heartbeat response isn't from an unused, forgotten tab among dozens.
* Therefore, after a connection has not seen packet requests for a set time interval, the server expires the connection to make it available to other visitors.
* The shorter the timeout, the more connections the server can handle.
> I would suggest a sensible default of 86400 seconds.
Let's say the server can handle 2048 simultaneous login sessions (I don't know the actual number). According to your suggestion, after the first 2048 visitors log in, the server would refuse to accept any new logins for 24 hours. This isn't realistic.
Here's how I handle the problem -- if I decide a reply is going to take more than a few seconds, I transfer my reply to a text editor and let the connection expire. When my long-winded reply is ready, I refresh the original page, paste my reply in, and click "Add Comment".
I hope this helps.
Because of the state information associated with logins, A Web server has a limited number of login connections it can handle.
You're making excuses for a particular setup - this is not something which holds true of all web servers. It's quite possible to store the state necessary to allow 10000 simultaneous logins if you use cookies and a db to store your state, rather than relying on in-memory state and fragile continuation links like this (created a few minutes ago):
https://news.ycombinator.com/x?fnid=l5qgIWgvnRbHE7owLEaPMw
or tying replies to stored state on the server which expires within minutes. Many large scale sites manage it, I suppose because they don't store data like this in memory or urls. I'm not sure about this idea of storing state in urls, it often backfires, and urls should identify resources and actions, not store state.
There is no good reason the server cannot receive a POST request from an authenticated user against a node id, and perform the requested action (post my reply please), no matter how many intervening logins/actions it has handled.
Looking at the form in question (reply forms) reveals why it does break:
It is annoying just how quickly reply links, more links etc expire on this site, and if the site used a different method to sign reply forms or construct other links they could get rid of this issue. Keeping data in memory like this will become more and more of a problem as the site scales - eventually things would break in seconds with enough load - that's clearly not acceptable.
You're making excuses for a particular setup - this is not something which holds true of all web servers. It's quite possible to store the state necessary to allow 10000 simultaneous logins if you use cookies and a db to store your state, rather than relying on in-memory state and fragile continuation links like this (created a few minutes ago):
https://news.ycombinator.com/x?fnid=l5qgIWgvnRbHE7owLEaPMw
or tying replies to stored state on the server which expires within minutes. Many large scale sites manage it, I suppose because they don't store data like this in memory or urls. I'm not sure about this idea of storing state in urls, it often backfires, and urls should identify resources and actions, not store state.
There is no good reason the server cannot receive a POST request from an authenticated user against a node id, and perform the requested action (post my reply please), no matter how many intervening logins/actions it has handled.
Looking at the form in question (reply forms) reveals why it does break:
<input type="hidden" name="fnid" value="UswPVmT3a8lZnqPBtcx359">
It is annoying just how quickly reply links, more links etc expire on this site, and if the site used a different method to sign reply forms or construct other links they could get rid of this issue. Keeping data in memory like this will become more and more of a problem as the site scales - eventually things would break in seconds with enough load - that's clearly not acceptable.
Then it's broken by design. There is no sense to do so, but a great way to anger visitors and destroy UX.
Although - other sites can manage user state and validating posts just fine without this 'feature.'
Even assuming it's all but impossible to fix given the implementation in Arc, surely it would be possible to automatically reassociate the post with whatever the current function id is? It seems to be as much a user experience problem as anything.
Even assuming it's all but impossible to fix given the implementation in Arc, surely it would be possible to automatically reassociate the post with whatever the current function id is? It seems to be as much a user experience problem as anything.
i always figured it was because they built each page and cached it to html, then that cache was set to expire after 5 minutes. then once the cache expired you couldn't go to the next page or what ever because the cached file was deleted.
IMO the reasons presented for this "Feature" should be changed to "lazy programming". cause that's all it is.
IMO the reasons presented for this "Feature" should be changed to "lazy programming". cause that's all it is.
Even then it would make more sense if the location that new comments were POSTed to was not necessarily one which could expire. Would it be impossible for the API (or whatever passes for an API) to present a static endpoint for receiving comments?
its not impossible, just appears to be coded by someone who over did it with caching in the beginning before the site needed it.
CSRF tokens != login sessions != persistent HTTP connections.
There is no need to keep an HTTP connection open in order for the server to log people in or recognize when a valid token has been submitted with a form.
There is no need to keep an HTTP connection open in order for the server to log people in or recognize when a valid token has been submitted with a form.
This is not the case. Websites, including hacker news, does not handle sessions like that.
A login system does introduce a notion of state between the server and client, but the use of the word 'connection' to describe that is probably a misnomer. I'm unaware of any basic HTTP servers that use a persistent socket connection (which seems to be the sort of connection you're describing, given the resource constraints you suggest) with clients in order to manage authentication.
Most often, the server just needs to set a flag associating an auth token with a given user (for whom it has a persistent representation, e.g. in a database.) Lookups against this token => user hash/map/table should be very cheap, especially if it's kept in memory. Further, even for tens of thousands of concurrent users, that's not a lot of memory overhead (1KB/authed user would be a generous upper bound, and that would still only be 10MB of RAM per 10k concurrent users.) Could be that HN is running on a 1990s-era desktop with 256MB of memory in someone's office and that this is an actual resource constraint issue--but my hunch is that this problem lies on the software side, not the hardware/resource side.
Most often, the server just needs to set a flag associating an auth token with a given user (for whom it has a persistent representation, e.g. in a database.) Lookups against this token => user hash/map/table should be very cheap, especially if it's kept in memory. Further, even for tens of thousands of concurrent users, that's not a lot of memory overhead (1KB/authed user would be a generous upper bound, and that would still only be 10MB of RAM per 10k concurrent users.) Could be that HN is running on a 1990s-era desktop with 256MB of memory in someone's office and that this is an actual resource constraint issue--but my hunch is that this problem lies on the software side, not the hardware/resource side.
I don't think this is what is going on here, it makes no sense for it to operate like this. To demonstrate this is not the case, I just momentarily unplugged my NIC and then submitted this comment. The connection was definitely dropped and re-associated, but my comment passed perfectly fine. This is more likely a very short cache time on CSRF tokens rather than anything to do with associated sockets.
The architectural model your comment and the initial post assumes to underlay the behavior is incorrect.
The reason for 'Unknown or Expired Link' is that HN runs on a webserver that uses continuations to maintain state. The continuations are stored in a table and based on load individual table entries get flushed to make room for new entries as needed. The message is actually always the result of an unknown link because the hash is no longer in the table.
Your experiment worked because the relevant continuation [state reference] was available after disconnect e.g. the 'route' so to speak was still valid.
Now obviously one might wonder why HN works this way. The reason is that the HN codebase came about as a way of field testing Arc [Graham's 100 year language]. Until rather recently, HN ran on a single server. It still uses relatively few lines of code.
The reason for 'Unknown or Expired Link' is that HN runs on a webserver that uses continuations to maintain state. The continuations are stored in a table and based on load individual table entries get flushed to make room for new entries as needed. The message is actually always the result of an unknown link because the hash is no longer in the table.
Your experiment worked because the relevant continuation [state reference] was available after disconnect e.g. the 'route' so to speak was still valid.
Now obviously one might wonder why HN works this way. The reason is that the HN codebase came about as a way of field testing Arc [Graham's 100 year language]. Until rather recently, HN ran on a single server. It still uses relatively few lines of code.
> Now obviously one might wonder why HN works this way. The reason is that the HN codebase came about as a way of field testing Arc [Graham's 100 year language].
This is really the key point: continuations are the wrong way to write this kind of application. It's significantly more work and there's no benefit from doing so but it allowed someone to test a pet idea and, having done so, fixing it would mean admitting that doing something because it's cool isn't good engineering.
This is really the key point: continuations are the wrong way to write this kind of application. It's significantly more work and there's no benefit from doing so but it allowed someone to test a pet idea and, having done so, fixing it would mean admitting that doing something because it's cool isn't good engineering.
The kind of application is specifically Hacker News, not a generic website or webserver. See the essay linked here for what kind of application this is:
https://news.ycombinator.com/item?id=7651525
https://news.ycombinator.com/item?id=7651525
Continuations are a questionable fit for web applications in general, however. HTTP is stateless and it requires a lot of extra work to build an abstraction which pretends otherwise. I prefer to embrace that from the beginning so the entire design matches the nature of the medium.
The explanation is mine. It wasn't an argument.
The application and implementation decisions are YC's. The essay explains their rationales.
The application and implementation decisions are YC's. The essay explains their rationales.
I know, I was demonstrating why that given explanation couldn't be true.
Http is stateless, so there are no "known sockets". Every request opens a new socket. Unplugging and plugging back in is a no-op in this context.
I'm not sure what's more disturbing; this reply, or the fact that only 2 out of 5 replies to this reply attempt to refute it.
This is largely correct. Each reply form includes a hidden input field with the name 'fnid' in their form. Many other pages include a parameter with the same name. This is the unique identifier of a closure that's associated with the page you're on. These closures are deleted after X time, at which point requests from that page can no longer be handled correctly.
Yes, obviously it would be great if X were increased. But doing so would require an increase in server resources or a significant change in the way HN's backend works; neither of these options is free. That is the reason why things are this way.
This is an old debate. See, for example: https://github.com/HackerNews/HN/issues/11
Yes, obviously it would be great if X were increased. But doing so would require an increase in server resources or a significant change in the way HN's backend works; neither of these options is free. That is the reason why things are this way.
This is an old debate. See, for example: https://github.com/HackerNews/HN/issues/11
fnids are evil, they just need to drop them and respond properly to posts without them. What more state needs to be recorded which is not already encoded in a POST reply request {user:n,post:7651593,text:'xxx'}?
The solution should be pretty straightforward:
Things like the more link are particularly pointless, they could just use something like:
The solution should be pretty straightforward:
Reply forms -> Remove fnid, authenticate on post, add CSRF
Flag links -> Remove fnid, authenticate on post, add CSRF
Things like the more link are particularly pointless, they could just use something like:
https://news.ycombinator.com/x?fnid=xxx -> https://news.ycombinator.com/news/2
The fnid is a CSRF token, just with an disadvantageous implementation that leads to a far too short expiry time.
I don't know much about the internals of Arc, but maybe it would be possible to serialise the continuation data, combine it with a reasonable expiry time, apply some form of authenticated encryption to this, and supply that as the fnid? And reverse the process - with appropriate integrity checks - when the fnid is submitted. Then you have your state distributed at the client-side instead of all being kept on the server, so it can scale more effectively.
EDIT: Nevermind, just realised that this very suggestion is addressed here, and it's more difficult than I anticipated https://github.com/HackerNews/HN/issues/11#issuecomment-3215...
I don't know much about the internals of Arc, but maybe it would be possible to serialise the continuation data, combine it with a reasonable expiry time, apply some form of authenticated encryption to this, and supply that as the fnid? And reverse the process - with appropriate integrity checks - when the fnid is submitted. Then you have your state distributed at the client-side instead of all being kept on the server, so it can scale more effectively.
EDIT: Nevermind, just realised that this very suggestion is addressed here, and it's more difficult than I anticipated https://github.com/HackerNews/HN/issues/11#issuecomment-3215...
A CSRF token doesn't have to be stored in memory on the server like these fnids, you only need a way to verify it like a secret for decoding it. This is a solved problem for many other servers so they could just look at some other implementation of CSRF, or start by getting rid of the many uses of fnid which are not required anyway (they are also used on GET requests).
I don't think they should try to serialise fnids, but just get rid of them completely. What state is required for a posted reply other than the three things outlined above + CSRF protection?
I don't think they should try to serialise fnids, but just get rid of them completely. What state is required for a posted reply other than the three things outlined above + CSRF protection?
You're right, I was overthinking the solution.
I think the most common technique I've seen for generating CSRF token is to compute an HMAC of the immutable request parameters.
I'm guessing that's what HN already implements for voting, as the token is dependent on user id and the id of the thing being voted on, and kind of looks like an SHA-1 hash.
I think the most common technique I've seen for generating CSRF token is to compute an HMAC of the immutable request parameters.
I'm guessing that's what HN already implements for voting, as the token is dependent on user id and the id of the thing being voted on, and kind of looks like an SHA-1 hash.
> * Because of the state information associated with logins, A Web server has a limited number of login connections it can handle.
a) this is technically incorrect – you can trivially handle this using signed cookies to store the logged in user
b) even using traditional database or filesystem based approaches, those limits would be aggressive even if the site was hosted on an iPhone – you're talking tens of bytes per session.
> To serve the maximum number of visitors, a server needs to recycle connections that are no longer in use.
Also technically wrong: if you aren't doing anything, you aren't causing any load whatsoever on the server. This would be different for something like a chat site using WebSockets but for a vanilla app like HN this is not an issue.
Also, again, even if you were using a persistent connection the absolute technical limits are much, much higher than the traffic HN sees - hundreds of thousands of simultaneous users.
> There's no reliable heartbeat method between browser and server that a server can use to determine whether a given connection is still in use.
Again, technically wrong: plain HTTP sites don't maintain persistent connections but in the case of something like a WebSocket connection you'd simply use TCP keepalives, which are a mature standard feature and very low overhead.
You appear to have confused the concept of a session with something which is actually happening at the network level. Try reading something about the HTTP request cycle (e.g. http://www.slideshare.net/cczona/full-stack-full-circle-what...) for basic background and after that move on to some discussion about the work done to make servers which can handle many simultaneous connections - the C10K problem was the original work tuning operating systems to handle tens of thousands of simultaneous connections and was large a non-issue at the network / basic HTTP leel by the mid-2000s:
http://bulk.fefe.de/scalability/
By now a single server can be assumed to handle hundreds of thousands of simultaneous connections at the network level – assuming, of course, that you have capacity to actual do whatever work those connections are used for. Idle users typing text into a form wouldn't be a problem even if it did use a persistent connection.
a) this is technically incorrect – you can trivially handle this using signed cookies to store the logged in user
b) even using traditional database or filesystem based approaches, those limits would be aggressive even if the site was hosted on an iPhone – you're talking tens of bytes per session.
> To serve the maximum number of visitors, a server needs to recycle connections that are no longer in use.
Also technically wrong: if you aren't doing anything, you aren't causing any load whatsoever on the server. This would be different for something like a chat site using WebSockets but for a vanilla app like HN this is not an issue.
Also, again, even if you were using a persistent connection the absolute technical limits are much, much higher than the traffic HN sees - hundreds of thousands of simultaneous users.
> There's no reliable heartbeat method between browser and server that a server can use to determine whether a given connection is still in use.
Again, technically wrong: plain HTTP sites don't maintain persistent connections but in the case of something like a WebSocket connection you'd simply use TCP keepalives, which are a mature standard feature and very low overhead.
You appear to have confused the concept of a session with something which is actually happening at the network level. Try reading something about the HTTP request cycle (e.g. http://www.slideshare.net/cczona/full-stack-full-circle-what...) for basic background and after that move on to some discussion about the work done to make servers which can handle many simultaneous connections - the C10K problem was the original work tuning operating systems to handle tens of thousands of simultaneous connections and was large a non-issue at the network / basic HTTP leel by the mid-2000s:
http://bulk.fefe.de/scalability/
By now a single server can be assumed to handle hundreds of thousands of simultaneous connections at the network level – assuming, of course, that you have capacity to actual do whatever work those connections are used for. Idle users typing text into a form wouldn't be a problem even if it did use a persistent connection.
Fortunately, I can usually hit the Back button, refresh the page, and resubmit my comment without losing the contents of the textarea. Anyway, I try to be extra careful and copy the comment to the clipboard before I hit "Submit" if I feel like it's been more than a few minutes since I started writing.
But this should not be necessary in the first place.
Please increase the timeout on your CSRF tokens (or whatever it is that triggers the error message above) to something more user-friendly. I would suggest a sensible default of 86400 seconds.
Thanks.