>If you write a crawler, you probably don't want it to waste time indexing a list of articles in every possible sort order, trying all "reply" buttons, things like that.
On the other hand, many websites (like wikipedia here) hide interesting pages behind a Disallow.
>Suppose scientists determined that 5G networks are harmful to people's health, what then? Are people gonna stop using cellphones? Are companies like T-Mobile and Verizon gonna just fold?