Show HN: Tiny Web Crawler(github.com)

2 pointsby l1feh4ck8 ปีที่แล้ว1 comments

https://github.com/indrajithi/tiny-web-crawler

1 comments

photonios8 ปีที่แล้ว

It's not entirely correct to remove query parameters from the URL. Query parameters are part of the URL and can make a website show entirely different content.

The Googlebot for example considers these two URL's different pages:

    /mypage?name=name1
    /mypage?name=name2