> No I don't want to receive push notification from your site...
A PWA does not equal push notifications. Those are completely optional and should of course only be added where that makes sense. And even then, you'd have to agree to receiving them first.
> ...or ability to visit your website offline
Fine, you don't have to. In that case, the service worker just improves page speed and reduces data usage.
I would argue that, while not every site should have to be converted to a PWA to work properly, every site can benefit from it. If a website is not a complex SPA, why should it not be fast, secure or offline-accessible?
HTTPS or Service Worker caching is not something a mobile browser can do for you by default.