UnifiedPush: A decentralized, open-source push notification protocol(f-droid.org)
f-droid.org
UnifiedPush: A decentralized, open-source push notification protocol
https://f-droid.org/2022/12/18/unifiedpush.html
23 comments
Sorry if I’m not understanding something but…
Doesn’t Apple and Google on the OS level only allow push notifications to come from its own services? Like, they look for the signature to be compatible with https certificate etc. Otherwise the phone could have a ton of push notifications incoming from anywhere on the Internet. The APN service for instance would only route notifications to a phone if it came from a registered website. How do you get around this?
As far as I know, even for VOIP permissions, iOS only wakes your code up if ITS APNS SERVICE sends the notification. How are you able to install a long-lived application in the background on iOS?
Similarly, the phone radio is listening for calls from the nearby cell towers in the network, not from absolutely any tower. I guess a Stingray can impersonate one, but they would probably need to fake the attestation, right?
Doesn’t Apple and Google on the OS level only allow push notifications to come from its own services? Like, they look for the signature to be compatible with https certificate etc. Otherwise the phone could have a ton of push notifications incoming from anywhere on the Internet. The APN service for instance would only route notifications to a phone if it came from a registered website. How do you get around this?
As far as I know, even for VOIP permissions, iOS only wakes your code up if ITS APNS SERVICE sends the notification. How are you able to install a long-lived application in the background on iOS?
Similarly, the phone radio is listening for calls from the nearby cell towers in the network, not from absolutely any tower. I guess a Stingray can impersonate one, but they would probably need to fake the attestation, right?
I believe the notifications used to be delivered by a very long-lived TCP connection, and there probably is no limit to those on Android.
The reason to use the google services is mostly touted as for battery savings, I believe.
The reason to use the google services is mostly touted as for battery savings, I believe.
>and there probably is no limit to those on Android.
Not true since Android Oreo. Long running background Service execution (without a foreground notification) has been dead for a while. Most of the useful system-wide config change updates (implicit Broadcasts) got killed as well. Basically, only Google (or GCM/Firebase Messaging) gets to do timely push notifications. You can try to roll your own MQTT server, but there’s a good chance the OS will kill your Service, in which case you would have to poll every 15 minutes with JobScheduler/WorkManager, with no guarantee that background task would even get run every 15 min.
Not true since Android Oreo. Long running background Service execution (without a foreground notification) has been dead for a while. Most of the useful system-wide config change updates (implicit Broadcasts) got killed as well. Basically, only Google (or GCM/Firebase Messaging) gets to do timely push notifications. You can try to roll your own MQTT server, but there’s a good chance the OS will kill your Service, in which case you would have to poll every 15 minutes with JobScheduler/WorkManager, with no guarantee that background task would even get run every 15 min.
Afaik Google and Apple just discount their own poller. It still has to wake up and check periodically.
They have special deals with the telcos that TCP connections to their servers can have longer TTLs. Plus of course, even if you do polling, if you have 5 different apps that connect every 10 minutes to their own respective services with different offsets, you have the device waking up every 2 minutes, with a potential latency of up to 10 minutes for each individual service. If it uses a central push service, you can set it to 5 minutes, the device would therefore wake up less than half of the time, and maximum latency would be 5 minutes, not 10.
Thus, it makes total sense to have one mechanism for push messages for the entire device. It's great to have degoogled alternatives here.
Thus, it makes total sense to have one mechanism for push messages for the entire device. It's great to have degoogled alternatives here.
> Plus of course, even if you do polling, if you have 5 different apps that connect every 10 minutes to their own respective services with different offsets, you have the device waking up every 2 minutes,
This is now impossible with background execution restraints set in Oreo. The OS clusters requests for background execution and does them with the same wake. Apps can only wake the device every 15 minutes at most (and requests to wake are not guaranteed).
This is now impossible with background execution restraints set in Oreo. The OS clusters requests for background execution and does them with the same wake. Apps can only wake the device every 15 minutes at most (and requests to wake are not guaranteed).
> This is now impossible with background execution restraints
Signature spoofing is similarly "impossible" there's no rule saying this stuff can't be changed by distributions that care about it. You already have to install a custom OS to get rid of gapps to begin with so aside from the maintenance burden for changes that will never get upstreamed this is really a nonissue.
Signature spoofing is similarly "impossible" there's no rule saying this stuff can't be changed by distributions that care about it. You already have to install a custom OS to get rid of gapps to begin with so aside from the maintenance burden for changes that will never get upstreamed this is really a nonissue.
Thanks for your great work creating this protocol. Hope more app developers add support for it, especially for privacy focused apps like Signal which by relying on Google servers are leaking metadata.
Regarding Signal, I don't think they have been approached yet... https://github.com/UnifiedPush/wishlist/issues/8
I think the consensus was that we wanted to wait for UnifiedPush to be more mature before approaching them, but it's probably about time. S1m made an interesting implementation for the (compatible) Signal fork Molly: https://github.com/mollyim/mollyim-android/pull/152
The implementation is quite interesting: it adds a linked device that does not receive encryption keys, but sends push notifications to the client.
I think the consensus was that we wanted to wait for UnifiedPush to be more mature before approaching them, but it's probably about time. S1m made an interesting implementation for the (compatible) Signal fork Molly: https://github.com/mollyim/mollyim-android/pull/152
The implementation is quite interesting: it adds a linked device that does not receive encryption keys, but sends push notifications to the client.
If you deliver notifications, you are always leaking metadata around timing.
A police unit parked outside a guy’s house and confirmed it was him in the real time chatroom, by cutting his internet and seeing him drop off.
If you want real anonymity on the Internet, never use push notifications. Always “pick up your mail” periodically from random endpoints on the Web.
A police unit parked outside a guy’s house and confirmed it was him in the real time chatroom, by cutting his internet and seeing him drop off.
If you want real anonymity on the Internet, never use push notifications. Always “pick up your mail” periodically from random endpoints on the Web.
In theory, if you were worried about this, I think you could get around that problem by just constantly send fixed sized encrypted messages large enough to hold any potential notifications. When a notification needs to get sent, you add it to the next outgoing encrypted message. Of course, now the delay for notifications is now coupled to the rate of these messages being sent.
The messages can still be traced to the client picking them up. Cut their networking and the client doesn’t receive the messages anymore… this shows up as a bounced message.
You can accumulate encrypted messages at random urls instead
You can accumulate encrypted messages at random urls instead
If nobody in the chat room can know that you lost connectivity, that means you can't do real time chat and might as well use email instead.
Note that a similar thing happens with any two-way conversation; if an immediate reply is expected and there's no reply, it's not a conversation anymore. (For more possible reasons, though.)
Disappearances can't have good error messages if you want plausible deniability. This will make any UI much less user-friendly.
Note that a similar thing happens with any two-way conversation; if an immediate reply is expected and there's no reply, it's not a conversation anymore. (For more possible reasons, though.)
Disappearances can't have good error messages if you want plausible deniability. This will make any UI much less user-friendly.
I guess I initially misunderstood what you meant by timing, but that makes sense.
This would be an interesting protocol. Almost like spread spectrum or frequency hopping in the radio space.
Have many servers who just accumulate encrypted packets and then only the host knows how to reassemble them in order and decrypt.
Could probably be somewhat realtime if executed correctly.
Have many servers who just accumulate encrypted packets and then only the host knows how to reassemble them in order and decrypt.
Could probably be somewhat realtime if executed correctly.
Simply use free Usenet Servers via Tor and post your anonymous* PGP
messages to the Usenet group alt.anonymous.messages.
*You send PGP encrypted messages via Mixmaster Remailers (via Tor) and preferably use a hashed subject (hsub) so that the intended receiver(s) can filter out their messages.
https://lists.gnupg.org/pipermail/gnupg-users/2019-March/061...
*You send PGP encrypted messages via Mixmaster Remailers (via Tor) and preferably use a hashed subject (hsub) so that the intended receiver(s) can filter out their messages.
https://lists.gnupg.org/pipermail/gnupg-users/2019-March/061...
> A police unit parked outside a guy’s house and confirmed it was him in the real time chatroom, by cutting his internet and seeing him drop off.
I guess I'm going off of too few details probably, but this doesn't seem like a very sound strategy to me. It's certainly not impossible that the real criminal lost internet around the same time as some suspect's was cut by the police. I know I've shown up as online to other people for up to several minutes after losing internet in real-time situations (e.g. games). I suppose police just need "reasonably likely" to take action though, not definite reasons.
I guess I'm going off of too few details probably, but this doesn't seem like a very sound strategy to me. It's certainly not impossible that the real criminal lost internet around the same time as some suspect's was cut by the police. I know I've shown up as online to other people for up to several minutes after losing internet in real-time situations (e.g. games). I suppose police just need "reasonably likely" to take action though, not definite reasons.
More than that, the criminal could have been a neighbour piggybacking on his WiFi.
The team has been working to get UP support added to Molly, a fork of Signal for Android.
https://molly.im/
https://github.com/mollyim/mollyim-android/pull/152
https://molly.im/
https://github.com/mollyim/mollyim-android/pull/152
Congrats! It's always surprised me that Amazon or one of the other Android vendors didn't do this themselves to make porting to FireOS or other services-included Android distributions easier, but I hope you get funding from them for doing their job for them.
Everyone wants to be Google, and I bet most managers see lock-in to their platform as a good thing, and ease of porting to competitors a bad thing.
At least, if they feel big enough to gain initial traction...
Huawei is pushing their own thing, alternative to Firebase[1]; Amazon as well[2].
There's a dozen services that provide cloud-based services to send push notifications to various services. I think the incentive to gain subscription revenue was greater than the opportunity to make these partially redundant.
I'm not sure why it took so long for something like UnifiedPush to appear. It took a fair bit of debugging, but ultimately, it was mostly up to one dedicated individual (S1m) with Android experience.
Before UnifiedPush, there was OpenPush[3], but it never materialized.
[1]: https://developer.huawei.com/consumer/en/hms/huawei-pushkit
[2]: https://docs.aws.amazon.com/sns/latest/dg/sns-mobile-applica...
[3]: https://bubu1.eu/openpush/
At least, if they feel big enough to gain initial traction...
Huawei is pushing their own thing, alternative to Firebase[1]; Amazon as well[2].
There's a dozen services that provide cloud-based services to send push notifications to various services. I think the incentive to gain subscription revenue was greater than the opportunity to make these partially redundant.
I'm not sure why it took so long for something like UnifiedPush to appear. It took a fair bit of debugging, but ultimately, it was mostly up to one dedicated individual (S1m) with Android experience.
Before UnifiedPush, there was OpenPush[3], but it never materialized.
[1]: https://developer.huawei.com/consumer/en/hms/huawei-pushkit
[2]: https://docs.aws.amazon.com/sns/latest/dg/sns-mobile-applica...
[3]: https://bubu1.eu/openpush/
I would imagine Huawei is doing that moreso due to sanctions, no?
Thank you very much for your hard and important work!
And a special thanks for not forcing us to use a slack channel!
Hopefully other OS projects will follow.
Good health and all the best for 2023!
And a special thanks for not forcing us to use a slack channel!
Hopefully other OS projects will follow.
Good health and all the best for 2023!
Does the user require to install a separate app to receive UnifiedPush messages?
Yes, they need to have a UnifiedPush distributor on their device, that acts like Google services.
OS makers could install a default UnifiedPush distributor though.
Lastly, the Embedded FCM UnifiedPush library (to be embeded into a client app) handles UnifiedPush over FCM (Google-provided push notifications), so that a server only has to handle the UnifiedPush API (with a rewrite proxy for UP->FCM).
https://unifiedpush.org/developers/embedded_fcm/
OS makers could install a default UnifiedPush distributor though.
Lastly, the Embedded FCM UnifiedPush library (to be embeded into a client app) handles UnifiedPush over FCM (Google-provided push notifications), so that a server only has to handle the UnifiedPush API (with a rewrite proxy for UP->FCM).
https://unifiedpush.org/developers/embedded_fcm/
We can't get users to understand that the ceo isn't sending them mail from [email protected] to go and buy giftcards, and not to ask any questions.
What's the actual likely hood that the average end user is going to install a random service?
What's the actual likely hood that the average end user is going to install a random service?
> What's the actual likely hood that the average end user is going to install a random service?
For now, UnifiedPush is targeted towards De-Googlers, self-hosters and FOSS enthusiasts, not average users.
For now, UnifiedPush is targeted towards De-Googlers, self-hosters and FOSS enthusiasts, not average users.
Except, in the threads here, this is called out repeatedly as a replacement for push notifications on phones, replacing webpush, apple push notifications, and whatever google is calling it.
Which implies the use case people are seeing is phone apps. Unless people are building one off apps for themselves, you've got end users in the mix.
I don't see how this can be positions as much outside of phone apps, as txt messages or emails would be easier and more universal. Being that they're probably going to be supported here.
To note, i haven't see anyone talking about this being replacement for the part of webpush that handles registration for notifications or whatnot, but as a whole separate channel/technology. Not 'install this to avoid using google/apple for webpush' but 'use this in your app, and have your users install the service'.
Which implies the use case people are seeing is phone apps. Unless people are building one off apps for themselves, you've got end users in the mix.
I don't see how this can be positions as much outside of phone apps, as txt messages or emails would be easier and more universal. Being that they're probably going to be supported here.
To note, i haven't see anyone talking about this being replacement for the part of webpush that handles registration for notifications or whatnot, but as a whole separate channel/technology. Not 'install this to avoid using google/apple for webpush' but 'use this in your app, and have your users install the service'.
Some apps just offer both, and even auto-detect. So if your phone is de-googled, the app will use UP, if it has the Google Services it will use that. For some apps it is available in the settings.
So yeah, the app has to support it, but the user can choose then.
So yeah, the app has to support it, but the user can choose then.
People are seeing it as a way to get efficient push notifications in an open, decentralized system.
That's pretty much it. Because there isn't really any possible alternative except something that looks like this.
GCM/APNS/ETC do not fit those requirements, so it needs to be something new.
That's pretty much it. Because there isn't really any possible alternative except something that looks like this.
GCM/APNS/ETC do not fit those requirements, so it needs to be something new.
Is there any plan to have a method available for patching proprietary apps to work with this? Notifications is the only reason I still have Goolag services on my phone's work profile, and having an alternative to that would be great for privacy.
As a sibling commenter said, those proprietary apps have a server part that sends notifications to Google's server.
If UnifiedPush were to gain enough traction (say it would be adopted on FireOS, HuaweiOS and whatever platform don't use Google services), those apps would likely start to support it directly.
Another possible avenue is to add UnifiedPush support to push libraries used by these apps both server and client-side. Multiple libraries (Google FCM included, I linked at least two others from Amazon and Huawei in another comment here) abstract push notifications away so the developer gets a single server to talk to, and a unified API across iOS, Android, Web, etc. If these libraries started supporting UnifiedPush, adoption could increase a lot, without involving any developer effort for these proprietary apps.
If UnifiedPush were to gain enough traction (say it would be adopted on FireOS, HuaweiOS and whatever platform don't use Google services), those apps would likely start to support it directly.
Another possible avenue is to add UnifiedPush support to push libraries used by these apps both server and client-side. Multiple libraries (Google FCM included, I linked at least two others from Amazon and Huawei in another comment here) abstract push notifications away so the developer gets a single server to talk to, and a unified API across iOS, Android, Web, etc. If these libraries started supporting UnifiedPush, adoption could increase a lot, without involving any developer effort for these proprietary apps.
The sender needs to know which push service to send the message to, so no.
> Notifications is the only reason I still have Goolag services on my phone's work profile
Wouldn't microG be enough for that?
Wouldn't microG be enough for that?
The home page you've given is a little confusing to me and I'm not sure if it's confusing or just not aimed at me.
Is this just for mobile (android only?) developers, or is it for any push notifications e.g. browser.
Could I use this with PHP or JavaScript for example?
Is this just for mobile (android only?) developers, or is it for any push notifications e.g. browser.
Could I use this with PHP or JavaScript for example?
Uh, the site doesn't load...
Looks like IPv6 is broken :/
The AAAA record was temporarily removed to try to mitigate.
The AAAA record was temporarily removed to try to mitigate.
Interested to know what happened here.
Shouldn't Happy eyeballs fall back to IPv4 when IPv6 is broken?
Shouldn't Happy eyeballs fall back to IPv4 when IPv6 is broken?
They should, which is why nobody noticed it was broken.
Someone told us IPv6 was broken after we submitted here, as they use an IPv6-only network, apparently. I just posted that up there in case that was the actual issue encountered by the commenter. We're not sure why it broke yet, but we'll fix it when there's less traffic.
Someone told us IPv6 was broken after we submitted here, as they use an IPv6-only network, apparently. I just posted that up there in case that was the actual issue encountered by the commenter. We're not sure why it broke yet, but we'll fix it when there's less traffic.
Off-topic: Sometimes I work on ships that have Wi-Fi for talking to local services, but due to strict metering, Internet access is usually disabled. So most of the time there's no way to get notifications from monitoring systems and your fellow crew, let alone shoreside team members.
It would be nice to have a solution for delivering pushes internally, and also hold open a best-effort connection to a remote server that allows some notifications through.
The last solution I considered was running a Matrix homeserver on the ship with some local channels, and another instance on shore, and using federation to connect them when the uplink is available. Monitoring systems could send their alerts as DMs or to a shared channel.
Then for the notifications, Apple has its Local Push Connectivity API you might be able to use to send notifications within the ship network. I'm not sure if this UnifiedPush would solve the problem on Android. Either way, though, it seems like a lot of engineering effort just for a chatroom.
It would be nice to have a solution for delivering pushes internally, and also hold open a best-effort connection to a remote server that allows some notifications through.
The last solution I considered was running a Matrix homeserver on the ship with some local channels, and another instance on shore, and using federation to connect them when the uplink is available. Monitoring systems could send their alerts as DMs or to a shared channel.
Then for the notifications, Apple has its Local Push Connectivity API you might be able to use to send notifications within the ship network. I'm not sure if this UnifiedPush would solve the problem on Android. Either way, though, it seems like a lot of engineering effort just for a chatroom.
One of the first uses of UnifiedPush was to self-host the whole Matrix messaging stack. In fact, we have already had someone in the UnifiedPush chat set this up on their ship [1]. Matrix + UnifiedPush would be perfect for a disconnected situation on ships (or Mars colonies :)
[1] https://matrix.to/#/!vwmBiTqilorqNCbGab:matrix.org/$hJpXxhVR...
[1] https://matrix.to/#/!vwmBiTqilorqNCbGab:matrix.org/$hJpXxhVR...
I'm a huge fan of Matrix and the idea of UnifiedPush, but I don't quite grok what this means (and I'm reluctant to fire up my Matrix client because I'm avoiding some things). Is there any chance you can elaborate on what "self-host the whole Matrix stack" means?
I read the thread just now:
~" Self hosted matrix + UP. Nothing special there. Only issue is cert expiration for setups without connections that result in expired certs for https. That requires some work. "~
ntfy [1] would work for you entirely in the LAN, if you self-host the server and the phones are connected to the same Wifi. It'll only work for Android phones though, since iOS forces APNS for push notifications.
ntfy is also a distributor for UnifiedPush, so you're not entirely off topic here. Hehe.
Disclaimer: I am the maintainer of ntfy.
[1] https://github.com/binwiederhier/ntfy
ntfy is also a distributor for UnifiedPush, so you're not entirely off topic here. Hehe.
Disclaimer: I am the maintainer of ntfy.
[1] https://github.com/binwiederhier/ntfy
Interesting thanks. Have you looked into Apple's Local Push Connectivity API? It's specifically designed for VoIP calls and text messages, but perhaps the latter is enough. Not sure if it requires you to bake in any keys at compile time, or if the iOS app could allow the end-user to configure the push server.
https://developer.apple.com/documentation/networkextension/l...
https://developer.apple.com/documentation/networkextension/l...
I have been using NTFY for months, love it!
In addition to what the others wrote (a local Matrix server and UnifiedPush provider such as ntfy would work fine), I suggest you look into p2p messaging systems such as Briar. I think Tox is also p2p?
Lastly, Secure Scuttlebutt was actually designed on a boat for sporadic internet connection and local connections, as well as sneakernet, so that would work too, but I don't think it's designed for IM. Perfect for blogs or facebook-like stuff though.
Have a member of the crew go onshore up to a Wi-Fi AP, it will act as an async data transfer for everyone. You can also exhange data with other travellers, including passing boats.
Interestingly, there are a few Matrix p2p experiments. I wish apple opened their Airdrop feature as that would make proximity networking easier, though that should be achievable with Wi-Fi NAN+P2P.
Lastly, Secure Scuttlebutt was actually designed on a boat for sporadic internet connection and local connections, as well as sneakernet, so that would work too, but I don't think it's designed for IM. Perfect for blogs or facebook-like stuff though.
Have a member of the crew go onshore up to a Wi-Fi AP, it will act as an async data transfer for everyone. You can also exhange data with other travellers, including passing boats.
Interestingly, there are a few Matrix p2p experiments. I wish apple opened their Airdrop feature as that would make proximity networking easier, though that should be achievable with Wi-Fi NAN+P2P.
Airdrop (AWDL) is open, within iOS and macOS at least, and P2P Matrix uses it :)
Ah, thanks for the correct term. I am not sure the specification is open enough for other operating systems to support it, which is what I meant.
Among my acquaintances, I can count Apple users on a single hand (<1% of the people I know), which severly limits the usefulness of the network.
There's OWL [1] which looks promising on Linux. I hope Android devices can get something soon. I'm still a bit confused by how Linux drives Wi-Fi radio, and what is and isn't implementable in software: could older Android devices gain support for NAN and OWL through a software update? What about old computers with old Wi-Fi cards?
I first got aware of Wi-Fi NAN by chatting with neilalexander on #yggdrasil:matrix.org :)
It looks promising for advertising capabilities to nearby hosts, allowing to negotiate p2p connections.
[1]: https://owlink.org/
Among my acquaintances, I can count Apple users on a single hand (<1% of the people I know), which severly limits the usefulness of the network.
There's OWL [1] which looks promising on Linux. I hope Android devices can get something soon. I'm still a bit confused by how Linux drives Wi-Fi radio, and what is and isn't implementable in software: could older Android devices gain support for NAN and OWL through a software update? What about old computers with old Wi-Fi cards?
I first got aware of Wi-Fi NAN by chatting with neilalexander on #yggdrasil:matrix.org :)
It looks promising for advertising capabilities to nearby hosts, allowing to negotiate p2p connections.
[1]: https://owlink.org/
You can self host a Gotify server (single file executable, web interface), install the app, set it to connect to the server on the local network. Sending a message is as easy as a curl to the server. It will show up in the apps.
https://gotify.net/
https://github.com/gotify/android
The problem is that there is no iOS app. You could use the browser there but maybe ntfy would be a better fit (Android + iOS.)
https://gotify.net/
https://github.com/gotify/android
The problem is that there is no iOS app. You could use the browser there but maybe ntfy would be a better fit (Android + iOS.)
ntfy is a Gotify alternative. It has an iOS app, but it still relies on Apple's push service, as it cannot open a persistent backgroung connection due to iOS restrictions.
> It would be nice to have a solution for delivering pushes internally, and also hold open a best-effort connection to a remote server that allows some notifications through.
... email?
... email?
I've been using NNTP as a Gossip-type protocol for eventually-consistent pub/sub telemetry, monitoring, and notifications between partially-connected nodes in an unstable mesh since about 1995.
Why use a new custom-built protocol instead of adding support for the Web Push protocol already supported by Firefox, Chrome, Safari and others? https://web.dev/push-notifications-web-push-protocol/. Now backend developers have Yet Another Push Notifications Protocol to build support for, instead of being able to just easily plug into their existing code. As a benefit, since all notifications through the Web Push protocol are end-to-end encrypted, there's no concern about "trusting" the distributing server like there seems to be in this protocol (based on https://unifiedpush.org/spec/server/)
UnifiedPush is compatible with WebPush: both just involve an HTTP POST to a specific URL. If your application server supports WebPush, it can send encrypted notifications to UnifiedPush servers (that is how we're planning to support Telegram, for example). You can then decrypt these notifications in the app [1]. However, some protocols only send a wake-up ping or random ID in the push notification, Trifa and Matrix respectively. There, encryption is unnecessary complexity.
However, we don't use the WebPush API between the push server and distributor since there is a lot of scope for innovation in that space (for example Google's FCM uses a custom XMPP based protocol).
[1] One example, https://github.com/UnifiedPush/dart-webpush-encryption
However, we don't use the WebPush API between the push server and distributor since there is a lot of scope for innovation in that space (for example Google's FCM uses a custom XMPP based protocol).
[1] One example, https://github.com/UnifiedPush/dart-webpush-encryption
> There, encryption is unnecessary complexity.
If you are sending a small amount of data it is likely a tiny overhead for some notable benefits.
1. If all data is encrypted then no data stands out for being encrypted.
2. It provides integrity which may prevent push servers from changing message to be invalid or triggering parsing bugs in applications.
3. If only encrypted messages are supported it is impossible for uses which do need encryption to forget to turn it on or accidentally turn it off.
For many apps the benefit here is indeed tiny, but so is the cost. So I think it is best to not support unencrypted messages altogether.
If you are sending a small amount of data it is likely a tiny overhead for some notable benefits.
1. If all data is encrypted then no data stands out for being encrypted.
2. It provides integrity which may prevent push servers from changing message to be invalid or triggering parsing bugs in applications.
3. If only encrypted messages are supported it is impossible for uses which do need encryption to forget to turn it on or accidentally turn it off.
For many apps the benefit here is indeed tiny, but so is the cost. So I think it is best to not support unencrypted messages altogether.
I agree that mandatory encryption would massively help privacy, integrity and even compliance [1]. However, the main cost is developer-effort involved in keeping track of keys and implementing encryption.
As a small project, ease of adoption has been our most important goal. However, as we work towards better WebPush compatibility [2], I personally absolutely want to promote RFC8291 encryption and make it easy to implement for developers using UnifiedPush, hopefully making it the norm.
[1] https://volkerkrause.eu/2022/11/12/kde-unifiedpush-push-noti...
[2] https://github.com/UnifiedPush/wishlist/issues/15
As a small project, ease of adoption has been our most important goal. However, as we work towards better WebPush compatibility [2], I personally absolutely want to promote RFC8291 encryption and make it easy to implement for developers using UnifiedPush, hopefully making it the norm.
[1] https://volkerkrause.eu/2022/11/12/kde-unifiedpush-push-noti...
[2] https://github.com/UnifiedPush/wishlist/issues/15
Making it the "norm" isn't good enough, because developers will always want to cut corners to ship, either based on pressure from management or (in open source projects) just not feeling comfortable with encryption or thinking that it's important enough to implement. Web push libraries are already widely available for major languages to handle implementing encryption, and keeping track of keys isn't any harder than keeping track of device IDs or endpoints would be without encryption—it's just one more row in your database.
WebPush doesn't really have keys to keep track of. Other than the VAPID signature (which is optional but may be enforced by the push service) all of the keys are ephemeral. They are used for just one subscription so storing them is just as easy as storing the rest of the notification URL.
> However, the main cost is developer-effort involved in keeping track of keys and implementing encryption.
As soon as you need encryption in at least one place in your software, it's going to happen.
As soon as you need encryption in at least one place in your software, it's going to happen.
I agree, though it of course it depends on the threat model. As a self hoster, encryption is pretty much irrelevant, as my push server is controlled by myself, and accessible with https. Not having encryption makes it easy to debug and experiment.
I agree encryption should be the default choice though. Maybe we can introduce it in a future protocol revision? Client side, I expect most people use libraries, making this easy. Server side? Not so much.
The main goal was to get it adopted as widely as possible though, and encryption certainly seemed like something that would slow the effort.
I agree encryption should be the default choice though. Maybe we can introduce it in a future protocol revision? Client side, I expect most people use libraries, making this easy. Server side? Not so much.
The main goal was to get it adopted as widely as possible though, and encryption certainly seemed like something that would slow the effort.
Even for self-hosted I'd rather trust the server as little as necessary. Just in case it is completed why let it see things that it doesn't need to see?
> However, some protocols only send a wake-up ping or random ID in the push notification, Trifa and Matrix respectively
While I understand that some developers may think of security as wasteful, having unencrypted data be the default relies on every developer to spend much more time considering the possible threat models and how . Case in point here is Matrix—the IDs they send aren't random at all, they actually send full room IDs and event IDs for each message, so you'd be able to build up someone's entire social graph just from reading their unencrypted push notification data and looking up room IDs on the Matrix server in question or even correlating room IDs across different Element installs. This also hurts users, since they're not in a position to inspect the source code and understand how or whether they need to trust their "push provider" with whatever data the app developers might have chosen to regard as "less sensitive"—WebPush's encryption-by-default design removes trust from the equation entirely and prevents of these sorts of "passive surveillance" attacks (except those based on timing or message length)
> You can then decrypt these notifications in the app
This pushes all of the complexity of supporting encryption onto the app developer, instead of having it be provided by the platform where it can be centrally audited and securely managed.
> However, we don't use the WebPush API between the push server and distributor since there is a lot of scope for innovation in that space (for example Google's FCM uses a custom XMPP based protocol).
Why not do this innovation in an open standards body like the IESG where other stakeholders can give feedback and review? The web push standard does allow for Push Servers to use other custom wire protocols besides the default RFC8030, but having open and standards-based discussion of the possible performance improvements would "lift all boats" in terms of being able to provide low-power devices with timely push notifications
While I understand that some developers may think of security as wasteful, having unencrypted data be the default relies on every developer to spend much more time considering the possible threat models and how . Case in point here is Matrix—the IDs they send aren't random at all, they actually send full room IDs and event IDs for each message, so you'd be able to build up someone's entire social graph just from reading their unencrypted push notification data and looking up room IDs on the Matrix server in question or even correlating room IDs across different Element installs. This also hurts users, since they're not in a position to inspect the source code and understand how or whether they need to trust their "push provider" with whatever data the app developers might have chosen to regard as "less sensitive"—WebPush's encryption-by-default design removes trust from the equation entirely and prevents of these sorts of "passive surveillance" attacks (except those based on timing or message length)
> You can then decrypt these notifications in the app
This pushes all of the complexity of supporting encryption onto the app developer, instead of having it be provided by the platform where it can be centrally audited and securely managed.
> However, we don't use the WebPush API between the push server and distributor since there is a lot of scope for innovation in that space (for example Google's FCM uses a custom XMPP based protocol).
Why not do this innovation in an open standards body like the IESG where other stakeholders can give feedback and review? The web push standard does allow for Push Servers to use other custom wire protocols besides the default RFC8030, but having open and standards-based discussion of the possible performance improvements would "lift all boats" in terms of being able to provide low-power devices with timely push notifications
This is partially explained in the blog post.
I'm personally using UnifiedPush HTTP endpoints as-if they were Web Push, and it just works. I handle decryption manually on the receiving end.
I'm personally using UnifiedPush HTTP endpoints as-if they were Web Push, and it just works. I handle decryption manually on the receiving end.
This isn't just for notifications to clients themselves, this would handle the federation of notification's.
Similar to how Apples push notification service works already (APN: https://en.wikipedia.org/wiki/Apple_Push_Notification_servic...)
Similar to how Apples push notification service works already (APN: https://en.wikipedia.org/wiki/Apple_Push_Notification_servic...)
Yes, the WebPush standard provides protocols for both. This is necessary because otherwise there wouldn't be an intercompatible way for backends to send notifications to different browsers. The protocol for the federation of notifications would be the most useful for other non-JS projects to reuse, so that backend developers can plug any native app in their existing webpush support without coding yet another system.
I think GP got it backwards, actually: Web Push seems to have a wire interface for submission and a JavaScript API for subscription, while the internals of how the notifications get from e.g. the FCM backend to the notification drawer on your Android phone remain unspecified (and partly secret). UnifiedPush has a wire interface for both submission and subscription+reception, plus an Android(?) intent API for interacting with the client for the latter that’s running on your phone.
It looks like the submission side of UnifiedPush could indeed have been Web Push-compatible but isn’t, though.
It looks like the submission side of UnifiedPush could indeed have been Web Push-compatible but isn’t, though.
> It looks like the submission side of UnifiedPush could indeed have been Web Push-compatible but isn’t, though.
Exactly. It was discussed a bit early on whether to mandate encryption, but simplicity was chosen instead.
Adjusting the server protocol to make it compatible with WebPush is also being discussed: https://github.com/UnifiedPush/wishlist/issues/15
That would be useful for supporting Telegram, for instance. I don't really see it becoming mandatory, however.
Exactly. It was discussed a bit early on whether to mandate encryption, but simplicity was chosen instead.
Adjusting the server protocol to make it compatible with WebPush is also being discussed: https://github.com/UnifiedPush/wishlist/issues/15
That would be useful for supporting Telegram, for instance. I don't really see it becoming mandatory, however.
> while the internals of how the notifications get from e.g. the FCM backend to the notification drawer on your Android phone remain unspecified (and partly secret)
Actually, the web push standard strongly recommends that you use RFC 8030 for this, but it does allow browsers to substitute other protocols as long as their semantics are the same https://datatracker.ietf.org/doc/html/rfc8030. I believe Mozilla and Edge both use HTTP Push, I'm not sure what Google uses on Desktop Chrome but I could see it going either way. Safari probably uses APNS.
Actually, the web push standard strongly recommends that you use RFC 8030 for this, but it does allow browsers to substitute other protocols as long as their semantics are the same https://datatracker.ietf.org/doc/html/rfc8030. I believe Mozilla and Edge both use HTTP Push, I'm not sure what Google uses on Desktop Chrome but I could see it going either way. Safari probably uses APNS.
I'm using Google Firebase Cloud Messaging [0] which allows pushing notifications to devices even if they're asleep, unlike local notification services where the device must be awake and the app must be active.
Will this service allow the same thing FCM does, including waking up the device? I'm looking at both iOS and Android devices, I see there's Flutter available but not iOS necessarily in the docs [1]. Related question, will the Flutter version also work for iOS devices? I know FCM (and its Flutter SDK) works for both, as well as web.
[0] https://firebase.google.com/products/cloud-messaging
[1] https://unifiedpush.org/developers/intro/
Will this service allow the same thing FCM does, including waking up the device? I'm looking at both iOS and Android devices, I see there's Flutter available but not iOS necessarily in the docs [1]. Related question, will the Flutter version also work for iOS devices? I know FCM (and its Flutter SDK) works for both, as well as web.
[0] https://firebase.google.com/products/cloud-messaging
[1] https://unifiedpush.org/developers/intro/
Yes, this is an FCM replacement on Android (and it can easily fall back to FCM if the user doesn't have UnifiedPush). There is no native UnifiedPush on iOS because Apple doesn't allow background services. The server side to support both is simple, but on the client you would need code for both FCM (for iOS) and UP (for Android). It would be interesting to build an FCM wrapper for iOS into the UP Flutter library.
I don't know too much about this, but is this not APNs [0]? I see there is a FCM integration (which is how I assume FCM does it anyway) [1] as well as a standalone Flutter implementation [2].
How would iOS native only apps on Swift do push notifications, surely Apple has their own service and everyone doesn't use Firebase?
[0] https://developer.apple.com/documentation/usernotifications/...
[1] https://firebase.flutter.dev/docs/messaging/apple-integratio...
[2] https://pub.dev/packages/flutter_apns
How would iOS native only apps on Swift do push notifications, surely Apple has their own service and everyone doesn't use Firebase?
[0] https://developer.apple.com/documentation/usernotifications/...
[1] https://firebase.flutter.dev/docs/messaging/apple-integratio...
[2] https://pub.dev/packages/flutter_apns
I recommended FCM for iOS instead of APNS because OP is already using it. For iOS clients, FCM just forwards notifications to APNS.
Interesting. This could have interest from China, since they have to use Android without the Google push service, causing battery drain issues. They are trying to come up with their own push notification service.
The issue is highlighted in this article [1].
[1] https://www.sixthtone.com/news/1011308/the-real-story-behind...
The issue is highlighted in this article [1].
[1] https://www.sixthtone.com/news/1011308/the-real-story-behind...
[deleted]
This is a nice concept, but I owned more than one Android phone that would ignore the battery optimization settings and kill the applications anyway.
I have been working and running ntfy [1] on my phone for a year now (ntfy is a UnifiedPush distributor), and it is true that Android does kill the app every now and then. But it is instantly restarted. Usually ntfy delivers notifications much much much faster than Google's FCM, especially in doze mode. FCM seems to try and conserve battery much more.
Disclaimer: I am the maintainer of ntfy.
[1] https://github.com/binwiederhier/ntfy
Disclaimer: I am the maintainer of ntfy.
[1] https://github.com/binwiederhier/ntfy
Sadly, I had phones where I couldn't keep anything running. Not sure what the manufacturer did there. Even the normal notifications didn't arrive at times. (E.g. Signal)
AFAIK Modern Android (>= 8.0) reserves the right to kill your app when running in the background unless there's a foreground toast bound (even then, can still be killed in low memory situations). How does this solution work around that? Is there a need to opt into the legacy "battery saver disabled" mode for the app which more closely mirrors <= 7.0 execution model?
Most distributors both show a foreground notification at all times and require exempting them from battery optimization. Because of its efficient design, something like ntfy only uses a few percent of battery per day [1].
System-level integration into Android ROMs would eliminate the need for those things.
[1] https://docs.ntfy.sh/faq/?h=battery#how-much-battery-does-th...
System-level integration into Android ROMs would eliminate the need for those things.
[1] https://docs.ntfy.sh/faq/?h=battery#how-much-battery-does-th...
Thanks! As I thought. Long lived sticky foreground notifications tend to annoy folks from my experience. Have you seen complaints?
I haven't seen any complaints, but folks who set this up are usually used to having persistent notifications (I currently have 6) without Google services.
At least I replaced the one from my Matrix client with the UnifiedPush-enabled ntfy. Hopefully it'll replace Signal too at some point. KDE Connect? Why not. K-9 mail? I hope, but that would require JMAP or IMAP adjustments.
One option would be to install the distributor as a system app.
At least I replaced the one from my Matrix client with the UnifiedPush-enabled ntfy. Hopefully it'll replace Signal too at some point. KDE Connect? Why not. K-9 mail? I hope, but that would require JMAP or IMAP adjustments.
One option would be to install the distributor as a system app.
You can easily remove them. There are instructions here: https://docs.ntfy.sh/subscribe/phone/#instant-delivery
The headline got my attention. I'm looking for a standard for subscribeable notifications / callbacks to update users of data changes in an API. Bonus point for federated. But server to server, not server to client. Double bonus points for connecting to consumer services like Zapier or IFTTT.
I'm planning to use WebSub and/or RSS, but on the look out for others. Any pointers?
I'm planning to use WebSub and/or RSS, but on the look out for others. Any pointers?
WebSub + RSS/Atom is a good option for relatively slowly changing data. (Maybe at most a couple of times an hour). It is nice because it is opt-in and an enhancement on the regular polling API.
Using WebSub or UnifiedPush would technically very similar but you need a way for the client to pass you the subscription information. WebSub has a standard discovery process to manage this.
Using WebSub or UnifiedPush would technically very similar but you need a way for the client to pass you the subscription information. WebSub has a standard discovery process to manage this.
The WebSub spec looks disarmingly simple but also complete for everything I can think of. I've not yet tried to implement it, but I'm waiting for the catch! Thanks for the confirmation.
That use-case sounds a bit similar to Webmentions: https://en.wikipedia.org/wiki/Webmention
See also... The See also section in that article :)
You say it would be server to server, but are both publicly reachable?
See also... The See also section in that article :)
You say it would be server to server, but are both publicly reachable?
We're running an API, we want to say to users e.g. "we finished processing your submission" or "this API resource changed". And offer pull (e.g. RSS) and push flavour.
> You say it would be server to server, but are both publicly reachable?
So yes, the user would have to run a public server. But only for practical reasons, otherwise how can you push.
Webmention isn't quite it, we're notifying about our own API Resources.
Websub (FKA PubSubHubub) does. It just doesn't seem to be widely adopted for some reason.
> You say it would be server to server, but are both publicly reachable?
So yes, the user would have to run a public server. But only for practical reasons, otherwise how can you push.
Webmention isn't quite it, we're notifying about our own API Resources.
Websub (FKA PubSubHubub) does. It just doesn't seem to be widely adopted for some reason.
Please don’t get me wrong but I wonder what is the use case for this. First of all this tries to solve a problem that has already been solved by the mobile device OS layer within IOS and Android and furthermore is only working for Android. Why should anyone use this?
It looks like this aims to be a replacement for people want to use an Android-based OS but don't want to rely on Google services, normally for reasons of philosophical objection to big tech's de facto surveillance capabilities. So presumably users of GrapheneOS, CalyxOS, LineageOS, etc.
Also since Google is blocked in China, Firebase Cloud Messaging doesn't work there. Android has something like a 75% market share in that market. Some vendors have their own offerings (e.g. Xiaomi MiPush, Huawei PushKit) etc, I could buy that a popular centralised solution would be more attractive than doing N integrations.
Also since Google is blocked in China, Firebase Cloud Messaging doesn't work there. Android has something like a 75% market share in that market. Some vendors have their own offerings (e.g. Xiaomi MiPush, Huawei PushKit) etc, I could buy that a popular centralised solution would be more attractive than doing N integrations.
This. It also works without internet connection if you want push notifications within a LAN.
The vision is that you only need to implement support for 1 integration: UnifiedPush, possibly by embedding one of the libraries that manage it and fallbacks. You automatically gain support for compatible alternatives.
There's a variety of reasons why people can't or don't want to rely on Google services. Before UnifiedPush, one would have to "hardcode" an alternative, or more commonly implement a persistent connection within the app.
This also has some potential for more efficient push implementation: for instance, if you have access to an SMS gateway, you could use that to push notification without enabling data connection.
The specification also makes it easier to support third-party operating systems and clients.
The vision is that you only need to implement support for 1 integration: UnifiedPush, possibly by embedding one of the libraries that manage it and fallbacks. You automatically gain support for compatible alternatives.
There's a variety of reasons why people can't or don't want to rely on Google services. Before UnifiedPush, one would have to "hardcode" an alternative, or more commonly implement a persistent connection within the app.
This also has some potential for more efficient push implementation: for instance, if you have access to an SMS gateway, you could use that to push notification without enabling data connection.
The specification also makes it easier to support third-party operating systems and clients.
Thanks for explaining and giving context - I was totally not aware of this total block of Google in the Chinese market.
Apps draining your battery to check for notifications is the worst thing about degoogled Android. With notifications enabled my telegram client can drain my battery in a very low number of hours (got suggestions here? any client supporting UnifiedPush?)
So this is welcome, but it requires application support. And if apps on your phone use both GCM and UnifiedPush you still end up with multiple providers. But at least I think I can trust UnifiedPush to be light on my battery.
So this is welcome, but it requires application support. And if apps on your phone use both GCM and UnifiedPush you still end up with multiple providers. But at least I think I can trust UnifiedPush to be light on my battery.
> telegram client can drain my battery in a very low number of hours (got suggestions here? any client supporting UnifiedPush?)
It's possible. There is nothing ready right now.
[1] https://github.com/UnifiedPush/wishlist/issues/18
[2] https://github.com/Telegram-FOSS-Team/Telegram-FOSS/issues/5...
It's possible. There is nothing ready right now.
[1] https://github.com/UnifiedPush/wishlist/issues/18
[2] https://github.com/Telegram-FOSS-Team/Telegram-FOSS/issues/5...
It's really not that bad. It uses 1% of battery for 17h. Sometimes even less than that and it shows 0% after the entire day.
So it's a hypothetical problem IMHO. There are very few cases or phones i see these days where people complain about battery: https://docs.ntfy.sh/faq/#how-much-battery-does-the-android-...
So it's a hypothetical problem IMHO. There are very few cases or phones i see these days where people complain about battery: https://docs.ntfy.sh/faq/#how-much-battery-does-the-android-...
The reason I said it is that it's a problem for me. Notifications on/off makes a night and day difference. Did you link ntfy because that's what Telegram FOSS uses?
[deleted]
Looks like a cool solution to one of the biggest issues in FOSS on Android.
the main issue isn't with the foss apps though. Those usually support polling. The actual problem is with other proprietary spyware apps that you're forced to use but they are locking you in with google's FCM. Until there's an easy way to patch those apps we can't really solve this problem
Yeah, I wouldn't even think about a degoogled phone at this point, none of my downloadable wiretaps would work, and I use a lot of them.
But this at least gives FOSS a chance to compete on battery life without FCM fallback like some do.
Most likely FOSS will never get full parity, Android might be gone and replaced long before then, but the ability to make an app that's F-droid legal is still a big deal.
But this at least gives FOSS a chance to compete on battery life without FCM fallback like some do.
Most likely FOSS will never get full parity, Android might be gone and replaced long before then, but the ability to make an app that's F-droid legal is still a big deal.
hmm, I think most spyware apps still work without google services (on GrapheneOS), but you'll have to go without notification, among a few other things. If you need a tracking devi.. cough.. googled phone for work only, you can consider using a separate phone for it, and a personal degoogled phone.
I use a googled for both home and work pretty extensively, including Google Calendar, Keep, and Assistant, plus Tile and YoLink(Which loses most of it's value without notifications).
Carrying two phones seems like a lot of stress, I generally like to minimize the number of objects in my life as much as I can.
Carrying two phones seems like a lot of stress, I generally like to minimize the number of objects in my life as much as I can.
This sounds like it could be the reverse of REST, so could be pretty important and perhaps it can grow into a W3C spec.
That's an interesting thought, but currently the Server-Server specification is quite trivial, the more complicated part is the Android (and Linux) API, which I don't think the W3C would cover.
There's definitely room for adoption within the freedesktop.org umbrella for the Linux part, once this part is a bit more mature.
There's definitely room for adoption within the freedesktop.org umbrella for the Linux part, once this part is a bit more mature.
Any possibility this will mean I can push a notification from my home automation that gets displayed on my Google TV?
If it can call a site, it can use ntfy.
Does anyone know how de-Googled Chinese phones work? How do programs like Wechat go about it?
Would this be vulnerable to spam?
Each application gets a unique URL, that is supposedly kept secret from any other party. They are also very easy to rotate, as the specification advises to ask for a new one (that could be the same) on every app launch.
Moreover, when something is sent to that URL, it gets forwarded to the application, which then interprets it before displaying a notification (for most apps).
By spam, I think you meant someone sending messages to random people's notification area. No, this is highly unlikely.
Moreover, when something is sent to that URL, it gets forwarded to the application, which then interprets it before displaying a notification (for most apps).
By spam, I think you meant someone sending messages to random people's notification area. No, this is highly unlikely.
I wish we could extend SMTP to support notifications as self-expiring emails
It would also be nice to extend IMAP to support UnifiedPush...
The use-case would be new mail notifications; it wouldn't be too hard to do, and would be quite useful, as I typically have multiple e-mail accounts.
You could however build a UnifiedPush-compliant SMTP-based or IMAP-based distributor.
The use-case would be new mail notifications; it wouldn't be too hard to do, and would be quite useful, as I typically have multiple e-mail accounts.
You could however build a UnifiedPush-compliant SMTP-based or IMAP-based distributor.
JMAP has a push implementation I really like, and it's natively compatible with UnifiedPush. Would be nice to see more adoption there.
https://jmap.io/spec-core.html#push
https://jmap.io/spec-core.html#push
Notifications would work (and does), maybe a sieve rule for deleting old messages?
>However, if each app actively maintains a server connection, the OS cannot suspend them.
This is a limitation with Android and not a fundamental network/OS limitation. As long as your protocol allows for long enough keep alive latency you can periodically wake up for a short period and allow apps to service their connections (which is what I'd imagine this does anyway just with a singular app.) This works on Linux if you don't have the extra Android crap getting in the way.
It's kind of a bummer to see bad OS architecture feed into over-complicated protocol/application design.
EDIT: I guess I wasn't clear here. In my experiments I had the background wake up service wake the device up once and applications had just that singular window for all of them to service their connections. The problem with Android is instead applications either keep the phone from suspending entirely or (apparently) have some API allowing them to register their own periodic wake up that isn't shared.
EDIT2: I also want to be clear that this isn't a criticism of NTFY. The developers behind it deserve congratulations for solving a serious problem with FOSS Android apps. My complaint is that the poor architecture and unescisary inflexibility of Android made this such a complex problem in the first place.
This is a limitation with Android and not a fundamental network/OS limitation. As long as your protocol allows for long enough keep alive latency you can periodically wake up for a short period and allow apps to service their connections (which is what I'd imagine this does anyway just with a singular app.) This works on Linux if you don't have the extra Android crap getting in the way.
It's kind of a bummer to see bad OS architecture feed into over-complicated protocol/application design.
EDIT: I guess I wasn't clear here. In my experiments I had the background wake up service wake the device up once and applications had just that singular window for all of them to service their connections. The problem with Android is instead applications either keep the phone from suspending entirely or (apparently) have some API allowing them to register their own periodic wake up that isn't shared.
EDIT2: I also want to be clear that this isn't a criticism of NTFY. The developers behind it deserve congratulations for solving a serious problem with FOSS Android apps. My complaint is that the poor architecture and unescisary inflexibility of Android made this such a complex problem in the first place.
> you can periodically wake up for a short period and allow apps to service their connections
For this to work, you need apps to somehow conform to a standard timing and respond to a signal that says "hey, the OS is awake for you to do network things". Now each developer needs to implement some special code that understands the situation. After a while, all the devs start using libSleepyNetwork because getting those details right is hard. The libSleepyNetwork people realize that they can use a system daemon to further coalesce work for greater efficiency (one event loop is better than N event loops), so now all the consumer program C APIs are now backed by IPCs to the daemon.
This is exactly what's already happened on Android, except libSleepyNetwork is implemented by either Google, or these people. I don't think it's bad OS design at all. It's chasing efficiency via abstraction that encapsulates some complex coordination.
For this to work, you need apps to somehow conform to a standard timing and respond to a signal that says "hey, the OS is awake for you to do network things". Now each developer needs to implement some special code that understands the situation. After a while, all the devs start using libSleepyNetwork because getting those details right is hard. The libSleepyNetwork people realize that they can use a system daemon to further coalesce work for greater efficiency (one event loop is better than N event loops), so now all the consumer program C APIs are now backed by IPCs to the daemon.
This is exactly what's already happened on Android, except libSleepyNetwork is implemented by either Google, or these people. I don't think it's bad OS design at all. It's chasing efficiency via abstraction that encapsulates some complex coordination.
> This is a limitation with Android and not a fundamental network/OS limitation. As long as your protocol allows for long enough keep alive latency you can periodically wake up for a short period and allow apps to service their connections (which is what I'd imagine this does anyway just with a singular app.)
The protocol does not specify how the distributor and push server communicate or keep a connection going. It's up to them to define an energy efficient way to do that. For ntfy [1], a single WebSocket or JSON stream connection is used to deliver messages to the device. This is very energy efficient and consumes <1% of battery on most phones.
As for regular wakeups (for polling), Android does not allow periodic work to be done more frequently than every 15 minutes, which is obviously not frequent enough to be useful.
> It's kind of a bummer to see bad OS architecture feed into over-complicated protocol/application design.
This is somewhat true. The restrictive Android eco-system is what led to the creation of UnifiedPush, but it's far from complicated IMHO
Disclaimer: I am the maintainer of ntfy [1], one of the UnifiedPush distributors.
[1] https://ntfy.sh
The protocol does not specify how the distributor and push server communicate or keep a connection going. It's up to them to define an energy efficient way to do that. For ntfy [1], a single WebSocket or JSON stream connection is used to deliver messages to the device. This is very energy efficient and consumes <1% of battery on most phones.
As for regular wakeups (for polling), Android does not allow periodic work to be done more frequently than every 15 minutes, which is obviously not frequent enough to be useful.
> It's kind of a bummer to see bad OS architecture feed into over-complicated protocol/application design.
This is somewhat true. The restrictive Android eco-system is what led to the creation of UnifiedPush, but it's far from complicated IMHO
Disclaimer: I am the maintainer of ntfy [1], one of the UnifiedPush distributors.
[1] https://ntfy.sh
could the 15 minute limit be circumvented with multiple identical apps (different package names) installed?
I mean yes, but if Google catches you, you're not going to have a good time. It's also incredibly inconvenient.
Also, you cannot specify the exact time when it's supposed to run, so it's likely they will just run at the same time.
Also, you cannot specify the exact time when it's supposed to run, so it's likely they will just run at the same time.
You can do that, but if keep alive are not synchronized (I don't think there is a mechanism to do so), you will wake more often if you keep more connections open.
I think the OS could synchronize them if keep alive packets are sent at the TCP level, but that would be hard to do at a higher level.
Letting a single program handle persistent connections leaves a lot more potential for energy efficiency optimizations.
I think the OS could synchronize them if keep alive packets are sent at the TCP level, but that would be hard to do at a higher level.
Letting a single program handle persistent connections leaves a lot more potential for energy efficiency optimizations.
Google famously tried allowing app keepalives to be synchronised... The scheduler allowed a wakeup in 'approximately' X minutes, and the OS would wake up and run all apps callbacks at once. If any app needed a wakeup at a precise time, then other callbacks at nearby approximate times would be run at the same time too.
Overall, the thought was that wakeups are expensive, wakeups that involve the network are particularly expensive, so might as well run them all at once (they tend not to be CPU bound - often much of the wakeup handler is spent waiting for the network to connect and send a packet or an ack to arrive back).
This turned out to be a very very bad idea. All it takes is some app to have a 'wake up exactly on the hour', and now all other app callbacks will also run exactly on the hour. End result: Mobile networks get 10 million phones all waking up within a few milliseconds of one another, and everyone wanting to connect, and the whole mobile network fails because it was never designed to have 10 million phones all trying to connect within a few milliseconds.
That idea was scrapped.
Overall, the thought was that wakeups are expensive, wakeups that involve the network are particularly expensive, so might as well run them all at once (they tend not to be CPU bound - often much of the wakeup handler is spent waiting for the network to connect and send a packet or an ack to arrive back).
This turned out to be a very very bad idea. All it takes is some app to have a 'wake up exactly on the hour', and now all other app callbacks will also run exactly on the hour. End result: Mobile networks get 10 million phones all waking up within a few milliseconds of one another, and everyone wanting to connect, and the whole mobile network fails because it was never designed to have 10 million phones all trying to connect within a few milliseconds.
That idea was scrapped.
Seems like adding a random deviation to wakeup times would solve that problem.
> if keep alive packets are sent at the TCP leve
Unfortunately there are lots of bits of hardware in mobile networks that fake acks - ie. TCP level acks will be received very quickly to indicate the mobile network has accepted the data, and will deliver it at some future time out to the internet.
The problem is they still send acks even if the remote end is no longer responsive.
So you can't reliably use TCP acks to know if a connection is still alive. You need to send actual data and have the application respond.
Unfortunately there are lots of bits of hardware in mobile networks that fake acks - ie. TCP level acks will be received very quickly to indicate the mobile network has accepted the data, and will deliver it at some future time out to the internet.
The problem is they still send acks even if the remote end is no longer responsive.
So you can't reliably use TCP acks to know if a connection is still alive. You need to send actual data and have the application respond.
Well that sounds utterly repulsive. And somewhat pointless.
In fact, I choose not to believe it. Is there some document discussing this kind of functionality?
In fact, I choose not to believe it. Is there some document discussing this kind of functionality?
Products like this: https://www.enea.com/solutions/traffic-management/tcp-accele...
I would think that in order to not have an obvious negative impact on battery life, it would be necessary for the OS to coalesce app notification service keepalives, performing them in scheduled batches instead of the moment an app requests one. That should be roughly equal to the current situation with APNS/GCN, where the OS can schedule network maintenance during periods where the device and antenna(s) are likely to be awake already.
this is NOT a limitation of Android. Android has supported wakelock based notification managers for a long time. And FCM works pretty much the same as was mentioned in the OP. The FCM manager maintains one long polled connection to Google notification servers and every other app subscribes to wakelocks from that manager.
Your app is only woken up by the OS if FCM receives a notification for you. For e.g. Xiaomi will not wake you up even if you get a notification, unless you're on a white list (the much glorified Xiaomi whitelist). When Android is put in power saving mode, the FCM manager optimized this even further.
Your app is only woken up by the OS if FCM receives a notification for you. For e.g. Xiaomi will not wake you up even if you get a notification, unless you're on a white list (the much glorified Xiaomi whitelist). When Android is put in power saving mode, the FCM manager optimized this even further.
Even if this were all synchronized, you’d still need to have all the apps memory state available - either taking up loads of RAM or requiring a lot of expensive paging.
There are two things I still don't get about software development:
- why is web development such a big thing
- why is app development a thing at all
- why is web development such a big thing
- why is app development a thing at all
Oh gosh, a wheel reinvented again? XMPP did it ~7 years ago [0] and we have used it for quite a few applications already.
[0]: https://xmpp.org/extensions/xep-0357.html
[0]: https://xmpp.org/extensions/xep-0357.html
It seems like you have misunderstood something.
I just read the specification you linked. If I understand it right, it just specifies what we call a "Push gateway" in our last diagram in the "under the hood" section.
As in, it specifies how multiple servers can connect to a single (XMPP-speaking) push server. This is also evident from the excerpt I'm quoting:
I just read the specification you linked. If I understand it right, it just specifies what we call a "Push gateway" in our last diagram in the "under the hood" section.
As in, it specifies how multiple servers can connect to a single (XMPP-speaking) push server. This is also evident from the excerpt I'm quoting:
>XMPP Push works between the user's XMPP server and two push notification services in tandem:
> The user's XMPP server publishes notifications to the XMPP Push Service of each of the user's client applications.
> The XMPP Push Service (as defined here) for a client application then delivers the notification to a third-party notification delivery service.
> The third-party (and potentially proprietary or platform-dependent) push service delivers the notification from the client application's backend service to the user's device.
You seem to not see the forest behind the trees. It doesn't do 100% of what your UnifiedPush does, sure, but it does 99%. All it needs to reach the final 1% is to send a message with an agreed payload format to a client. A day of work.
As I said, we did it as far back as 2016: a watered-down XMPP client (a "distributor", as you call it) was used to deliver push notifications to a few different apps on devices that didn't have GCM (as it was called then).
> Please make an effort to come across as less antagonizing in the future.
Someone has to antagonize people who ignore prior art.
As I said, we did it as far back as 2016: a watered-down XMPP client (a "distributor", as you call it) was used to deliver push notifications to a few different apps on devices that didn't have GCM (as it was called then).
> Please make an effort to come across as less antagonizing in the future.
Someone has to antagonize people who ignore prior art.
> You seem to not see the forest behind the trees
I almost used my sentence myself. The overlap is almost zero.
> All it needs to reach the final 1% is to send a message with an agreed payload format to a client.
This is the part we're interested in. We standardize this part and make it modular. The core issue is that multiple "client"s that do not interact with each other traditionally each implement their own payload format and background connection. Which drains battery. We propose a mechanism to delegate that to a third-party that can then multiplex those messages. That's all.
It's not a lot of work per se, but coming up with a modular API is not that trivial either, and we have to spread the word so that it can be used by multiple projects.
Lastly, you seem pretty focused on XMPP clients. Unfortunately (or maybe fortunately), not every app uses XMPP. On my phone, apps that could theoretically use UnifiedPush are: NextCloud, Etar Calendar, DavX5 caldav synchronization, K-9 mail, Element & others Matrix clients, HomeAssistant, FindMy & other device locators, covid tracker, Carnet notes application, Fennec (Firefox), KDE Connect, Mobilizon, Peertube Client, UnCiv, and various others, including chat apps and weather apps. XMPP would bring very little to those, especially as they just need that "1%" functionality you quoted.
There is some overlap indeed, as there is a server-server API. However, 99% of the spec you linked is completely useless or overkill for our use-case. We just accept POST requests to an endpoint and forward them to the app running on a user's device.
> As I said, we did it as far back as 2016: a watered-down XMPP client (a "distributor", as you call it) was used to deliver push notifications to a few different apps on devices that didn't have GCM (as it was called then).
That's pretty good indeed; was the specification open so that other apps could join in? Do you mind sharing a link? You can just implement the UnifiedPush specification in that distributor, make sure the server-side part accepts requests in the format specified, and you have a UnifiedPush-compliant application that other apps can use.
I've been looking for such a specification for a while, there's been some talks around the topic for years, including OpenPush [1], but nothing concrete or public that I know of. The link you shared in your previous comment does not address the "distributor" part.
> Someone has to antagonize people who ignore prior art.
Well, please at least try to understand the thing you're criticizing before, that's the bare minimum.
[1]: https://bubu1.eu/openpush/
I almost used my sentence myself. The overlap is almost zero.
> All it needs to reach the final 1% is to send a message with an agreed payload format to a client.
This is the part we're interested in. We standardize this part and make it modular. The core issue is that multiple "client"s that do not interact with each other traditionally each implement their own payload format and background connection. Which drains battery. We propose a mechanism to delegate that to a third-party that can then multiplex those messages. That's all.
It's not a lot of work per se, but coming up with a modular API is not that trivial either, and we have to spread the word so that it can be used by multiple projects.
Lastly, you seem pretty focused on XMPP clients. Unfortunately (or maybe fortunately), not every app uses XMPP. On my phone, apps that could theoretically use UnifiedPush are: NextCloud, Etar Calendar, DavX5 caldav synchronization, K-9 mail, Element & others Matrix clients, HomeAssistant, FindMy & other device locators, covid tracker, Carnet notes application, Fennec (Firefox), KDE Connect, Mobilizon, Peertube Client, UnCiv, and various others, including chat apps and weather apps. XMPP would bring very little to those, especially as they just need that "1%" functionality you quoted.
There is some overlap indeed, as there is a server-server API. However, 99% of the spec you linked is completely useless or overkill for our use-case. We just accept POST requests to an endpoint and forward them to the app running on a user's device.
> As I said, we did it as far back as 2016: a watered-down XMPP client (a "distributor", as you call it) was used to deliver push notifications to a few different apps on devices that didn't have GCM (as it was called then).
That's pretty good indeed; was the specification open so that other apps could join in? Do you mind sharing a link? You can just implement the UnifiedPush specification in that distributor, make sure the server-side part accepts requests in the format specified, and you have a UnifiedPush-compliant application that other apps can use.
I've been looking for such a specification for a while, there's been some talks around the topic for years, including OpenPush [1], but nothing concrete or public that I know of. The link you shared in your previous comment does not address the "distributor" part.
> Someone has to antagonize people who ignore prior art.
Well, please at least try to understand the thing you're criticizing before, that's the bare minimum.
[1]: https://bubu1.eu/openpush/
> Unfortunately (or maybe fortunately), not every app uses XMPP.
App doesn't need to 'use' XMPP to use push notifications. For developers, it is essentially identical to using FCM. XMPP part here simply offers a very effective way to decentralize the service, as federation server-to-server part in XMPP is wonderful, and top XMPP servers like ejabberd have great performance.
> Well, please at least try to understand the thing you're criticizing before, that's the bare minimum.
My company has created this very thing for a customer years ago, so I think I pretty much do understand what you are creating.
It wasn't open, yet, the effort to make it open would be pretty small. We could probably release it, but I don't see the real need for it, as it is a likely dead end: making apps run in the background on Android was what we did for years, and the writing is on the wall. Background running becomes more and more difficult with each new version of Android, even on pure Android phones. Biggest manufacturers, however, like Samsung, Huawei, Xiaomi simply kill off background processes without mercy. So it is more than likely we won't be able to run 'dispatcher' apps on Android, just like we can't on iOS, and we'll have to rely solely on push notification service which is built into the phone OS. If there is a fight worth fighting, it is the one to force phone OS developers to open this part of an OS to allow using custom push notifications services.
App doesn't need to 'use' XMPP to use push notifications. For developers, it is essentially identical to using FCM. XMPP part here simply offers a very effective way to decentralize the service, as federation server-to-server part in XMPP is wonderful, and top XMPP servers like ejabberd have great performance.
> Well, please at least try to understand the thing you're criticizing before, that's the bare minimum.
My company has created this very thing for a customer years ago, so I think I pretty much do understand what you are creating.
It wasn't open, yet, the effort to make it open would be pretty small. We could probably release it, but I don't see the real need for it, as it is a likely dead end: making apps run in the background on Android was what we did for years, and the writing is on the wall. Background running becomes more and more difficult with each new version of Android, even on pure Android phones. Biggest manufacturers, however, like Samsung, Huawei, Xiaomi simply kill off background processes without mercy. So it is more than likely we won't be able to run 'dispatcher' apps on Android, just like we can't on iOS, and we'll have to rely solely on push notification service which is built into the phone OS. If there is a fight worth fighting, it is the one to force phone OS developers to open this part of an OS to allow using custom push notifications services.
I am not trying to denigrate XMPP in any way, I also use it, though I am not very familiar with the internals. Here, the decentralized aspect doesn't seem to bring a lot of added value (what would you federate? Push senders and push servers? The relationship is mostly 1:1 in every case).
> It wasn't open
Then, I'm sorry, but even if the design was miles ahead of what is being proposed here, it might as well not have existed.
> If there is a fight worth fighting, it is the one to force phone OS developers to open this part of an OS to allow using custom push notifications services.
Well, we're proposing an API for it, and we will certainly push alternative OSes such as LineageOS, /e/ and others to adopt it; we hope it gets picked up by bigger fish, but we can't exactly pressure them into doing it, besides demonstrating that it works, documenting it and raising awareness, which is what we're doing here.
> It wasn't open
Then, I'm sorry, but even if the design was miles ahead of what is being proposed here, it might as well not have existed.
> If there is a fight worth fighting, it is the one to force phone OS developers to open this part of an OS to allow using custom push notifications services.
Well, we're proposing an API for it, and we will certainly push alternative OSes such as LineageOS, /e/ and others to adopt it; we hope it gets picked up by bigger fish, but we can't exactly pressure them into doing it, besides demonstrating that it works, documenting it and raising awareness, which is what we're doing here.
Please be nice. You can convey your information without the snarky tone. You may be factually correct, but your comment comes across quite rude.
From the HN guidelines [1]:
> Be kind. Don't be snarky. Have curious conversation; don't cross-examine. Please don't fulminate. Please don't sneer, including at the rest of the community. Edit out swipes.
[1] https://news.ycombinator.com/newsguidelines.html
From the HN guidelines [1]:
> Be kind. Don't be snarky. Have curious conversation; don't cross-examine. Please don't fulminate. Please don't sneer, including at the rest of the community. Edit out swipes.
[1] https://news.ycombinator.com/newsguidelines.html
From the blog post:
> One key feature of UnifiedPush is that the communication between the push server and the distributor is not specified. This means that a variety of technologies can be employed, such as WebSockets, Server-Sent Events, *XMPP*, raw TCP, or even SMS, whatever works best for the user.
> One key feature of UnifiedPush is that the communication between the push server and the distributor is not specified. This means that a variety of technologies can be employed, such as WebSockets, Server-Sent Events, *XMPP*, raw TCP, or even SMS, whatever works best for the user.
Sigh. What you miss is that this protocol introduces nothing that already isn't done by XMPP push notifications. It is decentralized, it can deliver to different endpoints, etc. You only need to create a dispatcher app that will relay the notifications to third party apps. Again, this too was done years ago.
> You only need to create a dispatcher app that will relay the notifications to third party apps.
That part is what specify UnifiedPush. It does not specify the communication between the push server and the distributor.
That part is what specify UnifiedPush. It does not specify the communication between the push server and the distributor.
So what! Few use that. I host a prosody server with lots of plugins and it never occurred to me to use it, let alone use cases outside if the XMPP ecosystem. Can the XMPP implementation fall back to google's push system. Does it have a blog entry, lovely diagrams, comprehensive docs, a nice on-boarding experience for developers? Ideas aren't worth much, implementation--getting things done is what matters.
UnifiedPush has been a thing for a while, see the official website: https://unifiedpush.org/
With this blog post, we† tried to clarify a few names. Feel free to point out things that are still unclear, the documentation is pretty much a WIP.
UnifiedPush aims at replacing the push notifications mechanism provided by Google services with something independent, that anyone can self-host or any OS can provide without compatibility issues. It does require small adjustments server-side and client-side for applications assuming Google services.
There is also a Matrix room at #unifiedpush:matrix.org and a Mastodon account at https://fosstodon.org/@unifiedpush
Note: we'll be monitoring this topic for a while, you don't have to reply to this thread.
† karmanyaahm and S1m are main authors