LocalStorage or WebSQL unexpectedly cleared(bugs.chromium.org)
bugs.chromium.org
LocalStorage or WebSQL unexpectedly cleared
https://bugs.chromium.org/p/chromium/issues/detail?id=1033655
13 comments
This actually hit me really hard. I made a PWA I used to track certain stats but the data wasn't so important that it needed to be stored in a database.
Instead I had an import/export for manual backups and assumed that the data inside indexedDb wouldn't be dropped.
Bad assumption.
Instead I had an import/export for manual backups and assumed that the data inside indexedDb wouldn't be dropped.
Bad assumption.
Yes, same for me. Also a bad assumption I made: People probably have enough storage space. The number of people running around with older devices that have 16 GB of internal storage is still like 5% or so. Add 32 GB devices, probably 10%. And most of them are constantly full.
Btw. the same is true of older iOS devices. They are constantly almost out of space and there is not enough free space to update to the most recent iOS version. It'd probably work with iTunes, but many people don't install iTunes. So they're stuck with iOS 10, although their device would still be capable of running iOS 13, simply because they don't have enough free space.
Btw. the same is true of older iOS devices. They are constantly almost out of space and there is not enough free space to update to the most recent iOS version. It'd probably work with iTunes, but many people don't install iTunes. So they're stuck with iOS 10, although their device would still be capable of running iOS 13, simply because they don't have enough free space.
I'm trying to figure out if they deserve the punishment they get from Apple for not installing itunes...
Ftw SQLite primary with Indexed DB and LokiJS wrapper for in-mem.
Apple taught this lesson a few years ago, when iOS 5.1 changed the local storage location: https://phonegap.com/blog/2012/04/18/ios-5-1-and-the-embedde...
But this behavior is within the range of spec compliance: "User agents may, possibly in a manner configured by the user, automatically delete stored data after a period of time." https://www.w3.org/TR/webdatabase/#privacy https://www.w3.org/TR/webstorage/#user-tracking
So these browser storage mechanisms must be considered cache, not primary storage of important data.
But this behavior is within the range of spec compliance: "User agents may, possibly in a manner configured by the user, automatically delete stored data after a period of time." https://www.w3.org/TR/webdatabase/#privacy https://www.w3.org/TR/webstorage/#user-tracking
So these browser storage mechanisms must be considered cache, not primary storage of important data.
> So these browser storage mechanisms must be considered cache, not primary storage of important data.
Maybe I’m an exception, but I’ve always viewed these storage mechanisms as unreliable. I primarily use them for storing things like user filter settings on a list view. It never even occurred to me to use them for anything critical.
Maybe I’m an exception, but I’ve always viewed these storage mechanisms as unreliable. I primarily use them for storing things like user filter settings on a list view. It never even occurred to me to use them for anything critical.
Blaming people for not using beta version for six weeks to test and inform them prior of the issue. Rich. Millions of apps affected.
That seems like an uncharitbale reading. A manager let people know that they'll be able to tell if a new chrome release will affect them by testing with beta. The overwhelming sentiment was "we're sorry".
When Google introduced Chrome in 2008 (via comic!), they claimed that within 20-30 minutes of a beta build, it was automatically tested on tens of thousands of the most popular webpages. Is that no longer true? Does that not include any which use local storage?
AFAIK it is still true, but this bug is specifically with upgrading. Basically all the testing occurs with an empty profile; migrations are specifically undertested.
Also note that this bug only affected WebViews and not Chrome. Not sure how testing of migrations compares between the two.
Also note that this bug only affected WebViews and not Chrome. Not sure how testing of migrations compares between the two.
[deleted]
I was bit by this in a cordova app on a past update. After that I backed up api keys, etc to native storage using a cordova plugin and added a wrapper that will fall back to it if localstorage lookups fail.
I would advise others to do the something similar since unless you are packaging the browser too (like crosswalk did), then that piece is really out of your control and you shouldn't expect it to be retained.
A similar issue can be seen on iOS under low storage conditions where the operating system will clear the browser cache to reclaim space.
I would advise others to do the something similar since unless you are packaging the browser too (like crosswalk did), then that piece is really out of your control and you shouldn't expect it to be retained.
A similar issue can be seen on iOS under low storage conditions where the operating system will clear the browser cache to reclaim space.
I can only recommend you switch fully to SQLite or native storage. localStorage has always been unreliable and could be cleared any time on iOS.
> a wrapper that will fall back to it if localstorage lookups fail.
If I understand correctly, the scenario is that saved data in localStorage is lost, so that won't help.
If I understand correctly, the scenario is that saved data in localStorage is lost, so that won't help.
“It” in this case is native storage, not localStorage
Yes, that was clear. You say it uses localstorage, and only fallback to native when localstorage is not available. This is not the scenario of the bug. Here, localstorage is available, it just has been wiped out.
Native storage protects against this scenario, though. It isn't safe to assume that localstorage will be retained.
Which plugin?
This plugin has worked reliably on both Android and iOS and has a simple api:
https://github.com/TheCocoaProject/cordova-plugin-nativestor...
https://github.com/TheCocoaProject/cordova-plugin-nativestor...
If would guess the cordova sqlite plugin
Not to downplay the issue, but what a ridiculous bug report:
> It causes such a big problem for our products!!!! Who is in charge?
I'd have expected more from a tech focused company like pocketpos.
> It causes such a big problem for our products!!!! Who is in charge?
I'd have expected more from a tech focused company like pocketpos.
i still don't understand the core issue: why is anyone saving essential data to localstorage? isn't that supposed to be a state-related, somewhat temporary (due to its nature) storage in-between the UI and the API/database?
don't you lose all your data when you use a different pc or smartphone?
i see a lot of mentions of cordova apps. are people publishing apps without a backend (and save everything in localstorage)?
i've worked on pwa's in the past but server sync was always a priority in my algorithms.
does anyone have any realistic use cases where localstorage is so essential that clearing it would somehow lose tons of data?
i see a lot of mentions of cordova apps. are people publishing apps without a backend (and save everything in localstorage)?
i've worked on pwa's in the past but server sync was always a priority in my algorithms.
does anyone have any realistic use cases where localstorage is so essential that clearing it would somehow lose tons of data?
Depends on the context. I've built Cordova/Phonegap apps in the past where you want things to work offline and sync once the user goes online. If the time from user going offline to online again is too long / device storage too little, that data that would eventually need to be synced, would be lost.
Always ways of working around it, like saving to files or whatever, but localStorage is a nice interface that works both for Cordova and client-side web apps.
Always ways of working around it, like saving to files or whatever, but localStorage is a nice interface that works both for Cordova and client-side web apps.
agree. for the use case that you mentioned there would be a problem.
but as you said, it could be overcome by using native apis that store state. it's even imperative in case of secrets, like access tokens and the like.
for client side web apps there's a bigger problem with localstorage. from my experience end users switch browsers and platforms. and they expect consistency from their web sites/apps. this expectation is much more intense than for native apps. so, while practical for developers, i'm not sure it's that welcomed in web land except for always having a local backup or as a temporary storage to alleviate network issues.
but as you said, it could be overcome by using native apis that store state. it's even imperative in case of secrets, like access tokens and the like.
for client side web apps there's a bigger problem with localstorage. from my experience end users switch browsers and platforms. and they expect consistency from their web sites/apps. this expectation is much more intense than for native apps. so, while practical for developers, i'm not sure it's that welcomed in web land except for always having a local backup or as a temporary storage to alleviate network issues.
>i see a lot of mentions of cordova apps. are people publishing apps without a backend (and save everything in localstorage)?
Yes. Why not? You can write a Cordova app that doesn't even have a backend.
Yes. Why not? You can write a Cordova app that doesn't even have a backend.
of course you can.
but since every device is as persistent as its weakest point (e.g. physical damage, loss/theft, etc) shouldn't apps that need to save essential/important data basically give up the argument of persistence because of using localstorage in the first place? your data is one breakdown/loss away from permanent destruction when using localstorage exclusively.
that is why, from my experience, you always rely on the server and only use localstorage as a backup in case the server is down, there's no internet, etc.
claiming massive loss of data because localstorage got emptied is basically saying "our app wasn't designed correctly".
what am i missing?
but since every device is as persistent as its weakest point (e.g. physical damage, loss/theft, etc) shouldn't apps that need to save essential/important data basically give up the argument of persistence because of using localstorage in the first place? your data is one breakdown/loss away from permanent destruction when using localstorage exclusively.
that is why, from my experience, you always rely on the server and only use localstorage as a backup in case the server is down, there's no internet, etc.
claiming massive loss of data because localstorage got emptied is basically saying "our app wasn't designed correctly".
what am i missing?
If my phone breaks, I lose my data, and that is my responsibility. You are saying that everything should be backed up on the cloud, I think that's crazy.
localstorage disappearing overnight IS a bug. If I put stuff in My Documents folder on Windows and Windows decides to delete it, that IS a bug too. You could argue that I should have backups, and you would be right, but that's no reason for Windows to delete my files, and that is no reason for me to never use my hard disk.
localstorage disappearing overnight IS a bug. If I put stuff in My Documents folder on Windows and Windows decides to delete it, that IS a bug too. You could argue that I should have backups, and you would be right, but that's no reason for Windows to delete my files, and that is no reason for me to never use my hard disk.
of course not everything should be backed up to the cloud.
but if it's important and only available on your phone, then yes please back it up, or better yet, have it up there in the cloud.
regarding your comparison between a windows folder and localstorage feels a bit exaggerated.
yes, the Chrome team really fumbled it. and yes localstorage and important data should be mutually exclusive.
regarding your comparison between a windows folder and localstorage feels a bit exaggerated.
yes, the Chrome team really fumbled it. and yes localstorage and important data should be mutually exclusive.
You're right that important data needs to be backed up, but I disagree that it is the app builder's responsibility to do that. It is the end-user's responsibility to think about backups, to choose where to backup and how often. From the app developer's perspective, it is commendable that, in this day and age, they allowed users to retain control of their own data, instead of automatically uploading the user's data to the developer's server.
Local storage and/or SQL DB were touted as features for exactly this use case. That Google did not consider their persistence as that important for the release is perhaps not surprising for a company used to storing all your data in their cloud (not claiming they did so knowingly, just that it's probably in their culture, just as in your comments above).
Local storage and/or SQL DB were touted as features for exactly this use case. That Google did not consider their persistence as that important for the release is perhaps not surprising for a company used to storing all your data in their cloud (not claiming they did so knowingly, just that it's probably in their culture, just as in your comments above).
interesting take. i agree with it in spirit.
but unfortunately i don't think your average user has any idea about what you wrote.
but unfortunately i don't think your average user has any idea about what you wrote.
...localstorage and important data should be mutually exclusive.
It seems like we could say "don't trust this" about any aspect of any platform. Categorizing every application as "really important, don't update the UI until securely written to three AZs" and "totally ephemeral purge every minute" excludes a lot of stuff in the middle. Sure, nothing made by humans is perfect, but many things could be improved. One improvement to localstorage would be to simply not delete all the data.
It seems like we could say "don't trust this" about any aspect of any platform. Categorizing every application as "really important, don't update the UI until securely written to three AZs" and "totally ephemeral purge every minute" excludes a lot of stuff in the middle. Sure, nothing made by humans is perfect, but many things could be improved. One improvement to localstorage would be to simply not delete all the data.
Are you serious? Maybe you missed the entire privacy conversation around cloud storage in the last couple of years. Also, with your reasoning, your entire laptop should merely be a client for a remote storage.
Saving stuff locally is good for privacy, for performance, for offline-capabilities. You can back up your entire phone LOCALLY, to have a solution of your stated problems (theft, physical damage).
Saving stuff locally is good for privacy, for performance, for offline-capabilities. You can back up your entire phone LOCALLY, to have a solution of your stated problems (theft, physical damage).
i am very serious. people lose their devices all the time.
never save important data on a single device. you simply delay the inevitable "i lost my data" scenario.
non-important data on the other hand should have no problems living on a single device. or even localstorage.
and as i said before, i don't think your average user has any ideas about backing up. that's why it almost always comes back to the developers to implement it.
never save important data on a single device. you simply delay the inevitable "i lost my data" scenario.
non-important data on the other hand should have no problems living on a single device. or even localstorage.
and as i said before, i don't think your average user has any ideas about backing up. that's why it almost always comes back to the developers to implement it.
I wrote a PWA which functioned as a calories counter and exercise tracker.
I stored the data on the client using indexedDB, offering an export/import option for manual backups.
I wasn't interested in the data and it's simpler to just store it on the client rather than setting up the infrastructure required to centrally persist this information.
It's really unfortunate that it's not a reliable storage medium.
I stored the data on the client using indexedDB, offering an export/import option for manual backups.
I wasn't interested in the data and it's simpler to just store it on the client rather than setting up the infrastructure required to centrally persist this information.
It's really unfortunate that it's not a reliable storage medium.
> It's really unfortunate that it's not a reliable storage medium.
That's not what it's for, you're just misusing the tool for something it isn't meant to be. It's not a database, it's a cache.
That's not what it's for, you're just misusing the tool for something it isn't meant to be. It's not a database, it's a cache.
In today's GDPR world, a lot of companies want to see as little of the users private data as possible.
If their app can avoid sending any data to a server and run fully in a web browser offline, they'll do that.
If their app can avoid sending any data to a server and run fully in a web browser offline, they'll do that.
Ouch! A recent Reddit post immediately came to mind:
https://www.reddit.com/r/androiddev/comments/ea86jt/users_re...
https://www.reddit.com/r/androiddev/comments/ea86jt/users_re...
looks like a big f_ckup...
From comment #26:
> why it was missed: test apps that don't happen to use local storage will not have a file created :(
From comment #26:
> why it was missed: test apps that don't happen to use local storage will not have a file created :(
Just how fast are Chrome rollouts on Android? Given how destructive this appears to have been, it seems amazing that they got to a 50% rollout before the first bug report.
Real life consequences (from comment #63):
>I heard from a company that uses localstorage for offline no-connection available that had local record of animals getting vaccionation. The update "erased" all the data. They don't know which animals got vaccine and can't repeat on all of them. Serious stuff.
https://bugs.chromium.org/p/chromium/issues/detail?id=103365...
>I heard from a company that uses localstorage for offline no-connection available that had local record of animals getting vaccionation. The update "erased" all the data. They don't know which animals got vaccine and can't repeat on all of them. Serious stuff.
https://bugs.chromium.org/p/chromium/issues/detail?id=103365...
You should apply a rule similar to 3-2-1:
3 copies of the data
2 copies on different storage mediums
1 backup in a separate location.
Ideally, a backup solution on the OS would target the browser profile itself at regular intervals, and you could allow the user to export the data into a file that can be downloaded by the browser (or copy/pasted).
It should be obvious that any form of browser storage is unreliable, and MUST be treated as such, and it should be EXPECTED to be deleted at a moments notice - setting such expectations encourages good practice at backing up your data incase of a failure.
Don't rely on others to ensure business critical data is stored and accessible - that's partially on the users/application to offer what they can.
3 copies of the data
2 copies on different storage mediums
1 backup in a separate location.
Ideally, a backup solution on the OS would target the browser profile itself at regular intervals, and you could allow the user to export the data into a file that can be downloaded by the browser (or copy/pasted).
It should be obvious that any form of browser storage is unreliable, and MUST be treated as such, and it should be EXPECTED to be deleted at a moments notice - setting such expectations encourages good practice at backing up your data incase of a failure.
Don't rely on others to ensure business critical data is stored and accessible - that's partially on the users/application to offer what they can.
Sounds like it's their fault for not backing up such important information.
"offline no-connection available"
Probably the veterinary work is not taking place at Starbucks...
Probably the veterinary work is not taking place at Starbucks...
This has been happening in Chrome extensions for years now. I’ve tried reporting it so many times. In our case it happens randomly so it’s impossible to reproduce, which inevitably leads to ”issue closed - couldn’t reproduce”.
Who uses localStorage for more than a local cache, or a short lived log of records to be synced somewhere into some serious database asap?
That people are storing/losing:
- years of financial data
- animal vaccination records
- etc.
because of this is so weird.
That people are storing/losing:
- years of financial data
- animal vaccination records
- etc.
because of this is so weird.
Even then, it can still be used for offline data... you collect/perform tasks without an internet connection and it syncs later. Chrome updates before data sync, boom, all your data is gone.
Now, as for years of data, I mostly agree, this should absolutely be backed up and/or synced more regularly.
Now, as for years of data, I mostly agree, this should absolutely be backed up and/or synced more regularly.
That still seems like a poor design. This would also happen if you cleared localStorage before the data is synced. If, for example, when you came back online and were trying to do something on an unrelated site, hit a bug and tech support asked you to clear your browser.
For something as critical as financial records or vaccine records, the sensible thing might be to not work in offline mode, or to persist to native storage.
I think it's fine as a cache for offline data for a lot of use cases (i.e. task trackers, note apps, shopping lists, etc). But it creates an unacceptable risk for business critical data, imo.
For something as critical as financial records or vaccine records, the sensible thing might be to not work in offline mode, or to persist to native storage.
I think it's fine as a cache for offline data for a lot of use cases (i.e. task trackers, note apps, shopping lists, etc). But it creates an unacceptable risk for business critical data, imo.
Developer writes POC/Demo/quick hack without the necessary warnings.
Much later a user (or less aware developer) finds the thing and starts using it.
Maybe software needs GHS labels like chemicals have.
Much later a user (or less aware developer) finds the thing and starts using it.
Maybe software needs GHS labels like chemicals have.
Storing financial data in localstorage. Lol how stupid can people be
This also servers as a warning for PWAs: despite all the hype and virtues (real or perceived), your are at the hands of very fickle gods
Don't you have the same problems with native? It's like like no OS update ever broke apps, or that Apple doesn't capriciously remove APIs, or reject you from the app store sometimes for unexplained reasons.
Hell, people get broken by OSS ecosystem libraries a lot.
Hell, people get broken by OSS ecosystem libraries a lot.
True to some extent, though I don't think there have been many OS updates which deleted all storage for all apps (well, moved to an inaccessible place)... This really is a massive failure of QA and dev.
Wish Apple and Google would put more browser collaboration time in, for the benefit of all..constant one upmanship hurts both and millions of man hours in bugs, pollyfills, API repairs, and on and on. Webview, WKWebview teams need more rigorous standards and safety checks clearly for consistency and reliability. Progressive development benefits both. Web-apps are a critical, large and important part of the app developer ecosystem.
Yup. In the end you are at the mercy of whoever controls the platform and its development.
That's not just true for PWAs.
That's why offsite data backups are a really good thing.
That's why offsite data backups are a really good thing.
Indeed, I don't mean to downplay the frustration of the developers and their customers. However, valuable data should really not be kept on a single mobile device, which is prone to damage, loss or stealing.
We all are at the fikle hands of Apple and Google sadly.
Wrong. Relying on being allowed on some appstore is much more like being "at the hands of very fickle gods". Pray that apple/google don't suddenly change their mind and lock you out, because it competes or goes against whatever new trend they adopt this year.
That's cynical. You must have your own bugs every now and then. This is no different.
PS: google employee, used to work with torne every now and then, but no stake in the conversation anymore.
PS: google employee, used to work with torne every now and then, but no stake in the conversation anymore.
Cynical? Yes. Is that the state of the world? Yes.
Browsers are platforms that are as complex as OSes these days. And any bugs or changes in how they work will affect you app if your app is PWA.
- It could be Chrome breaking audio: https://danshumway.com/blog/chrome-autoplay/
- It could be Chrome breaking LocalStorage: this discussion
- It could be Chrome removing support for a web standard that was implemented and shipped: https://www.chromestatus.com/feature/4642138092470272
I'm linking Chrome issues here because Chrome is the most visible of all browsers with the largest marketshare, but other browsers have their issues, too.
Browsers are platforms that are as complex as OSes these days. And any bugs or changes in how they work will affect you app if your app is PWA.
- It could be Chrome breaking audio: https://danshumway.com/blog/chrome-autoplay/
- It could be Chrome breaking LocalStorage: this discussion
- It could be Chrome removing support for a web standard that was implemented and shipped: https://www.chromestatus.com/feature/4642138092470272
I'm linking Chrome issues here because Chrome is the most visible of all browsers with the largest marketshare, but other browsers have their issues, too.
Just to set the record straight on the part I'm familiar with, "Custom Elements V0" was never a web standard. It was a proposal, which Chrome implemented and shipped without getting everyone else on board, and which turned out to have significant enough problems that a somewhat different thing got standardized and shipped in all browsers ("Custom Elements V1").
Once something is shipped, people will rely on it. For example, quite famously, Youtube did for a very long time (until this year IIRC). AMP used to use it.
There are several polyfills available. Several Polymer versions targeted v0. Etc. etc.
It's not specifically Google's fault that they are removing support for these, but, once again, that's the state of the world: browsers are lrage complex platforms that may and will break your app.
There are several polyfills available. Several Polymer versions targeted v0. Etc. etc.
It's not specifically Google's fault that they are removing support for these, but, once again, that's the state of the world: browsers are lrage complex platforms that may and will break your app.
Sure. Unshipping things is always hard, and likely to break people. But there's still a difference between unshipping support for a standard and unshipping support for a non-standard technology that only you ever implemented. In the latter case, people should be a lot more wary of depending on the technology in the first place.
While it's true that browser updates may break your app, it's also true that there are ways to write the app so as to minimize the risk of such breakage. Using Custom Elements V0 was _not_ a good way to avoid potential breakage...
While it's true that browser updates may break your app, it's also true that there are ways to write the app so as to minimize the risk of such breakage. Using Custom Elements V0 was _not_ a good way to avoid potential breakage...
On sane OSes native is nice.
On most consumer OSes I honestly would trust the chrome developers to be more well behaved.
On most consumer OSes I honestly would trust the chrome developers to be more well behaved.
It took them 6 weeks to push a fix. I used the opportunity to switch to Sqlite and can only recommend others the same.