UPSes suck and need to be disrupted(esr.ibiblio.org)
esr.ibiblio.org
UPSes suck and need to be disrupted
http://esr.ibiblio.org/?p=7839
14 comments
I think that what the author is really looking for is a battery inverter setup and this is quite a mature market. A lot of countries out there do not have stable electricity supply and these things work for all appliances. Something like this (https://bluettipower.co.uk/products/bluetti-ac500-b300s-home...) although there are a huge range of brands and prices in this space.
Pc UPS’s are underpowered and the market is so swamped with cheap garbage as the rant suggests that it is difficult to find anything decent.
Pc UPS’s are underpowered and the market is so swamped with cheap garbage as the rant suggests that it is difficult to find anything decent.
I mentioned this below in a reply to a similar suggestion - It is critical to remember that most of the 'battery house backup solutions' do not have fast enough switch time to switch without rebooting some computers.
Most of the battery systems (like EcoFlow, Bluetti) has switching times that are between 30 and 100ms. The larger Enphase ones are ~100ms, and the Tesla battery system is around 25ms.
Online conversion UPSs have 0ms, and switching ones are usually in the 4-8ms range. Some systems can handle 30ms, most can't handle 100ms.
Most of the battery systems (like EcoFlow, Bluetti) has switching times that are between 30 and 100ms. The larger Enphase ones are ~100ms, and the Tesla battery system is around 25ms.
Online conversion UPSs have 0ms, and switching ones are usually in the 4-8ms range. Some systems can handle 30ms, most can't handle 100ms.
I wonder if there is a market for flicker preventer which stops devices from restsrting on quick drops. I have more things that don’t want rebooting than ones need to keep running.
I don’t know if could use capacitor or need battery. If need lots of battery for load, might end up with UPS. It would need to be cheap.
I don’t know if could use capacitor or need battery. If need lots of battery for load, might end up with UPS. It would need to be cheap.
Well, a flicker preventer is a double on-line UPS, so the market has a device, albeit an expensive one. You could purchase a double on-line UPS that has small batteries and that would serve as a 'flicker preventer' while providing only a small bit of runtime.
It would be possible to build this kind of on-line conversion with a very very small battery that could only provide power for a few seconds, and that kind of device would serve the goal you mentioned. I suspect the cost for this device would quickly approach a UPS in cost, so it might not be the 'cheap' solution.
It would be possible to build this kind of on-line conversion with a very very small battery that could only provide power for a few seconds, and that kind of device would serve the goal you mentioned. I suspect the cost for this device would quickly approach a UPS in cost, so it might not be the 'cheap' solution.
There is a market, and it has players.
One way is small, inexpensive standby UPS: Something with fast switchover, but without the capacity to survive anything longer than necessary to avoid some flickering and to conduct an orderly shutdown if the badness persists. This is the cheapest way. (And for small low-voltage loads like Raspberry Pis or consumer routers, there are kits available using things like 18650 cells.)
Another way is a small-ish online UPS. These have no switchover time, but they're more expensive because the parts are bigger, and they're less efficient than a standby UPS.
Another way is a ferroresonant transformer. This isn't cheap, but it is lasting (no moving parts or chemistry to degrade) and can work through some flickering.
And finally: Motor-generators, with a flywheel to pick up the slack. These work fine, but they're big, alarmingly expensive, require maintenance sometimes, and there are good reasons to be scared of spinning flywheels.
One way is small, inexpensive standby UPS: Something with fast switchover, but without the capacity to survive anything longer than necessary to avoid some flickering and to conduct an orderly shutdown if the badness persists. This is the cheapest way. (And for small low-voltage loads like Raspberry Pis or consumer routers, there are kits available using things like 18650 cells.)
Another way is a small-ish online UPS. These have no switchover time, but they're more expensive because the parts are bigger, and they're less efficient than a standby UPS.
Another way is a ferroresonant transformer. This isn't cheap, but it is lasting (no moving parts or chemistry to degrade) and can work through some flickering.
And finally: Motor-generators, with a flywheel to pick up the slack. These work fine, but they're big, alarmingly expensive, require maintenance sometimes, and there are good reasons to be scared of spinning flywheels.
That is a lot more expensive than a UPS!
I did use PC UPSs quite successfully in the country with frequent power cuts.
On the other hand I do take the point that the inverter would not fail loudly as one of my UPSs did when my ex-wife decided to plug a hairdryer into it.
I did use PC UPSs quite successfully in the country with frequent power cuts.
On the other hand I do take the point that the inverter would not fail loudly as one of my UPSs did when my ex-wife decided to plug a hairdryer into it.
> UPSes could be designed with the kind of deep-cycle gel batteries used for marine applications like trolling motors to last even longer and be even more reliable.
Don't put a deep-cycle SLA or gel-cell battery in your UPS. It will last a much shorter amount of time (2-3 years vs 5+). I do not remember the specifics but it has to do with how big the plates are and how thick the internal connections are in the battery. Designing this chemistry of battery is about trade-offs and unless you greatly oversize the cells and do your own install externally it will wreck your battery longevity.
If you want to maintain spec in something like an APC UPS, buy CSB batteries (which is what APC slaps their sticker on) and get something from their GPL line https://www.csb-battery.com.tw/english/01_product/00_overvie...
The real disruption for UPS's are the switch from SLA to LFP cells, but power density isn't as good so they need to be sized up a decent amount to handle standard loads. But the chemistry is a ton better.
Don't put a deep-cycle SLA or gel-cell battery in your UPS. It will last a much shorter amount of time (2-3 years vs 5+). I do not remember the specifics but it has to do with how big the plates are and how thick the internal connections are in the battery. Designing this chemistry of battery is about trade-offs and unless you greatly oversize the cells and do your own install externally it will wreck your battery longevity.
If you want to maintain spec in something like an APC UPS, buy CSB batteries (which is what APC slaps their sticker on) and get something from their GPL line https://www.csb-battery.com.tw/english/01_product/00_overvie...
The real disruption for UPS's are the switch from SLA to LFP cells, but power density isn't as good so they need to be sized up a decent amount to handle standard loads. But the chemistry is a ton better.
LFP has much higher energy density (100 Wh/kg) than lead acid (30 Wh/kg). The lower weight and size is big advantage of LFP.
LFP also can be fully discharged while lead acid can only safely go to 50% which means more battery for same capacity. LFP is limited to 80% charging to preserve lifespan but still comes out ahead.
LFP also can be fully discharged while lead acid can only safely go to 50% which means more battery for same capacity. LFP is limited to 80% charging to preserve lifespan but still comes out ahead.
Poster said power density, and I don't really believe theyre' right (see:a123 batteries, and now there are plenty of competitors)
However that's different than you're saying, which is about energy capacity.
I believe they are mistaken because many lfp replacement batteries have BMS units that are usually the limiter. But you can also buy lfp starter batteries
However that's different than you're saying, which is about energy capacity.
I believe they are mistaken because many lfp replacement batteries have BMS units that are usually the limiter. But you can also buy lfp starter batteries
Yes, they are annoying. And you don't always know when your battery is reaching end of life. After having to take a last minute flight to fix a problem caused by a brief power outage I decided to mod the UPS to run off of deep cycle marine batteries. This is a 1500W APC unit. I picked up a couple of the biggest batteries they had at walmart, cut a hole in the side of the UPS, soldered in some 2 AWG wire, and connected the batteries in series with some ring terminals. The batteries charge up fine and they have gotten the servers through several multi-hour outages. It's coming up on five years now and they don't have the capacity that they used to. I'll probably replace them soon.
What the author wants exists, just not by the names that the author is likely searching for.
Victron makes combination charger-inverters that takes power from the grid and keeps separate batteries topped off and also supplies AC power via inverter. They sell these as their 'Multigrid' and 'MultiPlus-II' products and are populat setups for boats, where you have shore power but need to be able to unplug the shore power at any time and keep things going.
It's also a feature of quite a few cheap hybrid solar inverters now- they can charge batteries from mains if there is no PV input.
These work as a UPS but you do constantly run the load thru the inverter, thus your are taking a small efficiency hit; however it is truly uninterruptible. There is no 'switchover' time, which is also what makes a lot of UPS hardware so specialized- many are doing mains switching in essentially 1 cycle of 50/60hz mains power. Theres all sorts of stuff that goes with that.
Meanwhile, charger-inverter combos ditch having to deal with that by just going AC-DC to battery and then DC-AC. You always have clean uninterrupted power from the inverter. This used to fuel reliability concerns but these days much of this hardware is in applications running 24/7 for years now.
Victron makes combination charger-inverters that takes power from the grid and keeps separate batteries topped off and also supplies AC power via inverter. They sell these as their 'Multigrid' and 'MultiPlus-II' products and are populat setups for boats, where you have shore power but need to be able to unplug the shore power at any time and keep things going.
It's also a feature of quite a few cheap hybrid solar inverters now- they can charge batteries from mains if there is no PV input.
These work as a UPS but you do constantly run the load thru the inverter, thus your are taking a small efficiency hit; however it is truly uninterruptible. There is no 'switchover' time, which is also what makes a lot of UPS hardware so specialized- many are doing mains switching in essentially 1 cycle of 50/60hz mains power. Theres all sorts of stuff that goes with that.
Meanwhile, charger-inverter combos ditch having to deal with that by just going AC-DC to battery and then DC-AC. You always have clean uninterrupted power from the inverter. This used to fuel reliability concerns but these days much of this hardware is in applications running 24/7 for years now.
The funny thing here is that the "power station" market has this quite solved. Comes with an app minimum usually, which at least is a start for reverse engineering. If you're lucky you might even get a local diag socket. Unfortunately this comes with the usual IoT issues, but at least you can watch things. With better transfer hardware youd have a much better user experience for upses. I wonder why no one has done this yet.
Is using a power station as a UPS feasible? One of the issues I've read is that most don't have AC pass through, so it's always charging and discharging the battery.
However that seems a bit odd (with my limited electronics knowledge) as batteries don't have an 'in' and 'out' port. The AC charger input, inverter and battery are all connected to the same 12V (or whatever internal voltage it uses) rail. I imagine as long as the inverter output is less than the charger input, it won't touch the battery, as the resistance will be higher.
However that seems a bit odd (with my limited electronics knowledge) as batteries don't have an 'in' and 'out' port. The AC charger input, inverter and battery are all connected to the same 12V (or whatever internal voltage it uses) rail. I imagine as long as the inverter output is less than the charger input, it won't touch the battery, as the resistance will be higher.
The one I've used (ecoflow) has a passthrough and auto transfer option. 15ms which isn't great but I've tied it up to a small online ups and had no issues. The online ups eats the transfer time and otherwise keeps the power smooth
Think of it more that the battery sits on the side of the DC bus.
Double conversation UPS are generally the highest end (rectified AC > DC > inverted AC) because there's no switching time between the inverter taking power from the DC bus being supplied by the rectifier or the battery. It is always inverting. The power quality never changes too because of the always inverting setup.
Double conversation UPS are generally the highest end (rectified AC > DC > inverted AC) because there's no switching time between the inverter taking power from the DC bus being supplied by the rectifier or the battery. It is always inverting. The power quality never changes too because of the always inverting setup.
Even if it never touches the battery, I'd imagine there's a noticeable efficiency loss running from inverter at all times, which means higher power and cooling costs.
That is correct - double on-line conversion UPSs are usually in the 92-94% efficiency. Mine is almost always at 94% at 60% load. However that is the cost of on-line conversion and is well worth it.
> that could be assembled cheaply in a makerspace from off-the-shelf components, an Arduino-class microcontroller, and a PROM.
Excuse me? You can get off-the-shelf batteries (although LFP batteries with real BMSes are not actually interchangeable, sadly). But you also need the power electronics, and you should not be doing that as a DIY project unless you really know what you’re doing.
I would look to the off-grid/RV space for disruption. For example, here’s a “portable power station” for $600. It costs about 2x as much as a random 1500VA APC UPS I looked it, except that it can deliver twice the actual power (if your UPS is powering a modern power-factor-corrected PSU, you just need watts and the VA isn’t a big deal, and APC is cheating their specs by including unnecessary reactive power capacity in their headline number) and 20x the energy storage and thus runtime. APC tries hard to obscure very low capacity of their batteries.
No one will buy an off-grid setup that can barely power an espresso machine long enough for a single shot.
Excuse me? You can get off-the-shelf batteries (although LFP batteries with real BMSes are not actually interchangeable, sadly). But you also need the power electronics, and you should not be doing that as a DIY project unless you really know what you’re doing.
I would look to the off-grid/RV space for disruption. For example, here’s a “portable power station” for $600. It costs about 2x as much as a random 1500VA APC UPS I looked it, except that it can deliver twice the actual power (if your UPS is powering a modern power-factor-corrected PSU, you just need watts and the VA isn’t a big deal, and APC is cheating their specs by including unnecessary reactive power capacity in their headline number) and 20x the energy storage and thus runtime. APC tries hard to obscure very low capacity of their batteries.
No one will buy an off-grid setup that can barely power an espresso machine long enough for a single shot.
I'm running a Renogy inverter/charger 2000W with a single 12V 100Ah LiFePo battery that run 3 Odroid H2's (Proxmox cluster 1), 3 Beelink's (docker swarm cluster) and various switches and ISP device. There is also a dual gpu desktop that pulls 400-450 watts when running folding at home. It will carry that load for 6+ hours from an 80% charge which is were I keep it for long term battery health; though I do shutdown the dual gpu if the power is out for more than a few minutes.
The Renogy will pass 'shore power' directly through and switch (purportly) in <10ms although I'm not confident enough that it is quick enough (1/2 a cycle at 60hz) so I run the standard consumer ups downstream.
I'm using a pretty stock RV circuit layout. The one thing I did do is limit the max charging amps to 20 to ensure that the charging load plus the working load stayed well in the capacity of the 110V/15A line.
I did try a Bluetti 3BA but it never worked as expected; ie it would cut out at unexpected times and firmware updates are problematic.
The Renogy will pass 'shore power' directly through and switch (purportly) in <10ms although I'm not confident enough that it is quick enough (1/2 a cycle at 60hz) so I run the standard consumer ups downstream.
I'm using a pretty stock RV circuit layout. The one thing I did do is limit the max charging amps to 20 to ensure that the charging load plus the working load stayed well in the capacity of the 110V/15A line.
I did try a Bluetti 3BA but it never worked as expected; ie it would cut out at unexpected times and firmware updates are problematic.
Anedoctaly, just last month, my APC UPS started complaining that the battery was dead.
I bought a 3rd party battery replacement (12V), opened the unit (which is not made easy by APC) and was getting ready to replace the batteries, when I measured the current and it was 12.5V. The replacement one, measured 12.3V
I still have no idea why it's complaining the battery is going bad.
I bought a 3rd party battery replacement (12V), opened the unit (which is not made easy by APC) and was getting ready to replace the batteries, when I measured the current and it was 12.5V. The replacement one, measured 12.3V
I still have no idea why it's complaining the battery is going bad.
Battery voltage does not accurately reflect capacity. Only a load test will determine battery health. Much like a body of water can be silted up and shallow, a battery will accumulate internal damage that prevents it from providing sufficient power while under load.
The new battery may not have arrived fully charged. If it has been installed for a little while and given a chance to charge fully you should measure a voltage greater than 12.6. (disconnect battery and let it rest a few minutes before testing). If the voltage is below 12.6 you might have gotten a dud. Do a load test on the UPS to see how long it can power something like a 100 watt incandescent bulb.
The new battery may not have arrived fully charged. If it has been installed for a little while and given a chance to charge fully you should measure a voltage greater than 12.6. (disconnect battery and let it rest a few minutes before testing). If the voltage is below 12.6 you might have gotten a dud. Do a load test on the UPS to see how long it can power something like a 100 watt incandescent bulb.
Thanks for the explanation, I will try that.
That's a well-known problem with APC UPSes using non-APC batteries. The simple physical solution on one of my 1500s (that does not disrupt power to the clients) is to hold down the power button for about 10 seconds. It will beep, then beep again, at which point you can release the power button. If, after a clunking noise, the display still shows the same battery problem, repeat the procedure and it should go away until next time, whenever that may be. I have apcupsd running on a Raspberry Pi via USB and configured it to send a text to my mobile when the battery complaint happens so that I can perform the button hold solution. So far the periodicity is about a month between such events. I dislike having to do all this, but it is not a show stopper.
Pro-tip: don't buy your UPS online, buy at a local store with a reasonable return policy (e.g. CostCo).
I have had duds across three major brands (e.g. die within a year), and not having to ship the product back to the OEM will save you time and money.
I have had duds across three major brands (e.g. die within a year), and not having to ship the product back to the OEM will save you time and money.
As far as battery chemistry, SLAB is a cost tradeoff against your "threats". Our threat is midwestern storms in the spring that cause power interruptions. These are usually momentary (10s) or 30m or less. Anything longer and we'd send people home for the day. Preventing a network reboot during those momentary outages is the goal. Extended runtime just isn't needed, but a decently sized UPS can power one our typical floor racks for about 2-4 hours. We're likely only going to ever deploy SLAB because it's so danged cheap for something thats only used a handful a times a year.
I've found that Eaton UPSs pretty much knock out a lot of the concerns listed. I've got 9 of them deployed for floor/closet racks and a raspberry pi to interrogate the UPS with Telegraf, and those are shipped to an InfluxDB instance. Back in the day we would have used SNMP, but that isn't cool anymore... I also have recently deployed a few over the higher end Cyberpower UPSs to see how they do long term (these are a little bit of a meme brand it seems like).
That all being said, the lower end Eaton UPSs do not have a way to "Remote Test" (basically they lack a relay to isolate from mains and take the load onto battery). Higher end UPSs can do these commands over USB. One of the help desk tasks is once a quarter, visit each UPS and yank the power for 2m while Telegraf measures the battery drain stats. Not a huge problem, but a mundane task.
I've found that Eaton UPSs pretty much knock out a lot of the concerns listed. I've got 9 of them deployed for floor/closet racks and a raspberry pi to interrogate the UPS with Telegraf, and those are shipped to an InfluxDB instance. Back in the day we would have used SNMP, but that isn't cool anymore... I also have recently deployed a few over the higher end Cyberpower UPSs to see how they do long term (these are a little bit of a meme brand it seems like).
That all being said, the lower end Eaton UPSs do not have a way to "Remote Test" (basically they lack a relay to isolate from mains and take the load onto battery). Higher end UPSs can do these commands over USB. One of the help desk tasks is once a quarter, visit each UPS and yank the power for 2m while Telegraf measures the battery drain stats. Not a huge problem, but a mundane task.
APC has gone downhill in the time I've been a sysadmin. Eaton is the way to go for commercial units. Don't purchase anything without a bypass or else the UPS will be dead when the battery dies, as decent UPSs operate in on-line mode.
I am concerned about newish implementation of li-ion cells in commercial UPSs. They are not designed to stay at capacity for a long time. That coupled with the volatility of li-ion, means there may be more datacenter fires in the future.
I am concerned about newish implementation of li-ion cells in commercial UPSs. They are not designed to stay at capacity for a long time. That coupled with the volatility of li-ion, means there may be more datacenter fires in the future.
I think whole-home battery backup (charging from solar panels on your roof) using more modern battery chemistry is probably a better solution if you're able to do that.
I have this in my house, but it does not provide what a UPS does. I have 42kwh of Enphase batteries as well as 53 solar panels (20kw) with enphase microinverters, and the battery system is inline with the incoming power. If we lose power the batteries kick in, however the time for them to kick is in >100ms. Computers will reboot with that timeframe of power drop.
In my case I use a 16kw APC Symmetry UPS (which is datacenter UPS) for the interconnetion to my dataroom. The UPS provides double on-line conversion as well as battery backup. However the batteries in the UPS are only large enough to provide for about 30 mins of runtime at my current sustained power rate (~6kw)
My goal with the UPS is to have both the inline conversion protection as well as the protection from power drops.
As the author mentioned, UPSes do suck, and the Symmetra I have does as well. I have replaced 3 power modules (which are about $1k each), one network module, and all of the batteries. And it has an alarm right now about one of the battery banks being bad. ;)
(For those interested, there is a bit more detail about my power system in a garage build thread: https://www.garagejournal.com/forum/threads/jeffs-mountain-s...)
In my case I use a 16kw APC Symmetry UPS (which is datacenter UPS) for the interconnetion to my dataroom. The UPS provides double on-line conversion as well as battery backup. However the batteries in the UPS are only large enough to provide for about 30 mins of runtime at my current sustained power rate (~6kw)
My goal with the UPS is to have both the inline conversion protection as well as the protection from power drops.
As the author mentioned, UPSes do suck, and the Symmetra I have does as well. I have replaced 3 power modules (which are about $1k each), one network module, and all of the batteries. And it has an alarm right now about one of the battery banks being bad. ;)
(For those interested, there is a bit more detail about my power system in a garage build thread: https://www.garagejournal.com/forum/threads/jeffs-mountain-s...)
Jesus... 6kW...
I'd be paying more than 1000€/mo for that
I'd be paying more than 1000€/mo for that
Indeed, power in the NW US, which is significantly hydro-power, is relatively inexpensive. 6kw over a month is about $500/month(~4.3MWh), and my solar system generates about 2MWh/month.
Extravagant.
Biggest beef with consumer-grade is when battery dies, the entire UPS fails ... EVEN WHEN A/C POWER IS STILL BEING SUPPLIED.
100% of them.
100% of them.
I had to replace battery recently on APC UPS and it didn’t lose power. But did make a continuous alert tone that was so annoying that turned off UPS and pulled it out.
That's just not true.
on shitty ones maybe, decent ones with bypass not so much
I have one of those absolutely terrible cheap Cyberpower ones, and I actually use it with the battery completely removed for some weird reasons. Works fine, though it gets upset and wants to sound its buzzer even with that disabled in its configuration utility. A few dabs with the desoldering tool fixed that.
But really, don't buy those things they are awful as a UPS. Never did work reliably.
But really, don't buy those things they are awful as a UPS. Never did work reliably.
Shitty UPSes are the ones which does NOT ALLOW you to replace rhe battery in-situ while critical load is being applied without disruption of critical load.
Respectfully, no deep cycle AGM battery lasts for decades.
Sourcing a proper deep-cycle battery requires a bit of research.
Simply splicing a big-ass battery onto a consumer grade UPS won't work well, the little battery charger might not be able to fully recharge the new battery. The charge-discharge profile is all wrong.
Enterprise grade UPSs are not optimized for battery longevity, they are designed to recharge quickly and maximize runtime, both to the detriment of the batteries. APC in particular is notorious for overcharging batteries which can cause swelling and eventual rupture if they are not replaced on a regular basis. Also older APC "SmartUPS" would power off your load if you tried to use a regular serial cable to manage them. Surprise!
If you are looking for something robust for home use I suggest a reconditioned off-lease Eaton 5PX2000 with as many external battery packs as you want. They support network management, the front panel is actually useful, and they can be programmed to maximize battery life rather than runtime. They require a 20a 120v circuit but unless you are loading it up to more than 75% capacity you can get away with a regular 15a plug and an adapter.