Western Digital says contamination impacting production at Japanese facilities(reuters.com)
reuters.com
Western Digital says contamination impacting production at Japanese facilities
https://www.reuters.com/business/western-digital-says-contamination-impacting-production-japanese-facilities-2022-02-10/
10 comments
Reminds me of old stories about Intel - how, when they were taking a (chip fabrication) process which worked at plant X to (new or re-tooled) plant Y, they would duplicate EVERYTHING from X at Y. Right down to the exact brand / line / formulation of the paint on (say) the walls of the lobby, the type of toilet paper in the bathrooms, etc.
That process even has a name: "Copy Exactly!"
> The Copy Exactly! methodology focuses on matching the manufacturing site to the development site. Matching occurs at all levels for physical inputs and statistically-matched responses (outputs). This process enables continuous matching over time by using coordinated changes, audits, process control systems, and joint Fab management structures
https://en.wikipedia.org/wiki/Copy_Exactly!
> The Copy Exactly! methodology focuses on matching the manufacturing site to the development site. Matching occurs at all levels for physical inputs and statistically-matched responses (outputs). This process enables continuous matching over time by using coordinated changes, audits, process control systems, and joint Fab management structures
https://en.wikipedia.org/wiki/Copy_Exactly!
And here us software developers are still trying to do this and think we're the smartest people in the room for inventing it!
Back in the days of "classic" ASP pages, the usual model at the shop I worked in was that whenever we needed to add a page to the site we would copy a working page and then modify it. Sure it duplicated a lot of common code, but it wasn't without its advantages. You could be pretty certain that your page would not break anything else, and that later changes to a page would also be self-contained. It did suck when a change needed to be made to many pages, but if you knew about cygwin and grep, sed, and awk, those could often be automated.
So you got code reuse via copy and paste!
sometimes called copy stupid in extreme circumstances
See: Target's botched move into Canada. They basically copied everything they did in the US. This was compounded by an optimistic purchase of another company's locations to expand into. They bombed the project's runway on that and couldn't afford to keep it going while they sorted out all the problems resulting from Canada revealing all the flaws in their US systems.
https://en.wikipedia.org/wiki/Target_Canada
https://en.wikipedia.org/wiki/Target_Canada
As a Canadian whose partner was initially very excited about the Target launch, what killed it for her/us was primarily the logistics issues— you'd go into the store and like 20% of the shelves would be empty including everything that was supposed to be on markdown, and even some of the stuff that was there wouldn't scan properly at the till.
But apart from that, I think we got the impression that there were some market positioning issues too. Like, in the US, Wal-Mart has a reputation for being the absolute cheapest, so Target being the "step up" option is a pretty reasonable place to be. Whereas in Canada, Wal-Mart was already occupying that step-up space (relative to discount department stores like Giant Tiger and the old Zellers locations that Target Canada bought up), and those overseeing the Target push hadn't really committed on whether they wanted to fight Wal-Mart directly for that space, or try to go a level further up and take on the "nice" department stores like the Bay and Sears.
Neither option was great, given that Wal-Mart was well-entrenched and the nice department stores having been dying for decades anyway. But not choosing a path at all is often worse than either of two bad options might have been if they'd at least been fully committed-to.
But apart from that, I think we got the impression that there were some market positioning issues too. Like, in the US, Wal-Mart has a reputation for being the absolute cheapest, so Target being the "step up" option is a pretty reasonable place to be. Whereas in Canada, Wal-Mart was already occupying that step-up space (relative to discount department stores like Giant Tiger and the old Zellers locations that Target Canada bought up), and those overseeing the Target push hadn't really committed on whether they wanted to fight Wal-Mart directly for that space, or try to go a level further up and take on the "nice" department stores like the Bay and Sears.
Neither option was great, given that Wal-Mart was well-entrenched and the nice department stores having been dying for decades anyway. But not choosing a path at all is often worse than either of two bad options might have been if they'd at least been fully committed-to.
That wiki article makes it sound like they did change something compared to the US stores; prices went up. Canadian customers knew that from previously crossing the border to US Target stores and subsequently stopped going pretty much immediately after launch.
I believe part of the reason is they completely changed their enterprise resource planning & supply chain software. There were a LOT of changes, and yet not enough - not accounting for Canadian price sensitivity and the distances/warehousing difference.
This is a fascinating read I bookmarked at the time :)
http://www.canadianbusiness.com/the-last-days-of-target-cana...
I believe (but could be wrong) Home Depot did essentially the same thing - used Canadian market as test bed for SAP, except they already had an established stable presence here.
This is a fascinating read I bookmarked at the time :)
http://www.canadianbusiness.com/the-last-days-of-target-cana...
I believe (but could be wrong) Home Depot did essentially the same thing - used Canadian market as test bed for SAP, except they already had an established stable presence here.
That general philosophy seems to be intact. A very recent example: a large machine at their Hillsboro research facility was dismantled and sent to production in Ireland, with the same model with a very slight upgrade shipped in to replace it at the R&D location.
Ignore the red herring title: https://semianalysis.com/is-intel-shipping-tools-out-of-us-f...
Ignore the red herring title: https://semianalysis.com/is-intel-shipping-tools-out-of-us-f...
Docker for factories.
the original reproducible build in a way
I was watching a video about the gold coating on the JWST primary mirror segments and noticed a conspicuously placed sign warning employees not to bring anything containing silicone into the facility.[1] I had no idea it was such a problem contaminant! I wonder if they have issues with personnel using silicone based personal products at home.
[1] https://www.youtube.com/watch?v=fgn7bKs042Q&t=211s
[1] https://www.youtube.com/watch?v=fgn7bKs042Q&t=211s
Siloxanes are serious contaminates in several industrial processes. Another big one is powder coating "paint". It can be bad even if only comes from a personal care product used by an employee at home.
And you called out their near-infinite ability to be spread around.
Very useful and interesting class of chemicals.
And you called out their near-infinite ability to be spread around.
Very useful and interesting class of chemicals.
What is powder coating paint? Curious so as I can avoid it :)
https://en.m.wikipedia.org/wiki/Powder_coating
It's a common process used to coat all sorts of objects. But you don't buy "powder coating paint" and use it yourself, it is done in purpose built facilities.
It's a common process used to coat all sorts of objects. But you don't buy "powder coating paint" and use it yourself, it is done in purpose built facilities.
I think the parent is referring to something else, given the part about personal care products.
Scrapped or scraped? Did they really need to abandon the plant or the equipment, or just clean it?
Silicone oil is one of the most stubborn contaminants I have ever encountered. I use trisodium phosphate to clean it up, and even then it sometimes persists. Try pouring epoxy over wood that has been polished with Pledge, and you'll see what I mean.
I can believe "scrapped."
I can believe "scrapped."
Isn’t a silicone based oil used in oil diffusion vacuum pumps?
Seems like one should look for a different solution
Seems like one should look for a different solution
Not a clean room, but reminds me when I was working in a partner's small datacenter and found an electrician in there using a chop saw on metal conduit.
Are there sources for both of your stories? I would be very curious to learn the details.
What, you mean it’s not spelled “silicone valley”?
What does contamination even mean in this context?
In scientific terms - some chemical (used in the production) was supposed to be 99.999999% pure...but was not, and the offending impurity(ies) was (in terms of consequences for the uber-delicate production processes) Profoundly Unhelpful.
In poetic terms - a hundred thousand tiny demons got loose in the plant, and each and every one of them will need to be laboriously exorcised, before production can resume.
In poetic terms - a hundred thousand tiny demons got loose in the plant, and each and every one of them will need to be laboriously exorcised, before production can resume.
Exactly. Pretty poor news article not addressing this. At least they could have added "The manufacturer declined to comment on how the contamination occured" if that's the case.
Poor manufacturing process? (Of course poor is relative in today's silicon) Sabotage? Earthquake? (They have them all the time) Radiation from Fukushima? (Maybe a bit late 10 years after)
Poor manufacturing process? (Of course poor is relative in today's silicon) Sabotage? Earthquake? (They have them all the time) Radiation from Fukushima? (Maybe a bit late 10 years after)
FOD: https://en.wikipedia.org/wiki/Foreign_object_damage
Working in Communications/Medical/Aerospace manufacturing, this is a MASSIVE concern.
Working in Communications/Medical/Aerospace manufacturing, this is a MASSIVE concern.
Impurities in raw materials?
[deleted]
maybe dust?
6.5 exabytes short? It sounds like a lot, but does someone more informed know if that is going to make a dent?
6.5 million in equivalent 1-terabyte units (note that NAND chips tend to be 64-512GB, the unit is just to illustrate approximate loss). However, I'm reading it beyond the 6.5 exabytes since that the WD press release (or any Kioxia statement) did not state that the production has been resumed, so I'm assuming it's actually a downtime and not just bad chips.
Global flash shipments last year were estimated to be ~600EB, so that is ~1%.
Does anyone know what kind of products this kind of flash storage is used in? SSDs or NVMe SSD or something else?
Everything that uses NAND flash, which ranges from computers to phones to TVs. WD and Kioxia supply chips to the commodity market, not just for their own branded products.
For example 40% of "Brand X" SSD drives might have WD chips, 40% Toshiba chips, and 20% Samsung chips so the manufacturer can get a better price and/or diversify supply. If WD can only supply 25% next quarter orders to Toshiba and Samsung will increase. That will naturally push prices up to some degree.
For example 40% of "Brand X" SSD drives might have WD chips, 40% Toshiba chips, and 20% Samsung chips so the manufacturer can get a better price and/or diversify supply. If WD can only supply 25% next quarter orders to Toshiba and Samsung will increase. That will naturally push prices up to some degree.
In general Kioxia are making SSDs (NVME and otherwise) as well as USB drives and SD cards. I'm guessing this affects the whole range.
[deleted]
> said they’re working to get the plants in Yokkaichi and Kitakami back to normal operations as quickly as possible.
> Kioxia said its newer and more lucrative type of 3D flash was the product line impacted, and “the company does not anticipate that shipment of its conventional 2D NAND flash memory will be affected.”
Reads like NAND chips marketed as Kioxia BiCS4.
1: https://www.bloomberg.com/news/articles/2022-02-09/western-d...
Reads like NAND chips marketed as Kioxia BiCS4.
1: https://www.bloomberg.com/news/articles/2022-02-09/western-d...
Was i the only one expecting to read what was actually the contaminant?
What a happy coincidence. Just as prices of flash memory have stabilized.
Manufacturers of commodity parts like storage or LCD panels are infamous for price fixing and gouging. All the disasters are great excuse for unjustified price hikes.
I remember how a decade ago I was building my PC and HDD prices were painful.
Manufacturers of commodity parts like storage or LCD panels are infamous for price fixing and gouging. All the disasters are great excuse for unjustified price hikes.
I remember how a decade ago I was building my PC and HDD prices were painful.
That was because of floods, right?
yeah, building plants in regions with high seismic activity and high flooding potential was for sure the best idea of the century
now let's build our new plants in LA, that'll help a lot in the future :D
dumb western politicians are responsible of the current situation, shame on them for sacrificingmany generations of future asian-dependent citizens
now let's build our new plants in LA, that'll help a lot in the future :D
dumb western politicians are responsible of the current situation, shame on them for sacrificingmany generations of future asian-dependent citizens
[deleted]
Intel is making the right choice with a fab in ohio. So boring, people cannot find it on a map.
you don't look at the global picture, this explains your situation in the west
https://www.mercurynews.com/2009/01/22/intels-silicon-valley...
https://www.mercurynews.com/2009/01/22/intels-silicon-valley...
Was it a fly[0], by any chance?
[0]: https://www.youtube.com/watch?v=9rUEJwsXbnk
[0]: https://www.youtube.com/watch?v=9rUEJwsXbnk
You wouldn't think they'd ship buggy hard drives, would you?
I never really think about it, although I imagine the chip shortage is putting a big dent in the production of HDD's / SSD's.
With all this chip shortage, i was thinking that i still have the same laptop from 2014. It's not like it's slow and it's not like they make them much faster anymore. So maybe the chip shortage is not as big problem as rumored?
Someone crashed my car recently. I had to buy another one (the other was impossible to repair).
I ended buying a used base model Mitsibishi Lancer, for the same price a new Mitsibishi Lancer EVO used to cost 3 years ago. And I only bought that car specifically, because it was the cheapest sedan I could find, a Honda Civic or a Toyota Corolla was literally double the price.
The reason for that? The chip shortage. While shopping for cars I even found a guy that sold his Lancer dashboard and put in its place a crappy one. Then when checking if I should buy a new HB20, I found out they are coming with crappy dashboards and the ones from 2019 are not avaialble anymore... The reason for that, again is the chip shortage, coloured dashboards with complex functions require more chips, so in case of new cars, the manufacturers are shipping them with simpler monochromatic dashboards with less functions, and some used car owners have been literally selling their dashboards to richer people, because they now are crazy expensive.
Meanwhile the government of my country has been terrified of truckers, the truckers been giving them a hard time, specially because of the prices of trucks, both used and new, they are saying that trucking is not profitable anymore and want the government to give them money.
Know why trucking is not profitable anymore? Well... the chip shortage made trucks get crazy expensive.
The government also was going to switch the machines used for voting, to make election more secure and fair (the new machines would allow better methods of recounting not vulnerable to hacking), but this was aborted, because new machines had their price skyrocket... because of the chip shortage.
The list goes on, the chip shortage is not just causing inflation or making people not afford a new laptop, it is literally disrupting production, changing products, changing people behaviour, causing social changes, causing LEGAL changes... The chip shortage literally is affecting how the leader of 250 million people is chosen!
I ended buying a used base model Mitsibishi Lancer, for the same price a new Mitsibishi Lancer EVO used to cost 3 years ago. And I only bought that car specifically, because it was the cheapest sedan I could find, a Honda Civic or a Toyota Corolla was literally double the price.
The reason for that? The chip shortage. While shopping for cars I even found a guy that sold his Lancer dashboard and put in its place a crappy one. Then when checking if I should buy a new HB20, I found out they are coming with crappy dashboards and the ones from 2019 are not avaialble anymore... The reason for that, again is the chip shortage, coloured dashboards with complex functions require more chips, so in case of new cars, the manufacturers are shipping them with simpler monochromatic dashboards with less functions, and some used car owners have been literally selling their dashboards to richer people, because they now are crazy expensive.
Meanwhile the government of my country has been terrified of truckers, the truckers been giving them a hard time, specially because of the prices of trucks, both used and new, they are saying that trucking is not profitable anymore and want the government to give them money.
Know why trucking is not profitable anymore? Well... the chip shortage made trucks get crazy expensive.
The government also was going to switch the machines used for voting, to make election more secure and fair (the new machines would allow better methods of recounting not vulnerable to hacking), but this was aborted, because new machines had their price skyrocket... because of the chip shortage.
The list goes on, the chip shortage is not just causing inflation or making people not afford a new laptop, it is literally disrupting production, changing products, changing people behaviour, causing social changes, causing LEGAL changes... The chip shortage literally is affecting how the leader of 250 million people is chosen!
Where do you get the idea that trucking is unprofitable? The trucking industry has recovered from pandemic downturn and is making plenty of profits.
Say you need a new truck. Here's what the market dynamics look like: https://www.theverge.com/22923871/carvana-pandemic-used-car-...
Say you need your truck fixed. It may take months to get the parts needed.
This hasn't played out yet, not even close.
Say you need your truck fixed. It may take months to get the parts needed.
This hasn't played out yet, not even close.
The commercial heavy vehicle market is wildly different from consumer automobiles.
As far as parts availability, I see just as many trucks on the road as ever. If it were widespread that it took months to fix trucks, there would be a huge impact. There’s not.
Do you have actual proof of what you’re talking about?
As far as parts availability, I see just as many trucks on the road as ever. If it were widespread that it took months to fix trucks, there would be a huge impact. There’s not.
Do you have actual proof of what you’re talking about?
The commercial heavy vehicle market is wildly different from consumer automobiles.
How so?
How so?
Indeed a laptop built in 2014 is unaffected by a chip shortage in 2020-22.
Until it's accidentally dropped and you need a replacement.
Or you want to buy a webcam for it. Or a car, a TV, an oven, a dishwasher. Or you interact with any service which is itself impacted, and then you suffer longer wait times, higher prices or are denied service. Maybe your ISP can't ship you a replacement if your router just got fried by lightning.
And let's hope you don't need a hospital ER, where you would find out that semiconductors are not the only critical shortage going on.
Until it's accidentally dropped and you need a replacement.
Or you want to buy a webcam for it. Or a car, a TV, an oven, a dishwasher. Or you interact with any service which is itself impacted, and then you suffer longer wait times, higher prices or are denied service. Maybe your ISP can't ship you a replacement if your router just got fried by lightning.
And let's hope you don't need a hospital ER, where you would find out that semiconductors are not the only critical shortage going on.
I used to think that about my 2011 laptop as just looking at the CPU speeds were not really much faster. however, CPU is only a part of the system. GPU/RAM/SSD/etc have seen several generational leaps ahead and all add up to an improved experience. Also, even though the CPU speeds aren't much faster, the transistors in the chips are drastically changing. So some bits of code run faster because now they are on the chip.
Just as an aside, a CPU from 2022 will generally be much faster and efficient than one from 2014, even if the clock speed is similar, due to architecture improvements.
It's not rumoured, it's thousands of articles that explain how this is a big deal and how many industries have had to slow down because of it. Chips are used in everything, your personal laptop is really irrelevant in the equation.
How exactly does your old laptop have anything to do with a global shortage?
I haven't been to the hospital in years, so clearly the news about long wait times at the ER are bogus.
Their point was that most people already have the technology they need, and getting the latest shiniest thing isn't as much of an issue.
Well a lot of the demand that creates this shortage is for replacement of 'old' phones or laptops which are not truly old
The short explanation is that phone and computer chips are incompatible with cars (except in Tesla cars where they do use some, and even then Tesla cars still use chips that are literally have no comparison with computer chips).
The long explanation is that the "node" of the process used in phones and computers are different from those used in cars. Most computers can tolerate errors and correct them a bit later, but in cars that scenario can result to injury or even death. Car chips are built to totally different standards, and to borrow a terminology in software, written in assembly and not C. You cannot just reuse the processes intended for computer chips and apply them in cars, you need to manually check that everything still works out.
The long explanation is that the "node" of the process used in phones and computers are different from those used in cars. Most computers can tolerate errors and correct them a bit later, but in cars that scenario can result to injury or even death. Car chips are built to totally different standards, and to borrow a terminology in software, written in assembly and not C. You cannot just reuse the processes intended for computer chips and apply them in cars, you need to manually check that everything still works out.
There are multiple shortages going on. For leading edge chips going into phones and GPU's maybe we wait for our game console or new phone a few months. But there are also shortages for automotive sensors, microcontrollers, power delivery controllers. Not buying a new phone isn't going to solve the IC shortage for cars or white goods.
The worst part is that silicone oil will diffuse over surfaces so the whole plant got contaminated and needed to be scrapped.
I've also heard that some electronics plant had an issue where some metal tiles in the lobby of the plant were growing tiny whiskers of zinc, which got tracked in by workers and started shorting out the electronics.