We have multiple different pages. In our cluster we have 3 machines and if one of them is unavailable because of broken network, we do not page. In this case the page came as an application error that the application was not able to cope with. When we have issue that we have seen before and the server can handle it on its own, we do not page.
In this case it was un-queued TRIM (I forgot to mention it in the blogpost). We have reached to Samsung and although it looked good at the beginning now they are silent for more than a month without any progress.
The drives we have detected the issue had still un-queued TRIM. I have reached to one of the kernel I/O developers for help and he confirmed that it is not related.
With our architecture, HW requirements, the price of HW and the price of the cloud VMs, even working on this for a week or two saves us significant amount of money both short-term and long-term. The side effect is that we now have tools to recover servers way faster and allows us to do things we have not thought about before.