We're a distributed GPU cloud with 10k+ consumer GPUs on our network.
We can crank out 4500+ stable diffusion images per dollar (highest in the market), all on consumer 3060s.
The high-end GPUs definitely have a role but for a bulk of production/inference serving can be done on consumer GPUs at 80-90% of the cost.
This is what we've done at Salad (www.salad.com). SaladCloud (officially launching on July 11th) has 10k+ Nvidia GPUs at the lowest prices in the market. Not conducive for training but for serving inferences at massive scale.
Try www.salad.com. We've got 10k+ GPUs - from 8GB to 24GB. You get 10x more inferences per dollar compared to others. Our product team is pretty happy to help out on Discord. Some prices of interest.
RTX 3060 - 12 GB - $0.08/hr
RTX 3090 - 24 GB - $0.25/hr
We're a distributed GPU cloud with 10k+ consumer GPUs on our network. We can crank out 4500+ stable diffusion images per dollar (highest in the market), all on consumer 3060s.
The high-end GPUs definitely have a role but for a bulk of production/inference serving can be done on consumer GPUs at 80-90% of the cost.