Do inference servers for large models really need high-end GPUs?

Not necessarily. With model quantization, B10, A10 or even T4 cards can serve inference — choose by actual need instead of defaulting to costly cards like H100.