How do training cluster and inference server hardware differ?

Training: high-speed GPU interconnect (NVLink/IB), large memory, high power draw. Inference: lower interconnect requirements, can lean on CPU compute, lower power, and supports high-density deployment.