I Autoscaled My LLM on Queue Depth. It Scaled Down Under Full Load.
信息来源:DEV Community·
内容摘要
KEDA scaled my LLM server on waiting requests and removed a replica in the middle of a load test. A local kind lab, three measured runs, and the one-line metric fix.