According to Amazon AWS, the company has released Ray LLM Deep Learning Containers designed to run multi-node inference workloads on Amazon EKS. The solution enables users to deploy a single large language model across two GPU nodes, with the system utilizing separate prefill and decode nodes behind one OpenAI-compatible endpoint.
The pre-built container ships with the full inference stack, allowing users to configure the deployment without needing to assemble individual components. This approach simplifies the process of setting up distributed LLM inference infrastructure for organizations working with large language models that require multiple GPUs to operate efficiently.
The implementation on Amazon EKS provides a managed Kubernetes environment for running these distributed workloads, while the OpenAI-compatible endpoint ensures compatibility with existing applications and workflows built around OpenAI’s API specifications. By separating prefill and decode operations across different nodes, the architecture aims to optimize resource utilization during the inference process.