Amazon SageMaker HyperPod adds Global Inference Gateway for cross-region routing

AWS introduces Global Inference Gateway to route inference requests across multiple SageMaker HyperPod clusters and regions from a single endpoint, with automatic failover and load balancing based on real-time GPU utilization.

AI-generated summary · 1 source

How this is made ↗

Amazon Web Services has introduced the Global Inference Gateway for Amazon SageMaker HyperPod, allowing organizations to route machine learning inference requests across multiple clusters and AWS regions through a single managed endpoint.

The gateway addresses a capacity utilization problem: during traffic peaks, requests can queue on saturated clusters while GPUs in other clusters or regions sit idle. According to AWS, the new service automatically routes traffic based on real-time signals including key-value cache utilization and queue depth, sending requests to the best-suited cluster across registered regions.

The Global Inference Gateway works as a second tier above the existing Inference Gateway. While the original Inference Gateway routes requests to specific pods within a single cluster, the new global layer selects which cluster receives each request and carries it over the AWS backbone. Inside each cluster, the existing pod-level routing remains unchanged.

AWS says the service provides automatic failover when clusters fail and handles authentication through AWS Signature Version 4 and AWS Identity and Access Management. Organizations running models across multiple clusters previously needed to manage separate endpoints for each cluster, requiring custom load balancers, DNS configuration and cross-region networking.

AWS describes a scenario where a team serving four models across five HyperPod clusters in two regions previously tracked 20 endpoints—one per model per cluster. The Inference Gateway reduced that to five endpoints, one per cluster, but clients still had to select clusters themselves. The Global Inference Gateway consolidates routing to a single entry point.

Continue reading

All news →
Nov 21, 2026

Anthropic debuts on Nasdaq at $2 trillion valuation, then retreats as chip stocks slide

Nov 21, 2026

Meta Expands Digital Brain Project, Releases Faster Chemistry Simulation Model

Nov 21, 2026

Security experts link recent tech outages to AI-driven patching pressures