# Reaching an AWS Service That Isn't in Your Region

Hi guys! We run a small internal LLM gateway at work. It sits between our engineers and Bedrock, hands out scoped API keys, tracks spend per user and writes an audit log. Nothing fancy, but it takes real traffic.

Then the OpenAI models showed up on Bedrock and everyone wanted them. I figured it was a config change. It turned into a networking project.

So let's talk about it.

## Context and challenge

Our landing zone runs on Landing Zone Accelerator, home region ca-central-1. Same shape I described in my [secure hybrid network post](https://blogs.houessou.com/how-to-set-up-a-secure-hybrid-network-on-aws): a central network account owns the VPCs and shares subnets out to workload accounts, Transit Gateway in the middle, perimeter account handling egress behind a Network Firewall.

The piece that matters here is the endpoints VPC. Instead of every VPC paying for its own interface endpoints, one VPC holds about twenty of them, with private hosted zones that override the public AWS names and get associated out to every spoke. A workload calls `kms.ca-central-1.amazonaws.com` and gets back a private IP a few hops away.

And we have a rule. No workload traffic on the public internet. Not "must be encrypted", which is a given anyway. Private. It's the reason the endpoints VPC exists at all.

Cross-region inference had already solved the region problem for us, and it's worth being clear about why.

With cross-region inference you still call the Bedrock endpoint in your own region. Bedrock routes the request internally to wherever it has capacity. So our gateway called `bedrock-runtime.ca-central-1.amazonaws.com`, hit the interface endpoint over the Transit Gateway, stayed private, and the fact that the model was actually served from somewhere else was Bedrock's problem, not ours.

The OpenAI models don't work like that. They're only reachable through the newer `bedrock-mantle` endpoint, that endpoint only exists in us-east-1, and those models don't support cross-region inference.

So there was no in-region endpoint to call. Point the SDK at Mantle from ca-central-1 and it resolves a public name, gets a public IP, and heads out through the perimeter to the internet. Which is the one thing we're not allowed to do.

![](https://cdn.hashnode.com/uploads/covers/60aaa78720d1f45ceb7f046d/707a74d5-2f8c-4c65-b84d-b20728a0318a.png align="center")

## Alternative approaches

AWS added cross-region PrivateLink a while back, so my first move was to check whether we could just point an endpoint at the service in us-east-1.

```bash
aws ec2 describe-vpc-endpoint-services --region ca-central-1 \
    --service-region us-east-1 --filters Name=owner,Values=amazon \
    --query 'ServiceNames[?contains(@,`bedrock`)]'
```

```plaintext
[]
```

Empty. This one trips up a lot of people, because there are two features with almost the same name. The one from November 2024 covers **customer** endpoint services, the ones you build yourself behind an NLB. The one from November 2025 covers a list of **AWS** services: S3, KMS, ECS, Lambda and a few others. Mantle is on neither.

Keep the first one in mind though. It comes back.

Peering is what the docs suggest, and what the re:Post answer to this exact question says. Put a VPC in us-east-1, peer it, done. Works fine for one consumer.

It doesn't scale, and the reason catches people out: traffic arriving at a VPC over a Transit Gateway is never forwarded across a peering connection. So you can't peer once into the shared endpoints VPC and let the TGW spread it around. Every VPC that wants the service needs its own peering and its own routes. We have three environments, so that's three peerings on day one and another every time someone new shows up.

We nearly took it anyway. When this started only prod was in scope, and for one consumer peering is genuinely the cheapest thing you can do. Remembering that dev and test exist is what killed it.

The other option is a second Transit Gateway in us-east-1, peered inter-region. There's nothing wrong with this and it's what most teams would ship. It's transitive, it's expressible in our config, future workloads get it for free. It also costs roughly eight times the fixed monthly price of peering for exactly one destination, needs address space nobody had reserved, and forces a conversation about which route tables are allowed to reach the far region.

## Solution overview

Go back to that November 2024 feature. Cross-region PrivateLink works for customer endpoint services. Mantle isn't one, but nothing stops us from making one.

An endpoint service has to sit behind a Network Load Balancer. An NLB takes IP addresses as targets. And an interface endpoint is just a set of ENIs with private IPs in your subnets.

So: put a Mantle endpoint in a small VPC in us-east-1, put an NLB in front of it targeting those ENI addresses, publish the NLB as an endpoint service with cross-region enabled, then consume it with a single interface endpoint in the endpoints VPC we already run.

The NLB isn't balancing anything. It's an adapter. An endpoint service is the only thing AWS lets you share across regions, and an endpoint service needs an NLB, so that's the price of entry.

And because the consumer endpoint lives in the endpoints VPC, every workload VPC that already routes there can reach it. No peering, no second Transit Gateway, no route table changes anywhere.

![](https://cdn.hashnode.com/uploads/covers/60aaa78720d1f45ceb7f046d/2d6c5355-676d-4ad2-9706-142c99085d5b.png align="center")

### Certificates and TLS

First question I got was the right one. If you put a load balancer in the middle, who owns the certificate?

Nobody does, and it's worth seeing why.

The NLB listener is TCP, not TLS. That makes it pure Layer 4 passthrough: it copies bytes between two sockets and never looks inside them. The TLS handshake happens end to end between the client and the Mantle endpoint itself.

The bit that makes it work is SNI. When the client opens the connection it sends the hostname it wants in the clear in the ClientHello, before encryption starts. That travels through the NLB untouched, reaches the real endpoint, and the service answers with a certificate for that name.

Which also explains why a TLS listener on the NLB would be a bad idea. You'd terminate the handshake in the wrong place, break the certificate chain, and break SigV4 signing along with it.

## Testing

Nothing documents whether an NLB can target an AWS-managed endpoint's ENIs. The address rules allow it, and there's an AWS blog post doing the same trick for API Gateway private APIs, but nobody does it against a regional AWS service. So I built it in a sandbox instead of arguing about it.

Targets went healthy straight away:

```plaintext
192.168.240.10  us-east-1b  healthy
192.168.240.42  us-east-1c  healthy
```

Then I put a test host in the consumer VPC in ca-central-1, with no internet gateway and no NAT gateway at all, so a passing result couldn't be a false positive.

The name resolves to a private address in the host's own VPC:

```plaintext
$ getent ahostsv4 bedrock-mantle.us-east-1.api.aws
192.168.241.58
192.168.241.94
```

Both of those are the consumer endpoint's own ENIs. No hosts file, no override, just the private hosted zone doing its job. And the certificate comes back for the real hostname:

```plaintext
subject=CN=bedrock-mantle.us-east-1.api.aws
X509v3 Subject Alternative Name:
    DNS:bedrock-mantle.us-east-1.api.aws,
    DNS:*.bedrock-mantle.us-east-1.vpce.amazonaws.com
issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M04
```

Both names matter. The first is why TLS validates when a client asks for the real hostname. The second is why the private zone can alias that name at a `vpce` name without a mismatch. And the issuer is Amazon's public CA, so we own no certificate anywhere in this path.

Then an actual request:

```plaintext
$ curl -X POST -d '{}' https://bedrock-mantle.us-east-1.api.aws/v1/responses
status=401 tls=0
{"error":{"code":"invalid_api_key","message":"Missing 'authorization' or
 'x-api-key' header","type":"permission_denied_error"}}
```

That's the API telling me I forgot my credentials. A proxy error or a dead connection can't produce that body. The request left a host with no internet access, went through the endpoint in its own VPC, over PrivateLink to us-east-1, through the NLB and into Mantle.

## Key recommendations

**Use a TCP health check, not HTTP.** An AWS service endpoint serves no health path, so an HTTP check marks every healthy target as unhealthy and you conclude the whole pattern is blocked. It isn't. I'm fairly sure this is why the few community reports out there say this doesn't work.

**Read the DNS name from the service.** Mantle answers on `bedrock-mantle.us-east-1.api.aws`, not `.amazonaws.com`. If your automation builds zone names from a pattern, you get a private hosted zone that is live, correctly formed and resolves nothing, while your traffic quietly keeps going out the internet. The only symptom is that nothing changes.

**Check the AZs in the account you're deploying into.** Mantle is only in three of the six AZs in us-east-1, and cross-region PrivateLink doesn't work in a handful of zone IDs. Zone names map to different physical zones per account. In our sandbox `us-east-1a` is `use1-az6` and doesn't offer the service at all, so the obvious default of "a and b" would have deployed into nothing.

**The record sits at the zone apex.** The zone is named for the service and so is the record, which is where DNS forbids a CNAME. It has to be an A-record alias, pointing at the regional endpoint entry and never a zonal one.

**Your pipeline role needs** `vpce:AllowMultiRegion`**.** Both to publish the endpoint service to another region and to create the consumer endpoint. If an SCP denies it you get an explicit deny, and there's no console fallback on cutover day.

## Conclusion

The honest ending is that this is temporary. Mantle will almost certainly join the cross-region AWS services list at some point, and when it does the whole thing collapses into one interface endpoint with one extra argument on it.

I think that's the best argument for it rather than a weakness. Of everything we looked at, this leaves the least behind: an NLB, an endpoint service and a small VPC nothing else depends on. The second Transit Gateway would have left a gateway, three attachments and a routing decision written down in six places.

One caveat before you build on this. AWS says nothing about registering an AWS-managed endpoint's ENIs as load balancer targets. Not that you may, not that you may not. I measured it and it works, and that's the whole claim. If you're in a regulated environment, take that to your review board before you build on it, not after.

The code is an extension to the [secure hybrid network repo](https://github.com/hpfpv/aug-secure-hybrid-network), so it plugs into the modules that are already there. The new one is [vpc-endpoint-cross-region](https://github.com/hpfpv/aug-secure-hybrid-network/tree/main/modules/vpc-endpoint-cross-region), and the consumer endpoint sits in the existing [vpc-endpoint](https://github.com/hpfpv/aug-secure-hybrid-network/tree/main/modules/vpc-endpoint) module next to the in-region ones. The spoke VPCs don't change at all, which is the part I'm happiest with.

Happy networking!

**Good to read:**

*   [Architecture patterns for consuming private APIs cross-account](https://aws.amazon.com/blogs/compute/architecture-patterns-for-consuming-private-apis-cross-account/)
    
*   [Cross-region enabled AWS services](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html)
    
*   [Centralized access to VPC private endpoints](https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/centralized-access-to-vpc-private-endpoints.html)