# Cross Region Stack References ## Status accepted ## Context The CDK allows for you to natively (in code) reference resources between stacks. For example: ```ts const bucket = new s3.Bucket(stack1, 'Bucket'); const handler = new lambda.Function(stack2, 'Handler'); bucket.grantRead(handler); ``` Here we have create an S3 bucket in one stack and natively referenced the bucket from a resource in a different stack. This works because CDK knows that this is a cross stack reference and will create the appropriate stack exports and imports. In this case it would create an `Export` in `stack1`. ```json { "Outputs": { "ExportsOutputFnGetAttBucket83908E77Arn063C8555": { "Value": { "Fn::GetAtt": [ "Bucket83908E77", "Arn" ] }, "Export": { "Name": "stack1:ExportsOutputFnGetAttBucket83908E77Arn063C8555" } } } } ``` And an "Import" in stack2 ```json { "Resources": { "HandlerServiceRoleDefaultPolicyCBD0CC91": { "Type": "AWS::IAM::Policy", "Properties": { "PolicyDocument": { "Statement": [ { "Action": [ "s3:GetBucket*", "s3:GetObject*", "s3:List*" ], "Effect": "Allow", "Resource": [ { "Fn::ImportValue": "stack1:ExportsOutputFnGetAttBucket83908E77Arn063C8555" }, { "Fn::Join": [ "", [ { "Fn::ImportValue": "stack1:ExportsOutputFnGetAttBucket83908E77Arn063C8555" }, "/*" ] ] } ] } ], "Version": "2012-10-17" }, "PolicyName": "HandlerServiceRoleDefaultPolicyCBD0CC91", "Roles": [ { "Ref": "HandlerServiceRoleFCDC14AE" } ] } } } } ``` If these stack exist in different regions this is no longer possible. This is due to an underlying limitation with CloudFormation, namely that Stack Exports/Imports do not work cross-region. There are some cases where cross region references are _required_ by the AWS services themselves. A good example of this is AWS CloudFront. CloudFront is a global service and you can create a CloudFront distribution via CloudFormation in any AWS Region. Other resources that are used by CloudFront are required to be created in the `us-east-1` region. For example, lets say you have an application that you created in `us-east-2`. ```ts const appStack = new Stack(app, 'AppStack', { env: { region: 'us-east-2' } }); const service = new ApplicationLoadBalancedFargateService(appStack, 'Service'); ``` If I want to add CloudFront to this application. I can add the distribution in the same Stack, but if I also want to add a ACM Certificate (why wouldn't I?) it becomes more difficult. In order to use a ACM Certificate with CloudFront, the certificate must be created in `us-east-1` regardless of what region you create the CloudFront distribution from. ```ts const appStack = new Stack(app, 'AppStack', { env: { region: 'us-east-2' } }); const service = new ApplicationLoadBalancedFargateService(appStack, 'Service'); // this won't work!!! const certificate = new acm.Certificate(appStack, 'Cert'); const distribution = new Distribution(appStack, 'Distribution', { defaultBehavior: { origin: new LoadBalancerV2Origin(service.loadBalancer) }, certificate, }); ``` To workaround this issue we have created things like the `DnsValidatedCertificate` construct which uses custom resources to create the certificate in `us-east-1`. This requires us to essentially maintain our own `Certificate` resource that maintains feature parity with the official `AWS::CertificateManager::Certificate` resource. Another example is the `aws-cloudfront.experimental` `EdgeFunction` construct. This takes a different approach to managing cross region resources. Instead of creating the resources with a custom resource, we instead create a support stack in `us-east-1` which creates the lambda function. We then use a custom resource to "lookup" the function arn in the Stack that creates the CloudFront distribution. This is becoming our recommended pattern for creating cross-region resources, so why not add an officially supported method for doing this. ## Constraints The biggest constraint with implementing a solution is CloudFormation itself. A common request from CDK users is for the CDK to support [weak references](https://github.com/aws/aws-cdk-rfcs/issues/82). The reason we have not yet implemented this feature is that there are good reasons as to why strong references exist and are the only officially supported method. Let's walk through an example to illustrate. Suppose I had a Lambda function that referenced an S3 bucket in some way (read data, write data, etc). CloudFormation will create a "strong" reference between these two resources. ```ts const bucket = new s3.Bucket(stack1, 'Bucket', { bucketName: 'mybucket', }); const handler = new lambda.Function(stack2, 'Handler', { environment: { BUCKET_NAME: bucket.bucketName, }, }); bucket.grantRead(handler); ``` If I tried to update the bucket, for example changing the name from `mybucket` to `myNewNameBucket` CloudFormation will fail the deployment for stack1 and prevent the bucket from being recreated. This is because it _knows_ that `stack2` is using the bucket. If it allows the bucket to change and the export to change you could end up in an unrecoverable state for `stack2`. ```mermaid sequenceDiagram Note over Stack1,Stack2: Initial Deployment activate Stack2 Note over Stack1: Create export mybucket Stack2->>+Stack1: Read export mybucket Note over Stack2: Create Function deactivate Stack2 Note over Stack1,Stack2: Second Deployment activate Stack2 Note over Stack1: Delete export mybucket Note over Stack1: Create export myNewNameBucket Stack2->>+Stack1: Read export myNewNameBucket deactivate Stack2 Note over Stack2: Update Function Failed! Stack2-->>+Stack2: Rollback activate Stack2 Stack2->>+Stack1: Read export mybucket Note right of Stack1: Export doesn't exist! deactivate Stack2 Note over Stack2: Stack rollback failed! Note over Stack1,Stack2: We're stuck!! ``` For the CDK to implement it's own concept of references it needs to take this into account. ### Custom Resources This solution utilizes custom resources to manage outputs/imports, and custom resources come with their own constraints. - Custom resources are only executed when the properties change. There is no way to have the resource execute on every deploy. - Custom resources only know about the current update (via `ResourceProperties`) and the previous update (via `OldResourceProperties`). Custom resources cannot keep track of all prior updates, unless we were to implement some external state mechanism. ## Decision The CDK will natively support cross region stack references. ```ts const appStack = new Stack(app, 'AppStack', { env: { region: 'us-east-2' } }); const service = new ApplicationLoadBalancedFargateService(appStack, 'Service'); // this will work!!! const certificate = new acm.Certificate(appStack, 'Cert'); const distribution = new Distribution(appStack, 'Distribution', { defaultBehavior: { origin: new LoadBalancerV2Origin(service.loadBalancer) }, certificate, }); ``` Since it is not natively supported by CloudFormation we will use CloudFormation custom resources to perform the output/import. This behavior will not be enabled by default and will be controlled by an optional Stack property. ```ts new Stack(app, 'MyStack', { crossRegionReferences: true, }); ``` ### Outputs In order to "output" the value from the producing stack, a custom resource will be created in the producing stack which will create an SSM parameter with a generated name in the consuming region. For example the name might be `/cdk/exports/stack2/stack1useast1CertRefCert5C9F`. To implement strong references the custom resources will be allowed to create new outputs, but will only be allowed to update/delete existing outputs if the output has _not_ been imported. If it has been imported the stack update will fail (similar to the behavior of native exports). See [Imports](#imports) for how the import is performed. ### Imports The consuming stack will then "import" the value via a SSM dynamic reference. This is possible because we know the name of the SSM parameter that the producing stack creates. This will look something like `{{resolve:ssm:/cdk/exports/stack2/stack1useast1CertRefCert5C9F}}`. A custom resource will also be created in the consuming stack that will be responsible for marking the SSM parameter as having been "imported". It will do so by adding a tag to the parameter, something like `aws-cdk:strong-ref=stack2`. If the value is no longer imported by the stack then the tag will be removed. The producing stack will use the presence of the tag to determine whether or not the output can be updated/deleted. Since the imports for a stack are exported as SSM parameters with the stack name as part of the name prefix, when the importing stack is deleted it will clean up and remove any SSM parameters under that prefix. ## Alternatives This solution uses a push model where the producing stack "pushes" the output to the target region. One alternative that was considered was to use a pull model where the consuming stack would "pull" the output from the producing region. For example the producing stack could produce a normal CloudFormation export and then the consuming stack would have a custom resource the reads the exports. This alternative had several limitations: 1. No way to implement strong references. CloudFormation would not know the export is being used and would allow it to be updated/deleted. 2. The consuming custom resource would need to run every time the stack is deployed. This would require introducing a salt that would cause a template diff on every deploy (not ideal). ## Consequences If we add support for cross region references we will need to support cross region references going forward. We will not be tied to this implementation though.