--- feature name: remove-constructs-compat start date: 2020-07-05 rfc pr: [#195](https://github.com/aws/aws-cdk-rfcs/pull/195) related issue: [#192](https://github.com/aws/aws-cdk-rfcs/issues/192) --- # Removal of Construct Compatibility Layer ## Summary As part of our effort to broaden the applicability of the CDK's programming model to other domains such as [Kubernetes](https://cdk8s.io), we have extracted the base `Construct` class (and a few related types) to an independent library called [constructs](https://github.com/aws/constructs). To preserve backwards compatibility in AWS CDK 1.x, a "compatibility layer" ([construct-compat.ts]) has been added to the AWS CDK. This layer served as an API shim so that CDK code will continue to function without change. As part [AWS CDK v2.0], we plan to remove this redundant layer and make some improvements to a few APIs in "constructs" based on our learnings from CDK v1.x and new projects such as CDK for Kubernetes. This RFC describes the motivation, implications and plan for this project. [aws cdk v2.0]: https://github.com/aws/aws-cdk-rfcs/issues/156 [construct-compat.ts]: https://github.com/aws/aws-cdk/blob/fcca86e82235387b88371a0682cd0fc88bc1b67e/packages/%40aws-cdk/core/lib/construct-compat.ts ## Table of Contents - [Removal of Construct Compatibility Layer](#removal-of-construct-compatibility-layer) - [Summary](#summary) - [Table of Contents](#table-of-contents) - [Release Notes](#release-notes) - [00-DEPENDENCY: Declare a dependency on "constructs"](#00-dependency-declare-a-dependency-on-constructs) - [01-BASE-TYPES: Removal of base types](#01-base-types-removal-of-base-types) - [02-ASPECTS: Changes in Aspects API](#02-aspects-changes-in-aspects-api) - [03-DEPENDABLE: Changes to IDependable implementation](#03-dependable-changes-to-idependable-implementation) - [04-STACK-ROOT: Stacks as root constructs](#04-stack-root-stacks-as-root-constructs) - [05-METADATA-TRACES: Stack traces no longer attached to metadata by default](#05-metadata-traces-stack-traces-no-longer-attached-to-metadata-by-default) - [06-NO-PREPARE: The `prepare` hook is no longer supported](#06-no-prepare-the-prepare-hook-is-no-longer-supported) - [07-NO-SYNTHESIZE: The `synthesize` hook is no longer supported](#07-no-synthesize-the-synthesize-hook-is-no-longer-supported) - [08-VALIDATION: The `validate()` hook is now `node.addValidation()`](#08-validation-the-validate-hook-is-now-nodeaddvalidation) - [09-LOGGING: Logging API changes](#09-logging-logging-api-changes) - [Motivation](#motivation) - [1. Redundant layer](#1-redundant-layer) - [2. Multiple `Construct` types](#2-multiple-construct-types) - [3. Composability with other domains](#3-composability-with-other-domains) - [4. Non-intuitive dependency requirement](#4-non-intuitive-dependency-requirement) - [Design](#design) - [00-DEPENDENCY](#00-dependency) - [01-BASE-TYPES](#01-base-types) - [What can we do on 1.x](#what-can-we-do-on-1x) - [10-CONSTRUCT-NODE](#10-construct-node) - [What can we do in 1.x](#what-can-we-do-in-1x) - [Alternative Considered](#alternative-considered) - [02-ASPECTS](#02-aspects) - [What can we do on 1.x for 02-ASPECTS](#what-can-we-do-on-1x-for-02-aspects) - [03-DEPENDABLE](#03-dependable) - [What can we do on 1.x for 03-DEPENDABLE](#what-can-we-do-on-1x-for-03-dependable) - [04-STACK-ROOT](#04-stack-root) - [Preserving unique IDs using an ID of `Default`](#preserving-unique-ids-using-an-id-of-default) - [Alternative considered](#alternative-considered-1) - [Alternative 1: Breaking unique IDs](#alternative-1-breaking-unique-ids) - [Alternative 2: `node.relocate()`](#alternative-2-noderelocate) - [What can we do on 1.x for 04-STACK-ROOT](#what-can-we-do-on-1x-for-04-stack-root) - [05-METADATA-TRACES](#05-metadata-traces) - [What can we do on 1.x for 05-METADATA-TRACES](#what-can-we-do-on-1x-for-05-metadata-traces) - [06-NO-PREPARE](#06-no-prepare) - [What can we do on 1.x for 06-NO-PREPARE](#what-can-we-do-on-1x-for-06-no-prepare) - [07-NO-SYNTHESIZE](#07-no-synthesize) - [Implications on CDK code base](#implications-on-cdk-code-base) - [Implications on end-users](#implications-on-end-users) - [What can we do on 1.x for 07-NO-SYNTHESIZE](#what-can-we-do-on-1x-for-07-no-synthesize) - [08-VALIDATION](#08-validation) - [What can we do on 1.x for 08-VALIDATION](#what-can-we-do-on-1x-for-08-validation) - [09-LOGGING](#09-logging) - [What can we do on 1.x for 09-LOGGING](#what-can-we-do-on-1x-for-09-logging) - [Drawbacks](#drawbacks) - [User migration effort](#user-migration-effort) - [CDK codebase migration efforts](#cdk-codebase-migration-efforts) - [Staging of the 2.x fork](#staging-of-the-2x-fork) - [Rationale and Alternatives](#rationale-and-alternatives) - [Alternatives considered](#alternatives-considered) - [Adoption Strategy](#adoption-strategy) - [Unresolved questions](#unresolved-questions) - [Future Possibilities](#future-possibilities) - [Implementation Plan](#implementation-plan) - [Preparation of 1.x](#preparation-of-1x) - [constructs 10.x](#constructs-10x) - [2.x Work](#2x-work) - [Rejected Work](#rejected-work) - [10-CONSTRUCT-NODE](#10-construct-node-1) ## Release Notes > This section "works backwards" from the v2.0 release notes in order to > describe the user impact of this change. **BREAKING CHANGE**: As part of CDK v2.0, all types related to the _constructs programming model_ have been removed from the AWS CDK and should be used directly from the [constructs](https://github.com/aws/constructs) library. For most CDK libraries and apps, you will likely just need change this: ```ts import { Construct } from '@aws-cdk/core'; ``` With this: ```ts import { Construct } from 'constructs'; ``` --- The following table summarizes the API changes between 1.x and 2.x. The following sections describe all the related breaking changes and details migration strategies for each change. | 1.x | 2.x | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ | | `@aws-cdk/*` | `aws-cdk-lib` and `constructs@^4` | | `import { Construct } from '@aws-cdk/core'` | `import { Construct } from 'constructs'` | | `@aws-cdk/core.Construct` | `constructs.Construct` | | `@aws-cdk/core.IConstruct` | `constructs.IConstruct` | | `@aws-cdk/core.ConstructOrder` | `constructs.ConstructOrder` | | `@aws-cdk/core.ConstructNode` | `constructs.Node` | | `myConstruct.node.applyAspect(aspect)` | `Aspects.of(myConstruct).add(aspect)` | | `@aws-cdk/core.IDependable` | `constructs.IDependable` | | `@aws-cdk/core.DependencyTrait` | `constructs.Dependable` | | `@aws-cdk.core.DependencyTrait.get(x)` | `constructs.Dependable.of(x)` | | `myConstruct.node.dependencies` | Is now non-transitive | | `myConstruct.addMetadata()` | Stack trace not attached by default | | `ConstructNode.prepareTree()`,`node.prepare()`,`onPrepare()`,`prepare()` | Not supported, use aspects instead | | `ConstructNode.synthesizeTree()`,`node.synthesize()`,`onSynthesize()`,`synthesize()` | Not supported | | `myConstruct.onValidate()`,`myConstruct.validate()`hooks | Implement `constructs.IValidation` and call `myConstruct.node.addValidation()`instead | | `ConstructNode.validate(node)` | `myConstruct.node.validate()` | ### 00-DEPENDENCY: Declare a dependency on "constructs" As part of migrating your code to AWS CDK 2.0, you will need to declare a dependency on the `constructs` library (in addition to the `aws-cdk-lib` library which now includes the entire AWS CDK). For libraries, this should be a peer dependency, similarly to your dependency on the AWS CDK. You will likely also want to declare those as `devDependencies` in order to be able to run tests in your build environment. To increase interoperability of your library, the recommendation is to use the lowest possible **caret** version: ```json { "peerDependencies": { "aws-cdk-lib": "^2.0.0", "constructs": "^10.0.0" }, "devDependencies": { "aws-cdk-lib": "^2.0.0", "constructs": "^10.0.0" } } ``` For apps, you should declare these as direct dependencies, and you would normally want to use the highest version available: ```json { "dependencies": { "aws-cdk-lib": "^2.44.0", "constructs": "^10.787.0" } } ``` NOTE: Due to it's foundational nature, the `constructs` library is committed to never introduce breaking changes. Therefore, it's version will be `10.x`. ### 01-BASE-TYPES: Removal of base types The following `@aws-cdk/core` types have stand-in replacements in `constructs`: - The `@aws-cdk/core.Construct` class has been replaced with `constructs.Construct` - The `@aws-cdk/core.IConstruct` type has been replaced with `constructs.IConstruct` - The `@aws-cdk/core.ConstructOrder` class has been replaced with `constructs.ConstructOrder` - The `@aws-cdk/core.ConstructNode` class has been replaced with `constructs.Node` See [examples](https://github.com/aws/aws-cdk/pull/9056/commits/e4dff913d486592b1899182e9a928765553654fa). ### 02-ASPECTS: Changes in Aspects API Aspects are not part of the "constructs" library, and therefore instead of `construct.node.applyAspect(aspect)` use `Aspects.of(construct).add(aspect)`. The `Tag.add(scope, name, value)` API has been removed. To apply AWS tags to a scope, use: ```ts Tags.of(scope).add(name, value); ``` See [examples](https://github.com/aws/aws-cdk/pull/9056/commits/e4dff913d486592b1899182e9a928765553654fa). ### 03-DEPENDABLE: Changes to IDependable implementation If you need to implement `IDependable`: - The `@aws-cdk/core.IDependable` type has been replaced with `constructs.IDependable` - The `@aws-cdk/core.DependencyTrait` class has been replaced with `constructs.Dependable` - `@aws-cdk.core.DependencyTrait.get(x)` is now `constructs.Dependable.of(x)` - `c.node.dependencies` is now **non-transitive** and returns only the dependencies added to the current node. The method `c.node.addDependency(otherConstruct)` **did not change** and can be used as before. > You can use the new `c.node.dependencyGraph` to access a rich object > model for reflecting on the node's dependency graph. See [examples](https://github.com/aws/aws-cdk/pull/9056/commits/e4dff913d486592b1899182e9a928765553654fa). ### 04-STACK-ROOT: Stacks as root constructs It is common in unit tests to use `Stack` as the root of the tree: ```ts const stack = new Stack(); const myConstruct = new MyConstruct(stack, 'MyConstruct'); // make assertions ``` This is still a supported idiom, but in 2.x these root stacks will have an implicit `App` parent. This means that `stack.node.scope` will be an `App` instance, while previously it was `undefined`. The "root" stack will have a construct ID of `Default` unless otherwise specified. Please note that this also means that the value of `construct.path` for all constructs in the tree would now have a `Default/` prefix (if it was `Foo/Bar` it will now be `Default/Foo/Bar`). > In contrast, the value of `construct.uniqueId` will _not_ change because > `Default` is a special ID that is ignored when calculating unique IDs (this > feature already exists in 1.x). ### 05-METADATA-TRACES: Stack traces no longer attached to metadata by default For performance reasons, the `c.node.addMetadata()` method will _not_ attach stack traces to metadata entries. Stack traces will still be associated with all `CfnResource` constructs and can also be added to custom metadata using the `stackTrace` option: ```ts c.node.addMetadata(key, value, { stackTrace: true }); ``` See [examples](https://github.com/aws/aws-cdk/pull/9056/commits/e4dff913d486592b1899182e9a928765553654fa). ### 06-NO-PREPARE: The `prepare` hook is no longer supported The **prepare** hook (`construct.onPrepare()` and `construct.prepare()`) is no longer supported as it can easily be abused and cause construct tree corruption when the tree is mutated during this stage. Consider a design where you mutate the tree in-band, or use `Lazy` values or Aspects if appropriate. See [examples](https://github.com/aws/aws-cdk/pull/9056/commits/3d4fcb5ab72ca0777f3abfa2c4aa10e0d7deba6b). Although we recommend that you rethink the use of "prepare", you can use this idiom to implement "prepare" using aspects: ```ts Aspects.of(this).add({ visit: () => this.prepare() }); ``` The `ConstructNode.prepare(node)` method no longer exists. To realize references & dependencies in a scope call `Stage.of(scope).synth()`. ### 07-NO-SYNTHESIZE: The `synthesize` hook is no longer supported The `synthesize()` overload (or `onSynthesize()`) is no longer supported. Synthesis is now implemented only at the app level. If your use case for overriding `synthesize()` was to emit files into the cloud assembly directory, you can now find the current cloud assembly output directory during initialization using `Stage.of(this).outdir`. See [examples](https://github.com/aws/aws-cdk/pull/9056/commits/02b45b50c27fffae1bfbc4f61f6adc1f8fd92672). The `ConstructNode.synthesize(node)` method no longer exists. However, since now `Stage.of(scope)` is always defined and returns the enclosing stage/app, you can can synthesize a construct node through `Stage.of(scope).synth()`. For additional questions/guidance on how to implement your use case without this hook, please post a comment on [this GitHub issue](https://github.com/aws/aws-cdk/issues/8909). ### 08-VALIDATION: The `validate()` hook is now `node.addValidation()` To add validation logic to a construct, use `c.node.addValidation()` method instead of overriding a protected `validate()` method: Before: ```ts class MyConstruct extends Construct { protected validate(): string[] { return ['validation-error']; } } ``` After: ```ts class MyConstruct extends Construct { constructor(scope: Construct, id: string) { super(scope, id); this.node.addValidation({ validate: () => ['validation-error'] }); } } ``` The static method `ConstructNode.validate(node)` is no longer available. You can use `c.node.validate()` which only validates the _current_ construct and returns the list of all error messages returned from calling `validation.validate()` on all validations added to this node. ### 09-LOGGING: Logging API changes The `construct.node.addInfo()`, `construct.node.addWarning()` and `construct.node.Error()` methods are now available under the `Annotations.of(construct)` API: Instead of: ```ts myConstruct.node.addWarning('my warning'); ``` Use: ```ts import { Annotations } from '@aws-cdk/core'; Annotations.of(construct).addWarning('my warning'); ``` See [examples](https://github.com/aws/aws-cdk/pull/9056/commits/e4dff913d486592b1899182e9a928765553654fa). ## Motivation There are various motivations for this change: 1. Removal of a redundant layer 2. User confusion caused by multiple `Construct` types 3. Inability to compose AWS CDK constructs into other domains 4. Non-intuitive dependency requirement on `constructs` ### 1. Redundant layer The current compatibility layer does not have any logic in it. It is pure glue introduced in order to avoid breaking v1.x users. As we release v2.0 we are able to clean up this layer, and with it, improve maintainability and code hygiene. ### 2. Multiple `Construct` types The current situation is error-prone since we have two `Construct` classes in the type closure. For example, when a developer types `"Construct"` and uses VSCode to _automatically add import statements_, the IDE will actually add an import for `constructs.Construct`. If they define a custom construct class which extends this type instead of the `core.Construct` type, it won't be possible to pass an instance of this class as a scope to AWS CDK constructs such as `Stack` for example. ### 3. Composability with other domains The main motivation for this change is to enable composition of AWS CDK constructs with constructs from other domains. It is currently impossible to define AWS CDK constructs within a non-AWS-CDK construct scope. For example, consider the [CDK for Terraform](https://github.com/hashicorp/terraform-cdk) or a similar project, which uses constructs to define stacks through Terraform. We are currently working with HashiCorp to enable the following use case in the Terraform CDK: ```ts import * as cdktf from 'cdktf'; // <=== Terraform CDK import * as s3 from '@aws-cdk/aws-s3'; // <=== AWS CDK const stack = new cdktf.TerraformStack(...); // COMPILATION ERROR: `this` is of type `constructs.Construct` and not a `@aws-cdk/core.Construct`. new s3.Bucket(this, 'my-bucket'); ``` In order to enable this usage, we will need `s3.Bucket` to accept any object that implements `constructs.Construct` as its `scope`. At the moment, this will fail compilation with the error above, because the `scope` in `s3.Bucket` is `core.Construct`. Being able to create compositions from multiple CDK domains is a powerful direction for the CDK ecosystem, and this change is required in order to enable these use case. ### 4. Non-intuitive dependency requirement As we transition to [monolithic packaging] as part of v2.x, CDK users will have to take a _peer dependency_ on both the CDK library (`aws-cdk-lib`) and `constructs`. > Currently, the AWS CDK also takes `constructs` as a normal dependency (similar > to all dependencies), but this is about to change with mono-cdk. The reason `constructs` will have to be defined as a peer-dependency of the AWS CDK, whether we leave the compatibility layer or not, is due to the fact that all AWS CDK constructs eventually extend the base `constructs.Construct` class. This means that this type is part of their public API, and therefore must be defined as a peer dependency (otherwise, there could be incompatible copies of `Construct` in the dependency closure). The removal of the compatibility layer means that now anyone who uses the AWS CDK will need to explicitly use the `constructs.Construct` type (even for trivial apps), and therefore it would "make sense" for them to take a dependency on the `constructs` library. See the RFC for [monolithic packaging] for more details. [monolithic packaging]: https://github.com/aws/aws-cdk-rfcs/blob/master/text/0006-monolothic-packaging.md ## Design This section analysis the required changes and discusses the implementation approach and alternatives. For each change, we added a **What can we do on 1.x?** section which discusses our strategy for front-loading the change into the 1.x branch to reduce forking costs and/or alert users of the upcoming deprecation. This design is based on this [proof of concept](https://github.com/aws/aws-cdk/pull/9056). ### 00-DEPENDENCY In order to enable composability with other CDK domains (Terraform, Kubernetes), all constructs must use the same version of the `Construct` base class. As long as all libraries in a closure take a **peer dependency** on a compatible version of `constructs`, the npm package manger will include a single copy of the library, and therefore all constructs will derive from the same `Construct` (and more importantly, accept the same `Construct` for `scope`). Practically this means that we can never introduce a major version of `constructs` because any major version will require a new major version of all CDKs, and that is impossible to require or coordinate given the decentralized nature of the ecosystem. We propose to take a commitment to **never introduce breaking changes in "constructs"**. This implies that we will never introduce another major version. To symbolize that to users, we will use the major version `10.x`. ### 01-BASE-TYPES Once `construct-compat.ts` is removed from `@aws-cdk/core`, all CDK code (both the framework itself and user code) would need to be changed to use the types from `constructs`. Since the APIs are similar in almost all cases, this is a simple mechanical change as shown in the [release notes](#release-notes). #### What can we do on 1.x The main concern for the CDK codebase is maintaining this change alongside a 1.x branch until we switch over to 2.x. Since this change includes modifications to all `import` sections in almost all of our files, merge conflicts are imminent. To reduce these costs, we propose to modify the `scope` argument on all 1.x to accept `constructs.Construct` instead of `core.Construct`. This will provide the following benefits (enforced by an `awslint` rule): 1. The `import` statement for `import { Construct } from 'constructs'` would already exist which will reduce merge conflicts. 2. It will unlock composition of framework constructs into other domains (e.g. it will be possible to pass an AWS CDK L2 to a terraform stack). Note that we will _not_ change the base classes to `constructs.Construct` because it is technically (and practically) a breaking API change (we must maintain the invariant that "`s3.Bucket` is a `core.Construct`". The remaining change in 2.x will be to update any base classes to use `constructs.Construct` but this is actually not very prevalent in the framework code because the majority of constructs actually extend `core.Resource`. Alternatives considered: - **Do nothing in 1.x**: will incur an ongoing maintenance cost of the 1.x -> 2.x merge conflicts. - **Automatic merge resolution and import organization**: requires research and development, not necessarily worth it. ### 10-CONSTRUCT-NODE Since we won't be able to release additional major versions of the "constructs" library (in order to ensure interoperability between domains is always possible), we need to closely examine the API of this library. In particular, the API of the `Construct` class, which is the base of all constructs in all CDKs, should be as small as possible in order not to "pollute" domain-specific APIs introduced in various domains. In many cases (cdk8s, cdktf), constructs are _generated_ on-demand from domain-specific API specifications. In such cases, we need to ensure that the API in `Construct` does not conflict with generated property names or methods. The current API of `Construct` in the base class (2.x, 3.x) only includes a few protected `onXxx` methods (`onPrepare`, `onValidate` and `onSynthesize`). Those methods will be removed in 10.x ([prepare](#06-no-prepare) and [synthesize](#07-no-synthesize) are no longer supported and [validate](#08-validation) will be supported through `addValidation()`). In AWS CDK 1.x the construct API is available under `myConstruct.node`. This API has been intentionally removed when we extracted "constructs" from the AWS CDK in order to allow the compatibility layer in AWS CDK 1.x to use the same property name and expose the shim type (jsii does not allow changing the type of a property in a subclass). The base library currently offers `Node.of(scope)` as an alternative - but this API is cumbersome to use and not discoverable. In evidence, in CDK for Terraform, they chose to offer [`constructNode`] in `TerraformElement` as a sugar for `Node.of()`. [`constructnode`]: https://github.com/hashicorp/terraform-cdk/blob/5becfbc699180adfe920dec794200bbf56dda0a7/packages/cdktf/lib/terraform-element.ts#L21 Another downside of `Node.of()` is that it means that the `IConstruct` interface is now an empty interface, which is a very weak type in TypeScript due to structural typing (it's structurally identical to `any`). As we evaluate this use case for constructs 10.x, we would like to restore the ability to access the construct API from a property of `Construct`, and use that property as the single marker that represents a construct type (`IConstruct`). To reduce the risk of naming conflicts (e.g. see [Terraform CDK issue](https://github.com/hashicorp/terraform-cdk/pull/230)) between `node` and domain-specific APIs, we propose to introduce this API under the name `construct` (of type `Node`). This has a few benefits: 1. It's semantically signals that "this is the construct API". 2. The chance for conflicts with domain-specific names is low ("construct" is not prevalent). 3. We can introduce this API while deprecating `node` in AWS CDK 1.x. The main downside is that it is **a breaking change** in AWS CDK 2.x. There is likely quite a lot of code out there (a [few hundred](https://github.com/search?q=cdk+node.addDependency++extension%3Ats&type=Code&ref=advsearch&l=&l=) results for an approximated GitHub code search). > We also considered the name `constructNode` as an alternative but there is no > additional value in the word "node" being included, especially given the type > is `Node`. #### What can we do in 1.x As mentioned above, since this is a new name for this property, we can technically introduce it in 1.x and announce that `node` is deprecated. This will allow users to migrate to the new API before 2.x is released and hopefully will reduce some of the friction from the 2.x migration. To encourage users to migrate, we will consider introducing this deprecation through a runtime warning message (as well as the @deprecated API annotation). #### Alternative Considered See [discussion](https://github.com/aws/aws-cdk-rfcs/pull/195/files#r460718960) over the RFC PR. ### 02-ASPECTS Aspects are actually a form of "prepare" and as such, if they mutate the tree, their execution order becomes critical and extremely hard to get right. To that end, we decided to remove them from the `constructs` library as they pose a risk to the programming model. However, we are aware that aspects are used by AWS CDK apps and even 3rd party libraries such as [cdk-watchful](https://github.com/eladb/cdk-watchful). Therefore, we propose to continue to support aspects in 2.x, with the goal of rethinking this programming model for a future major version. One future direction is to turn aspects into "reactive" so that they subscribe to tree events and react in-band during initialization, and not as a separate phase. Since aspects are no longer part of the base programming model, we need a way to apply aspects to scopes for AWS CDK apps. To do that, we propose to use the "trait" pattern, which is becoming a common idiom for offering APIs over CDK scopes: ```ts Aspects.of(scope).add(aspect); ``` The major downside of this change is discoverability, but `construct.node.applyAspect` is not necessarily more discoverable. We will make sure documentation is clear. We will use this opportunity to normalize the tags API and change it to use the same pattern: `Tags.of(x).add(name, value)`. #### What can we do on 1.x for 02-ASPECTS - We will migrate the 1.x branch to `Aspects.of(x)` and add a deprecation warning to `this.node.applyAspect()`. - Introduce `Tags.of(x).add()` and add a deprecation warning to `Tag.add()`. ### 03-DEPENDABLE The `constructs` library supports dependencies through `node.addDependency` like in 1.x, but the API to implement `IDependable` has been changed. The `constructs` library also introduces `DependencyGroup` which is a mix between `CompositeDependable` and `ConcreteDependable`. #### What can we do on 1.x for 03-DEPENDABLE It should be possible to migrate the 1.x codebase to use the new APIs without any breaking changes to users. ### 04-STACK-ROOT If we move staging of assets to the initialization phase, it means we need to know at that time where is the cloud assembly output directory. As mentioned above, in production this information is available from the enclosing `Stage`. This works in production but introduces a minor issue with unit tests which use a `Stack` as the root construct. This is a very common pattern in the CDK codebase today: ```ts const stack = new Stack(); const foo = new MyFoo(stack, 'MyFoo'); expect(stack).to(haveResource('AWS::Foo')); ``` In such cases, assets will not work because the output directory is only determined later (in the `expect` call). One alternative would be to simply modify these unit tests so that stacks are no longer used as roots (basically add `new App()` as the scope). This approach would require a change in many unit tests across the code base with no clear value to users. We propose to modify the `Stack` construct such that if a stack is created without an explicit `scope`, an `App` instance will automatically be created and used as it's scope. ```ts const stack = new Stack(); assert(stack.node.scope instanceof App); // previously it was `undefined` ``` Since only the root construct may have an empty ID, we will also need to assign an ID. We propose to use `"Stack"` since we already have fallback logic that uses this as the stack name when the stack does not have an ID (see [stack.ts](https://github.com/aws/aws-cdk/blob/8c0142030dce359591aa76fe314f19fce9eddbe6/packages/%40aws-cdk/core/lib/stack.ts#L920)). This change will allow us to remove any special casing we have for stacks in the testing framework and throughout the synthesis code path (we have quite a lot of that), because we will be able to assume that `Stage.of(construct.node.root)` is never `undefined` and has a `synth()` method which returns a cloud assembly. Unit tests sometimes use "incremental tests" for synthesized templates. For example: ```ts const stack = new Stack(); const c1 = new MyConstruct(stack, 'c1', { foos: ['bar'] }); expect(stack).toHaveResource('AWS::Resource', { Foos: ['bar'], }); // now add a "foo" and verify that the synthesized template contains two items c1.addFoo('baz'); expect(stack).toHaveResource('AWS::Resource', { Foos: ['bar', 'baz'], }); ``` Since `stage.synth()` (which is called by `expect(stack)`) would reuse the synthesized output if called twice, we will also need to introduce a `stage.synth({ force: true })` option. This will be the default behavior when using `expect(stack)` or `SynthUtils.synth()`. #### Preserving unique IDs using an ID of `Default` A side effect of adding a `App` parent to "root" stacks is that we now have an additional parent scope for all constructs in the tree. The location of the construct in the tree is taken into account when calculating `node.path` and `node.uniqueId`. Since `uniqueId` is used in several places throughout the AWS Construct Library to allocate names for resources, and we have multiple unit tests that expect these values, we will use the ID `Default` for the root stack. The `uniqueId` algorithm in the constructs library (see [reference](https://github.com/aws/constructs/blob/d5c3ee0a89e09318838f78bfb89832c0548d846d/src/private/uniqueid.ts#L33)) ignores any node with the ID `Default` for the purpose of calculating the unique ID, which allows us to perform this change without breaking unique IDs. We will accept the fact that `node.path` is going to change for this specific use case (only relevant in tests). #### Alternative considered ##### Alternative 1: Breaking unique IDs We explored the option of fixing all these test expectations throughout the CDK code base and back port this change over the 1.x behind a feature flag in order to reduce the potential merge conflicts between 1.x and 2.x. The downsides of this approach are: 1. This is technically a breaking (behavioral) change for end-users since `node.path` and `node.uniqueId`, and their derivatives, will change for trees rooted by a `Stack`, and unit tests will need to be updated. 1. We currently don't have a way to implicitly run all our unit tests behind a feature flag, and it is not a trivial capability to add. ##### Alternative 2: `node.relocate()` We also explored the option of introducing an additional capability to constructs called `node.relocate(newPath)` which allows modifying the path of a scope such that all child scopes will automatically be "relocated" to a new path. This would have allowed avoiding the breakage in `node.path` but would have also introduced several other idiosyncrasies and potential violations of invariants such as the fact that a path is unique within the tree. #### What can we do on 1.x for 04-STACK-ROOT We will introduce this change over the 1.x branch as-is, acknowledging that we are technically breaking the behavior of `node.path` in unit tests which use `Stack` as the root. Since we are not breaking `uniqueId`, we expect this to be tolerable over the 1.x branch. ### 05-METADATA-TRACES Since stack traces are not attached to metadata entries by default in constructs 4.x, we will need to pass `stackTrace: true` for `CfnResource`s. This will preserve the deploy-time stack traces which are very important for users. Other metadata entries will not get stack traces by default, and that's a reasonable behavioral change. #### What can we do on 1.x for 05-METADATA-TRACES No need to introduce over 1.x as the change is very local to `CfnResource` and therefore can be applies over 2.x without risk. ### 06-NO-PREPARE The "prepare" hook was removed from constructs since it is a very fragile API. Since the tree can be mutated during prepare, the order of `prepare` invocations becomes critical, and almost impossible to get right without a rich model of relationships between these "prepare" calls. The prepare hook was used in the CDK in a few cases: 1. Resolution of references across stacks and nested stacks 2. Resolution of dependencies between constructs across stacks 3. Calculation of logical IDs based on hashes of other resources (API GW Deployment, Lambda Version). The first two use cases have already been addressed by centralizing the "prepare" logic at the stage level (into [prepare-app.ts](https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/core/lib/private/prepare-app.ts)). #### What can we do on 1.x for 06-NO-PREPARE - The 3rd item can be addressed using `Lazy` tokens (see [example](https://github.com/aws/aws-cdk/pull/8962/files#diff-51d435d71a31c2607f923fc4d96cac56R140)), and will be addressed on 1.x prior to the 2.x fork. - We will also add a deprecation warning on 1.x which will identify constructs that implement "prepare" and refer users to a GitHub issue for details and consultation. ### 07-NO-SYNTHESIZE Version 4.x of the `constructs` library does not contain a lifecycle hook for synthesis as described [above](#the-synthesize-hook-is-no-longer-supported). The reason this is not available at the base class is because the abstraction did not "hold water" as the AWS CDK evolved and new CDKs emerged. In the AWS CDK, we eventually ended up with a centralized synthesis logic at the `Stage`-level ([synthesis.ts](https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/core/lib/private/synthesis.ts)). The main reason was that we needed to "break" the recursion in various domain-specific points (e.g. stages, nested stacks) which meant that the generic logic of "traverse the entire tree and call `synthesize`" did not hold. In `cdk8s`, the support for [chart dependencies](https://github.com/awslabs/cdk8s/blob/ef95b9ffce8a39200e028c2fe8acc55a9915161c/packages/cdk8s/src/app.ts#L39) required that the name of the output manifest will be determined based on the topologic order at the app level. Here again, the generic approach failed. In lieu of those failures, we decided that there is no additional value in actually offering a synthesis mechanism at the `constructs` level. Each CDK-domain implements synthesis at the "right" level. This does not mean that specific domains can't offer a decentralized approach (i.e. call a method called "synthesize" on all constructs in the tree), it just means that this is not provided at the base `Construct` class. #### Implications on CDK code base In the AWS CDK itself, `synthesize()` was used in three locations: 1. `Stack` - creates a CFN template and adds itself to the cloud assembly manifest. 2. `AssetStaging` - stages an asset into the cloud assembly. 3. `TreeMetadata` - creates `tree.json` with information about the construct tree. For `Stack` and `TreeMetadata`, we will convert the generic `synthesize()` method to `_synthesizeTemplate()` and `_synthesizeTree()` and will call them from the centralized synthesis function. The `AssetStaging` construct does not really need to wait until synthesis in order to stage the asset. In fact, all the information required already exists during initialization. The only missing information is the cloud assembly output directory, and this information is actually known during initialization (we know this as soon as the CDK app is created). Therefore, the solution for `AssetStaging` is to move the staging logic to the constructor and use `Stage.of(this).outdir` to find the output directory of the current stage. #### Implications on end-users Participation in synthesis is an "advanced" feature of the CDK framework and we assume most end-users don't use this directly. If they need "last minute processing", they can add an aspect to the node which will be applied before synthesis (the alternative to "prepare"). The use case of emitting arbitrary files into the cloud assembly directory is weak. The cloud assembly is a well-defined format, and is currently "closed". There are no extension points that tools can identify. To that end, just writing files to the cloud assembly output directory does not make tons of sense. Yet, if there is still a use case for writing files during initialization, it is possible to find out the output directory through `Stage.of(scope).outdir`. This is how asset staging will be implemented. #### What can we do on 1.x for 07-NO-SYNTHESIZE 1. The framework changes should be done on the 1.x branch as they are non-breaking. 2. We will also add a deprecation notice that identifies the existence of a `synthesize()` method on a construct (during synthesis) and warns users that this hook will no longer be available in 2.x, offering a GitHub issue for details consultation. ### 08-VALIDATION Since construct validation is quite rare and we want to encourage users to validate in entry points, in constructs 4.x, the `validate()` protected method was removed and `node.addValidation()` can be used to add objects that implement `IValidation`. An error will be thrown if a `validate()` method is found on constructs with instructions on how to implement validation in 2.x. #### What can we do on 1.x for 08-VALIDATION We can introduce this change over the 1.x as long as we continue to support `validate()` alongside a deprecation warning with instructions on how to migrate to the new API. ### 09-LOGGING We decided that logging is not generic enough to include in `constructs`. It emits construct metadata that is very CLI specific (e.g. `aws:cdk:warning`) and currently there is no strong abstraction. To continue to enable logging, we will utilize the `Annotations.of(x).addWarning()` pattern. #### What can we do on 1.x for 09-LOGGING We can introduce this change on 1.x and add a deprecation warning. ## Drawbacks ### User migration effort The main drawback from users' point of view is the introduction of the aforementioned breaking changes as part of the transition to CDK 2.0. As mentioned above, for the majority of users, the migration will be trivial and mechanical (import from "constructs" instead of "@aws-cdk/core"). The removal of the "prepare" and "synthesize" hooks may require users to rethink their design in very advanced scenarios. We will create a GitHub issue to consult users on alternative designs. ### CDK codebase migration efforts The AWS CDK codebase itself utilizes all of these APIs, and the migration effort is quite substantial. Having said that, the majority of this work is already complete and a branch is being maintained with these changes as a pre-cursor to the v2.x fork. ### Staging of the 2.x fork Since this change involves modifications to the CDK's source code, it may cause merge conflicts as during the period in which we need to forward-port or back-port code between the v1.x branch and the v2.x branches. The key would be to continuously merge between the branches. ## Rationale and Alternatives As a general rule, software layers which do not provide value to users should not exist. The constructs compatibility layer was added as solution for maintaining backwards compatibility within the v1.x version line while we extract `constructs` into an independent library. Redundant layers are expensive to maintain and are prone to idiosyncrasies as they evolve over time (for example, a CDK engineer may be tempted to add an AWS-specific feature in this layer, making it harder to clean up later). If we consider the various [reasons](#drawbacks) not to take this change, the main reason would be to simplify the [migration for users from 1.x to 2.x](#user-migration-effort). The major version 2.x is already required to introduce monolithic packaging, and this change, for most users, is likely to be trivial (see [above](#breaking-changes)). Therefore, we believe this is probably not the correct motivation to reject this proposal. The [repository migration](#repository-migration-efforts) efforts and [co-existence of 2.x/1.x](#co-existence-of-2x1x) are both one-off costs this proposal suggests ways to reduce the chance for merge conflicts across these branches. ### Alternatives considered At a high-level, we may consider to postpone this change to v3.x or to never take it, leaving this compatibility layer in place for eternity. If we examine the various [motivations](#motivation) for this change, we may come up with various alternatives, all of which eventually cause a breaking change to our users. For example, we considered only changing the type of the `scope` argument to all CDK constructs to use `constructs.Construct`, while the base class will still extend `cdk.Construct`. This will likely confuse users who design their own constructs as they won't know which construct to extend, the two base classes will slowly diverge from each other as both layers evolve. Another alternative is to rename `cdk.Construct` to something like `AwsConstruct`. This, would take up most of the cost of this change (which is the CDK codebase change and merge risks against the fork). Postponing to v3.x will leave us with the set exact set of problems, only with a more mature ecosystem which is harder to migrate off of. The [Design](#design) section describes alternatives for various aspects of this project. ## Adoption Strategy See [Release Notes](#release-notes). ## Unresolved questions - [ ] Automation of `import` conflict resolution. > - What parts of the design do you expect to resolve through the RFC process > before this gets merged? > - What parts of the design do you expect to resolve through the implementation > of this feature before stabilization? > - What related issues do you consider out of scope for this RFC that could be > addressed in the future independently of the solution that comes out of this > RFC? ## Future Possibilities > Think about what the natural extension and evolution of your proposal would be > and how it would affect CDK as whole. Try to use this section as a tool to > more fully consider all possible interactions with the project and ecosystem > in your proposal. Also consider how this fits into the roadmap for the > project. > > This is a good place to "dump ideas", if they are out of scope for the RFC you > are writing but are otherwise related. > > If you have tried and cannot think of any future possibilities, you may simply > state that you cannot think of anything. ## Implementation Plan ### Preparation of 1.x We will try to front load as much of this change to 1.x in order to reduce the merge conflict potential. To that end, we will continuously merge from "master" into [the POC branch](https://github.com/aws/aws-cdk/pull/9056) and slowly back port changes from it into `master` as much as possible. The goal is the minimize the changes between master and the POC branch which will be merged into the 2.x branch once created. - [01-BASE-TYPES](#01-base-types) - [ ] Normalize reference to base types (`cdk.Construct` => `Construct`). - [ ] Use an `awslint` rule to modify the `scope` argument on all 1.x to accept `constructs.Construct` instead of `core.Construct` - [10-CONSTRUCT-NODE](#10-construct-node) - [ ] Introduce `c.construct` as an alias to `c.node` - [ ] Deprecate `c.node` (with a warning message) - [ ] Replace all `c.node` with `c.construct`. - [02-ASPECTS](#02-aspects) - [ ] Introduce `Aspects.of(x)` and deprecate `applyAspect` - [ ] Introduce `Tags.of()` and deprecate `Tag.add()` - [03-DEPENDABLE](#03-dependable) - [ ] Introduce `Dependable` as an alias to `DependencyTrait`, introduce `DependencyGroup` - [04-STACK-ROOT](#04-stack-root) - [ ] Introduce `node.relocate()` in constructs 3.x - [ ] Implement implicit `App` for root `Stack`s, relocated to "". - [05-METADATA-TRACES](#05-metadata-traces) - N/A - [06-NO-PREPARE](#06-no-prepare) - [ ] Back port [this commit](https://github.com/aws/aws-cdk/pull/9056/commits/3d4fcb5ab72ca0777f3abfa2c4aa10e0d7deba6b) to master - [ ] Add a deprecation warning if `onPrepare()` or `prepare()` is identified on a construct during synthesis - [07-NO-SYNTHESIZE](#07-no-synthesize) - [ ] Back port the changes related to synthesis from [this commit](https://github.com/aws/aws-cdk/pull/9056/commits/02b45b50c27fffae1bfbc4f61f6adc1f8fd92672) - [ ] Add a deprecation warning if `onSynthezize()` or `synthesize()` is declared on a construct - [08-VALIDATION](#08-validation) - [ ] Introduce `node.addValidation()` and deprecate `validate()` and `onValidate()` by back porting [this commit](https://github.com/aws/aws-cdk/pull/9056/commits/42bd929aa36dc97ae39b15b77cf1f69d754c4a92) to master - [09-LOGGING](#09-logging) - [ ] Introduce `Annotations.of()` deprecate `node.addWarning/error/info`. ### constructs 10.x [This branch](https://github.com/aws/constructs/pull/133) is the staging branch for constructs 10.x. - [00-DEPENDENCY](#00-dependency) - [ ] Document API compatibility assurance and the 10.x version number. - [01-BASE-TYPES](#01-base-types) - [x] Reintroduce `Construct.isConstruct()`. - [02-ASPECTS](#02-aspects) - [x] Remove aspects (`IAspect` and `node.applyAspect`). - [03-DEPENDABLE](#03-dependable) - [x] Reintroduce dependencies (`IDependable`, `Dependable`, `DependencyGroup`) - [x] Change `node.dependencies` to return the list of node dependency (non recursive) and add `node.depgraph` which returns a `Graph` object from cdk8s. - [x] Change `addDependency` to accept `IDependable` instead of `IConstruct`. - [x] Return only local dependencies in `node.dependencies` - [ ] Migrate [DependencyGraph](https://github.com/awslabs/cdk8s/blob/master/packages/cdk8s/src/dependency.ts) from cdk8s into `constructs`. - [04-STACK-ROOT](#04-stack-root) - N/A - [05-METADATA-TRACES](#05-metadata-traces) - [x] Do not emit stack traces in `addMetadata` (`{ stackTrace: true }`). - [06-NO-PREPARE](#06-no-prepare) - [x] Removal of `onPrepare` and `node.prepare()` - [07-NO-SYNTHESIZE](#07-no-synthesize) - [x] Removal of `onSynthesize` and `node.synthesize()` - [ ] Expose `lock()` and `unlock()`. - [08-VALIDATION](#08-validation) - [ ] Introduce `IValidation`, `addValidation()` and `node.validate()`. - [09-LOGGING](#09-logging) - [x] Remove `node.addWarning()`, `node.addError()`, ... ### 2.x Work - [ ] Once the 2.x branch will be created, we will merge the remaining changes from the POC branch into it. - [ ] Write a migration guide with guidance on `synthesize` and `prepare` in - [ ] Updates to Developer Guide (add 2.x section) - [ ] Updates to READMEs across the library (add 2.x section) ## Rejected Work The following section includes sections that were rejected from inclusion in this project. ### 10-CONSTRUCT-NODE Originally, we wanted to rename the `node` property to `construct` (`bucket.construct` instead of `bucket.node`). The motivation is described below in the original text. We eventually decided not to include this change because languages like .NET do not allow properties to have the same name as the class `Construct` and `Construct` (this name is preserved in .NET to the constructor), and jsii does not have any support for per-language naming. We also considered the name `constructNode` as an alternative but there is no additional value in the word "node" being included, especially given the type is `Node`. References: - Pull request to change all occurances of `node` to `construct` in the AWS CDK codebase: - Commit which reverts this change after jsii packaging failure: The original text follows: > Since we won't be able to release additional major versions of the > "constructs" library (in order to ensure interoperability between domains is > always possible), we need to closely examine the API of this library. > > In particular, the API of the `Construct` class, which is the base of all > constructs in all CDKs, should be as small as possible in order not to > "pollute" domain-specific APIs introduced in various domains. > > In many cases (cdk8s, cdktf), constructs are _generated_ on-demand from > domain-specific API specifications. In such cases, we need to ensure that the > API in `Construct` does not conflict with generated property names or methods. > > The current API of `Construct` in the base class (2.x, 3.x) only includes a > few protected `onXxx` methods (`onPrepare`, `onValidate` and `onSynthesize`). > Those methods will be removed in 10.x ([prepare](#06-no-prepare) and > [synthesize](#07-no-synthesize) are no longer supported and > [validate](#08-validation) will be supported through `addValidation()`). > > In AWS CDK 1.x the construct API is available under `myConstruct.node`. This > API has been intentionally removed when we extracted "constructs" from the AWS > CDK in order to allow the compatibility layer in AWS CDK 1.x to use the same > property name and expose the shim type (jsii does not allow changing the type > of a property in a subclass). > > The base library currently offers `Node.of(scope)` as an alternative - but > this API is cumbersome to use and not discoverable. In evidence, in CDK for > Terraform, they chose to offer [`constructNode`] in `TerraformElement` as a > sugar for `Node.of()`. > > Another downside of `Node.of()` is that it means that the `IConstruct` > interface is now an empty interface, which is a very weak type in TypeScript > due to structural typing (it's structurally identical to `any`). > > As we evaluate this use case for constructs 10.x, we would like to restore the > ability to access the construct API from a property of `Construct`, and use > that property as the single marker that represents a construct type > (`IConstruct`). > > To reduce the risk of naming conflicts (e.g. see > [Terraform CDK issue](https://github.com/hashicorp/terraform-cdk/pull/230)) > between `node` and domain-specific APIs, we propose to introduce this API > under the name `construct` (of type `Node`). > > This has a few benefits: > > 1. It's semantically signals that "this is the construct API". > 2. The chance for conflicts with domain-specific names is low ("construct" is > not prevalent). > 3. We can introduce this API while deprecating `node` in AWS CDK 1.x. > > The main downside is that it is **a breaking change** in AWS CDK 2.x. There is > likely quite a lot of code out there (a > [few hundred](https://github.com/search?q=cdk+node.addDependency++extension%3Ats&type=Code&ref=advsearch&l=&l=) > results for an approximated GitHub code search). > > See > [discussion](https://github.com/aws/aws-cdk-rfcs/pull/195/files#r460718960) > over the RFC PR. [`constructnode`]: https://github.com/hashicorp/terraform-cdk/blob/5becfbc699180adfe920dec794200bbf56dda0a7/packages/cdktf/lib/terraform-element.ts#L21