<!--
Copyright 2015-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"). You may
not use this file except in compliance with the License. A copy of the
License is located at

http://aws.amazon.com/apache2.0/

or in the "license" file accompanying this file. This file is distributed
on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied. See the License for the specific language governing
permissions and limitations under the License.
-->

# Support Custom ECS Fields in Compose commands

## Introduction
([Issue #267](https://github.com/aws/amazon-ecs-cli/issues/267))

For testing containers in a local development environment, the ECS CLI currently depends on docker compose, with support for [Composefiles up through V2](https://docs.docker.com/compose/compose-file/compose-file-v2).

API calls that use the compose file include the following and their subcommands:
* [ecs-cli compose](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose.html)
* [ecs-cli compose service](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose-service.html)

The Composefile is translated into an [ECS Task Definition](http://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_TaskDefinition.html). We depend on [libcompose](https://github.com/docker/libcompose) to read/parse the Composefile.

Currently, our task definitions allow us to specify:
* Family: Generated by the CLI using a prefix and project name
* ContainerDefinitions: Parsed from `services` field in Composefile
* Volumes: Parsed from `volumes` fields in Composefile
* TaskRoleArn: specified by flag `--task-role-arn`

## Parameters currently unsupported (or partially supported) by ECS CLI:
(See: [Task definition parameters](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html), [Task Definition Template](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-task-definition.html))

### Top-level fields
* `networkMode` [Issue # 153](https://github.com/aws/amazon-ecs-cli/issues/153)
	* The Docker networking mode to use for the containers in the task. The valid values are none, bridge, and host.
	* If the network mode is none, the containers do not have external connectivity. The default Docker network mode is bridge. The host network mode offers the highest networking performance for containers because it uses the host network stack instead of the virtualized network stack provided by the bridge mode.
* `placementConstraints`
	* Array of [TaskPlacementConstraint objects](http://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_TaskDefinitionPlacementConstraint.html).
* `taskRoleArn` [Issue #188](https://github.com/aws/amazon-ecs-cli/issues/188) (currently uses `--task-role-arn` flag, submitted in PR [#247](https://github.com/aws/amazon-ecs-cli/pull/247))

### ContainerDefinition fields:
* `essential` (within container definition)

## Possible Approaches

### Option 1: Add flags
Support for taskRoleArn was added with the use of a flag ([#247](https://github.com/aws/amazon-ecs-cli/pull/247)). We could add all the fields not currently supported as flags. We could take the same approach for other ECS fields.

*Pros*:
* Relatively easy to implement
* Modular

*Cons*:
* Would be difficult/impossible to deprecate the flags once implemented. Opens up question of whether we want to add versioning the client.
* Adding a new flag any time ECS adds a new field to Task Def could get clunky, though the Docker CLI seems to take this approach.
* Adds verbosity to the CLI's commands

### Option 2: Use multiple files
Since we now support multiple Composefiles thanks to [PR#263](https://github.com/aws/amazon-ecs-cli/pull/263), we could pass in custom fields through a file with a specific naming convention. This option continues to maintain a separation between using the ecs-cli to bridge the docker-compose workflow with a native ECS developer experience.

*Pros*:
* Keeps all ECS-specific fields separate from Composefile

*Cons*:
* Less streamlined experience
* Requires parsing a possibly new file format

### Option 3: Use single Composefile, and/or allow passing in Task Definition as a primitive
Since the Composefile is translated into an ECS Task Definiion, we could specify our own template format that would be more in line with our own Task Definitions. This would either remove the dependency on docker-compose or drastically abstract away the coupling to docker-compose.

*Pros*:
* More streamlined user experience that would be more "ECS native"
* De-couples us from docker-compose and libcompose
* Would be a step towards the larger ECS Local DevX goal

*Cons*:
* May require a shim to continue working directly with docker-compose and maintain backwards compatibility
* Would be difficult to integrate with libcompose, and possibly require implementing a different parsing tool.
* May break existing workflows that use docker-compose
* Broadens scope of current feature request

## Issues

We currently depend on [libcompose](https://github.com/docker/libcompose) to parse the Composefile. This dependency ties us to only being able to support versions of the Composefile that libcompose also supports.

## Related work:
* Create new API that allows converting an ECS Task Definiton (json or yaml) into a Docker Composefile
* container-transform: allows converting Composefile to Task Def

# Proposal: Option 2

While Option 1 would likely be the fastest solution to achieve the desired functionality, its impact on the extensibility of the CLI made it not the ideal route. Option 3 is more aligned with the broader vision of designing a new local development experience using the ECS CLI, but was not compatible enough with existing workflows based on docker-compose and would have a much broader scope than just delivering the desired functionality.

Currently, the local development experience of a large number of our customers is based around using docker-compose. We want to continue supporting customers who use docker-compose as they are using it today, while allowing ECS custom fields to be used in production. This keeps ECS fields separate from fields that can only really be tested locally, as the fields not currently supported are dependent on calling the ECS backend.

While we currently provide the ability to pass in more than one compose file (cf: [PR#263](https://github.com/aws/amazon-ecs-cli/pull/263)), the use case for that feature was to allow for overrides to a base compose file in order for customers to more easily configure different environments. These overrides, however, were on fields only supported by Docker Compose. The fields we are trying to support here are specific to ECS Task Definitions. Passing them in through a separate file would allow customers to maintain the integrity of existing composefiles as well as clearly communicate any differences between Docker services and ECS container definitions. It would also allow for more modular validation of any ECS-specific data.

## Implementation

* Pass in a file with ECS task definition fields via a new flag, `--task-def-fields` (Other possible names: `--ecs-fields`, `ecs-task-def-fields`, `ecs-compose`, `--ecs-extensions`, `ecs-task-definition`).
* File format would be YAML and only include the following fields:
    * `task_role_arn` as a top-level field.
    * `ecs_network_mode` as a top-level field.

       This corresponds to [network_mode](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#network_mode) in an ECS Task Definition, but with the added prefix to disambiguate from the existing [network_mode](https://docs.docker.com/compose/compose-file/#network_mode) field specified by Docker Compose. This is because the docker compose network mode is specified per container, and ECS does not provide that level of granularity for this field in our [ContainerDefinition](http://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html). Valid values for this field are "none", "host", "bridge" and other custom network modes defined by ECS. Any invalid string will return an error, which is consistent with the validation behavior of Docker Compose.
    * `essential` as a nested field.

       `essential` is part of an ECS Container Definition, which is corresponds to the `services` field in a normal docker compose file. We will mirror the YAML hierarchy and allow the customer to specify which services are essential under the service name. While we could specify `essential` as a top-level field that would take a collection of service names as its value, mirroring the existing hierarchy has the benefit of being more consistent with the Composefile format as well as being more easily extensible should we decide to add other Container Definition fields on individual services.
* **NOTE:** We will not be supporting placementConstraints for this feature.

Example `task-def-fields.yml` file:

```
task_definition:
  task_role_arn: string
  ecs_network_mode: string # valid values: "none", "host", "bridge", and custom ECS network modes.
  services:
    <serviceName>:
      essential: boolean
```