# Adding new projects All projects in this repo follow a common set of processes and organization. * container images base off of one of the eks-distro-base [variants](https://gallery.ecr.aws/eks-distro-build-tooling/) * reproducible builds with checked in CHECKSUMS file for validation during build/release * projects uploading to s3 define an `expected_artifacts` file for validation during build/release * ATTRIBUTION.txt files are generated using `go-licenses` and checked into the repo * ATTRIBUTION.txt and LICENSES, gathered from dependencies of project, are added to tarballs and container images * (some) projects have different versions for different Kubernetes versions, referred to as `release branches` ## Typical folder structure The following example assumes a RELEASE_BRANCH folder, but for projects without release branches those files would be in the project root. ``` +-- projects// | +-- _output (generated) +-- (generated) +-- 1-21 | +-- ATTRIBUTION.txt +-- CHECKSUMS +-- GIT_TAG +-- GOLANG_VERSION +-- build | +-- create_binaries.sh +-- docker/linux | +-- Dockerfile +-- expected_artifacts_ (if s3 artifacts) +-- Makefile +-- README.md ``` ## New project When adding a new project, start by copying a Makefile from one of the simpler projects, such as [kube-rbac-proxy](../../projects/brancz/kube-rbac-proxy/Makefile). The key pieces to setup in the Makefile are: * GIT_TAG - This should contain the tag in the upstream repo which is to be built. The Makefile should get this from the GIT_TAG file either in the project root or the release branch. * GOLANG_VERSION - This should contain the version of golang which the upstream repo uses for building this project. This can be a chore to figure out, typically we go with the verison in the go.mod file, however, sometimes you need to look in the upstream Dockerfile or Makefile. The Makefile should get this from the GOLANG_VERSION file either in the project root or the release branch. * REPO - upstream repo name. * COMPONENT - upstream org/repo. * BINARY_TARGET_FILES - The binary file names to be built, automatically appended with $(OUTPUT_BIN_DIR) and generated for each $(BINARY_PLATFORMS). Passed to `go build -o`. * SOURCE_PATTERNS - The patterns to pass to `go build`. Must follow the same ordering as $(BINARY_TARGET_FILES) * IMAGE_NAMES (optional) - When building multiple images define them via `IMAGE_NAMES` and `local-images` and `images` will automatically be created to build the `amd64` platform in `local-images` and `push` for images. If not set, will default to `REPO`. These names need to match `IMAGE_NAME` in the below variable `_IMAGE`. * _IMAGE_COMPONENT (optional) - Common Makefile will use this if set to override the default component in the `IMAGE` variable. By default `IMAGE` will be set to `$(IMAGE_REPO)/$(COMPONENT):$(IMAGE_TAG)` * GO_MOD_PATHS (optional) - Use this config if your targeted go.mod files are not in the project root.` ### Scripts #### create-binaries.sh The only required script when adding a new project. By default, the common `binaries` target will call create-binaries.sh from the project build directory with the following args * $TAG - GIT_TAG * $BIN_PATH - intended output directory * $OS - OS to pass to go build via GOOS * $ARCH - ARCH to pass to go build via GOARCH It will be called once per os/arch combination we are building for. By default this is `linux/amd64 linux/arm64`, but can be overridden in a project Makefile via `BINARY_PLATFORMS`. ### Dockerfile For projects building container images, docker/linux/Dockerfile is required. `DOCKERFILE_FOLDER` can be used to override the directory containing the Dockerfile when necessary. This may be necessary if building multiple container images in the same project. Refer to [cert-manager](../../projects/cert-manager/cert-manager/Makefile) as an example of this. By default the base image will be `eks-distro-base`, but this can be overridden by setting `BASE_IMAGE_NAME` in the Makefile. This base image name should be one of the `eks-distro-base` variants. For curated packages, the base image should be one of the `eks-distro-minimal-base` variants if possible. This may require a multi-stage builds for some packages. Refer to [hello-eks-anywhere](https://github.com/aws/eks-anywhere-build-tooling/blob/main/projects/aws-containers/hello-eks-anywhere/docker/linux/Dockerfile) as an example. Since we building outside of docker, most Dockerfiles look like: ``` ARG BASE_IMAGE FROM ${BASE_IMAGE} ARG TARGETARCH ARG TARGETOS COPY LICENSES /LICENSES COPY ATTRIBUTION.txt /ATTRIBUTION.txt COPY bin/$TARGETOS/$TARGETARCH/ /usr/local/bin/ ENTRYPOINT ["/usr/local/bin/"] CMD ["/usr/local/bin/"] ``` ### Gitignore During every build process, the upstream Github repository for the project is cloned into the respective project directory under `projects//`. Hence, it is standard practice to have a `.gitignore` in each project directory with the upstream repository, to prevent Git from tracking the cloned repository. Also, each build generates the `_output` folder which stores the binaries, tarballs and other related artifacts generated by the build. To avoid checking these in, the `_output` folder is also added to `.gitignore`. ## Readme Every project must have a README.md that contains a basic description of the project and provides instructions on updating the project (if applicable). The README also includes other useful information such as the version or tag being built and a badge displaying the latest build status for the project. ### Other files After setting up the Makefile and create-binaries.sh, we need to generate the CHECKSUMS and ATTRIBUTION.txt. Please refer to [building-locally](building-locally.md) for resources for generating these two files. ## Other scenarios ### s3-artifacts/tarballs Some projects deliver tarballs to s3 and to support this there are additional targets necessary. [cluster-api](../../projects/kubernetes-sigs/cluster-api/Makefile) is a good example for reference of this. Setting `HAS_S3_ARTIFACTS` to true in the project's Makefile will automatically add the necessary pre-reqs to the `build` and `release` targets. Projects with this requirement need to supply the `expected_artifacts_` file in their root. ### multiple containers For projects which build multiple containers, DOCKERFILE_FOLDER generally needs overridden and additional _NAME variables need to be set in the Makefile, one per container image to be built. Refer to [cert-manager](../../projects/cert-manager/cert-manager/Makefile) as an example of this. ### additional pre-reqs for image builds Sometimes projects need to run additional targets before building images. [release](../../projects/kubernetes/release/Makefile) is a good example to refer to for this situation. To add additional pre-reqs to the image builds: ``` $(call IMAGE_TARGETS_FOR_NAME, kube-proxy-base): ``` ### fixing licenses As documented in [attribution-files](attribution-files.md) there are situations where licenses need to be fixed before running attribution generation. [helm-controller](../../projects/fluxcd/helm-controller/Makefile) is an example of this. To handle these cases, projects generally add the following to the Makefile: ``` FIX_LICENSES_TARGET= $(FIX_LICENSES_TARGET): ``` ### patching There are several projects in EKS Anywhere that require patching, such as Kind, Cluster API, Flux, etc. To support these, the Common.mk will automatically add the following if a `patches` directory exists. ``` GIT_PATCH_TARGET=$(REPO)/eks-anywhere-patched $(BINARY_TARGETS): | $(GIT_PATCH_TARGET) $(GIT_PATCH_TARGET): $(GIT_CHECKOUT_TARGET) git -C $(REPO) am $(MAKE_ROOT)/patches/* @touch $@ ``` ## Upstream projects tracker This repository has a [script](../../build/lib/generate_projects_list.sh) that generates a [list](../../UPSTREAM_PROJECTS.yaml) of the Github projects that are being built from this repository, along with the tag or commit built. When adding a new project or bumping up the tag for a project, the `generate` target is to be used to re-generate this list and check in the updated file. This file must only be modified through the script and not edited by hand.