= Kubernetes - Enforcing Network Security Policies with Calico
:toc:


This exercise will walk you through configuring Calico.

== Prerequisites

This chapter uses a cluster with 3 master nodes and 5 worker nodes as described here: link:../../cluster-install#multi-master-multi-node-multi-az-gossip-based-cluster[multi-master, multi-node gossip based cluster].

Alternatively, you can create a new cluster and use kops CLI options to specify the networking mode as shown below:

  kops create cluster \
    --name example2.cluster.k8s.local \
    --master-count 1 \
    --node-count 2 \
    --zones ${AWS_AVAILABILITY_ZONES} \
    --networking calico \
    --yes

This will create a 1 master node and 2 worker nodes cluster. It uses `--networking calico` which tells the cluster to use calico networking instead of the default networking provided by kubenet. Note, the name here is `example2.cluster.k8s.local` instead of the usual `example.cluster.k8s.local` name. This is just to make sure, in case, the two clusters can coexist.

View the cluster configuration using the following command:

  kops edit cluster example2.cluster.k8s.local

This will show the following fragment under `.spec`:

  networking:
    calico: {
      crossSubnet: true
    }

Rest of the chapter goes with the presumption that you are using an existing cluster.

NOTE: A smaller sized cluster, with 1 master node and 2 worker nodes, is chosen here. This is to minimize the amount of time for rolling update the cluster later. It typically takes 5-6 mins per node for rolling update.

All configuration files for this chapter are in the `calico` directory. Make sure you change to that directory before giving any commands in this chapter.

== Installation

Edit the Kubernetes cluster configuration using the command:

  kops edit cluster example.cluster.k8s.local

By default, Kubernetes uses `kubenet` for networking. This is captured in the configuration file as:

  networking:
    kubenet: {}

Update the networking configuration to use `calico` by setting the following property:

  networking:
    calico:
      crossSubnet: true

View changes that will be applied to the cluster:

  kops update cluster example.cluster.k8s.local

It shows output as:

```
I1025 15:55:45.512183    3454 apply_cluster.go:420] Gossip DNS: skipping DNS validation
W1025 15:55:45.534000    3454 firewall.go:202] Opening etcd port on masters for access from the nodes, for calico.  This is unsafe in untrusted environments.
I1025 15:55:45.536302    3454 executor.go:91] Tasks: 0 done / 72 total; 34 can run
I1025 15:55:46.365905    3454 executor.go:91] Tasks: 34 done / 72 total; 14 can run
I1025 15:55:46.841194    3454 executor.go:91] Tasks: 48 done / 72 total; 20 can run
I1025 15:55:47.760698    3454 executor.go:91] Tasks: 68 done / 72 total; 3 can run
I1025 15:55:48.379152    3454 executor.go:91] Tasks: 71 done / 72 total; 1 can run
I1025 15:55:48.511060    3454 executor.go:91] Tasks: 72 done / 72 total; 0 can run
Will create resources:
  ManagedFile/cluster.k8s.local-addons-networking.projectcalico.org-k8s-1.6
    Location              addons/networking.projectcalico.org/k8s-1.6.yaml

  ManagedFile/cluster.k8s.local-addons-networking.projectcalico.org-pre-k8s-1.6
    Location              addons/networking.projectcalico.org/pre-k8s-1.6.yaml

  SecurityGroupRule/node-to-master-protocol-ipip
    SecurityGroup         name:masters.cluster.k8s.local id:sg-887938fa
    Protocol              4
    SourceGroup           name:nodes.cluster.k8s.local id:sg-ec74359e

  SecurityGroupRule/node-to-master-tcp-1-4001
    SecurityGroup         name:masters.cluster.k8s.local id:sg-887938fa
    Protocol              tcp
    FromPort              1
    ToPort                4001
    SourceGroup           name:nodes.cluster.k8s.local id:sg-ec74359e

Will modify resources:
  LaunchConfiguration/master-us-east-1d.masters.cluster.k8s.local
    UserData
                          ...
                            - _aws
                            - _kubernetes_master
                          + - _networking_cni
                            channels:
                            - s3://kubernetes-aws-io/cluster.k8s.local/addons/bootstrap-channel.yaml
                          ...


  LaunchConfiguration/nodes.cluster.k8s.local
    UserData
                          ...
                            - _automatic_upgrades
                            - _aws
                          + - _networking_cni
                            channels:
                            - s3://kubernetes-aws-io/cluster.k8s.local/addons/bootstrap-channel.yaml
                          ...


  LoadBalancer/api.cluster.k8s.local
    Lifecycle              <nil> -> Sync

  LoadBalancerAttachment/api-master-us-east-1d
    Lifecycle              <nil> -> Sync

  ManagedFile/cluster.k8s.local-addons-bootstrap
    Contents
                          ...
                                  k8s-addon: storage-aws.addons.k8s.io
                                version: 1.6.0
                          +   - id: pre-k8s-1.6
                          +     kubernetesVersion: <1.6.0
                          +     manifest: networking.projectcalico.org/pre-k8s-1.6.yaml
                          +     name: networking.projectcalico.org
                          +     selector:
                          +       role.kubernetes.io/networking: "1"
                          +     version: 2.1.2-kops.1
                          +   - id: k8s-1.6
                          +     kubernetesVersion: '>=1.6.0'
                          +     manifest: networking.projectcalico.org/k8s-1.6.yaml
                          +     name: networking.projectcalico.org
                          +     selector:
                          +       role.kubernetes.io/networking: "1"
                          +     version: 2.1.2-kops.1


Must specify --yes to apply changes
```

Apply the changes using the command:

  kops update cluster example.cluster.k8s.local --yes

It shows the output:

```
I1025 15:56:26.679683    3458 apply_cluster.go:420] Gossip DNS: skipping DNS validation
W1025 15:56:26.701541    3458 firewall.go:202] Opening etcd port on masters for access from the nodes, for calico.  This is unsafe in untrusted environments.
I1025 15:56:27.214980    3458 executor.go:91] Tasks: 0 done / 72 total; 34 can run
I1025 15:56:27.973367    3458 executor.go:91] Tasks: 34 done / 72 total; 14 can run
I1025 15:56:28.427597    3458 executor.go:91] Tasks: 48 done / 72 total; 20 can run
I1025 15:56:30.010284    3458 executor.go:91] Tasks: 68 done / 72 total; 3 can run
I1025 15:56:30.626483    3458 executor.go:91] Tasks: 71 done / 72 total; 1 can run
I1025 15:56:30.934673    3458 executor.go:91] Tasks: 72 done / 72 total; 0 can run
I1025 15:56:31.545416    3458 update_cluster.go:247] Exporting kubecfg for cluster
kops has set your kubectl context to example.cluster.k8s.local

Cluster changes have been applied to the cloud.


Changes may require instances to restart: kops rolling-update cluster
```

Determine if any of the nodes will require a restart using the command:

  kops rolling-update cluster example.cluster.k8s.local

Output from this command is shown:

```
$ kops rolling-update cluster example.cluster.k8s.local
NAME              STATUS      NEEDUPDATE  READY MIN MAX NODES
master-us-east-1d NeedsUpdate 1           0     1   1   1
nodes             NeedsUpdate 2           0     2   2   2

Must specify --yes to rolling-update.
```

The `STATUS` column shows that both master and worker nodes need to be updated.

Perform the rolling update using the command shown:

  kops rolling-update cluster example.cluster.k8s.local --yes

Output from this command is shown:

```
NAME              STATUS      NEEDUPDATE  READY MIN MAX NODES
master-us-east-1d NeedsUpdate 1           0     1   1   1
nodes             NeedsUpdate 2           0     2   2   2
I1025 16:16:31.978851    3733 instancegroups.go:350] Stopping instance "i-0cdcb2e51e5656b44", node "ip-172-20-44-219.ec2.internal", in AWS ASG "master-us-east-1d.masters.cluster.k8s.local".
I1025 16:21:32.411639    3733 instancegroups.go:350] Stopping instance "i-060b2c9652e2075ac", node "ip-172-20-54-182.ec2.internal", in AWS ASG "nodes.cluster.k8s.local".
I1025 16:23:32.973648    3733 instancegroups.go:350] Stopping instance "i-0baffcbc9a758a6c4", node "ip-172-20-94-82.ec2.internal", in AWS ASG "nodes.cluster.k8s.local".
I1025 16:25:33.784129    3733 rollingupdate.go:174] Rolling update completed!
```

== Advanced Topics

=== Preventing Outbound access

NetworkPolicy resources in Kubernetes versions prior to 1.8 allow you to isolate inbound traffic only.  To filter outbound traffic, you need to configure Calico directly using the `calicoctl` tool.  Refer to the section https://docs.projectcalico.org/v2.5/getting-started/kubernetes/tutorials/advanced-policy[Prevent outgoing connections from pods] for further information.

Kubernetes is an evolving project and for Kubernetes versions 1.8 and newer NetworkPolicy is growing to support egress traffic, so users of Kubernetes 1.8+ should refer to the section https://docs.projectcalico.org/v2.6/getting-started/kubernetes/tutorials/advanced-policy[Prevent outgoing connections from pods], which the same section as above but in the newer Calico version's docs updated for this upgrade and allows only using `kubectl`.

The https://kubernetes.io/docs/concepts/services-networking/network-policies/[Kubernetes official Network Policies Concepts Documentation] contains more information and examples around the egress support. Currently these changes are in beta state, with 1.10 the goal for general availability. Work towards completing egress support for NetworkPolicy can be tracked at https://github.com/kubernetes/features/issues/366[Kubernetes/Features: GA Egress support for Network Policy] and https://github.com/kubernetes/kubernetes/issues/22469[Kubernetes/Kubernetes: Kubernetes Network Policy].

