== Best practices for using {partner-product-short-name} on AWS
// Provide post-deployment best practices for using the technology on AWS, including considerations such as migrating data, backups, ensuring high performance, high availability, etc. Link to software documentation for detailed information.

See the following guidelines and best practices for the Secure Firewall Cloud Native.

=== Deployment Guidelines

* Deploy Cisco Secure Firewall Cloud Native with a minimum of two enforcement points.
* For resiliency, Cisco recommends using RAVPN with SFCN redirector and elasticache for Redis.
* Cisco recommends using EFS storage. 

=== Service Roles

Service Roles are used to define and separate duties between enforcement points in the cluster. The mandatory "sfcn.cisco.com/service-role" label is used to match and apply configurations to their respective pods. When you have two service roles deployed, "default" and "vpnredirector" for example, then configuration(s) will only be applied to the enforcer pods with the same label.

This allows you to apply elements such as ACLs, routes, etc. to your default enforcer while limiting the vpnredirector configuration to only to what is needed for its role in the cluster. The use of service roles in segmenting workloads in the Secure Firewall Cloud Native is further explained in the https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/cloud-native/getting-started/secure-firewall-cloud-native-gsg.html[Secure Firewall Getting Started Guide^].

=== Full and Delta Deployments

When modifying ASAConfigurations, the control plane will make a best-effort attempt to conduct a delta deployment. This means that only the CLI lines that were modified within the spec are operated upon in order to minimize disruption.

If a delta deployment fails, by default a full deployment will be attempted afterward. This involves the full desired set of ASAConfigurations being reapplied onto the enforcement points.

*Note*: Full deployments may lead to traffic disruption and/or VPN sessions getting dropped. When testing or experimenting with configuration changes, we recommend that you apply them outside of peak traffic hours.

=== Scaling

Secure Firewall Cloud Native utilizes the https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[Horizontal Pod Autoscaler (HPA)^] and https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler[Cluster Autoscaler^] in concert to bring CNFW instances up and down based on the needs of your network. After you deploy the SFCN cluster, use the steps mentioned here to modify/configure https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/cloud-native/getting-started/secure-firewall-cloud-native-gsg/sfcn-advanced.html#Cisco_Concept.dita_ebd1a5e4-186c-43ec-a71e-7570f7579235[Horizontal Pod Scaling^]

The Quick Start provides separate fields for these options. It also lets you configure Classless Inter-Domain Routing (CIDR) blocks, instance types, and {partner-product-short-name} settings, as discussed later in this guide.

=== Explore Other Configuration Options

In addition to scaling, you can find information about these other configuration options:

* https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/cloud-native/getting-started/secure-firewall-cloud-native-gsg/sfcn-basic.html[Basic Configuration^] includes information about namespace, service roles, networking, and access lists.
* https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/cloud-native/getting-started/secure-firewall-cloud-native-gsg/sfcn-advanced.html[Advanced Configuration^] includes information about multiple tenants.
* https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/cloud-native/getting-started/secure-firewall-cloud-native-gsg/sfcn-licensing.html[Licensing^] provides guidelines about how to license the Secure Firewall Cloud Native. 

The Secure Firewall Cloud Native Getting Started Guide is updated often with expanded configuration topics, including SAML and Redis datastore.

=== Use Case Examples and Scenarios

Once the Cisco Secure Fireweall Cloud Native is built using this quickstart, SFCN can be used for the following use cases: 

* https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/cloud-native/getting-started/secure-firewall-cloud-native-gsg/sfcn-vpn-route53.html[Scalable Remote Access VPN with Route53 Integration^]
* https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/cloud-native/getting-started/secure-firewall-cloud-native-gsg/m_ra-vpn-with-session-reconnect.html[Scalable Remote Access VPN with Session Reconnect^]
* https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/cloud-native/getting-started/secure-firewall-cloud-native-gsg/sfcn-advanced.html#Cisco_Concept.dita_c425d95d-3609-45c5-9f43-1cddc1c32db9[Enable Multi-tenancy^]
* https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/cloud-native/getting-started/secure-firewall-cloud-native-gsg/sfcn-vpn-l2l.html[Backhaul Design for Site-to-Site VPN^]
* https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/cloud-native/getting-started/secure-firewall-cloud-native-gsg/sfcn-vpn-inside-vpc.html[Backhaul Design for Inside VPC^]

== Other useful information
//Provide any other information of interest to users, especially focusing on areas where AWS or cloud usage differs from on-premises usage.

=== Cisco CLI Usage Guidelines

Cisco CNGW (containerized firewall) uses Cisco ASA configuration. You can use ASA CLI in the YAML file.

*Note*: The use of shorthand ASA CLI syntax is not supported in this release. In other words, using int instead of interface or net instead of network will cause delta deployments to fail. We recommend you use full syntax when working on production clusters.

*Example:*

apiVersion: cnfw.cisco.com/v1
kind: ASAConfiguration
metadata:
  name: interface-example
  labels:
    sfcn.cisco.com/service-role: default
spec:
  order: 1
  description: "DHCP interfaces taking the AWS-assigned IPs"
  cliLines: |

    interface Management0/0
    nameif management
    ip address dhcp
    no shutdown

    interface TenGigabitEthernet0/0
    nameif outside
    ip address dhcp
    security-level 100
    no shutdown

    interface TenGigabitEthernet0/1
    nameif inside
    ip address dhcp
    no shutdown

`kubectl apply -f interfaces.yaml -n $NS`

Before you implement a Kubernetes deployment strategy, you’ll need to understand the pieces of a Kubernetes deployment and how they all function together.
You can review samples and example scripts available at the https://github.com/CiscoDevNet/sfcn[Cisco Secure Firewall Cloud Native GitHub repository^].

=== Custom IAM Policy for CloudFormation Users

Perform steps 1 -3 if you prefer to use a custom IAM policy to deploy the CloudFormation stack instead of the AdminstratorAccess managed policy mentioned in the link:#_iam_permissions[IAM Permissions] section of this guide.

** *Step 1 - Create IAM user*
* Create the user in the AWS Management Console, the AWS CLI, Tools for Windows PowerShell, or by using an AWS API operation. If you create the user in the AWS Management Console, then most of the steps are handled automatically via a wizard, based on your choices. If you create the users programmatically, then you must perform each of those steps individually.
* Create credentials for the user, depending on the type of access the user requires:
*** **Programmatic access**: The IAM user might need to make API calls, use the AWS CLI, or use the Tools for Windows PowerShell. In that case, create an access key (access key ID and a secret access key) for that user. 
*** **AWS Management Console access**: If the user needs to access the AWS Management Console, create a password for the user. Disabling console access for a user prevents them from signing in the to the AWS Management Console using their user name and password. It does not change their permissions or prevent them from accessing the console using an assumed role.
 
** *Step 2 - Create IAM Access Policy* Users who require programmatic access enabled to perform all the deployment activities must create and use the IAM policy (SFCNFullAccess) to deploy Cisco Secure Firewall Cloud Native.

* Sign in to the AWS Management Console and open the IAM console.
* In the navigation pane on the left, choose Policies.
* Choose Create Policy.
* Choose the JSON tab.
* Type or paste the following JSON policy document. 
[source,SFCNFullAccess IAM Policy,options="nowrap"]
SFCNFullAccess IAM Policy 
  { 
      "Version": "2012-10-17", 
      "Statement": [ 
          { 
              "Action": [ 
                  "ec2:*", 
                  "elasticfilesystem:*", 
                  "elasticache:*", 
                  "kms:DescribeKey", 
                  "kms:ListAliases", 
                  "s3:*", 
                  "iam:*", 
                  "elasticloadbalancing:*", 
                  "cloudwatch:*", 
                  "autoscaling:*", 
                  "eks:*", 
                  "ec2-instance-connect:SendSSHPublicKey", 
                  "lambda:*", 
                  "states:DescribeStateMachine", 
                  "states:ListStateMachines", 
                  "tag:GetResources", 
                  "xray:GetTraceSummaries", 
                  "xray:BatchGetTraces", 
                  "ds:CreateComputer", 
                  "ds:DescribeDirectories", 
                  "logs:*", 
                  "ssm:*", 
                  "ec2messages:*", 
                  "cloudformation:*", 
                  "ssmmessages:CreateControlChannel", 
                  "ssmmessages:CreateDataChannel", 
                  "ssmmessages:OpenControlChannel", 
                  "ssmmessages:OpenDataChannel" 
              ], 
              "Effect": "Allow", 
              "Resource": "*" 
          }, 
          { 
              "Action": "iam:CreateServiceLinkedRole", 
              "Effect": "Allow", 
              "Resource": "*", 
              "Condition": { 
                  "StringLike": { 
                      "iam:AWSServiceName": [ 
                          "elasticfilesystem.amazonaws.com" 
                      ] 
                  } 
              } 
          }, 
          { 
              "Action": "iam:CreateServiceLinkedRole", 
              "Effect": "Allow", 
              "Resource": "arn:aws:iam::*:role/aws-service-role/elasticache.amazonaws.com/AWSServiceRoleForElastiCache", 
              "Condition": { 
                  "StringLike": { 
                      "iam:AWSServiceName": "elasticache.amazonaws.com" 
                  } 
              } 
          }, 
          { 
              "Effect": "Allow", 
              "Action": "iam:CreateServiceLinkedRole", 
              "Resource": "*", 
              "Condition": { 
                  "StringEquals": { 
                      "iam:AWSServiceName": [ 
                          "autoscaling.amazonaws.com", 
                          "ec2scheduled.amazonaws.com", 
                          "elasticloadbalancing.amazonaws.com", 
                          "spot.amazonaws.com", 
                          "spotfleet.amazonaws.com", 
                          "transitgateway.amazonaws.com" 
                      ] 
                  } 
              } 
          }, 
          { 
              "Effect": "Allow", 
              "Action": "iam:PassRole", 
              "Resource": "*", 
              "Condition": { 
                  "StringEquals": { 
                      "iam:PassedToService": "lambda.amazonaws.com" 
                  } 
              } 
          }, 
          { 
              "Effect": "Allow", 
              "Action": "iam:CreateServiceLinkedRole", 
              "Resource": "arn:aws:iam::*:role/aws-service-role/ssm.amazonaws.com/AWSServiceRoleForAmazonSSM*", 
              "Condition": { 
                  "StringLike": { 
                      "iam:AWSServiceName": "ssm.amazonaws.com" 
                  } 
              } 
          }, 
          { 
              "Effect": "Allow", 
              "Action": [ 
                  "iam:DeleteServiceLinkedRole", 
                  "iam:GetServiceLinkedRoleDeletionStatus" 
              ], +
              "Resource": "arn:aws:iam::*:role/aws-service-role/ssm.amazonaws.com/AWSServiceRoleForAmazonSSM*" 
          } 
      ] 
  } 
* Resolve any security warnings, errors, or general warnings generated during policy validation, and then choose Review policy.
* When you are finished, choose Next: Tags.
* On the Review policy page:
*** Name ― Type the name for this policy: SFCNFullAccess.
*** Description ― Optionally, enter a description for the policy that you are creating.
* Review the policy Summary to see the permissions that are granted by your policy. Then choose Create policy to save your work.

** *Step 3 - Attach IAM access policy to the IAM user* IAM users must explicitly be given permissions to administer credentials or IAM resources.
* Sign in to the AWS Management Console and open the IAM console.
* Choose Users in the navigation pane, choose the name of the user whose permissions you want to modify, and then choose the Permissions tab.
* Choose Add permissions, and then choose Attach existing policies directly to user.
* Select the SFCNFullAccess managed policy that you created for the SFCN administrator user:
*** Use the Search feature to filter the policies by name.
* You can also create a new managed policy by choosing Create policy. If you do, return to this browser tab or window when the new policy is done. Choose Refresh; and then select the check box for the new policy to attach it to your user. For more information, see Creating IAM policies.
* Choose Next: Review to see the list of policies that are to be attached to the user. Then choose Add.

** *Step 4 - Create Cisco Secure Firewall Cluster Stack*; refer to link:#_deployment_options[Deployment options] for a desciption of the options available and link:#_launch_the_quick_start[Launch the Quick Start] for deployment steps.

== Troubleshooting

*Q.* I want to attach a console session.

*A.* Use kubectl to attach a particular CNFW's console session. From there you can run any command that would not result in a change in the running configuration of CNFW. For example: capture, packet-tracer, show run, etc.

`kubectl attach -it <sfcn-enforcer-pod-name> -n $NS -c cnfw`

Where `$NS` represents the namespace of the pod.

You might be required to run the enable command before being able to execute any of the show commands. The enable command would require you to enter a password when run for the first time. This password is not persistent and would be reset once the sfcn-enforcer pod restarts or unscheduled.

*Q.* I want to SSH to EKS nodes.

*A.* Install ec2instanceconnect (pip install ec2instanceconnectcli), and add the following to your shell profile:

`[source,*EKS mssh*,options="nowrap"]`

EKS mssh
```

function sshn() {
  PS3="Select a node to ssh: "
  select node in $(kubectl get nodes | grep -v NAME | awk '{print $1}')
  do
    echo "Selected:  $node"
    INSTANCE_ID=$(kubectl describe node $node | awk -F'/' '/ProviderID/{print $5}')
    REGION=$(kubectl describe nodes $node | awk -F'=' '/topology.kubernetes.io\/region/{print $2}')
    mssh --region $REGION ubuntu@$INSTANCE_ID
    break
    done
    return $?
}
```
Execute `sshn`, and select the node you wish to SSH to when prompted.

*Q.* I want to check the deployment status.

*A.* After creating an ASAConfiguration object, you can track the status of its deployment using the status field. See the ASAConfiguration CRD for detailed information on the status field.

The ASAConfiguration status field currently has 2 status conditions:

*Valid:* This condition provides information if the CLI lines present in the ASAConfiguration object are valid or not. There are certain CLI commands that are blocked from using to maintain system integrity. If ASAConfiguration contains any of those commands, the Valid condition would be false. Valid condition's message field would provide detailed information on which command is blocked.

*Deployed:* This condition provides information if the CLI lines have been deployed to all applicable nodes in the cluster. This condition becomes true only when the CLI lines are deployed successfully across all applicable nodes. This status condition shows a summary on how many nodes to which the cluster has been deployed.

In the event the deployed status shows the configuration has been deployed to only x nodes of y nodes, check the NodeConfiguration's status field of the nodes present in the cluster.

`kubectl describe nodeconfiguration <node_name> -n $NS`

*Q.* I want to view container logs.

*A.* By default, the system forwards the combined logs for all SFCN-related containers running in the cluster to a central logging service. Use the following kubectl command to view the logs:

```
[source,$ POD=$(grep -m 1 logging-server,options="nowrap"]
POD=$(grep -m 1 logging-server <<< "`kubectl get pods -n $NS --no-headers=true -o custom-columns=':metadata.name'`")
kubectl logs -n $NS pod/$POD -c logging-server
```
Optionally, the parameter `-f` can be added to continuously tail the log. Note that $NS represents the namespace of the pod.

Alternatively, it is possible to login to the logging server:

`kubectl exec -n $NS -it $POD -c logging-server -- /bin/sh`

[source,$ cd /var/log/logs,options="nowrap"]
$ cd /var/log/logs
$ ls
  messages-kv.log  messages.log
$ grep asac messages-kv.log
  Mar 30 21:30:27 ip-192-168-118-112 [2021/03/30 21:30:27] [ info] inotify_fs_add(): inode=1536390 watch_fd=62 name=/var/log/containers/sfcn-cli-validation-<truncated>.log
  Mar 30 21:33:10 ip-192-168-118-112 [2021/03/30 21:33:10] [ info] inotify_fs_add(): inode=1536539 watch_fd=64 name=/var/log/containers/sfcn-cli-validation-<truncated>.log
  

Logs stored on the logging server are routinely rotated and cleaned up. If you need to preserve logs, save the files before the house-cleaning jobs run.

Unified container logs are formatted as follows:
```
<year-month-day> <time> <node name> <pod namespace> <pod name> <container name> -- <log message> -- <extra JSON-formatted log metadata>

For example, an entry from a configuration pod may look like this:
[source,2021-03-18 21:28:00 ,options="nowrap"]
2021-03-18 21:28:00 ip-1-1-1-1.ec2.internal sfcn-system sfcn-configurer-1111111111-22222 manager --
2021-03-18T21:28:00.806Z INFO controllers.SNMPExporter Reconciliation started {"namespace": "sfcn-system"} -- {"_SDATA":{"k8s":{
```

The unified logs are also stored and periodically rotated on the node where the logging-server is running at /home/ubuntu/var-log/.

Standard output and standard error logs for individual containers can also be viewed using the kubectl log command:

`kubectl logs pod/<pod_name> -c <container_name>`