# Scenario Description Result
1 Check inputs PASS
2 Check current HA installation (prep / bridge tasks) PASS
3 Stop Primary database with HDB Stop and check failover result PASS
4 Stop NEW Primary database with HDB Stop and check failover result PASS
5 Stop Secondary database with HDB Stop and check result PASS
6 Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result PASS
7 Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE PASS
8 Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result PASS
9 Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE PASS

Common Steps
Gathering Facts
2023-01-06T18:04:52.355065Z Check inputs
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-ohbfstobieiwzsyphrsbkoskuzbzbsdz ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028288.759115-47339-115177343985838/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-elpumwbsxjbrlrionkmbdzaecwnihihm ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028288.752331-47340-79010110780512/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-ywgbssclvtnldtzxvwufhkgousmrkitc ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028288.7735271-47341-242523053383017/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-rsnqpbwamaqihjbpwacfipapiaocsvsg ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028288.808212-47344-214765278922649/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-check-inputs : Check if OS is supported
2023-01-06T18:04:52.355065Z Check inputs
ansible-role-check-inputs : Check if OS is supported
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-check-inputs : Check input parameters
2023-01-06T18:04:52.355065Z Check inputs
ansible-role-check-inputs : Check input parameters
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-check-inputs : Ensure non mandatory input parameters were sent as required
2023-01-06T18:04:52.355065Z Check inputs
ansible-role-check-inputs : Ensure non mandatory input parameters were sent as required
HANA Pri: fail - Changed: false
See full message
The conditional check 'not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined' failed. The error was: error while evaluating conditional (not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined): No variable found with this name: INPUT_EC2_USER The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-check-inputs/tasks/check_parameters.yml': line 12, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Ensure non mandatory input parameters were sent as required ^ here

HANA Sec: fail - Changed: false
See full message
The conditional check 'not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined' failed. The error was: error while evaluating conditional (not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined): No variable found with this name: INPUT_EC2_USER The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-check-inputs/tasks/check_parameters.yml': line 12, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Ensure non mandatory input parameters were sent as required ^ here

ASCS: fail - Changed: false
See full message
The conditional check 'not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined' failed. The error was: error while evaluating conditional (not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined): No variable found with this name: INPUT_EC2_USER The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-check-inputs/tasks/check_parameters.yml': line 12, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Ensure non mandatory input parameters were sent as required ^ here

PAS: fail - Changed: false
See full message
The conditional check 'not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined' failed. The error was: error while evaluating conditional (not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined): No variable found with this name: INPUT_EC2_USER The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-check-inputs/tasks/check_parameters.yml': line 12, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Ensure non mandatory input parameters were sent as required ^ here

Common Steps
Gathering Facts
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-rdpuwxgzuoqnmzvqfwtpixlylbtwoxbc ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028292.609022-47416-233106058009648/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-ggsithtfzsdbrejvtpcwhbopcbmblsbh ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028292.575712-47417-207003330308627/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-xurmfakkvfrhtzpcpeeidlshfmgpfvkp ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028292.575453-47419-78882287454772/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-wfypwenawjcxupzdrqphfbzdrodhphjv ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028292.5756621-47421-73789626846448/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:05:03.629739
End date: 2023-01-06 12:05:03.678182
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:05:03.617932
End date: 2023-01-06 12:05:03.663448
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:05:05.810125
End date: 2023-01-06 12:05:06.021445
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:05:05.800218
End date: 2023-01-06 12:05:05.988742
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:05:08.185271
End date: 2023-01-06 12:05:08.401418
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:05:08.171657
End date: 2023-01-06 12:05:08.365068
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:05:08.106446
End date: 2023-01-06 12:05:08.109366
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:05:08.111801
End date: 2023-01-06 12:05:08.114862
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Check if this node is secondary
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:05:10.579927
End date: 2023-01-06 12:05:10.796414
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:05:10.568071
End date: 2023-01-06 12:05:10.761007
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:05:10.574876
End date: 2023-01-06 12:05:10.577730
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:05:10.577762
End date: 2023-01-06 12:05:10.580871
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:05:15.556577
End date: 2023-01-06 12:05:15.773908
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:05:15.322930
End date: 2023-01-06 12:05:15.507868
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : System is not ready for HA testing
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:05:17.841481
End date: 2023-01-06 12:05:18.048096
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code


ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check HA configuration status after self-healing actions

HANA Sec: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here

Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check current status for PRIMARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 12:05:20.656070
End date: 2023-01-06 12:05:20.673242
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 1673028258
* master-SAPHana_AD0_00 : 150


ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check current status for SECONDARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 12:05:22.639928
End date: 2023-01-06 12:05:22.655406
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SOK
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 30
* master-SAPHana_AD0_00 : 100

Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:05:24.752834
End date: 2023-01-06 12:05:24.768297
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK

Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:05:26.760258
End date: 2023-01-06 12:05:26.791202
Task duration: < 1min
Errors:
Standard output:


Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:05:28.878393
End date: 2023-01-06 12:05:29.969633
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:05:29.161651
End date: 2023-01-06 12:05:30.145788
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 12:05:32.416706
End date: 2023-01-06 12:05:32.479837
Task duration: < 5s
Errors:
Standard output:
1

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 12:05:36.730050
End date: 2023-01-06 12:05:36.917594
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana"
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana\'
Start date: 2023-01-06 12:05:38.878892
End date: 2023-01-06 12:05:38.883559
Task duration: < 5s
Errors:
Standard output:
ignore_errors
4 ETW000 [ dev trc,00000] INFO : Database 'AD0/00' instance is running on 'demo-hana'

Common Steps
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana"
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana"
ansible-role-bridge-task : Show current step
2023-01-06T18:05:39.257263Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED BRIDGE TASK"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-ifebrvgdkozwikwdxjzslyszgasrxzpr ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028339.503111-47849-28894626655997/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-aboioebivdoluhedtapefvsrzgmdwsqk ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028339.492218-47850-181410668467247/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-jpfbgglisjdewkorffvsxqmwlymrufui ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028339.507146-47852-124894267509782/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-lcvxayivmbkzsrsgjcphjutasrhkzdww ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028339.502562-47854-233517865147371/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop primary database
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop primary database
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop HDB
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop HDB
HANA Pri: include_tasks - Changed: false




ansible-role-ha-test-scenarios : Stop HANA database
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop HANA database
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB stop
Start date: 2023-01-06 12:05:44.223344
End date: 2023-01-06 12:07:12.350738
Task duration: < 5s
Errors:
Standard output:
hdbdaemon will wait maximal 300 seconds for NewDB services finishing.
Stopping instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function Stop 400

06.01.2023 12:05:44
Stop
OK
Waiting for stopped instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function WaitforStopped 600 2


06.01.2023 12:07:12
WaitforStopped
OK
hdbdaemon is stopped.

Common Steps
ansible-role-ha-test-scenarios : Wait for failover to happen
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for failover to happen
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep PROMOTED
Start date: 2023-01-06 12:07:36.509050
End date: 2023-01-06 12:07:36.524794
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : PROMOTED

Common Steps
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep DEMOTED
Start date: 2023-01-06 12:08:41.624606
End date: 2023-01-06 12:08:41.642948
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : DEMOTED


Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY hana_{{ hana_sid_lower }}_sync_state to be {{ crm_mon_prim }}
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY hana_{{ hana_sid_lower }}_sync_state to be {{ crm_mon_prim }}

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep PRIM
Start date: 2023-01-06 12:08:44.642129
End date: 2023-01-06 12:08:44.659031
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : PRIM

Common Steps
ansible-role-ha-test-scenarios : Wait for PRIMARY to be SOK
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be SOK
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for PRIMARY hana_{{ hana_sid_lower }}_sync_state to be {{ crm_mon_sok }}
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY hana_{{ hana_sid_lower }}_sync_state to be {{ crm_mon_sok }}
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:09:36.476711
End date: 2023-01-06 12:09:36.493064
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK


Common Steps
ansible-role-ha-test-scenarios : Include counter checker
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Include counter checker
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get secondary hostname
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:09:38.584292
End date: 2023-01-06 12:09:38.614122
Task duration: < 1min
Errors:
Standard output:


Common Steps
ansible-role-ha-test-scenarios : Check SAP ASCS queue length
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Check SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-ha-test-scenarios : Get Enqueue lock number
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 12:09:41.652796
End date: 2023-01-06 12:09:41.717189
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
ASCS: debug - Changed: false
See full message
0

ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
ASCS: debug - Changed: false
See full message
1

ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
ansible-role-ha-test-scenarios : Start HDB
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Start HDB
HANA Pri: include_tasks - Changed: false




ansible-role-ha-test-scenarios : Start HANA database
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Start HANA database
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB start && echo "ignore_errors"
Start date: 2023-01-06 12:09:43.867042
End date: 2023-01-06 12:09:54.006510
Task duration: < 5s
Errors:
Impromptu CCC initialization by 'rscpCInit'.
See SAP note 1266393.
Standard output:


StartService
OK
OK
Starting instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function StartWait 2700 2


06.01.2023 12:09:43
Start
OK

06.01.2023 12:09:54
StartWait
OK
ignore_errors

Common Steps
ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:09:54.632235Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-wxflaqpaustrfekdpdbnryajaueehqyv ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028594.843526-48354-100702773666933/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-rqtsvukyltanfacrgjwmiglpjruylhyq ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028595.646305-48356-121861383341204/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-bwlvnfyejobazplqpweonxsvyagbhihv ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028594.863643-48357-262858958363237/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-iquoieddtsvzhfppesbsncmuzhjvjrrn ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028595.646361-48359-118718757955689/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:10:06.675379
End date: 2023-01-06 12:10:06.722260
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:10:06.703921
End date: 2023-01-06 12:10:06.751344
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:10:08.877959
End date: 2023-01-06 12:10:09.075275
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:10:08.880176
End date: 2023-01-06 12:10:09.089819
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:10:11.086136
End date: 2023-01-06 12:10:11.282445
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:10:12.216287
End date: 2023-01-06 12:10:12.427093
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:10:11.091834
End date: 2023-01-06 12:10:11.094652
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:10:11.116412
End date: 2023-01-06 12:10:11.119594
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Check if this node is secondary
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:10:14.472051
End date: 2023-01-06 12:10:14.675456
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:10:14.887505
End date: 2023-01-06 12:10:15.095731
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:10:14.495895
End date: 2023-01-06 12:10:14.498731
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:10:14.500056
End date: 2023-01-06 12:10:14.503109
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:10:19.729128
End date: 2023-01-06 12:10:19.931563
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:10:19.772951
End date: 2023-01-06 12:10:19.993905
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : System is not ready for HA testing
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:10:22.069486
End date: 2023-01-06 12:10:22.274053
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check HA configuration status after self-healing actions
HANA Pri: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here


Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check current status for PRIMARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 12:10:24.542487
End date: 2023-01-06 12:10:24.558701
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 1673028572
* master-SAPHana_AD0_00 : 150

ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check current status for SECONDARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 12:10:26.553005
End date: 2023-01-06 12:10:26.570922
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SOK
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 30
* master-SAPHana_AD0_00 : 100


Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:10:28.696877
End date: 2023-01-06 12:10:28.714155
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK


Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:10:30.720709
End date: 2023-01-06 12:10:30.750227
Task duration: < 1min
Errors:
Standard output:

Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:10:32.870416
End date: 2023-01-06 12:10:33.971760
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:10:32.892265
End date: 2023-01-06 12:10:33.902116
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 12:10:36.173627
End date: 2023-01-06 12:10:36.233934
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 12:10:40.417793
End date: 2023-01-06 12:10:40.717504
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana2\'
Start date: 2023-01-06 12:10:42.730370
End date: 2023-01-06 12:10:42.734592
Task duration: < 5s
Errors:
Standard output:
ignore_errors
4 ETW000 [ dev trc,00000] INFO : Database 'AD0/00' instance is running on 'demo-hana2'

Common Steps
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
ansible-role-bridge-task : Show current step
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED BRIDGE TASK"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop primary database
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop primary database
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop HDB
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop HDB

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Stop HANA database
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop HANA database
HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB stop
Start date: 2023-01-06 12:10:45.014164
End date: 2023-01-06 12:11:55.126040
Task duration: < 5s
Errors:
Standard output:
hdbdaemon will wait maximal 300 seconds for NewDB services finishing.
Stopping instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function Stop 400

06.01.2023 12:10:45
Stop
OK
Waiting for stopped instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function WaitforStopped 600 2


06.01.2023 12:11:55
WaitforStopped
OK
hdbdaemon is stopped.

Common Steps
ansible-role-ha-test-scenarios : Wait for failover to happen
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for failover to happen
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep PROMOTED
Start date: 2023-01-06 12:12:54.836393
End date: 2023-01-06 12:12:54.854025
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : PROMOTED


Common Steps
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep DEMOTED
Start date: 2023-01-06 12:13:56.955255
End date: 2023-01-06 12:13:56.971414
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : DEMOTED

Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be PRIM
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be PRIM
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep PRIM
Start date: 2023-01-06 12:13:59.907559
End date: 2023-01-06 12:13:59.925492
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : PRIM


Common Steps
ansible-role-ha-test-scenarios : Wait for PRIMARY to be SOK
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be SOK
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for PRIMARY hana_ad0_sync_state to be SOK
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY hana_ad0_sync_state to be SOK

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:14:51.487493
End date: 2023-01-06 12:14:51.503589
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK

Common Steps
ansible-role-ha-test-scenarios : Include counter checker
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Include counter checker
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get secondary hostname
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY

HANA Sec: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:14:53.889005
End date: 2023-01-06 12:14:53.918509
Task duration: < 1min
Errors:
Standard output:

Common Steps
ansible-role-ha-test-scenarios : Check SAP ASCS queue length
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Check SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-ha-test-scenarios : Get Enqueue lock number
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 12:14:56.963267
End date: 2023-01-06 12:14:57.025045
Task duration: < 5s
Errors:
Standard output:
1

Common Steps
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
ASCS: debug - Changed: false
See full message
1

ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
ASCS: debug - Changed: false
See full message
0

ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
ansible-role-ha-test-scenarios : Start HDB
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Start HDB

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Start HANA database
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Start HANA database
HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB start && echo "ignore_errors"
Start date: 2023-01-06 12:14:59.215362
End date: 2023-01-06 12:15:09.353478
Task duration: < 5s
Errors:
Impromptu CCC initialization by 'rscpCInit'.
See SAP note 1266393.
Standard output:


StartService
OK
OK
Starting instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function StartWait 2700 2


06.01.2023 12:14:59
Start
OK

06.01.2023 12:15:09
StartWait
OK
ignore_errors

Common Steps
ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:15:09.988380Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-wlaxwwcudrqjmtffjgrvwdwdjvsygmjw ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028910.953807-49200-77458765745127/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-uuhekbhrnxrrvtsnvzukwcioyvgmwvsx ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028910.219928-49201-161872134845200/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-llnefjggyzbwpejrizvygqipgqaulzsm ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028910.2199192-49203-264564258180743/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-eznogbrbgevjjmelbcskjsqaoyhyrgxk ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673028910.985895-49205-147020729126283/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:15:22.581282
End date: 2023-01-06 12:15:22.630217
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:15:22.269732
End date: 2023-01-06 12:15:22.315573
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:15:24.936567
End date: 2023-01-06 12:15:25.150534
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:15:24.865935
End date: 2023-01-06 12:15:25.059410
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:15:27.244725
End date: 2023-01-06 12:15:27.446599
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:15:27.257641
End date: 2023-01-06 12:15:27.447903
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:15:27.248393
End date: 2023-01-06 12:15:27.251151
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:15:27.268775
End date: 2023-01-06 12:15:27.271651
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Check if this node is secondary
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:15:29.885562
End date: 2023-01-06 12:15:30.093849
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:15:29.642237
End date: 2023-01-06 12:15:29.834666
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:15:29.646707
End date: 2023-01-06 12:15:29.649327
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:15:29.637533
End date: 2023-01-06 12:15:29.640453
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:15:34.740368
End date: 2023-01-06 12:15:34.960312
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:15:34.731912
End date: 2023-01-06 12:15:34.924559
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : System is not ready for HA testing
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:15:36.982692
End date: 2023-01-06 12:15:37.200991
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code


ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check HA configuration status after self-healing actions

HANA Sec: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here

Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check current status for PRIMARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 12:15:39.328128
End date: 2023-01-06 12:15:39.346397
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 1673028886
* master-SAPHana_AD0_00 : 150


ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check current status for SECONDARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 12:15:41.455136
End date: 2023-01-06 12:15:41.470842
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SOK
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 30
* master-SAPHana_AD0_00 : 100

Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:15:43.523338
End date: 2023-01-06 12:15:43.539459
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK

Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:15:45.550806
End date: 2023-01-06 12:15:45.581822
Task duration: < 1min
Errors:
Standard output:


Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:15:47.634224
End date: 2023-01-06 12:15:48.703295
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:15:47.701390
End date: 2023-01-06 12:15:48.699538
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 12:15:50.955629
End date: 2023-01-06 12:15:51.014969
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 12:15:55.482801
End date: 2023-01-06 12:15:55.748420
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana"
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana\'
Start date: 2023-01-06 12:15:57.884674
End date: 2023-01-06 12:15:57.888967
Task duration: < 5s
Errors:
Standard output:
ignore_errors
4 ETW000 [ dev trc,00000] INFO : Database 'AD0/00' instance is running on 'demo-hana'

Common Steps
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana"
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana"
ansible-role-bridge-task : Show current step
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED BRIDGE TASK"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop secondary database
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Stop secondary database
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop HDB
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Stop HDB

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Stop HANA database
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Stop HANA database
HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB stop
Start date: 2023-01-06 12:16:00.210664
End date: 2023-01-06 12:16:22.306496
Task duration: < 5s
Errors:
Standard output:
hdbdaemon will wait maximal 300 seconds for NewDB services finishing.
Stopping instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function Stop 400

06.01.2023 12:16:00
Stop
OK
Waiting for stopped instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function WaitforStopped 600 2


06.01.2023 12:16:22
WaitforStopped
OK
hdbdaemon is stopped.

Common Steps
ansible-role-ha-test-scenarios : Check SAP ASCS queue length
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Check SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-ha-test-scenarios : Get Enqueue lock number
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 12:16:24.331744
End date: 2023-01-06 12:16:24.392513
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
ASCS: debug - Changed: false
See full message
0

ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
ASCS: debug - Changed: false
See full message
0

ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
ansible-role-ha-test-scenarios : Start HDB
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Start HDB

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Start HANA database
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Start HANA database
HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB start && echo "ignore_errors"
Start date: 2023-01-06 12:16:26.679579
End date: 2023-01-06 12:17:10.880567
Task duration: < 5s
Errors:
Impromptu CCC initialization by 'rscpCInit'.
See SAP note 1266393.
Standard output:


StartService
OK
OK
Starting instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function StartWait 2700 2


06.01.2023 12:16:26
Start
OK

06.01.2023 12:17:10
StartWait
OK
ignore_errors

Common Steps
ansible-role-ha-test-scenarios : Wait for HDB to restablish HA
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for HDB to restablish HA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep DEMOTED
Start date: 2023-01-06 12:17:12.991884
End date: 2023-01-06 12:17:13.008726
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : DEMOTED

Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY to be SOK
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be SOK
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be SOK
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be SOK

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:17:57.796697
End date: 2023-01-06 12:17:57.812476
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK

Common Steps
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get secondary hostname
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-ha-test-scenarios : Wait for counter of PRIMARY be bigger than SECONDARY
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for counter of PRIMARY be bigger than SECONDARY
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(echo $sec_count | tr -d '[:space:]'); re='^[0-9]+$'; if ! [[ $sec_count =~ $re ]] ; then exit 1; fi; if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:18:50.799082
End date: 2023-01-06 12:18:50.834356
Task duration: < 1min
Errors:
Standard output:


Common Steps
ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:18:51.442700Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-doeccyintflzxgxzcoubsvdbdfcourrz ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029131.637883-49896-128841437883132/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-jaigxcouiuxixxeadyavziciuozqcqnw ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029131.656074-49897-149958030933945/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-krgwwxpmmboyjdcdwzbdcwcmjyhdjzzb ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029132.452596-49899-28132922220231/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-llkaqfyidduuptfppvrsleznckqbgqak ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029132.426383-49901-240330919674555/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:19:04.183867
End date: 2023-01-06 12:19:04.235334
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:19:03.947126
End date: 2023-01-06 12:19:03.992698
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:19:06.419764
End date: 2023-01-06 12:19:06.639299
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:19:06.432881
End date: 2023-01-06 12:19:06.615710
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:19:08.890175
End date: 2023-01-06 12:19:09.108874
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:19:08.819668
End date: 2023-01-06 12:19:09.014856
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:19:08.823059
End date: 2023-01-06 12:19:08.825815
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:19:08.884889
End date: 2023-01-06 12:19:08.887954
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Check if this node is secondary
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:19:11.293480
End date: 2023-01-06 12:19:11.504731
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:19:11.287021
End date: 2023-01-06 12:19:11.467707
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:19:11.560047
End date: 2023-01-06 12:19:11.562771
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:19:11.292305
End date: 2023-01-06 12:19:11.295189
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:19:16.559418
End date: 2023-01-06 12:19:16.771222
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:19:16.199530
End date: 2023-01-06 12:19:16.394598
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : System is not ready for HA testing
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:19:19.343125
End date: 2023-01-06 12:19:19.550014
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code


ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check HA configuration status after self-healing actions

HANA Sec: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here

Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check current status for PRIMARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 12:19:21.911893
End date: 2023-01-06 12:19:21.929529
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 1673029135
* master-SAPHana_AD0_00 : 150


ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check current status for SECONDARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 12:19:23.964732
End date: 2023-01-06 12:19:23.980595
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SOK
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 30
* master-SAPHana_AD0_00 : 100

Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:19:26.162462
End date: 2023-01-06 12:19:26.178709
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK

Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:19:28.219782
End date: 2023-01-06 12:19:28.250876
Task duration: < 1min
Errors:
Standard output:


Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:19:30.457524
End date: 2023-01-06 12:19:31.526657
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:19:30.468219
End date: 2023-01-06 12:19:31.463811
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 12:19:33.726679
End date: 2023-01-06 12:19:33.788881
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 12:19:38.422339
End date: 2023-01-06 12:19:38.584859
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana"
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana\'
Start date: 2023-01-06 12:19:40.751556
End date: 2023-01-06 12:19:40.755725
Task duration: < 5s
Errors:
Standard output:
ignore_errors
4 ETW000 [ dev trc,00000] INFO : Database 'AD0/00' instance is running on 'demo-hana'

Common Steps
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana"
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana"
ansible-role-bridge-task : Show current step
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED BRIDGE TASK"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING TEST SCENARIO: CRASH_NODE_PROC_PRE"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Run test for CRASH_NODE with "echo 'b' > /proc/sysrq-trigger" scenario
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Run test for CRASH_NODE with "echo 'b' > /proc/sysrq-trigger" scenario
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Gather AWS info for instances
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Gather AWS info for instances
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get instances information from AWS
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Get instances information from AWS
HANA Pri: amazon.aws.ec2_metadata_facts - Changed: false

HANA Sec: amazon.aws.ec2_metadata_facts - Changed: false

ASCS: amazon.aws.ec2_metadata_facts - Changed: false

PAS: amazon.aws.ec2_metadata_facts - Changed: false

ansible-role-ha-test-scenarios : Save instance ID of node to be crashed
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Save instance ID of node to be crashed
HANA Pri: set_fact - Changed: false
See full message
All items completed

HANA Sec: set_fact - Changed: false
See full message
All items completed

ASCS: set_fact - Changed: false
See full message
All items completed

PAS: set_fact - Changed: false
See full message
All items completed

ansible-role-ha-test-scenarios : Crash primary node
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Crash primary node
HANA Pri: include_tasks - Changed: false




ansible-role-ha-test-scenarios : Crash instance using command "echo 'b' > /proc/sysrq-trigger"
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Crash instance using command "echo 'b' > /proc/sysrq-trigger"
HANA Pri: shell - Changed: false
See full message
Failed to connect to the host via ssh: Shared connection to 3.224.135.0 closed.

Common Steps
ansible-role-ha-test-scenarios : Wait for primary node to be in "stopped" state in AWS
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for primary node to be in "stopped" state in AWS
HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Wait for instance i-002c4f7bc22f2412b to be in stopped state
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for instance i-002c4f7bc22f2412b to be in stopped state
HANA Sec: amazon.aws.ec2_instance_info - Changed: false

ansible-role-ha-test-scenarios : Wait for failover to happen
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for failover to happen
HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep PROMOTED
Start date: 2023-01-06 12:23:55.602735
End date: 2023-01-06 12:23:55.618092
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : PROMOTED



Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be PRIM
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be PRIM
HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep PRIM
Start date: 2023-01-06 12:24:47.064179
End date: 2023-01-06 12:24:47.079833
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : PRIM



Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY counter to be bigger than 0
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY counter to be bigger than 0
HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get secondary hostname
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Get secondary hostname
HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than 0
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than 0
HANA Sec: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); zero=0; if (( $prim_count > $zero )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:24:49.215992
End date: 2023-01-06 12:24:49.232594
Task duration: < 1min
Errors:
Standard output:



Common Steps
ansible-role-ha-test-scenarios : Adds restarted instance to HANA group
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Adds restarted instance to HANA group
HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Gather crashed instance status
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Gather crashed instance status
HANA Sec: amazon.aws.ec2_instance_info - Changed: false

ansible-role-ha-test-scenarios : Start crashed instance back
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Start crashed instance back
HANA Sec: shell - Changed: true
Command details Command: aws ec2 start-instances --region us-east-1 --instance-ids i-002c4f7bc22f2412b --profile default
Start date: 2023-01-06 13:24:52.318646
End date: 2023-01-06 13:24:53.825326
Task duration: < 5s
Errors:
Standard output:
{
"StartingInstances": [
{
"CurrentState": {
"Code": 0,
"Name": "pending"
},
"InstanceId": "i-002c4f7bc22f2412b",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
}
]
}

Common Steps
ansible-role-ha-test-scenarios : Wait for instance to be in running state
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for instance to be in running state
HANA Sec: amazon.aws.ec2_instance_info - Changed: false

ansible-role-ha-test-scenarios : Add new instance to hosts group
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Add new instance to hosts group
HANA Sec: add_host - Changed: true

ansible-role-ha-test-scenarios : Run post actions for test "CRASH_NODE" with "echo 'b' > /proc/sysrq-trigger" scenario
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Run post actions for test "CRASH_NODE" with "echo 'b' > /proc/sysrq-trigger" scenario
ansible-role-ha-test-scenarios : Run test for "Reboot" scenario
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Run test for "Reboot" scenario
ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:25:28.651677Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Show current step
HANA Sec: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED TEST SCENARIO: CRASH_NODE_PROC_PRE"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-fiapcalmudxmphsxllwdcajwmyvaqdqb ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029529.578683-50911-9853729036343/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-hroimpgakvsfjgeotbcbxzcsnwtqhjqm ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029528.8655932-50913-185119309289729/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-rngmczdesetqkgbydtkufdxyvgforxtf ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029529.8446589-50914-106425577108924/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-btikvupndgahzfibhyrooubvnfrxcszo ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029529.615629-50916-169805994507244/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:30:26.170843
End date: 2023-01-06 12:30:26.216339
Task duration: 3min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:25:41.311613
End date: 2023-01-06 12:25:41.357859
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:30:35.779948
End date: 2023-01-06 12:30:36.013482
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:30:29.349964
End date: 2023-01-06 12:30:29.543911
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------------ | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |

status system replication site "1": UNKNOWN
overall system replication status: UNKNOWN

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:30:38.080004
End date: 2023-01-06 12:30:38.305728
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:30:38.075836
End date: 2023-01-06 12:30:38.294892
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | STARTING | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | STARTING | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | STARTING | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | STARTING | UNKNOWN | UNKNOWN | |

status system replication site "1": UNKNOWN
overall system replication status: UNKNOWN

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:30:39.102585
End date: 2023-01-06 12:30:39.105411
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:30:39.106894
End date: 2023-01-06 12:30:39.110050
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Check if this node is secondary
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:30:41.255812
End date: 2023-01-06 12:30:41.449064
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:30:41.264417
End date: 2023-01-06 12:30:41.466310
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ------------ | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | STARTING | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": UNKNOWN
overall system replication status: UNKNOWN

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:30:42.035858
End date: 2023-01-06 12:30:42.038643
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:30:41.684075
End date: 2023-01-06 12:30:41.686975
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:30:47.660814
End date: 2023-01-06 12:30:47.841280
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:30:47.663245
End date: 2023-01-06 12:30:47.872827
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ------------ | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": INITIALIZING
overall system replication status: INITIALIZING

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : System is not ready for HA testing

HANA Sec: debug - Changed: false
See full message
An ERROR was found while checking for current HANA installation. Starting now some self-healing actions

ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:31:04.603183
End date: 2023-01-06 12:31:04.807069
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check HA configuration status after self-healing actions
HANA Pri: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here


Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check current status for PRIMARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 12:31:07.103446
End date: 2023-01-06 12:31:07.120416
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 1673029859
* master-SAPHana_AD0_00 : 150

ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check current status for SECONDARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 12:31:09.181629
End date: 2023-01-06 12:31:09.202316
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SFAIL
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 30
* master-SAPHana_AD0_00 : -INFINITY


Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:32:08.026823
End date: 2023-01-06 12:32:08.044598
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK


Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:32:11.006689
End date: 2023-01-06 12:32:11.036175
Task duration: < 1min
Errors:
Standard output:

Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:32:13.312204
End date: 2023-01-06 12:32:14.296344
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:32:13.116357
End date: 2023-01-06 12:32:14.108299
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 12:32:17.462437
End date: 2023-01-06 12:32:17.528862
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 12:32:22.744050
End date: 2023-01-06 12:32:22.999392
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana2\'
Start date: 2023-01-06 12:32:25.125069
End date: 2023-01-06 12:32:25.129348
Task duration: < 5s
Errors:
Standard output:
ignore_errors
4 ETW000 [ dev trc,00000] INFO : Database 'AD0/00' instance is running on 'demo-hana2'

Common Steps
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
ansible-role-bridge-task : Show current step
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED BRIDGE TASK"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING TEST SCENARIO: CRASH_NODE_PROC_POST"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Run post actions for test "CRASH_NODE" with "echo 'b' > /proc/sysrq-trigger" scenario
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Run post actions for test "CRASH_NODE" with "echo 'b' > /proc/sysrq-trigger" scenario
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for new SECONDARY instance to be up
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for new SECONDARY instance to be up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep DEMOTED
Start date: 2023-01-06 12:32:27.730664
End date: 2023-01-06 12:32:27.747618
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : DEMOTED




Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY to be SOK
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for SECONDARY to be SOK
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be SOK
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be SOK
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:32:29.882716
End date: 2023-01-06 12:32:29.898332
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK




Common Steps
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get secondary hostname
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Wait for counter of PRIMARY be bigger than SECONDARY
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for counter of PRIMARY be bigger than SECONDARY

HANA Sec: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(echo $sec_count | tr -d '[:space:]'); re='^[0-9]+$'; if ! [[ $sec_count =~ $re ]] ; then exit 1; fi; if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:32:31.987524
End date: 2023-01-06 12:32:32.017894
Task duration: < 1min
Errors:
Standard output:



Common Steps
ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:32:32.370543Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED TEST SCENARIO: CRASH_NODE_PROC_POST"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-swhzvnuxrtagkcudznkpkdugtxyspobw ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029952.5782988-51979-233572218354727/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-zthjbrotypjhpxrictdapvandvlsoimx ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029952.578327-51981-84516852173192/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-gpmchzxetxbljgxznyeeigdrwlohchrh ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029952.5906901-51982-45719757972214/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-wukabxionaoycvfcepyrsanejflpkqch ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673029952.595339-51984-27327424164355/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:32:44.484277
End date: 2023-01-06 12:32:44.529957
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:32:44.531295
End date: 2023-01-06 12:32:44.577518
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:32:46.779477
End date: 2023-01-06 12:32:46.984172
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:32:46.779605
End date: 2023-01-06 12:32:46.987987
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:32:49.089208
End date: 2023-01-06 12:32:49.269836
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:32:49.086761
End date: 2023-01-06 12:32:49.292093
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:32:49.090368
End date: 2023-01-06 12:32:49.093107
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:32:49.097307
End date: 2023-01-06 12:32:49.100356
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Check if this node is secondary
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:32:51.408206
End date: 2023-01-06 12:32:51.601032
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:32:51.407764
End date: 2023-01-06 12:32:51.614696
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:32:51.413066
End date: 2023-01-06 12:32:51.415946
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:32:51.429078
End date: 2023-01-06 12:32:51.432148
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:32:57.324705
End date: 2023-01-06 12:32:57.518123
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:32:57.320413
End date: 2023-01-06 12:32:57.526386
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : System is not ready for HA testing
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:32:59.633778
End date: 2023-01-06 12:32:59.839591
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check HA configuration status after self-healing actions
HANA Pri: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here


Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check current status for PRIMARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 12:33:02.030586
End date: 2023-01-06 12:33:02.046607
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 1673029921
* master-SAPHana_AD0_00 : 150

ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check current status for SECONDARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 12:33:04.082853
End date: 2023-01-06 12:33:04.098674
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SOK
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 30
* master-SAPHana_AD0_00 : 100


Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:33:06.306588
End date: 2023-01-06 12:33:06.322382
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK


Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:33:08.343330
End date: 2023-01-06 12:33:08.373541
Task duration: < 1min
Errors:
Standard output:

Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:33:10.464592
End date: 2023-01-06 12:33:11.448516
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:33:10.462729
End date: 2023-01-06 12:33:11.454776
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 12:33:13.636558
End date: 2023-01-06 12:33:13.697765
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 12:33:17.918161
End date: 2023-01-06 12:33:18.094214
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana2\'
Start date: 2023-01-06 12:33:20.175486
End date: 2023-01-06 12:33:20.179797
Task duration: < 5s
Errors:
Standard output:
ignore_errors
4 ETW000 [ dev trc,00000] INFO : Database 'AD0/00' instance is running on 'demo-hana2'

Common Steps
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
ansible-role-bridge-task : Show current step
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED BRIDGE TASK"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING TEST SCENARIO: CRASH_NODE_PROC_PRE"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Run test for CRASH_NODE with "echo 'b' > /proc/sysrq-trigger" scenario
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Run test for CRASH_NODE with "echo 'b' > /proc/sysrq-trigger" scenario
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Gather AWS info for instances
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Gather AWS info for instances
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get instances information from AWS
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Get instances information from AWS
HANA Pri: amazon.aws.ec2_metadata_facts - Changed: false

HANA Sec: amazon.aws.ec2_metadata_facts - Changed: false

ASCS: amazon.aws.ec2_metadata_facts - Changed: false

PAS: amazon.aws.ec2_metadata_facts - Changed: false

ansible-role-ha-test-scenarios : Save instance ID of node to be crashed
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Save instance ID of node to be crashed
HANA Pri: set_fact - Changed: false
See full message
All items completed

HANA Sec: set_fact - Changed: false
See full message
All items completed

ASCS: set_fact - Changed: false
See full message
All items completed

PAS: set_fact - Changed: false
See full message
All items completed

ansible-role-ha-test-scenarios : Crash primary node
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Crash primary node

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Crash instance using command "echo 'b' > /proc/sysrq-trigger"
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Crash instance using command "echo 'b' > /proc/sysrq-trigger"
HANA Sec: shell - Changed: false
See full message
Failed to connect to the host via ssh: Shared connection to 3.88.185.79 closed.

Common Steps
ansible-role-ha-test-scenarios : Wait for primary node to be in "stopped" state in AWS
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for primary node to be in "stopped" state in AWS
HANA Pri: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Wait for instance i-08aab2be1a109d26f to be in stopped state
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for instance i-08aab2be1a109d26f to be in stopped state
HANA Pri: amazon.aws.ec2_instance_info - Changed: false

ansible-role-ha-test-scenarios : Wait for failover to happen
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for failover to happen
HANA Pri: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Pri: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep PROMOTED
Start date: 2023-01-06 12:37:28.174722
End date: 2023-01-06 12:37:28.189973
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : PROMOTED



Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
HANA Pri: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be PRIM
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be PRIM
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep PRIM
Start date: 2023-01-06 12:38:27.326362
End date: 2023-01-06 12:38:27.341284
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : PRIM



Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY counter to be bigger than 0
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY counter to be bigger than 0
HANA Pri: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get secondary hostname
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Get secondary hostname
HANA Pri: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than 0
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than 0
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); zero=0; if (( $prim_count > $zero )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:38:29.548137
End date: 2023-01-06 12:38:29.564410
Task duration: < 1min
Errors:
Standard output:



Common Steps
ansible-role-ha-test-scenarios : Adds restarted instance to HANA group
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Adds restarted instance to HANA group
HANA Pri: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Gather crashed instance status
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Gather crashed instance status
HANA Pri: amazon.aws.ec2_instance_info - Changed: false

ansible-role-ha-test-scenarios : Start crashed instance back
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Start crashed instance back
HANA Pri: shell - Changed: true
Command details Command: aws ec2 start-instances --region us-east-1 --instance-ids i-08aab2be1a109d26f --profile default
Start date: 2023-01-06 13:38:32.601805
End date: 2023-01-06 13:38:34.823661
Task duration: < 5s
Errors:
Standard output:
{
"StartingInstances": [
{
"CurrentState": {
"Code": 0,
"Name": "pending"
},
"InstanceId": "i-08aab2be1a109d26f",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
}
]
}

Common Steps
ansible-role-ha-test-scenarios : Wait for instance to be in running state
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for instance to be in running state
HANA Pri: amazon.aws.ec2_instance_info - Changed: false

ansible-role-ha-test-scenarios : Add new instance to hosts group
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Add new instance to hosts group
HANA Pri: add_host - Changed: true

ansible-role-ha-test-scenarios : Run post actions for test "CRASH_NODE" with "echo 'b' > /proc/sysrq-trigger" scenario
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Run post actions for test "CRASH_NODE" with "echo 'b' > /proc/sysrq-trigger" scenario
ansible-role-ha-test-scenarios : Run test for "Reboot" scenario
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Run test for "Reboot" scenario
ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:39:09.830480Z Stop Secondary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED TEST SCENARIO: CRASH_NODE_PROC_PRE"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-frbalwefezqdscuxdhvwrgjuiwbdwrfm ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673030350.047987-53007-112492081183456/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-lufjxqagzcxmrxvfythpncljsivdinpw ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673030350.815446-53008-201836923242271/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-jievomeflnyjapxnajqjzkojnphtrtlv ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673030350.827363-53011-53372938517721/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-jbcczipxxgraraoyyowobfzbduzzjqxd ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673030350.8573241-53012-182769885537410/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:39:23.138987
End date: 2023-01-06 12:39:23.184642
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 12:43:58.953587
End date: 2023-01-06 12:43:58.999329
Task duration: 3min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:44:02.567394
End date: 2023-01-06 12:44:02.789248
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------------ | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |

status system replication site "2": UNKNOWN
overall system replication status: UNKNOWN

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:44:15.756094
End date: 2023-01-06 12:44:15.948224
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:44:18.046536
End date: 2023-01-06 12:44:18.243918
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ------------ | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | STARTING | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": UNKNOWN
overall system replication status: UNKNOWN

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:44:18.039567
End date: 2023-01-06 12:44:18.232390
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:44:19.019629
End date: 2023-01-06 12:44:19.022380
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:44:19.018192
End date: 2023-01-06 12:44:19.021326
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Check if this node is secondary
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:44:21.182834
End date: 2023-01-06 12:44:21.419809
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ------------ | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": INITIALIZING
overall system replication status: INITIALIZING

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:44:21.134068
End date: 2023-01-06 12:44:21.326963
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:44:21.148970
End date: 2023-01-06 12:44:21.151768
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:44:21.169926
End date: 2023-01-06 12:44:21.172769
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:44:27.865548
End date: 2023-01-06 12:44:28.061579
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | --------------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | SYNCING | Preparing persistence |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": SYNCING
overall system replication status: SYNCING

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 12:44:27.876838
End date: 2023-01-06 12:44:28.076872
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : System is not ready for HA testing
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 12:44:37.644612
End date: 2023-01-06 12:44:37.850053
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code


ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check HA configuration status after self-healing actions

HANA Sec: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here

Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check current status for PRIMARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 12:44:40.201934
End date: 2023-01-06 12:44:40.217687
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 1673030673
* master-SAPHana_AD0_00 : 150


ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check current status for SECONDARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 12:44:42.289973
End date: 2023-01-06 12:44:42.308338
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SFAIL
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 30
* master-SAPHana_AD0_00 : -INFINITY

Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:45:41.192955
End date: 2023-01-06 12:45:41.208827
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK

Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:45:44.951856
End date: 2023-01-06 12:45:44.982164
Task duration: < 1min
Errors:
Standard output:


Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:45:48.193809
End date: 2023-01-06 12:45:49.167921
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 12:45:47.254065
End date: 2023-01-06 12:45:48.236631
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 12:45:52.417898
End date: 2023-01-06 12:45:52.481184
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 12:45:57.643399
End date: 2023-01-06 12:45:57.884747
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana"
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana\'
Start date: 2023-01-06 12:45:59.927092
End date: 2023-01-06 12:45:59.931226
Task duration: < 5s
Errors:
Standard output:
ignore_errors
4 ETW000 [ dev trc,00000] INFO : Database 'AD0/00' instance is running on 'demo-hana'

Common Steps
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana"
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana"
ansible-role-bridge-task : Show current step
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED BRIDGE TASK"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING TEST SCENARIO: CRASH_NODE_PROC_POST"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Run post actions for test "CRASH_NODE" with "echo 'b' > /proc/sysrq-trigger" scenario
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Run post actions for test "CRASH_NODE" with "echo 'b' > /proc/sysrq-trigger" scenario
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for new SECONDARY instance to be up
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for new SECONDARY instance to be up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep DEMOTED
Start date: 2023-01-06 12:46:02.572821
End date: 2023-01-06 12:46:02.588635
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : DEMOTED



Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY to be SOK
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for SECONDARY to be SOK
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be SOK
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be SOK

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 12:46:04.669586
End date: 2023-01-06 12:46:04.685097
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK



Common Steps
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get secondary hostname
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Wait for counter of PRIMARY be bigger than SECONDARY
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Wait for counter of PRIMARY be bigger than SECONDARY
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(echo $sec_count | tr -d '[:space:]'); re='^[0-9]+$'; if ! [[ $sec_count =~ $re ]] ; then exit 1; fi; if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 12:46:06.921752
End date: 2023-01-06 12:46:06.952628
Task duration: < 1min
Errors:
Standard output:




Common Steps
ansible-role-ha-test-scenarios : Show current step
2023-01-06T18:46:07.353915Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED TEST SCENARIO: CRASH_NODE_PROC_POST"
"------------------------------------------------------"