# Scenario Description Result
1 Check inputs PASS
2 Check current HA installation (prep / bridge tasks) PASS
3 Stop Primary database with HDB Stop and check failover result PASS
4 Stop NEW Primary database with HDB Stop and check failover result PASS
5 Stop Secondary database with HDB Stop and check result PASS
6 Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result PASS
7 Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE FAIL

Common Steps
Gathering Facts
2023-01-06T16:21:59.053212Z Check inputs
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-jxrbrfclfgngnhumoiinaokobtmcombw ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022115.489471-39384-75282095398106/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-cngumbsmtazfduligdekbzfrroqsnzbf ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022115.500909-39385-209564777467711/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-pdogqbpcdczqeimefqmxgaqrvmbsrzql ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022115.501177-39387-67886740969969/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-yobemdfijtqtaffoxqnjydihullsrole ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022115.5016031-39389-106682733674378/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-check-inputs : Check if OS is supported
2023-01-06T16:21:59.053212Z Check inputs
ansible-role-check-inputs : Check if OS is supported
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-check-inputs : Check input parameters
2023-01-06T16:21:59.053212Z Check inputs
ansible-role-check-inputs : Check input parameters
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-check-inputs : Ensure non mandatory input parameters were sent as required
2023-01-06T16:21:59.053212Z Check inputs
ansible-role-check-inputs : Ensure non mandatory input parameters were sent as required
HANA Pri: fail - Changed: false
See full message
The conditional check 'not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined' failed. The error was: error while evaluating conditional (not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined): No variable found with this name: INPUT_EC2_USER The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-check-inputs/tasks/check_parameters.yml': line 12, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Ensure non mandatory input parameters were sent as required ^ here

HANA Sec: fail - Changed: false
See full message
The conditional check 'not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined' failed. The error was: error while evaluating conditional (not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined): No variable found with this name: INPUT_EC2_USER The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-check-inputs/tasks/check_parameters.yml': line 12, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Ensure non mandatory input parameters were sent as required ^ here

ASCS: fail - Changed: false
See full message
The conditional check 'not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined' failed. The error was: error while evaluating conditional (not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined): No variable found with this name: INPUT_EC2_USER The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-check-inputs/tasks/check_parameters.yml': line 12, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Ensure non mandatory input parameters were sent as required ^ here

PAS: fail - Changed: false
See full message
The conditional check 'not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined' failed. The error was: error while evaluating conditional (not non_mandatory_input_variables_to_check is undefined and lookup('vars', item) is undefined): No variable found with this name: INPUT_EC2_USER The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-check-inputs/tasks/check_parameters.yml': line 12, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Ensure non mandatory input parameters were sent as required ^ here

Common Steps
Gathering Facts
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-awbcciqcbntwpkycjqwisbkhhjrbyhej ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022119.267262-39448-54416685347753/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-cpledtkcbwwxetuozlkxdhyhhowqrtii ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022119.267273-39449-116600665571963/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-ysdzmnuwkzkmmjlckbfqtadixvlhlzev ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022119.267367-39451-110993028209959/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-cwswfpjipljhrvrnxspfpridukntqstv ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022119.28257-39453-205444542457490/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 10:22:53.052716
End date: 2023-01-06 10:22:53.100874
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 10:22:10.778464
End date: 2023-01-06 10:22:10.826546
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:24:26.056942
End date: 2023-01-06 10:24:26.275433
Task duration: 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:22:55.221453
End date: 2023-01-06 10:22:55.416342
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:24:28.194420
End date: 2023-01-06 10:24:28.402646
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:24:29.089163
End date: 2023-01-06 10:24:29.301457
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ------------ | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | STARTING | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": UNKNOWN
overall system replication status: UNKNOWN

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:24:29.089278
End date: 2023-01-06 10:24:29.092067
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:24:29.095136
End date: 2023-01-06 10:24:29.098183
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Check if this node is secondary
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:24:31.243948
End date: 2023-01-06 10:24:31.448215
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:24:31.253838
End date: 2023-01-06 10:24:31.464597
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ------------ | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": INITIALIZING
overall system replication status: INITIALIZING

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:24:31.299100
End date: 2023-01-06 10:24:31.301742
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:24:31.257367
End date: 2023-01-06 10:24:31.260336
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:24:35.798445
End date: 2023-01-06 10:24:35.993037
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:24:35.803541
End date: 2023-01-06 10:24:36.002567
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ------------ | --------------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | SYNCING | Preparing persistence |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": INITIALIZING
overall system replication status: INITIALIZING

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : System is not ready for HA testing

HANA Sec: debug - Changed: false
See full message
An ERROR was found while checking for current HANA installation. Starting now some self-healing actions

ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:24:52.071503
End date: 2023-01-06 10:24:52.280461
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check HA configuration status after self-healing actions
HANA Pri: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here


Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check current status for PRIMARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 10:24:54.578053
End date: 2023-01-06 10:24:54.594168
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 1673022237
* master-SAPHana_AD0_00 : 150

ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check current status for SECONDARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 10:24:56.480482
End date: 2023-01-06 10:24:56.498684
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SFAIL
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 10
* master-SAPHana_AD0_00 : 0


Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 10:25:05.445846
End date: 2023-01-06 10:25:05.462935
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK


Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 10:25:07.666979
End date: 2023-01-06 10:25:07.696779
Task duration: < 1min
Errors:
Standard output:

Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 10:25:09.665696
End date: 2023-01-06 10:25:10.754577
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 10:25:09.777595
End date: 2023-01-06 10:25:10.784457
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 10:25:12.851405
End date: 2023-01-06 10:25:12.913957
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 10:25:16.860945
End date: 2023-01-06 10:25:17.153855
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana2\'
Start date: 2023-01-06 10:25:19.090234
End date: 2023-01-06 10:25:19.094478
Task duration: < 5s
Errors:
Standard output:
ignore_errors
4 ETW000 [ dev trc,00000] INFO : Database 'AD0/00' instance is running on 'demo-hana2'

Common Steps
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
ansible-role-bridge-task : Show current step
2023-01-06T16:25:19.416685Z Check current HA installation (prep / bridge tasks)
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED BRIDGE TASK"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-lvcrrcaggzlqumwehljszdxmpdpbesqr ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022319.621557-40081-231258746885993/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-gjyapxpuobzkkvznytjbicbxkkbufqxp ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022319.621557-40082-268599933591787/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-xjgdmcgmbzfkebdxtgoualubkemmwzqz ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022319.64145-40084-150461046873136/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-fjufgcyzjgggeujyggwecdyqtonvieba ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022319.642317-40086-243866214014389/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-ha-test-scenarios : Show current step
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop primary database
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop primary database
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop HDB
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop HDB

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Stop HANA database
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop HANA database
HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB stop
Start date: 2023-01-06 10:25:24.251755
End date: 2023-01-06 10:26:20.360628
Task duration: < 5s
Errors:
Standard output:
hdbdaemon will wait maximal 300 seconds for NewDB services finishing.
Stopping instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function Stop 400

06.01.2023 10:25:24
Stop
OK
Waiting for stopped instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function WaitforStopped 600 2


06.01.2023 10:26:20
WaitforStopped
OK
hdbdaemon is stopped.

Common Steps
ansible-role-ha-test-scenarios : Wait for failover to happen
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for failover to happen
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep PROMOTED
Start date: 2023-01-06 10:27:17.375718
End date: 2023-01-06 10:27:17.392641
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : PROMOTED


Common Steps
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep DEMOTED
Start date: 2023-01-06 10:28:21.349553
End date: 2023-01-06 10:28:21.367259
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : DEMOTED

Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY hana_{{ hana_sid_lower }}_sync_state to be {{ crm_mon_prim }}
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY hana_{{ hana_sid_lower }}_sync_state to be {{ crm_mon_prim }}
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep PRIM
Start date: 2023-01-06 10:28:24.156230
End date: 2023-01-06 10:28:24.174182
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : PRIM


Common Steps
ansible-role-ha-test-scenarios : Wait for PRIMARY to be SOK
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be SOK
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for PRIMARY hana_{{ hana_sid_lower }}_sync_state to be {{ crm_mon_sok }}
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY hana_{{ hana_sid_lower }}_sync_state to be {{ crm_mon_sok }}

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 10:29:14.600264
End date: 2023-01-06 10:29:14.616732
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK

Common Steps
ansible-role-ha-test-scenarios : Include counter checker
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Include counter checker
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get secondary hostname
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY

HANA Sec: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 10:29:16.561484
End date: 2023-01-06 10:29:16.590926
Task duration: < 1min
Errors:
Standard output:

Common Steps
ansible-role-ha-test-scenarios : Check SAP ASCS queue length
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Check SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-ha-test-scenarios : Get Enqueue lock number
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 10:29:19.367581
End date: 2023-01-06 10:29:19.428329
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
ASCS: debug - Changed: false
See full message
0

ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
ASCS: debug - Changed: false
See full message
0

ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
ansible-role-ha-test-scenarios : Start HDB
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Start HDB

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Start HANA database
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Start HANA database
HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB start && echo "ignore_errors"
Start date: 2023-01-06 10:29:21.488795
End date: 2023-01-06 10:29:31.627384
Task duration: < 5s
Errors:
Impromptu CCC initialization by 'rscpCInit'.
See SAP note 1266393.
Standard output:


StartService
OK
OK
Starting instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function StartWait 2700 2


06.01.2023 10:29:21
Start
OK

06.01.2023 10:29:31
StartWait
OK
ignore_errors

Common Steps
ansible-role-ha-test-scenarios : Show current step
2023-01-06T16:29:32.247669Z Stop Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-qnvzzobqtjjfbyqnqqibyzfmjihtmivv ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022573.220038-40503-85328191724774/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-hhajehugeyocluedqpyydaligtgynady ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022572.477261-40504-105583705663855/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-xbtbfhmxcehssctjgpsnkmyigitqkytg ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022572.488192-40506-263015331621560/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-eevanpizycbprkoeinkcjkxwicrajqcx ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022573.2246132-40508-222513831544221/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 10:29:44.066405
End date: 2023-01-06 10:29:44.115888
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 10:29:44.034928
End date: 2023-01-06 10:29:44.081356
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:29:46.155993
End date: 2023-01-06 10:29:46.370370
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:29:46.141859
End date: 2023-01-06 10:29:46.335735
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:29:48.332332
End date: 2023-01-06 10:29:48.547527
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:29:48.352733
End date: 2023-01-06 10:29:48.546605
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:29:48.343337
End date: 2023-01-06 10:29:48.346090
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:29:48.405968
End date: 2023-01-06 10:29:48.409064
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Check if this node is secondary
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:29:50.495492
End date: 2023-01-06 10:29:50.714319
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:29:50.461925
End date: 2023-01-06 10:29:50.656390
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:29:50.526645
End date: 2023-01-06 10:29:50.529337
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:29:50.504478
End date: 2023-01-06 10:29:50.507452
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:29:55.126540
End date: 2023-01-06 10:29:55.344695
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:29:55.117127
End date: 2023-01-06 10:29:55.301140
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : System is not ready for HA testing
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:29:57.297759
End date: 2023-01-06 10:29:57.517047
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code


ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check HA configuration status after self-healing actions

HANA Sec: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here

Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check current status for PRIMARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 10:29:59.537433
End date: 2023-01-06 10:29:59.554528
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 1673022552
* master-SAPHana_AD0_00 : 150


ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check current status for SECONDARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 10:30:01.459151
End date: 2023-01-06 10:30:01.475819
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SOK
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 30
* master-SAPHana_AD0_00 : 0

Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 10:30:03.481585
End date: 2023-01-06 10:30:03.496887
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK

Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 10:30:05.385512
End date: 2023-01-06 10:30:05.416400
Task duration: < 1min
Errors:
Standard output:


Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 10:30:07.397039
End date: 2023-01-06 10:30:08.422708
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 10:30:07.399053
End date: 2023-01-06 10:30:08.416930
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 10:30:10.533541
End date: 2023-01-06 10:30:10.596553
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 10:30:14.596762
End date: 2023-01-06 10:30:14.903935
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana"
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana\'
Start date: 2023-01-06 10:30:16.819960
End date: 2023-01-06 10:30:16.824448
Task duration: < 5s
Errors:
Standard output:
ignore_errors
4 ETW000 [ dev trc,00000] INFO : Database 'AD0/00' instance is running on 'demo-hana'

Common Steps
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana"
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana"
ansible-role-bridge-task : Show current step
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED BRIDGE TASK"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Show current step
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop primary database
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop primary database
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop HDB
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop HDB
HANA Pri: include_tasks - Changed: false




ansible-role-ha-test-scenarios : Stop HANA database
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Stop HANA database
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB stop
Start date: 2023-01-06 10:30:19.011979
End date: 2023-01-06 10:31:39.133836
Task duration: < 5s
Errors:
Standard output:
hdbdaemon will wait maximal 300 seconds for NewDB services finishing.
Stopping instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function Stop 400

06.01.2023 10:30:19
Stop
OK
Waiting for stopped instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function WaitforStopped 600 2


06.01.2023 10:31:39
WaitforStopped
OK
hdbdaemon is stopped.

Common Steps
ansible-role-ha-test-scenarios : Wait for failover to happen
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for failover to happen
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep PROMOTED
Start date: 2023-01-06 10:32:32.136011
End date: 2023-01-06 10:32:32.152546
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : PROMOTED

Common Steps
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be DEMOTED
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep DEMOTED
Start date: 2023-01-06 10:33:36.194398
End date: 2023-01-06 10:33:36.211152
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : DEMOTED


Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be PRIM
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be PRIM

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep PRIM
Start date: 2023-01-06 10:33:39.063978
End date: 2023-01-06 10:33:39.080544
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : PRIM

Common Steps
ansible-role-ha-test-scenarios : Wait for PRIMARY to be SOK
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY to be SOK
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for PRIMARY hana_ad0_sync_state to be SOK
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for PRIMARY hana_ad0_sync_state to be SOK
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 10:34:30.348100
End date: 2023-01-06 10:34:30.365609
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK


Common Steps
ansible-role-ha-test-scenarios : Include counter checker
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Include counter checker
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get secondary hostname
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 10:34:32.351675
End date: 2023-01-06 10:34:32.384283
Task duration: < 1min
Errors:
Standard output:


Common Steps
ansible-role-ha-test-scenarios : Check SAP ASCS queue length
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Check SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-ha-test-scenarios : Get Enqueue lock number
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 10:34:35.299868
End date: 2023-01-06 10:34:35.362423
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
ASCS: debug - Changed: false
See full message
0

ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
ASCS: debug - Changed: false
See full message
0

ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
ansible-role-ha-test-scenarios : Start HDB
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Start HDB
HANA Pri: include_tasks - Changed: false




ansible-role-ha-test-scenarios : Start HANA database
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Start HANA database
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB start && echo "ignore_errors"
Start date: 2023-01-06 10:34:37.537916
End date: 2023-01-06 10:34:47.684601
Task duration: < 5s
Errors:
Impromptu CCC initialization by 'rscpCInit'.
See SAP note 1266393.
Standard output:


StartService
OK
OK
Starting instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function StartWait 2700 2


06.01.2023 10:34:37
Start
OK

06.01.2023 10:34:47
StartWait
OK
ignore_errors

Common Steps
ansible-role-ha-test-scenarios : Show current step
2023-01-06T16:34:48.274925Z Stop NEW Primary database with HDB Stop and check failover result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-bxwichbaxladobainbddgirpxfaplchp ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022888.479876-41736-227806660782160/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-nrejajrbenxnnwydzmejrmxqhqdzfifp ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022889.2133088-41737-154903312404009/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-tsfimbwhthkqnpdzelovmptljozofaub ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022888.49727-41739-8587241951619/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-abwojtycxtwcjmsuxtytxcxlzsmzekua ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673022889.234497-41741-60207275342878/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 10:35:00.101937
End date: 2023-01-06 10:35:00.150306
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 10:35:00.069472
End date: 2023-01-06 10:35:00.116696
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:35:02.328553
End date: 2023-01-06 10:35:02.521008
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:35:02.317373
End date: 2023-01-06 10:35:02.527654
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:35:04.623752
End date: 2023-01-06 10:35:04.825846
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:35:04.600066
End date: 2023-01-06 10:35:04.808139
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:35:04.619581
End date: 2023-01-06 10:35:04.622492
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:35:04.569524
End date: 2023-01-06 10:35:04.572572
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Check if this node is secondary
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:35:06.876100
End date: 2023-01-06 10:35:07.083633
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:35:06.888724
End date: 2023-01-06 10:35:07.097112
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:35:06.878328
End date: 2023-01-06 10:35:06.881240
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:35:06.865827
End date: 2023-01-06 10:35:06.868841
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:35:11.567102
End date: 2023-01-06 10:35:11.774736
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:35:11.542710
End date: 2023-01-06 10:35:11.760369
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : System is not ready for HA testing
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:35:13.949033
End date: 2023-01-06 10:35:14.165822
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check HA configuration status after self-healing actions
HANA Pri: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here


Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check current status for PRIMARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 10:35:16.583091
End date: 2023-01-06 10:35:16.599296
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 1673022865
* master-SAPHana_AD0_00 : 150

ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check current status for SECONDARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 10:35:18.624750
End date: 2023-01-06 10:35:18.642313
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SOK
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 30
* master-SAPHana_AD0_00 : 0


Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 10:35:20.746697
End date: 2023-01-06 10:35:20.763950
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK


Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 10:35:22.773636
End date: 2023-01-06 10:35:22.803239
Task duration: < 1min
Errors:
Standard output:

Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 10:35:24.845877
End date: 2023-01-06 10:35:25.940591
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 10:35:24.831100
End date: 2023-01-06 10:35:25.897297
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 10:35:28.099187
End date: 2023-01-06 10:35:28.168276
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 10:35:32.231635
End date: 2023-01-06 10:35:32.512551
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana2\'
Start date: 2023-01-06 10:35:34.769459
End date: 2023-01-06 10:35:34.773759
Task duration: < 5s
Errors:
Standard output:
ignore_errors
4 ETW000 [ dev trc,00000] INFO : Database 'AD0/00' instance is running on 'demo-hana2'

Common Steps
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
ansible-role-bridge-task : Show current step
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED BRIDGE TASK"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Show current step
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Run test for "HDB Stop" scenario
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop secondary database
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Stop secondary database
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Stop HDB
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Stop HDB
HANA Pri: include_tasks - Changed: false




ansible-role-ha-test-scenarios : Stop HANA database
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Stop HANA database
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB stop
Start date: 2023-01-06 10:35:37.072641
End date: 2023-01-06 10:35:57.173615
Task duration: < 5s
Errors:
Standard output:
hdbdaemon will wait maximal 300 seconds for NewDB services finishing.
Stopping instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function Stop 400

06.01.2023 10:35:37
Stop
OK
Waiting for stopped instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function WaitforStopped 600 2


06.01.2023 10:35:57
WaitforStopped
OK
hdbdaemon is stopped.

Common Steps
ansible-role-ha-test-scenarios : Check SAP ASCS queue length
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Check SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-ha-test-scenarios : Get Enqueue lock number
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 10:35:59.188710
End date: 2023-01-06 10:35:59.249532
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Save current ASCS enqueue count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Show ASCS enqueue count post cut
ASCS: debug - Changed: false
See full message
0

ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Show ASCS enqueue count pre cut
ASCS: debug - Changed: false
See full message
0

ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Find ASCS enqueue drift count
ASCS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Fail if enqueue count drift is bigger than param 5
ansible-role-ha-test-scenarios : Start HDB
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Start HDB
HANA Pri: include_tasks - Changed: false




ansible-role-ha-test-scenarios : Start HANA database
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Start HANA database
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh; HDB start && echo "ignore_errors"
Start date: 2023-01-06 10:36:01.374902
End date: 2023-01-06 10:36:47.589455
Task duration: < 5s
Errors:
Impromptu CCC initialization by 'rscpCInit'.
See SAP note 1266393.
Standard output:


StartService
OK
OK
Starting instance using: /usr/sap/AD0/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function StartWait 2700 2


06.01.2023 10:36:01
Start
OK

06.01.2023 10:36:47
StartWait
OK
ignore_errors

Common Steps
ansible-role-ha-test-scenarios : Wait for HDB to restablish HA
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for HDB to restablish HA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be DEMOTED
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep DEMOTED
Start date: 2023-01-06 10:36:49.728001
End date: 2023-01-06 10:36:49.745678
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : DEMOTED


Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY to be SOK
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be SOK
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be SOK
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be SOK
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 10:37:33.883185
End date: 2023-01-06 10:37:33.901505
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK


Common Steps
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than PRIMARY
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get secondary hostname
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-ha-test-scenarios : Wait for counter of PRIMARY be bigger than SECONDARY
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Wait for counter of PRIMARY be bigger than SECONDARY

HANA Sec: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(echo $sec_count | tr -d '[:space:]'); re='^[0-9]+$'; if ! [[ $sec_count =~ $re ]] ; then exit 1; fi; if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 10:38:33.483145
End date: 2023-01-06 10:38:33.515133
Task duration: < 1min
Errors:
Standard output:

Common Steps
ansible-role-ha-test-scenarios : Show current step
2023-01-06T16:38:34.056023Z Stop Secondary database with HDB Stop and check result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED TEST SCENARIO: HDB_STOP"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-vgrmeftdsueyzybuhwlkrzudlnidzhuu ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673023115.060611-42477-151151820226822/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-rleubptlxslptysppyhyyfytwhkfibdl ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673023114.268301-42478-225177839865592/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-pftshcpsyfsacizxexzsyijhvqfifhrq ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673023115.065462-42480-93598619258359/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-ctyjkcgnqpqteebbjltjakugbyuefopo ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673023115.065691-42482-46781386835485/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 10:38:46.401059
End date: 2023-01-06 10:38:46.449729
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 10:38:46.645911
End date: 2023-01-06 10:38:46.692127
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:38:48.839114
End date: 2023-01-06 10:38:49.033108
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:38:48.840331
End date: 2023-01-06 10:38:49.046821
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:38:51.510195
End date: 2023-01-06 10:38:51.720898
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:38:51.148224
End date: 2023-01-06 10:38:51.355737
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:38:51.119807
End date: 2023-01-06 10:38:51.122637
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:38:51.108690
End date: 2023-01-06 10:38:51.111677
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Check if this node is secondary
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:38:53.789191
End date: 2023-01-06 10:38:53.996212
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:38:53.706090
End date: 2023-01-06 10:38:53.914335
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:38:53.761634
End date: 2023-01-06 10:38:53.764329
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:38:53.745906
End date: 2023-01-06 10:38:53.748897
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:38:58.506494
End date: 2023-01-06 10:38:58.710785
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 1
site name: demo-hana
active primary site: 2
primary masters: demo-hana2
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:38:58.477505
End date: 2023-01-06 10:38:58.673800
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : System is not ready for HA testing
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:39:00.633680
End date: 2023-01-06 10:39:00.829116
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana2 | 30001 | nameserver | 1 | 2 | demo-hana2 | demo-hana | 30001 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30007 | xsengine | 2 | 2 | demo-hana2 | demo-hana | 30007 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30040 | docstore | 5 | 2 | demo-hana2 | demo-hana | 30040 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30003 | indexserver | 3 | 2 | demo-hana2 | demo-hana | 30003 | 1 | demo-hana | YES | SYNC | ACTIVE | |
| AD0 | demo-hana2 | 30011 | dpserver | 4 | 2 | demo-hana2 | demo-hana | 30011 | 1 | demo-hana | YES | SYNC | ACTIVE | |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: demo-hana2
See full message
non-zero return code

ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check HA configuration status after self-healing actions
HANA Pri: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here


Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check current status for PRIMARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 10:39:02.848177
End date: 2023-01-06 10:39:02.864120
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 1673023113
* master-SAPHana_AD0_00 : 150

ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check current status for SECONDARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 10:39:04.965748
End date: 2023-01-06 10:39:04.982407
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SOK
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 30
* master-SAPHana_AD0_00 : 100


Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 10:39:06.999165
End date: 2023-01-06 10:39:07.015971
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK


Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 10:39:08.922465
End date: 2023-01-06 10:39:08.952009
Task duration: < 1min
Errors:
Standard output:

Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 10:39:10.870320
End date: 2023-01-06 10:39:11.914419
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 10:39:10.865990
End date: 2023-01-06 10:39:11.869077
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 1
site name: demo-hana

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 2

primary masters: demo-hana2

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: demo-hana2

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana2 (primary/primary)
|---demo-hana (sync/logreplay)

Tier of demo-hana2: 1
Tier of demo-hana: 2

Replication mode of demo-hana2: primary
Replication mode of demo-hana: sync

Operation mode of demo-hana2: primary
Operation mode of demo-hana: logreplay

Mapping: demo-hana2 -> demo-hana
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 10:39:13.914688
End date: 2023-01-06 10:39:13.976144
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 10:39:17.893330
End date: 2023-01-06 10:39:18.066525
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana2\'
Start date: 2023-01-06 10:39:20.037232
End date: 2023-01-06 10:39:20.041398
Task duration: < 5s
Errors:
Standard output:
ignore_errors
4 ETW000 [ dev trc,00000] INFO : Database 'AD0/00' instance is running on 'demo-hana2'

Common Steps
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
ansible-role-bridge-task : Show current step
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-bridge-task : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED BRIDGE TASK"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Show current step
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING TEST SCENARIO: CRASH_NODE_PROC_PRE"
"------------------------------------------------------"

ansible-role-ha-test-scenarios : Run test for CRASH_NODE with "echo 'b' > /proc/sysrq-trigger" scenario
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Run test for CRASH_NODE with "echo 'b' > /proc/sysrq-trigger" scenario
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Gather AWS info for instances
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Gather AWS info for instances
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get instances information from AWS
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Get instances information from AWS
HANA Pri: amazon.aws.ec2_metadata_facts - Changed: false

HANA Sec: amazon.aws.ec2_metadata_facts - Changed: false

ASCS: amazon.aws.ec2_metadata_facts - Changed: false

PAS: amazon.aws.ec2_metadata_facts - Changed: false

ansible-role-ha-test-scenarios : Save instance ID of node to be crashed
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Save instance ID of node to be crashed
HANA Pri: set_fact - Changed: false
See full message
All items completed

HANA Sec: set_fact - Changed: false
See full message
All items completed

ASCS: set_fact - Changed: false
See full message
All items completed

PAS: set_fact - Changed: false
See full message
All items completed

ansible-role-ha-test-scenarios : Crash primary node
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Crash primary node

HANA Sec: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Crash instance using command "echo 'b' > /proc/sysrq-trigger"
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Crash instance using command "echo 'b' > /proc/sysrq-trigger"
HANA Sec: shell - Changed: false
See full message
Failed to connect to the host via ssh: Shared connection to 184.72.76.117 closed.

Common Steps
ansible-role-ha-test-scenarios : Wait for primary node to be in "stopped" state in AWS
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for primary node to be in "stopped" state in AWS
HANA Pri: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Wait for instance i-08aab2be1a109d26f to be in stopped state
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for instance i-08aab2be1a109d26f to be in stopped state
HANA Pri: amazon.aws.ec2_instance_info - Changed: false

ansible-role-ha-test-scenarios : Wait for failover to happen
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for failover to happen
HANA Pri: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Pri: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PROMOTED
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep PROMOTED
Start date: 2023-01-06 10:40:51.027513
End date: 2023-01-06 10:40:51.049599
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_clone_state : PROMOTED



Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY to be PRIM
HANA Pri: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be PRIM
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY hana_ad0_sync_state to be PRIM
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | grep PRIM
Start date: 2023-01-06 10:41:30.159197
End date: 2023-01-06 10:41:30.176306
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : PRIM



Common Steps
ansible-role-ha-test-scenarios : Wait for SECONDARY counter to be bigger than 0
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for SECONDARY counter to be bigger than 0
HANA Pri: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-ha-test-scenarios : Get secondary hostname
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Get secondary hostname
HANA Pri: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than 0
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for counter of SECONDARY be bigger than 0
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); zero=0; if (( $prim_count > $zero )); then exit 0; else exit 1; fi
Start date: 2023-01-06 10:41:32.270411
End date: 2023-01-06 10:41:32.291956
Task duration: < 1min
Errors:
Standard output:



Common Steps
ansible-role-ha-test-scenarios : Adds restarted instance to HANA group
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Adds restarted instance to HANA group
HANA Pri: include_tasks - Changed: false



ansible-role-ha-test-scenarios : Gather crashed instance status
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Gather crashed instance status
HANA Pri: amazon.aws.ec2_instance_info - Changed: false

ansible-role-ha-test-scenarios : Start crashed instance back
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Start crashed instance back
HANA Pri: shell - Changed: true
Command details Command: aws ec2 start-instances --region us-east-1 --instance-ids i-08aab2be1a109d26f --profile default
Start date: 2023-01-06 11:41:35.165880
End date: 2023-01-06 11:41:36.737024
Task duration: < 5s
Errors:
Standard output:
{
"StartingInstances": [
{
"CurrentState": {
"Code": 0,
"Name": "pending"
},
"InstanceId": "i-08aab2be1a109d26f",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
}
]
}

Common Steps
ansible-role-ha-test-scenarios : Wait for instance to be in running state
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Wait for instance to be in running state
HANA Pri: amazon.aws.ec2_instance_info - Changed: false

ansible-role-ha-test-scenarios : Add new instance to hosts group
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Add new instance to hosts group
HANA Pri: add_host - Changed: true

ansible-role-ha-test-scenarios : Run post actions for test "CRASH_NODE" with "echo 'b' > /proc/sysrq-trigger" scenario
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Run post actions for test "CRASH_NODE" with "echo 'b' > /proc/sysrq-trigger" scenario
ansible-role-ha-test-scenarios : Run test for "Reboot" scenario
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Run test for "Reboot" scenario
ansible-role-ha-test-scenarios : Show current step
2023-01-06T16:42:11.535665Z Stop Primary database with "echo 'b' > /proc/sysrq-trigger" and check result
ansible-role-ha-test-scenarios : Show current step
HANA Pri: debug - Changed: false
See full message
"------------------------------------------------------"
"FINISHED TEST SCENARIO: CRASH_NODE_PROC_PRE"
"------------------------------------------------------"

Common Steps
Gathering Facts
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
Gathering Facts
HANA Pri: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-gpnkncuhqokytmrnfpbmmcvcpofzcnkc ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673023331.745881-43370-108415426654289/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.9.172 192.168.10.10]
ansible_all_ipv6_addresses: [fe80::7:3eff:fe91:4497]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana.sapgspteam.net
ansible_hostname: demo-hana
ansible_nodename: demo-hana
ansible_os_family: RedHat

HANA Sec: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-aavutcjdwzkzaopleteavfseintdyqfx ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673023332.5032792-43368-229863102054431/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.31.149]
ansible_all_ipv6_addresses: [fe80::844:79ff:fe32:6a11]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-hana2.sapgspteam.net
ansible_hostname: demo-hana2
ansible_nodename: demo-hana2
ansible_os_family: RedHat

ASCS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-lhkzjewmealjktwazrvtrerhghxagacj ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673023332.510019-43371-155519735906289/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.11.24]
ansible_all_ipv6_addresses: [fe80::4b:4dff:fe66:146f]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-ascs.sapgspteam.net
ansible_hostname: demo-ascs
ansible_nodename: demo-ascs
ansible_os_family: RedHat

PAS: gather_facts - Changed: false
Instance state details
Environment variables HOME: /root
PATH: /sbin:/bin:/usr/sbin:/usr/bin
SUDO_GID: 1000
TERM: xterm-256color
USER: root
SUDO_UID: 1000
LOGNAME: root
PWD: /home/ec2-user
SUDO_COMMAND: /bin/sh -c echo BECOME-SUCCESS-xqplrqpjjybjdnbpuxpyjnftgqgzgfiq ; /usr/libexec/platform-python /home/ec2-user/.ansible/tmp/ansible-tmp-1673023332.503359-43373-163336745059442/AnsiballZ_setup.py
SHELL: /bin/bash
ansible_all_ipv4_addresses: [172.31.15.152]
ansible_all_ipv6_addresses: [fe80::9b:e9ff:fe8e:a41d]
ansible_distribution: RedHat
ansible_domain: sapgspteam.net
ansible_fqdn: demo-pas.sapgspteam.net
ansible_hostname: demo-pas
ansible_nodename: demo-pas
ansible_os_family: RedHat

ansible-role-bridge-task : Show current step
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Show current step
HANA Sec: debug - Changed: false
See full message
"------------------------------------------------------"
"STARTING BRIDGE TASK"
"------------------------------------------------------"

ansible-role-bridge-task : Common preparation steps
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Common preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if custom sapenv.sh exists
HANA Pri: stat - Changed: false

HANA Sec: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if custom sapenv.sh exists
ASCS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure custom .sapenv.sh file exists



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if custom sapenv.sh exists
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if custom sapenv.sh exists
PAS: stat - Changed: false

ansible-role-bridge-task : Create copy of original .sapenv.sh file
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Create copy of original .sapenv.sh file
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Remove all "tset" statements from the file. PS! This is important when calling Ansible from Jenkins
ansible-role-bridge-task : Adding HANA preparation steps
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Adding HANA preparation steps
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if HANA is fully up
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if HANA is fully up on both nodes with GetProcessList. Waits for it to be up if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 10:42:24.111710
End date: 2023-01-06 10:42:24.160144
Task duration: < 1min
Errors:
Standard output:
0
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && sapcontrol -nr 00 -function GetProcessList | grep -c YELLOW
Start date: 2023-01-06 10:46:58.413348
End date: 2023-01-06 10:46:58.459665
Task duration: 3min
Errors:
Standard output:
0
See full message
non-zero return code



Common Steps
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register which node is Primary and Secondary for HANA
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ASCS: include_tasks - Changed: false

PAS: include_tasks - Changed: false

ansible-role-bridge-task : Erase all PRIMARY IPs
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Erase all PRIMARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Erase all SECONDARY IPs
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Erase all SECONDARY IPs
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Waits for status message if required by a past failover event
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Waits for status message if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:47:01.538988
End date: 2023-01-06 10:47:01.768769
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------------ | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | CONNECTION TIMEOUT | UNKNOWN | UNKNOWN | |

status system replication site "2": UNKNOWN
overall system replication status: UNKNOWN

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:47:07.832118
End date: 2023-01-06 10:47:08.041677
Task duration: < 1min
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code



ansible-role-bridge-task : Check if this node is primary
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if this node is primary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:47:10.049602
End date: 2023-01-06 10:47:10.276387
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | NO | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | NO | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | NO | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | NO | UNKNOWN | UNKNOWN | |

status system replication site "2": UNKNOWN
overall system replication status: UNKNOWN

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:47:10.061294
End date: 2023-01-06 10:47:10.256500
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:47:10.968223
End date: 2023-01-06 10:47:10.970991
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:47:10.972234
End date: 2023-01-06 10:47:10.975169
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register THIS NODE ip as PRIMARY
HANA Pri: set_fact - Changed: false




ansible-role-bridge-task : Check if this node is secondary
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if this node is secondary
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:47:12.991027
End date: 2023-01-06 10:47:13.238794
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | --------------------------------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | SYNCING | Missing Log: 100 % (360448/360448 Byte) |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | STARTING | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | STARTING | UNKNOWN | UNKNOWN | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | SYNCING | Missing Log: 100 % (352256/352256 Byte) |

status system replication site "2": UNKNOWN
overall system replication status: UNKNOWN

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:47:12.978792
End date: 2023-01-06 10:47:13.171634
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

ASCS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:47:13.003809
End date: 2023-01-06 10:47:13.006568
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

PAS: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:47:13.005970
End date: 2023-01-06 10:47:13.008947
Task duration: < 5s
Errors:
/bin/sh: /usr/sap/AD0/home/.sapenv-ansible.sh: No such file or directory
Standard output:
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register THIS NODE ip as SECONDARY

HANA Sec: set_fact - Changed: false



ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register primary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register secondary public HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public IP for all nodes
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register primary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public IP for all nodes
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register secondary public IP for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register primary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Register secondary public INVENTORY_HOSTNAME for all nodes
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ASCS: set_fact - Changed: false

PAS: set_fact - Changed: false

ansible-role-bridge-task : Check HA config status
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check HA config status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check SYNC replication status
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check SYNC replication status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false

ansible-role-bridge-task : Check HA configuration status
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check HA configuration status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:47:18.738246
End date: 2023-01-06 10:47:18.956478
Task duration: < 5s
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ------------ | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | STARTING | UNKNOWN | INITIALIZING | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": INITIALIZING
overall system replication status: INITIALIZING

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py && echo "ignore_errors"
Start date: 2023-01-06 10:47:18.723865
End date: 2023-01-06 10:47:18.904304
Task duration: < 5s
Errors:
Standard output:
this system is either not running or not primary system replication site

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: SYNC
site id: 2
site name: demo-hana2
active primary site: 1
primary masters: demo-hana
See full message
non-zero return code

Common Steps
ansible-role-bridge-task : System is not ready for HA testing
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : System is not ready for HA testing
HANA Pri: debug - Changed: false
See full message
An ERROR was found while checking for current HANA installation. Starting now some self-healing actions


ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Wait for replication to be active before triggering self-healing actions
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && cd /usr/sap/AD0/HDB00/exe/python_support && python systemReplicationStatus.py
Start date: 2023-01-06 10:47:35.376282
End date: 2023-01-06 10:47:35.581506
Task duration: < 1min
Errors:
Standard output:
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication |
| | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details |
| -------- | --------- | ----- | ------------ | --------- | ------- | --------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | demo-hana | 30001 | nameserver | 1 | 1 | demo-hana | demo-hana2 | 30001 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30007 | xsengine | 2 | 1 | demo-hana | demo-hana2 | 30007 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30040 | docstore | 5 | 1 | demo-hana | demo-hana2 | 30040 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30003 | indexserver | 3 | 1 | demo-hana | demo-hana2 | 30003 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |
| AD0 | demo-hana | 30011 | dpserver | 4 | 1 | demo-hana | demo-hana2 | 30011 | 2 | demo-hana2 | YES | SYNC | ACTIVE | |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: demo-hana
See full message
non-zero return code


ansible-role-bridge-task : Check HA configuration status after self-healing actions
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check HA configuration status after self-healing actions

HANA Sec: shell - Changed: false
See full message
The conditional check 'mode_unknown in output_before.stdout' failed. The error was: error while evaluating conditional (mode_unknown in output_before.stdout): 'dict object' has no attribute 'stdout' The error appears to be in '/Users/sesterhg/AWS/code/awsgithubpubliccode/automate-sap-hana-high-availability-test/roles/ansible-role-bridge-task/tasks/hana/check_ha_current_status.yaml': line 9, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Check HA configuration status after self-healing actions ^ here

Common Steps
ansible-role-bridge-task : Fail if HA setup is not fully active
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if HA setup is not fully active
ansible-role-bridge-task : Check the current replication and overall system status
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check the current replication and overall system status
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check current status for PRIMARY
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check current status for PRIMARY
HANA Pri: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12
Start date: 2023-01-06 10:47:37.937440
End date: 2023-01-06 10:47:37.954984
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana:
* hana_ad0_clone_state : PROMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana2
* hana_ad0_roles : 4:P:master1:master:worker:master
* hana_ad0_site : demo-hana
* hana_ad0_srmode : sync
* hana_ad0_sync_state : PRIM
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana
* lpa_ad0_lpt : 1673023600
* master-SAPHana_AD0_00 : 150


ansible-role-bridge-task : Check current status for SECONDARY
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check current status for SECONDARY

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12
Start date: 2023-01-06 10:47:39.945332
End date: 2023-01-06 10:47:39.963650
Task duration: < 5s
Errors:
Standard output:
* Node: demo-hana2:
* hana_ad0_clone_state : DEMOTED
* hana_ad0_op_mode : logreplay
* hana_ad0_remoteHost : demo-hana
* hana_ad0_roles : 4:S:master1:master:worker:master
* hana_ad0_site : demo-hana2
* hana_ad0_srmode : sync
* hana_ad0_sync_state : SFAIL
* hana_ad0_version : 2.00.050.00.1592305219
* hana_ad0_vhost : demo-hana2
* lpa_ad0_lpt : 30
* master-SAPHana_AD0_00 : 0

Common Steps
ansible-role-bridge-task : Show primary output
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Show primary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Show secondary output
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Show secondary output
HANA Pri: debug - Changed: false

HANA Sec: debug - Changed: false

ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if PRIMARY is not PROMOTED
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if SECONDARY is not DEMOTED
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if PRIMARY hana_ad0_sync_state is not PRIM
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if hana_ad0_sync_state is SOK for SECONDARY, and retries if required by a past failover event

HANA Sec: shell - Changed: true
Command details Command: crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | grep SOK
Start date: 2023-01-06 10:47:49.117788
End date: 2023-01-06 10:47:49.134217
Task duration: < 1min
Errors:
Standard output:
* hana_ad0_sync_state : SOK

Common Steps
ansible-role-bridge-task : Get secondary hostname
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get secondary hostname
HANA Pri: set_fact - Changed: false

HANA Sec: set_fact - Changed: false

ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if counter of PRIMARY is bigger than SECONDARY and retries if required by a past failover event
HANA Pri: shell - Changed: true
Command details Command: prim_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); sec_count=$(crm_mon -A1 --output-as text | sed -n '/Node: demo-hana2/,$p' | head -n 12 | tail -1 | sed 's/^.*://'); if (( $prim_count > $sec_count )); then exit 0; else exit 1; fi
Start date: 2023-01-06 10:47:51.460187
End date: 2023-01-06 10:47:51.504979
Task duration: < 1min
Errors:
Standard output:


Common Steps
ansible-role-bridge-task : Check if replication mode is sync
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if replication mode is sync
HANA Pri: include_tasks - Changed: false

HANA Sec: include_tasks - Changed: false



ansible-role-bridge-task : Check replication status
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check replication status
HANA Pri: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 10:47:53.615453
End date: 2023-01-06 10:47:54.730582
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: shell - Changed: true
Command details Command: source /usr/sap/AD0/home/.sapenv-ansible.sh && hdbnsutil -sr_state
Start date: 2023-01-06 10:47:53.603004
End date: 2023-01-06 10:47:54.587482
Task duration: < 5s
Errors:
Standard output:

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

Common Steps
ansible-role-bridge-task : Print current replication status
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Print current replication status
HANA Pri: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: demo-hana

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false
is primary suspended: false

Host Mappings:
~~~~~~~~~~~~~~

demo-hana -> [demo-hana2] demo-hana2
demo-hana -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

HANA Sec: debug - Changed: false
See full message

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: sync
operation mode: logreplay
site id: 2
site name: demo-hana2

is source system: false
is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
is primary suspended: false
is timetravel enabled: false
replay mode: auto
active primary site: 1

primary masters: demo-hana

Host Mappings:
~~~~~~~~~~~~~~

demo-hana2 -> [demo-hana2] demo-hana2
demo-hana2 -> [demo-hana] demo-hana


Site Mappings:
~~~~~~~~~~~~~~
demo-hana (primary/primary)
|---demo-hana2 (sync/logreplay)

Tier of demo-hana: 1
Tier of demo-hana2: 2

Replication mode of demo-hana: primary
Replication mode of demo-hana2: sync

Operation mode of demo-hana: primary
Operation mode of demo-hana2: logreplay

Mapping: demo-hana -> demo-hana2
done.

ansible-role-bridge-task : Check if replication status is SYNC
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if replication status is SYNC
ansible-role-bridge-task : Update replication mode to be SYNC if needed
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Update replication mode to be SYNC if needed
ansible-role-bridge-task : Check if replication status is SYNCMEM
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if replication status is SYNCMEM
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Update replication mode to be SYNCMEM if needed
ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get current SAP ASCS queue length


ASCS: include_tasks - Changed: false


ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get current SAP ASCS queue length
ASCS: include_tasks - Changed: false

ansible-role-bridge-task : Get Enqueue lock number
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get Enqueue lock number
ASCS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh && sapcontrol -nr 00 -function EnqGetStatistic | grep locks_now: | grep -Eo '[0-9]'
Start date: 2023-01-06 10:47:56.899011
End date: 2023-01-06 10:47:56.960485
Task duration: < 5s
Errors:
Standard output:
0

Common Steps
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Save ASCS enqueue count to compare after cut
ASCS: set_fact - Changed: false

ansible-role-bridge-task : Check if PAS is connected to the primary database
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if PAS is connected to the primary database



PAS: include_tasks - Changed: false

ansible-role-bridge-task : Get current SAP ASCS queue length
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Get current SAP ASCS queue length
PAS: include_tasks - Changed: false

ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if no primary DB hostname is found in variables
ansible-role-bridge-task : Make sure no trans log file exists
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Make sure no trans log file exists
PAS: file - Changed: true

ansible-role-bridge-task : Generate R3trans data
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Generate R3trans data
PAS: shell - Changed: true
Command details Command: source /home/ad0adm/.sapenv-ansible.sh; source /home/ad0adm/.dbenv.sh; cd /home/ad0adm; R3trans -d && echo "ignore_errors"
Start date: 2023-01-06 10:48:01.253315
End date: 2023-01-06 10:48:01.520191
Task duration: < 5s
Errors:
Standard output:
This is R3trans version 6.26 (release 777 - 13.08.20 - 20:20:08).
unicode enabled version
R3trans finished (0000).
ignore_errors

ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Check if PAS is connected to database "demo-hana2"
PAS: shell - Changed: true
Command details Command: echo "ignore_errors" && cat /home/ad0adm/trans.log | grep \'demo-hana2\'
Start date: 2023-01-06 10:48:03.563803
End date: 2023-01-06 10:48:03.568063
Task duration: < 5s
Errors:
Standard output:
ignore_errors
See full message
non-zero return code

ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
2023-01-06T16:48:03.914233Z Run POST ACTIONS for scenario CRASH_NODE_PROC_PRE
ansible-role-bridge-task : Fail if PAS is not connected to database "demo-hana2"
PAS: fail - Changed: false
See full message
PAS is not connected to the database it is supposed to be: demo-hana2. Please check and try again