Health Scenarios

What Is a Health Scenario?

Health Scenario is the rule that reduces the result of one or more Health Checks to a single logical decision with AND / OR groups and, when necessary, with UNLESS (reversing). The decision of the scenario is read in the Disaster Recovery (DR) condition of the DNS records and in the conditions of the record answers; the scenario that reports the internet access of the local data center is also selected from here. In this way, composite conditions such as "both the Ping and the HTTP check failed" or "there is no WAN access, UNLESS the maintenance mode" are defined in a single scenario and are re-evaluated as the check results change.

The decision of the scenario occurs in two phases: the result of the condition in the Health Check field is the basic status of the scenario; the passing from this status to the final decision is managed with the Activation and Deactivation settings. When Automatic is selected, the final decision follows the basic status; when With Trigger is selected, the Trigger Actions are run at the moment of the transition and the final decision is held until a separate Activation Check / Deactivation Check condition is met; when Off is selected in the deactivation, the scenario does not return to passive by itself after it becomes active once.

AND / OR

Condition builder

The health checks are combined in groups; they are reversed with UNLESS

Three-valued

Decision logic

OK is true, Error is false; the other statuses are considered unknown and the final decision is preserved

2 gates

Activation / Deactivation

Automatic, With trigger or (in the deactivation) Off

DR

The decision consumers

The DR condition and the answer conditions of the DNS records, the internet scenario of the local data center

Always / Never

Automatic scenarios

Two ready scenarios that are always active and always passive

Cluster

Independent evaluation

Every device gives its own decision; only the primary device runs the trigger actions

How does it work?

1. When one of the health checks in the Health Check condition changes status, the scenario is re-evaluated; the result of the condition becomes the basic status of the scenario (met / not met / unknown). A selected check meets the condition while it is in the OK status; for the "the check failed" condition, the UNLESS form of the check is selected.
2. When the basic status changes, the Trigger Actions records of the direction in which With Trigger is selected are run in order.
3. The final decision is determined according to the gates: the Automatic gate follows the basic status; the With Trigger gate preserves the previous decision until the related check condition is met; the Off deactivation gate never opens.
4. When the status of the scenario changes, the configuration of the DNS records bound to the scenario is refreshed and the transition is written into the GTM record. If there is an unknown input, the decision is not changed and the final decision is preserved.

The Automatic Health Scenario Records

The second table on the list screen, Automatic Health Scenarios, shows the two scenarios that are produced by the system and cannot be edited:

Scenario	Description
Always	It is always considered active. When it is assigned to the DR condition of a DNS record, the record works in the DR status continuously.
Never	It is always considered passive. The condition to which it is assigned is never met.

The decision of these two scenarios is fixed and never changes. Both of them are listed, together with the user scenarios, in the selectors of the DR condition of the DNS record and of the internet scenario of the local data center; in the scenario selector of the record answer conditions only the user scenarios take place. The Always and Never automatic Health Check records that carry the same names are separate concepts: one is a ready scenario and the other is a ready check that can be selected in the condition builder.

The Status Values

The status circle in the list, the State column and the State row in the preview use the values below. The main text shows the final decision; the addition in parentheses states the intermediate status of the gates.

Status text	Color	Meaning
Activation action	green	The scenario is active: the final decision is met.
Deactivation action	gray	The scenario is passive: the final decision is not met.
Deactivation action (Awaiting activation check)	blue	The condition is met, the With Trigger activation is waiting for the Activation Check condition; the scenario is still passive.
Activation action (Awaiting deactivation check)	blue	The condition is broken, the With Trigger deactivation is waiting for the Deactivation Check condition; the scenario is still active.
… (Activation/deactivation collision)	gray	The activation and deactivation gates resolved to the same result; the final decision is held at its previous value. In the Off deactivation, this status is permanent when the condition is broken.
… (no health data, last state kept)	gray	A check in the condition does not produce a definite result (Preparing / No data); the final decision is preserved.

In a cluster the list circle is divided into two halves and every half shows the decision of one device; the State column writes only the decision of its own device, and the decision of the other device is seen on the preview panel.

Operational Best Practices

• More than one health check must be combined. A scenario bound to a single check is affected by the temporary network fluctuations; when the UNLESS form of more than one check is selected together in an AND group and the "all of them failed" condition is established, a more reliable decision is produced.
• The maintenance mode must be excluded with UNLESS. The is under maintenance automatic check of the data center is added to the condition in the UNLESS form and the planned maintenance starting a DR transition is prevented.
• A verification condition must be defined with With trigger in the DR transitions. In the transitions in which the external systems must be ready, With Trigger is selected; the Activation Check condition is bound to a check that shows that the transition is completed. The success of the trigger action does not activate the scenario on its own.
• Off must be used only in the one-way transitions. The Off deactivation keeps the scenario active even if the condition improves after the scenario becomes active; for the return to normal, the scenario must be edited.
• The unknown statuses must be taken into account. A newly added (Preparing) or non-result-producing (No data) check leaves the condition unknown and the scenario stays at its previous decision; it must be verified that the checks added to the condition produce a result.
• The references must be reviewed before deleting. The DNS records and the local data center setting that refer to the deleted scenario are not updated by themselves; as long as the reference remains, the last decision of the scenario before it was deleted continues to be used.
• The status transitions must be monitored from the records. The scenario transitions, the gate changes and the trigger action results are listed on the GTM Logs screen and in the Log window of the scenario; the collision and no health data additions must be noticed early.

How to Add?

A new Health Scenarios scenario is added through the TR7 web management interface with the steps below:

1. The Configuration Menu is opened by clicking the TR7 logo at the bottom left.
2. The GTM → Health Scenarios → Add steps are followed.
3. A unique and descriptive name is entered for the scenario.
4. The Health Check condition is established; the Activation and Deactivation settings are made.
5. The scenario is added by clicking the Add button.

TIP

Reaching the page via quick search

Instead of navigating the menu step by step, the related page is reached directly by using the search bar in the bottom left corner of the screen. The search bar opens with the Alt + T shortcut on Windows and Opt + T on macOS; typing the name of the feature being searched for is then sufficient.

The created scenario can be used in one or more DNS records and in the data center setting.

Health Scenarios Listing Screen

By following the GTM → Health Scenarios path, the list of all the health scenarios added on the TR7 device is reached. Scenario adding, editing and deleting operations are performed through this screen.

Health Scenarios Listing Form

The table opens with the following columns by default:

Column	Description
Status	The final decision of the scenario is shown with a colored circle; when it is hovered over, the status text appears (see The Status Values). In a cluster the circle is divided into two halves; every half shows the decision of one device.
Scenario Name	The name of the scenario.
State	The status text (for its own device).
Trigger Settings	It shows the Present or Absent value. Since the setting that feeds this column is not present in the interface, the cell displays Absent in every row.
Log	The Log text and the file icon; the icon opens the record window of the scenario ("<name>" health scenarios logs). The window lists the scenario transitions, the status changes of the checks in the condition and the trigger action events.

There is no column hidden by default; these five columns are in the column selector. The table is not sorted by default; when a column header is clicked, descending sorting is applied first, then ascending, and on the third click the sorting is removed. The list updates by itself as the check results arrive.

The items in the toolbar are arranged from left to right in the following order; the last two items (Page Information and Page Size) are in the information strip under the table. There is no filter card and no Usage Filter on this screen.

Columns

Which columns are shown in the table is selected; all five columns are on by default. In the same panel, columns are reordered by dragging and alignment is selected per column; the row height is fitted to the page with the Fit rows to the page button, and the column layout is reset to default with the Reset button. When the width badge next to a manually widened column is clicked, the column returns to the automatic width. The layout is stored per user and is common to the two tables.

Export

The table is exported as Excel, CSV or PDF by clicking the Export button. In the window that opens, the Filename, Format and Data Source, together with the Columns to be exported, are determined; the remaining fields are displayed according to the format: CSV Separator for CSV, Excel Sheet Name for Excel, and Page Orientation and Page Size for PDF.

Add

By clicking the Add button, the form for creating a new scenario opens in the Add tab.

Edit

By clicking the Edit button, the editing form of the selected scenario opens in the Edit tab. The button becomes enabled only when a single scenario is selected. The editing form is the same as the add form; all the fields including the name can be changed. The change is saved without asking for confirmation and the scenario is re-evaluated. The automatic scenarios cannot be edited.

Delete

When one or more scenarios are selected and the Delete button is clicked, the related scenarios are deleted. Before deleting, the "Are you sure?" confirmation is asked. The button is passive while there is no selection.

WARNING

A scenario that is in use can also be deleted and the confirmation window does not show where the scenario is used; before deleting, the Used in tab on the preview panel must be checked.

The scenario and its decision are removed; the DNS records (the DR condition, the answer conditions) and the internet scenario setting of the local data center that refer to the scenario are not updated by themselves, and the remaining references must be corrected manually. The Always and Never scenarios cannot be deleted.

Search

Used to search all information in the table.

RegExp Search

Used to search all information in the table with a regular expression (RegExp).

Column Based Search

Used to perform a column-based search on the information in the table; a separate search line opens for every column and next to every box there are the Negate button that reverses the result and its own RegExp switch.

Page Information

Contains information about the total number of defined scenarios and the number of scenarios displayed in the list: Showing N-M of T records and the number of selected scenarios (N selected, or Nothing selected when there is no selection).

Page Size

The maximum number of scenarios to be displayed on the page is selected. The default is 20 (100 in the automatic scenarios table); the options are 5, 10, 20, 50, 100, 200, 500 and All. With the arrow icons next to it, the pages are switched between.

There are two tables on the screen

The upper table lists the scenarios that the user defines; adding, editing and deleting are done from the toolbar of this table. The lower Automatic Health Scenarios table shows the Always and Never scenarios (see The Automatic Health Scenario Records); its title is written at the left of the toolbar and in this table the selection box and the Add, Edit, Delete buttons are not present, only the column settings, the export and the search take place. When a row is clicked, the preview opens; the Edit operation is not present in the preview title.

Right-Click Menu

When a row is right-clicked, a menu with the name of the scenario in its title opens. Both operations in the menu apply only to the right-clicked scenario; a selection made in the list does not affect the menu.

Operation	What it does
Edit	Opens the editing form of the scenario in the Edit tab; it is the same as the form opened by the Edit button in the toolbar.
Delete	Deletes the right-clicked scenario; the same confirmation question as the deletion in the toolbar is displayed.

The menu also opens in the automatic scenarios table; however, Always and Never cannot be edited or deleted, and on these rows both operations return an error.

Preview of the Selected Scenario

When a row in the list is clicked, the preview of the scenario opens on the right of the screen; when the selected row is clicked again, the panel closes. In the title there are a status circle, the name of the scenario and the Health Scenario subtitle, and at the top right there are two operations.

Preview of the Selected Scenario

Operation	What it does
Edit	Opens the editing form of the scenario in the Edit tab; this operation is not present on the automatic scenarios.
Log	Opens all the GTM event records and the user operations that belong to the scenario in a window, without leaving the page. The window that the icon in the Log column opens, on the other hand, is limited to the scenario transitions, the check status changes and the trigger action events.

The panel has three tabs; the last one is displayed only if there is a definition that uses the scenario:

Details

Lists the decision, the condition and the activation / deactivation settings of the scenario (the table below).

Log

Lists the status transitions of the scenario and the user operations performed on it as a timeline.

Used in

Lists the definitions that select the scenario (the DR condition and the answer conditions of the DNS record, the internet scenario of the local data center); the badge next to the tab name shows the number of usages. When a row is clicked, it is passed to the related definition.

The rows in the Details tab are as follows, in order. In a cluster the State row is divided into two columns only when the status text of the two devices is different; the Latest Actions rows are shown separately for every device and the other rows in a single column.

Row	Description	Visibility condition
State	The status text and the file icon that opens the record window of the scenario.	Always
Latest Actions	The last record of the last final decision change and its date.	Always
Health Check	The condition is shown in a read-only condition builder; the background is painted with the color of the basic status for every device.	If the condition is defined; it is not displayed on the automatic scenarios
Activation	The selected activation setting (icon + Automatic / With Trigger) and the description box.	On the user scenarios
Activation Trigger →	The assigned trigger actions; the pencil icon next to every action opens the form of the action.	Activation = With trigger
Activation Check	The activation verification condition, in a read-only builder; the background is painted with the color of the gate status.	Activation = With trigger
Latest Actions	The last change of the activation gate or the last activation trigger action event.	Activation = With trigger
Deactivation	The selected deactivation setting (Automatic / With Trigger / Off) and the description box.	On the user scenarios
Deactivation Trigger ←	The assigned trigger actions.	Deactivation = With trigger
Deactivation Check	The deactivation verification condition.	Deactivation = With trigger
Latest Actions	The last change of the deactivation gate or the last deactivation trigger action event.	Deactivation = With trigger

Scenario Add Form

A new health scenario is added on the TR7 device by following the GTM → Health Scenarios → Add steps.

Scenario Add Form

INFO

Scenario Name Required

The name of the scenario is entered. The name identifies the scenario in the list, in the scenario selectors and in the GTM records. This screen does not prevent the same name from being used in a second scenario; giving a distinctive name is recommended.

INFO

Health Check

The activation condition of the scenario: which health checks will be combined with which logic is defined in the condition builder. The field is not marked with an asterisk, but the record cannot be made without the condition being established: in an empty builder the Empty condition error is displayed and the Add button stays passive. When the condition is met, the basic status of the scenario becomes met, and when it is broken, not met.

The condition builder:

• Every line is a group. One or more health checks are added to the selector (Select HC) in the line; while the group is empty, No HC selected is written.
• When there is more than one check in the group, the and / or switch of the group appears at the right end of the selector, to the left of the arrow icon (the default is and): with and all the members and with or any one of them.
• The plus icon at the right of the line adds a new group (it is visible only in the first line; at most 10 groups); the trash can icon deletes the group (it is not visible while there is a single group). Between the groups there is a second and / or switch (the default is or in the new groups).
• In the selector list, first the Always and Never ready checks (only in the plain form) and then every user and automatic check take place in two forms: with its name and in the bold UNLESS <name> form (reversed). The plain form is met while the check is OK; the UNLESS form is met while the check is Error. The Add button of the selector lets a new Health Check be defined without leaving the page and adds the saved check to this group.
• Every selected check appears as a label that carries a live status circle; the eye icon on the label opens the quick preview of the check, the pencil icon (on the checks that are not automatic) opens the form of the check and the cross icon removes the check from the group.
• The background of the builder is painted separately for every device according to the instant status of the selected checks and shows whether the condition is met at that moment or not. This preview is an approximate calculation; for the definite decision, the Health Check row and the State value on the preview panel are taken as the basis.

For the evaluation rule (consecutive and groups are considered a single block, or starts a new block; three-valued logic), the Evaluation Mechanism section can be looked at.

An example

Group 1: UNLESS Ping DC-A and UNLESS HTTP DC-A · or · Group 2: UNLESS Access to DC-A is successful → the condition is met if the two checks on DC-A failed together or if there is no access to DC-A. In order to exclude the maintenance window, UNLESS DC-A is under maintenance is added to every group.

INFO

Activation Default: Automatic

How the scenario will become active when the condition is met is selected; under the option buttons there is a description box that changes according to the selection.

Option	Behavior
Automatic	When the condition is met, the scenario becomes active by itself. In the description box it says When the activation check turns ●, the activation action is done automatically.
With Trigger	When the condition is met, the actions in the Activation Trigger → list are run in order; the scenario waits in the Awaiting activation check status until the Activation Check condition is met and becomes active then.

Activation - With Trigger

The fields that become visible when With Trigger is selected:

Activation Trigger →

The Trigger Actions records to be run are determined with a multiple selection; while the list is empty, Nothing chosen is written in the selector. The Add button of the selector lets a new action be defined in a window and adds it to the list. The actions run in the order in the list; a failed action stops the remaining ones.

Activation Check

The condition that verifies that the activation has taken place is defined in the same builder as Health Check. The scenario does not become active until this condition is met; the condition is evaluated with the health check results independently of the result of the trigger actions.

In a setting change

When it is returned from With Trigger to another option and saved, the trigger action list and the verification condition that belong to this direction are deleted from the scenario; the same rule is valid for Automatic and Off on the deactivation side as well.

INFO

Deactivation Default: Automatic

How the scenario will return to passive when the condition is broken is selected.

Option	Behavior
Automatic	When the condition is broken, the scenario becomes passive by itself. In the description box it says When the activation check turns ●, the deactivation action is automatically done.
With Trigger	When the condition is broken, the actions in the Deactivation Trigger ← list are run in order; the scenario stays active in the Awaiting deactivation check status until the Deactivation Check condition is met. In the description box the When the activation check turns ●, the deactivation trigger ← is executed, and When the deactivation check turns ●, the deactivation action is done lines take place.
Off	The deactivation gate never opens: after the scenario becomes active once, it stays active even if the condition is broken (with the Activation/deactivation collision status the final decision is preserved). For the return to normal, the scenario is edited. In the description box it says The deactivation action is never done independently of the activation check.

Deactivation - With Trigger

The fields that become visible when With Trigger is selected:

Deactivation Trigger ←

The Trigger Actions records to be run; they are selected in the same form as Activation Trigger → and run in the same order.

Deactivation Check

The condition that verifies that the deactivation has taken place; the scenario stays active until it is met.

Off

A scenario in which Off is selected does not return to passive without human intervention. This option must be used only in the scenarios that require a permanent, one-way transition.

INFO

Add

The scenario is added by clicking the Add button; while there is an erroneous field in the form, the button is passive. The adding does not ask for confirmation; the record returns immediately, the Add tab closes and the scenario comes into the list with its status with its first evaluation. The operation is written into the user operation record ("<name>" health scenario is created).

Operating Mechanism

INFO

• The three-valued logic. The OK status of a check is considered true and the Error status false; every status other than these two (Preparing, No data, etc.) and an undefined check identifier are unknown. If a check reversed with UNLESS is unknown, the reversed result is unknown as well. In an AND group one definite false and in an OR group one definite true determines the result; otherwise a single unknown member makes the group unknown.
• The groups. Every line is a group; the members of the group are combined with the and / or switch in the line and the groups with the and / or switch between them. The groups connected with consecutive and are taken into a single block; or starts a new block. The result is the OR of the blocks. An empty group is considered false.
• The basic status and the gates. The result of the condition is the basic status. If the Activation gate is Automatic, it follows the basic status, and if it is With Trigger, it takes the result of the Activation Check condition. If the Deactivation gate is Automatic, it follows the reverse of the basic status, if it is With Trigger, it takes the Deactivation Check condition, and if it is Off, it never opens.
• The final decision. If the activation gate is open and the deactivation gate is closed, the scenario becomes active, and the reverse makes it passive. If the two gates resolve to the same result (collision) or if the basic status or one of the gates is unknown, the final decision is preserved at its previous value. If there is no history, in an unknown status it is considered passive, and in a collision the instant result of the condition is taken as the basis.
• The trigger actions. They run only when the basic status changes: when it is met, the Activation Trigger → list, and when it is broken, the Deactivation Trigger ← list. The success or the failure of the actions does not affect the final decision directly; the way that verifies that the transition has taken place is the related check condition. While the scenario does not have a recorded previous evaluation (when the scenario is first added), no action is run. When the same scenario is triggered again, the actions that are in progress are stopped first.
• The cluster. Every device evaluates the scenario independently with its own check results and keeps its own decision; the decisions can be different between the devices. Only the primary device runs the trigger actions; the other device takes over if its peer remains inaccessible for longer than one hour. The decisions are stored on the device and are restored at a restart.
• The record. The basic status, gate and final decision changes are written into the GTM record: "<name>" scenario is activated / is deactivated (the final decision), "<name>" activation check succeeded / failed (separate lines for the basic condition and the activation gate), "<name>" deactivation check succeeded / failed, "<name>" has no health data — the last decision is kept. The basic condition line is written in every mode; the Automatic gates do not write their own lines.

Scope and Limits

INFO

• The screen requires a GTM license and full GTM authorization. The GTM tab is visible with the display authorization; this screen opens only on the users who have full authorization in the GTM area, and if there is no GTM module in the license, the Not covered by your license card is displayed.
• The builder has two levels. There are groups and group members; there is no nested group. The reversing (UNLESS) is per check, not per group.
• There is no filter. In this list there is no filter card according to the status or another field; the search box is used.
• The name uniqueness is not verified in the web interface. A second scenario that carries the same name can be saved from this screen.
• The automatic scenarios cannot be managed. Always and Never are only monitored and selected.
• The deletion does not clean the dependencies. The records that refer to the deleted scenario continue to use the last decision; they must be updated manually.
• The logs. The scenario transitions, the gate changes, the status changes of the checks in the condition and the trigger action events are monitored in the window in the Log column, on the preview panel and on the GTM Logs screen.

Usage Location

The scenarios are listed by name on the screens that use their decisions; Always and Never also take place in the same lists together with the user scenarios.

The place where it is used	What it does
The DR condition of the DNS record on the Domains screen	While the scenario is active, the record passes to the disaster recovery (DR) answer.
The answer conditions of the DNS record	Under which condition the answer will be given is bound to the decision of the scenario.
The internet scenario of the local data center on the Service Management screen	Whether the internet access of the data center will be considered healthy or not is determined with the decision of the scenario.

In which definitions a scenario is used is seen from the Used in tab on the preview panel and must be reviewed before it is deleted; the deletion does not clean the remaining references.