Routing chains
==============
The routing chains enables to configure different parts of the fabric with
different routing engines. The routings would be applied on different parts
of the fabric one after the other (hence the name chains). So if there is
a node in the fabric that is configured in more than one part will be left with
the routing updated by the last routing engine it was part of.

The process of configuring routing chains can consist of the following steps:
1. define port groups.
2. define topology based on port groups.
3. define routing for each topology.

1 Port groups
=============
The idea behind the port groups is the ability to define sub groups of the
fabric and give each group an identifier that can be used to relate to all
nodes in this group, the ports group is a mandatory prerequisite for supporting
the routing chains feature as it will be used in order to define the
participants in each one of the routing algorithms.

In order to define a port group policy file set the parameter
pgrp_policy_file filename in the opensm configuration file.

The port groups policy file details the port groups in the fabric, the general
structure of the policy file shall be one or more paragraphs which defines a
group, each paragraph shall begin with the line port-group and end with the
line end-port-group, for example:

    port-group
    ...port group qualifiers...
    end-port-group

1.1 Port group qualifiers
=========================
Note that all qualifiers must end with colon (:), this is unlike the port group
start and end which does not require colon, also  colon is a predefined mark
which must not be used inside qualifier values, including a colon in the name
or use of a port group will cause the policy to fail.

1.1.1 name
==========
Each group must have a name, without name qualifier for each group the policy
will fail.
For example:

    name: grp1

1.1.2 use
=========
use is an optional qualifier which one can define in order to describe the
usage of this port group (if not defined empty string is used as default).
For example:

    use: first port group

1.2 Rule Qualifier
==================
There are several qualifiers which describes a rule which determine which ports
will be added to the group, for each port group policy must contain exactly one
rule qualifier (without any rule no ports can be chosen and more than one rule
will conflict), therefore each group shall contain only one of the following
rule qualifiers:

1.2.1 guid-list
===============
Comma separated list of guids to include in the group:

    port-guid: 0x283, 0x286, 0x289

If no specific physical ports where configured, all physical ports of each guid
will be chosen, However, for each guid one can detail specific physical ports
of this guid which will be included in the group, this can be done using the
following syntax:

Specify specific port in a guid to be chosen:

    port-guid: 0x283@3

Specify specific list of ports in a guid to be chosen:

    port-guid: 0x286@1/5/7

Specify specific range of ports in a guid to be chosen:

    port-guid: 0x289@2-5

Specify list of specific ports and ports ranges in a guid to be chosen:

    port-guid: 0x289@2-5/7/9-13/18

Complex rule example:

    port-guid: 0x283@5-8/12/14, 0x286, 0x289/6/8/12

1.2.2 port-guid-range
=====================
It is possible to configure a range of guids to be chosen to the group,
however while using the range qualifier it is impossible to details specific
physical ports.
For example:

    port-guid-range: 0x283-0x289

Note: it is impossible to specify a list of ranges, the below example is
invalid and will cause the policy to fail:

    port-guid-range: 0x283-0x289, 0x290-0x295

1.2.3 port-name
===============
It is possible to configure a list of hostnames as a rule, hosts with node
description which is built out of this hostnames will be chosen, since the node
description contain also the network card index it is also possible to specify
a specific network card index and in addition it is possible to specify
specific physical port that will be chosen, for example:

The below configuration will cause only physical port 2 of a host with node
description kuku HCA-1 to be chosen

    port-name: hostname=kuku; port=2; hca_idx=1

- Port and hca_idx parameters are optional, if port is not specified all
  physical ports will be chosen, if hca_idx is not specified all card numbers
  will be chosen, hostname is mandatory.

- One can configure a list of hostname/port/hca_idx sets in the same qualifier
  as follow:

    port-name: hostname=kuku; port=2; hca_idx=1 , hostname=host1; port=3, hostname=host2

Notice: port-name qualifier is relevant only for HCAs and not for switches.

1.2.4 port-regexp
=================
It is possible to define a regular expression which only nodes that their node
description match this regular expression will be chosen to the group:

    port-regexp: SW*

It is also possible to specify a specific physical port that will be chosen for
nodes who match (but this must be one specific port, there is no option to
define a list or a range of ports), the below example will cause only physical
port 3 of nodes who match to be added to the group:

    port-regexp: SW*:3

1.2.5 union-rule
================
It is possible to define a rule which unite two other port groups, which means
all ports from both groups will be included in the united group:

    union-rule: grp1, grp2

1.2.6 subtract-rule
===================
It is possible to define a rule which subtract one port group from another, the
below rule for example will cause all the ports which are part of grp1 but not
included in grp2 to be chosen:

    subtract-rule: grp1, grp2

In subtract (unlike union) the order does matter since the second group will be
subtracted from the first one.

In union and subtract rules it is possible to unite/subtract only two groups,
however it is possible to unite/subtract groups which are union or subtraction
themselves, as shown in the port groups policy file example (at end of the
appendix).

1.3 Predefined port groups
==========================
There are 3 predefined port groups which are automatically created and
available for use and cant be defined in the policy file (if a group in the
policy is configured with the name of one of those predefined groups the
policy will fail):

    ALL  A group which include all nodes in the fabric
    ALL_SWITCHES  A group which include all switches in the fabric.
    ALL_CAS  A group which include all HCAs in the fabric.

1.4 Port groups policy - example file
=====================================
    port-group
        name: grp3
        use: Subtract of groups grp1 and grp2
        subtract-rule:  grp1, grp2
    end-port-group

    port-group
        name: grp1
        port-guid: 0x281, 0x282, 0x283
    end-port-group

    port-group
        name: grp2
        port-guid-range: 0x282-0x286
    end-port-group

    port-group
        name: grp4
        port-name: hostname=kika port=1 hca_idx=1
    end-port-group

    port-group
        name: grp3
        union-rule: grp3, grp4
    end-port-group

2 Topologies policy configuration
=================================
This section describes the syntax to be used in order to configure topologies
policy.

In order to define a port group policy file set the parameter
topo_policy_file filename in the opensm configuration file.

2.1 Configuring a policy
========================
The topologies policy file details a list of topologies, the general structure
of the policy file shall be one or more paragraphs which defines a topology,
each paragraph shall begin with the line topology and end with the line
end-topology, for example:

    topology
    ...topology qualifiers...
    end-topology

2.2 Topology qualifiers
=======================
Note  All qualifiers must end with colon (:), this is unlike the topology
and end-topology which does not require colon, also  colon is a predefined
mark which must not be used inside qualifier values, including a colon in the
qualifiers value will cause the policy to fail.

Absence of any mandatory qualifier listed in this section, will cause policy
file parser to fail.

2.2.1 id
========
Mandatory qualifier.
For example:

    id: 1

- Legal Values  Any positive value.
- Must be unique.

2.2.2 sw-grp
============
Name of port group which shall include all switches to be used in this
topology.
Mandatory qualifier.
For example:

    sw-grp: ys_switches

2.2.3 hca-grp
=============
Name of port group which shall include all hcas to be used in this topology.
Optional qualifier when auto_join_hca_wo_topo is enabled, Otherwise mandatory.

When hca-grp is not specified, and auto_join_hca_wo_topo is enabled, all CAs
that are not set to any other topology and connected to the switches specified
by sw-grp qualifier, are added to this topology.

For example:

    hca-grp: ys_hosts

3 Routing Chain Policy Configuration
====================================
This section describes the syntax to be used in order to configure routing
chain policy.
Defining a routing chain policy file
In order to define a port group policy file set the parameter
rch_policy_file filename in the opensm configuration file

3.1 Configuring a policy
========================
The routing chains policy file details the routing engines (and their fallback
engines) to be used in order to route the fabric, the general structure of
the policy file shall be one or more paragraphs which defines an engine (or a
fallback engine), each paragraph shall begin with the line unicast-step and
end with the line end-unicast-step, for example:

    unicast-step
    ...routing engine qualifiers...
    end-unicast-step

3.2 Routing engine qualifiers
=============================
Note  All qualifiers must end with colon (:), this is unlike the unicast-step
and end-unicast-step which does not require colon, also  colon is a predefined
mark which must not be used inside qualifier values, including a colon in the
qualifiers value will cause the policy to fail.

3.2.1 id
========
id is mandatory, without id qualifier for each engine the policy will fail.
For example:

    id: 1

- Legal Values  size_t value (0 is not legal).
- The engines in the policy chain will be set according to ascending id order
  so it is highly crucial to verify that the id that is given to the engines
  match the order in which you would like the engines to be set.

3.2.2 engine
============
This is a Mandatory qualifier which describes the routing algorithm to be used
within this unicast step.
For example:

    engine: minhop

Currently on first phase of routing chains legal values are minhop/ftree/updn.

3.2.3 use
=========
This is an optional qualifier which enables to comments as for the usage of
this unicast step, if not defined empty string will be used as default.
For example:

    use: ftree routing for for yellow stone nodes

3.2.4 config
============
This is an optional qualifier which enables to define separate opensm config
file for a specific unicast step, if not defined then the config file which the
opensm was run with will take effect.
For example:

    config: /etc/config/opensm2.cfg

3.2.5 topology
==============
Define the topology that this engine uses.

- Legal value  id of an exist topology which is defined in topologies policy
  (or zero which represents the entire fabric and not a specific topology).
- Default value  If not specified, a routing engine will relate to the entire
  fabric (as if topology zero was defined).
- Notice: The first routing engine (the engine with the lowest id) MUST be
  configured with topology: 0 (entire fabric) or else the routing chain parser
  will fail.

For example:

    topology: 1

3.2.6 fallback-to
=================
This is an optional qualifier which enables to define the current unicast step
as fallback to another unicast step by defining the id of the unicast step
which this step is fallback to.

- If not defined then the current unicast step is not a fallback
- If the value of this qualifier is a non-exist engine id then this step will
  be ignored.
- Fallback step has no meaning as long as the step it is fallback to did not
  fail.
- It is not possible to define fallback to a fallback step (possible to define
  but it will be ignored)

3.2.7 path-bit
==============
This qualifier is deprecated. If used, min-path-bit and max-path-bit
are set to match path-bit value automatically.

3.2.8 min-path-bit,max-path-bit
==============
These are an optional qualifiers which allow range of LMC path-bit LIDs to be used
by the current unicast step, the LID is described by offset from the base LID.
For ports which lmc is greater than 0,
only the LID in range between base LID + min-path-bit
and base LID + max-path_bit will be relevant for this unicast step.
For example:

    min-path-bit: 1
    max-path-bit: 3

- If min-path-bit/max-path-bit are  not defined,
min-path-bit will be set to 0 and max-path-bit will be set to 2^LMC - 1
- Routing chain will fail in the following cases:
	- min-path-bit is greater than max-path-bit
	- the range min-path-bit:max-path-bit is not fitting fabric lmc lid range
	(e.g: lmc=1 and min-path-bit=2) then the policy will fail.

3.3 Routing Compulsions
=======================
The first unicast step of any routing chain must include all switches and HCAs
(topology id must be 0). min-path-bit of the first unicast step must be 0 (default)
if mentioned, max-path-bit must be either 0 or 0xFF (default).

3.4 Routing Chain Policy - example File
=======================================
    unicast-step
        topology: 0
        engine: minhop
        id: 1
    end-unicast-step

    unicast-step
        topology: 1
        engine: ftree
        config: /etc/opensm/opensm_ftree.cfg
        min-path-bit: 1
	max-path-bit: 1
        id: 2
    end-unicast-step

    unicast-step
        topology: 2
        engine: updn
        config: /etc/opensm/opensm_updn.cfg
        min-path-bit: 2
	max-path-bit: 3
        id: 3
        fallback-to: 2
    end-unicast-step
