                    Router Policy in OpenSM

==============================================================================
 Table of contents
==============================================================================

1. Overview
2. Router Policy
3. Load Share policy
4. FLID Policy
5. Router, Load Share and FLID Policy File Syntax Guidelines
6. Examples of Router Policy or FLID feature enable
7. Examples of Router and Load Share Policy
8. Examples of FLID Policy File

==============================================================================
 1. Overview
==============================================================================

IB subnet routing feature enables OpenSM to route packets based on the learned
remote IB subnet from the IB routers or router policy configured in the
routing policy file. IB routing policy feature enables OpenSM to apply the
static routing policies to alter the behavior dynamic IB routing. Subnet
routing is enabled using two configuration parameters defined in the opensm.conf
file.
	1) router_policy_enabled
		- By default router policy feature is disabled.
		- To enable router policy set parameter to TRUE.
		- Defined as "router_policy_enabled TRUE".
	2) router_policy_file
		- provides path to router policy file.
		- Definded as "router_policy_file <router policy file path>".

First router_policy_enabled should set to TRUE to enable routing feature
(dynamic routing and policy routing). If policy file is defined then OpenSM loads
router policy file. Any changes to the router policy file needs OpenSM restart.

With this feature Opensm supports dynamic and/or static routing. In dynamic
routing case next hop IB router is provided by the subnet routers in the subnet
to the Opensm using Router_info MAD message. In static routing case routes
are configures through the router policy file. When dynamic and static routes
exits both route tables are looked up and best destination router selected.
OpenSM resolves the destination IB router for the remote subnet prefix and applies
the routing policy in response to the Path Request from the nodes in the subnet.

Load share policy feature enables session based load sharing among available paths
between IB subnets. Load share policy is defined in the routing policy file and
configures the load share percentage to each path that has reachability to a
remote subnet. In case of any failure in exiting path configured load is
redistributed to configured remaining paths. Load share policy should be
configured on both IB subnets.

Alternatively, the Floating LID feature may be enabled. The Floating LID (FLID) is a LID used
for routing from a local subnet to leaf switches on a remote subnet.
In this case router_policy_enabled must also be set to TRUE and a router_policy_file must
be provided.
If valid FLID configurations were provided in the router_policy_file, regardless
of whether the file contains other router policy rules or load share rules, then the
above routing feature will be overridden and FLIDs will take effect.

The following section describes router_policy_file route policy, load share
policy and FLID policy definition details.

==============================================================================
 2. Router Policy
==============================================================================

Router policy is defined in the router policy file and has the following route rule
elements and syntaxes:
        - Static routing rule - statically configured prefix (s), router guid and weight.
                - name - name of the route rule static/dynamic (name dynamic is for feature use)
                - subnet-prefix - single or range of remote prefixes.
                - router-guid - router port guid to reach the remote prefixes.
                - Weight - highest route weight wins.
        - Static routing rule - Default route per IB subnet
                - subnet-prefix - Default (*)
                - guid - router port guid

Router policy syntax:
    router-rules - router policy section start
        router-rules - route rule start
            name:           <rule name "static" or "default">
            subnet-prefix:  <subnet-prefix, and/or range of subnet-prefixes>
            router-guid:    <port-guid of the router>
            weight:         <route weight value 0 to 10>
        end-router-rule - router rule end
    end-router-rules - router policy section end

If router_policy_enabled set then routing is resolved based on route priority
(weight) were route with highest priority wins. If router_policy_enabled set
and router_policy_file defined routes are resolved as defined below.
        - if dynamic route exist and no static route policy, then
	        dynamic route with highest priority selected
        - if dynamic route exist and static route policy configured then
                route with highest priority selected. It can be static or dynamic route
                if dynamic and static routes tied then static route gets the priority.
        - if no dynamic route exist and static route policy configured
                static route with highest priority selected.
                no prefix match and default rule configured then default route selected
Note:
    1) Dynamic policy name ("dynamic") defined for the future use. Currently
        there is no dynamic routing policy definition through policy file. Dynamic
        routing is expected to work with out any routing policy definition file.
    2) subnet-prefix should be site-local address
    3) Guid should be port guid of the router.

==============================================================================
 3. Load Share Policy
==============================================================================
Load share policy also defined in the router policy file and has following
rule elements and syntaxes:

    load-share-rules - load share policy start
        load-share-rule - load share rule start
            subnet-prefix:  <subnet prefix of the remote IB subnet>
            router-guid:    <router port-guid> - <percentage>
        end-load-share-rule - load share rule end
    end-load-share-rules - load share policy end

Load sharing enables to configure multiple user case scenarios:

Use case 1: Single path load share in multi path configuration. In this use case
            configuration has three paths (A, B, C) and path A configured to
            load share 40% load and remaining 60% load is not configured on
            either path B or path C. In this scenario load-share algorithm loads
            path A 40% and 60% to path B or path C depending on the best path
            to the remote IB subnet. If path C has best route then remaining 60%
            will be shared on path C. In case of path C failure 60% load will
            be redistributed to path B, since after path C failure path B is
            the best path to the subnet.

Use case 2: Two path load share. In this use case each path assigned specific
            percentage of load. Path A with 70% and path B with 30% then algorithm
	    distributes the load exactly configured. In this case even path C with
	    best path to the remote subnet exist no load is distributed to path C.

Use Case 3: Three path load share. In this use case each path assigned specific
            percentage of load. Path A with 40%, path B with 30% and path C 30%
            The load share algorithm distributes the load exactly configured.
            Also algorithm supports load share up to 10 different paths.

Note:
    1) Dynamic routing feature should be supported in the IB routers
    2) subnet-prefix should be site-local address
    3) Guid should be port guid of the router.
    4) Total percentage of each load-rule should be less than equal to 100.

==============================================================================
 4. FLID Policy
==============================================================================
The FLID policy is also defined in the router policy file and has the
following rules:

    flid-start - FLID policy start
	global: <global FLID range start> - <global FLID range end>
	<subnet prefix>: <local FLID range start> - <local FLID range end>
    flid-end - FLID policy end

The ranges specified are inclusive. For example in range 1 - 2 both 1 and 2
are included.
Ranges should not be empty (left value must be smaller or equal to right value).

Global range guidelines:
	-Range values should not be 0.
	-The values should be within the unicast LID range currently defined as 1 - 49151.
	-The number of elements in the range should not exceed total allowed FLIDs
	 currently defined as 1792.
	-Total should be divisible by 256.

Local range guidelines:
	-Amount of local ranges must not exceed maximum supported subnets which
	 is currently defined as 8.
	-Range of 0 - 0 signifies subnet is not participating in FLIDs.
	-Every local range must be a subset of the global range.
	-Local range must not overlap with any other local ranges.

Any error in the above guidelines will result in the FLIDs routing feature being disabled.

==============================================================================
 5. Router, Load Share and FLID Policy File Syntax Guidelines
==============================================================================

- Empty lines are ignored.
- Leading and trailing blanks, as well as empty lines, are ignored, so
  the indentation in the example is just for better readability.
- Comments are started with the pound sign (#) and terminated by EOL.
- Any keyword should be the first non-blank in the line, unless it's a
  comment.
- Keywords that denote section/subsection start have matching closing
  keywords.

==============================================================================
 6. Examples of Router Policy or FLID Feature Enable
==============================================================================
Here's a Router Policy feature enabling configuration in opensm.conf file
# Router Policy Options
router_policy_enabled TRUE
router_policy_file <router policy file path>

==============================================================================
 7. Examples of Router and Load Share Policy File
==============================================================================
Here's an example of the Router and Load Share policy file:

#  IB Subnet routing file

# Router rule policy section
# Router policy with single subnet prefix
router-rules
    router-rule
	    name: static
	    subnet-prefix: 0xfec000000000001
	    router-guid: 0x0145162300c10b01
	    weight: 1  // (1-10) highest weight wins
    end-router-rule
end-router-rules

# Route policy with range of subnet prefixes
router-rules
    router-rule
	    name: static
	    subnet-prefix: 0xfec0000000000011- 0xfec0000000000024
	    router-guid: 0x0145162300c10b01
	    weight: 1  // (1-10) highest weight wins
    end-router-rule
end-router-rules

#default route policy
router-rules
    router-rule
	    name: static
	    subnet-prefix:  *
	    router-guid: 0x0145162300c10b02
    end-router-rule
end-router-rules


# Load Share policy section
# Single path load share
load-share-rules
    load-share-rule
        subnet-prefix: 0xfec0000000000022
        router-guid: 0x0145041300c10b01 - 40
    end-load-share-rule
end-load-share-rules

# Two path load share
load-share-rules
    load-share-rule
        subnet-prefix: 0xfec0000000000022
        router-guid: 0x0145041300c10b01 - 60
        router-guid: 0x0145162300c10b01 - 40
    end-load-share-rule
end-load-share-rules

# Three path load share
load-share-rules
    load-share-rule
        subnet-prefix: 0xfec0000000000022
        router-guid: 0x0145041300c10b01 - 40
        router-guid: 0x0145162300c10b01 - 30
        router-guid: 0x0145163300c10b01 - 30
    end-load-share-rule
end-load-share-rules


# Multiple subnet load share configuration
load-share-rules
    load-share-rule
        subnet-prefix: 0xfec0000000000022
        router-guid: 0x0145041300c10b01 - 60
        router-guid: 0x0145162300c10b01 - 40
    end-load-share-rule

    load-share-rule
        subnet-prefix: 0xfec0000000000011
        router-guid: 0x0145045300c10b01 - 40
    end-load-share-rule

    load-share-rule
        subnet-prefix: 0xfec0000000000033
        router-guid: 0x0145046300c10b01 - 40
        router-guid: 0x0145166300c10b01 - 30
        router-guid: 0x0145167300c10b01 - 30
    end-load-share-rule
end-load-share-rules

==============================================================================
 8. Examples of FLID Policy File
==============================================================================

# FLID Policy File

flid-start
    global: 256 - 1023
    0xfec0000000000001: 521 - 841
    0xfec0000000000009: 843 - 951
flid-end
