Routing with NSX using multiple sites (with iBGP between the UDLR and the ESG's)

Routing with NSX using multiple sites (with iBGP between the UDLR and the ESG's)

Contents

Introduction

In one of my previous articles I demonstrated the routing between two Data Centers with using eBGP.

This article will be about doing the same, but now with iBGP between the DLR control VM and the ESG's of both sites.

Diagram

The following diagram will be used for our setup.

File:LAB100 - NSX Routing based on iBGP.pdf

Nsx-routing-blog2-1.png

Components

I am still using the same components as in this previous article.

Use-Cases

The use-case that I will be testing is the same use-case as in the previous article (only now with iBGP)

Tenant one

Tenant one will have workloads in DC1 and in DC2 and because it is the routing protocol to determine the egress path the traffic will exit (based on the configuration) from the primary site. When the primary site is down the traffic should exit from the secondary site.

Routing protocol options

All possible routing options are described in the previous article.

The routing option in this article will be:

Option 2

  1. iBGP peering between the UDLR and the ESG’s
  2. eBGP peering between the ESG’s and the external routers
  3. eBGP peering between the external routers and the CORE

Option 2

Because of the many options available in the setup I want to start with the following: Tenant 1 workloads with the use of UDLR-01 and option 1 routing. The other options will be outlined in other articles.

I have only documented the changes below compared to the previous article. The "show" outputs are given fully again of all the routing tables and the route path verification as well.

UDLR configuration

DC1

The UDLR is in AS# 65530 and it will stay this way. We will change the ESG's to be in this same AS.

  1. Change the peering AS towards the ESG

Nsx-routing-blog2-2.png

DC2

The UDLR control VM does not exist in DC2.

ESG configuration

DC1

  1. Change the Local AS
  2. Configure the BGP peers towards the external routers and the UDLR.
  3. Disable BGP
  4. Enable BGP

Nsx-routing-blog2-3.png

DC2

  1. Change the Local AS
  2. Configure the BGP peers towards the external routers and the UDLR.
  3. Disable BGP
  4. Enable BGP

Nsx-routing-blog2-4.png

Cisco1000V configuration

DC1

!
hostname rt-a-01
!
no router bgp 65511
router bgp 65511
 bgp router-id 10.11.11.31
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.100.19.2 remote-as 65530
 neighbor 10.100.19.2 description ESG-A
 !
 address-family ipv4
  neighbor 10.11.11.253 activate
  neighbor 10.100.19.2 activate
 exit-address-family
!
ip route 10.200.19.0 255.255.255.0 10.11.11.253
ip route 10.200.21.0 255.255.255.0 10.11.11.253
!


!
hostname rt-a-02
!
no router bgp 65511
router bgp 65511
 bgp router-id 10.11.11.32
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.100.21.2 remote-as 65530
 neighbor 10.100.21.2 description ESG-A
 !
 address-family ipv4
  neighbor 10.11.11.253 activate
  neighbor 10.100.21.2 activate
 exit-address-family
!
ip route 10.200.19.0 255.255.255.0 10.11.11.253
ip route 10.200.21.0 255.255.255.0 10.11.11.253
!


DC2

!
hostname rt-b-01
!
no router bgp 65512
router bgp 65512
 bgp router-id 10.11.11.33
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.200.19.2 remote-as 65530
 neighbor 10.200.19.2 description ESG-B
 !
 address-family ipv4
  neighbor 10.11.11.253 activate
  neighbor 10.200.19.2 activate
 exit-address-family
!
ip route 10.100.19.0 255.255.255.0 10.11.11.253
ip route 10.100.21.0 255.255.255.0 10.11.11.253
!


!
hostname rt-b-02
!
no router bgp 65512
router bgp 65512
 bgp router-id 10.11.11.34
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.200.21.2 remote-as 65530
 neighbor 10.200.21.2 description ESG-B
 !
 address-family ipv4
  neighbor 10.11.11.253 activate
  neighbor 10.200.21.2 activate
 exit-address-family
!
ip route 10.100.19.0 255.255.255.0 10.11.11.253
ip route 10.100.21.0 255.255.255.0 10.11.11.253
!


Core configuration

Between the Core and the Cisco 1000V CSR's the AS did not change and we are still doing eBGP there.

UDLR route peering verification

DC1

REGX-UDLR01-0> show ip bgp neighbors

BGP neighbor is 172.39.39.1,   remote AS 65530,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 10 messages, Sent 11 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 1 Identifier 0x57572d5c
         Route refresh request:received 0 sent 0
         Prefixes received 6 sent 7 advertised 7
Connections established 1, dropped 2
Local host: 172.39.39.13, Local port: 179
Remote host: 172.39.39.1, Remote port: 21815


BGP neighbor is 172.39.39.2,   remote AS 65530,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 10 messages, Sent 10 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 2 Identifier 0x57572d5c
         Route refresh request:received 0 sent 0
         Prefixes received 6 sent 7 advertised 7
Connections established 1, dropped 2
Local host: 172.39.39.13, Local port: 179
Remote host: 172.39.39.2, Remote port: 41244


DC2

The UDLR control VM does not exist in DC2.

ESG route peering verification

DC1

REGA-ESG01-0> show ip bgp neighbors

BGP neighbor is 10.100.19.1,   remote AS 65511,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 16 messages, Sent 14 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 1 Identifier 0xc2a60eac
         Route refresh request:received 0 sent 0
         Prefixes received 2 sent 10 advertised 10
Connections established 1, dropped 1
Local host: 10.100.19.2, Local port: 61962
Remote host: 10.100.19.1, Remote port: 179


BGP neighbor is 10.100.21.1,   remote AS 65511,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 17 messages, Sent 14 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 2 Identifier 0xc2a60eac
         Route refresh request:received 0 sent 0
         Prefixes received 2 sent 10 advertised 10
Connections established 1, dropped 1
Local host: 10.100.21.2, Local port: 27456
Remote host: 10.100.21.1, Remote port: 179


BGP neighbor is 172.39.39.13,   remote AS 65530,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 15 messages, Sent 18 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 3 Identifier 0xc2a60eac
         Route refresh request:received 0 sent 0
         Prefixes received 7 sent 6 advertised 6
Connections established 2, dropped 2
Local host: 172.39.39.1, Local port: 21815
Remote host: 172.39.39.13, Remote port: 179


DC2

REGB-ESG01-0> show ip bgp neighbors

BGP neighbor is 10.200.19.1,   remote AS 65512,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 18 messages, Sent 17 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 1 Identifier 0x359d76ec
         Route refresh request:received 0 sent 0
         Prefixes received 2 sent 10 advertised 10
Connections established 1, dropped 1
Local host: 10.200.19.2, Local port: 43262
Remote host: 10.200.19.1, Remote port: 179


BGP neighbor is 10.200.21.1,   remote AS 65512,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 18 messages, Sent 15 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 2 Identifier 0x359d76ec
         Route refresh request:received 0 sent 0
         Prefixes received 2 sent 10 advertised 10
Connections established 1, dropped 1
Local host: 10.200.21.2, Local port: 19183
Remote host: 10.200.21.1, Remote port: 179


BGP neighbor is 172.39.39.13,   remote AS 65530,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 18 messages, Sent 20 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 3 Identifier 0x359d76ec
         Route refresh request:received 0 sent 0
         Prefixes received 7 sent 6 advertised 6
Connections established 2, dropped 2
Local host: 172.39.39.2, Local port: 41244
Remote host: 172.39.39.13, Remote port: 179


Cisco1000V route peering verification

DC1

rt-a-01#show ip bgp summary
BGP router identifier 10.11.11.31, local AS number 65511
BGP table version is 54, main routing table version 54
15 network entries using 3720 bytes of memory
15 path entries using 1800 bytes of memory
3/3 BGP path/bestpath attribute entries using 720 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 6328 total bytes of memory
BGP activity 15/0 prefixes, 34/19 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.11.11.253    4        65510      22      24       54    0    0 00:13:57        5
10.100.19.2     4        65530      17      18       54    0    0 00:10:26       10
rt-a-01#


rt-a-02#show ip bgp summary
BGP router identifier 10.11.11.32, local AS number 65511
BGP table version is 54, main routing table version 54
15 network entries using 3720 bytes of memory
15 path entries using 1800 bytes of memory
3/3 BGP path/bestpath attribute entries using 720 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 6328 total bytes of memory
BGP activity 15/0 prefixes, 34/19 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.11.11.253    4        65510      22      23       54    0    0 00:13:34        5
10.100.21.2     4        65530      17      20       54    0    0 00:11:03       10
rt-a-02#


DC2

rt-b-01#show ip bgp summary
rt-b-01#show ip bgp summary
BGP router identifier 10.11.11.33, local AS number 65512
BGP table version is 48, main routing table version 48
15 network entries using 3720 bytes of memory
22 path entries using 2640 bytes of memory
3/3 BGP path/bestpath attribute entries using 720 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 7168 total bytes of memory
BGP activity 15/0 prefixes, 38/16 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.11.11.253    4        65510      22      24       48    0    0 00:13:39       12
10.200.19.2     4        65530      19      20       48    0    0 00:12:11       10
rt-b-01#


rt-b-02#show ip bgp summary
BGP router identifier 10.11.11.34, local AS number 65512
BGP table version is 28, main routing table version 28
15 network entries using 3720 bytes of memory
22 path entries using 2640 bytes of memory
3/3 BGP path/bestpath attribute entries using 720 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 7168 total bytes of memory
BGP activity 15/0 prefixes, 38/16 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.11.11.253    4        65510      21      21       28    0    0 00:13:29       12
10.200.21.2     4        65530      19      21       28    0    0 00:12:42       10
rt-b-02#


Core route peering verification

cs-01#show ip bgp summary 
BGP router identifier 10.11.11.253, local AS number 65510
BGP table version is 402, main routing table version 402
15 network entries using 2040 bytes of memory
42 path entries using 2184 bytes of memory
3/3 BGP path/bestpath attribute entries using 372 bytes of memory
2 BGP AS-PATH entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 4644 total bytes of memory
BGP activity 63/48 prefixes, 644/602 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.11.11.31     4        65511      27      25      402    0    0 00:16:20       10
10.11.11.32     4        65511      25      24      402    0    0 00:15:20       10
10.11.11.33     4        65512      25      23      402    0    0 00:14:43       10
10.11.11.34     4        65512      21      22      402    0    0 00:14:02       10
cs-01#


UDLR routing tables

DC1

REGX-UDLR01-0> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 15

B       10.11.11.0/24        [200/0]       via 172.39.39.1
B       10.22.22.0/24        [200/0]       via 172.39.39.1
B       10.100.19.0/24       [200/0]       via 172.39.39.1
B       10.100.21.0/24       [200/0]       via 172.39.39.1
B       10.200.19.0/24       [200/0]       via 172.39.39.2
B       10.200.21.0/24       [200/0]       via 172.39.39.2
C       172.20.1.0/24        [0/0]         via 172.20.1.254
C       172.20.2.0/24        [0/0]         via 172.20.2.254
C       172.20.3.0/24        [0/0]         via 172.20.3.254
C       172.20.8.0/24        [0/0]         via 172.20.8.254
C       172.20.9.0/24        [0/0]         via 172.20.9.254
C       172.20.10.0/24       [0/0]         via 172.20.10.254
C       172.39.39.0/28       [0/0]         via 172.39.39.13
B       172.39.39.16/28      [200/0]       via 172.39.39.1
B       172.39.39.32/28      [200/0]       via 172.39.39.2


DC2

The UDLR control VM does not exist in DC2.

ESG routing tables

DC1

REGA-ESG01-0> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 12

B       10.11.11.0/24        [20/0]        via 10.100.19.1
B       10.11.11.0/24        [20/0]        via 10.100.21.1
B       10.22.22.0/24        [20/0]        via 10.100.19.1
B       10.22.22.0/24        [20/0]        via 10.100.21.1
C       10.100.19.0/24       [0/0]         via 10.100.19.2
C       10.100.21.0/24       [0/0]         via 10.100.21.2
B       172.20.1.0/24        [200/0]       via 172.39.39.14
B       172.20.2.0/24        [200/0]       via 172.39.39.14
B       172.20.3.0/24        [200/0]       via 172.39.39.14
B       172.20.8.0/24        [200/0]       via 172.39.39.14
B       172.20.9.0/24        [200/0]       via 172.39.39.14
B       172.20.10.0/24       [200/0]       via 172.39.39.14
C       172.39.39.0/28       [0/0]         via 172.39.39.1
C       172.39.39.16/28      [0/0]         via 172.39.39.17


DC2

REGB-ESG01-0> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 12

B       10.11.11.0/24        [20/0]        via 10.200.19.1
B       10.11.11.0/24        [20/0]        via 10.200.21.1
B       10.22.22.0/24        [20/0]        via 10.200.19.1
B       10.22.22.0/24        [20/0]        via 10.200.21.1
C       10.200.19.0/24       [0/0]         via 10.200.19.2
C       10.200.21.0/24       [0/0]         via 10.200.21.2
B       172.20.1.0/24        [200/0]       via 172.39.39.14
B       172.20.2.0/24        [200/0]       via 172.39.39.14
B       172.20.3.0/24        [200/0]       via 172.39.39.14
B       172.20.8.0/24        [200/0]       via 172.39.39.14
B       172.20.9.0/24        [200/0]       via 172.39.39.14
B       172.20.10.0/24       [200/0]       via 172.39.39.14
C       172.39.39.0/28       [0/0]         via 172.39.39.2
C       172.39.39.32/28      [0/0]         via 172.39.39.33
REGB-ESG01-0>


Cisco1000V routing tables

DC1

rt-a-01#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.31/32 is directly connected, GigabitEthernet2
B        10.22.22.0/24 [20/0] via 10.11.11.253, 00:17:32
C        10.100.1.0/24 is directly connected, GigabitEthernet1
L        10.100.1.101/32 is directly connected, GigabitEthernet1
C        10.100.19.0/24 is directly connected, GigabitEthernet3
L        10.100.19.1/32 is directly connected, GigabitEthernet3
B        10.100.21.0/24 [20/0] via 10.100.19.2, 00:14:45
S        10.200.19.0/24 [1/0] via 10.11.11.253
S        10.200.21.0/24 [1/0] via 10.11.11.253
      172.20.0.0/24 is subnetted, 6 subnets
B        172.20.1.0 [20/0] via 10.100.19.2, 00:14:02
B        172.20.2.0 [20/0] via 10.100.19.2, 00:14:02
B        172.20.3.0 [20/0] via 10.100.19.2, 00:14:02
B        172.20.8.0 [20/0] via 10.100.19.2, 00:14:02
B        172.20.9.0 [20/0] via 10.100.19.2, 00:14:02
B        172.20.10.0 [20/0] via 10.100.19.2, 00:14:02
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.100.19.2, 00:14:45
B        172.39.39.16 [20/0] via 10.100.19.2, 00:14:45
B        172.39.39.32 [20/0] via 10.11.11.253, 00:14:47
rt-a-01#


rt-a-02#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.32/32 is directly connected, GigabitEthernet2
B        10.22.22.0/24 [20/0] via 10.11.11.253, 00:16:46
C        10.100.1.0/24 is directly connected, GigabitEthernet1
L        10.100.1.102/32 is directly connected, GigabitEthernet1
B        10.100.19.0/24 [20/0] via 10.100.21.2, 00:15:18
C        10.100.21.0/24 is directly connected, GigabitEthernet3
L        10.100.21.1/32 is directly connected, GigabitEthernet3
S        10.200.19.0/24 [1/0] via 10.11.11.253
S        10.200.21.0/24 [1/0] via 10.11.11.253
      172.20.0.0/24 is subnetted, 6 subnets
B        172.20.1.0 [20/0] via 10.100.21.2, 00:14:35
B        172.20.2.0 [20/0] via 10.100.21.2, 00:14:35
B        172.20.3.0 [20/0] via 10.100.21.2, 00:14:35
B        172.20.8.0 [20/0] via 10.100.21.2, 00:14:35
B        172.20.9.0 [20/0] via 10.100.21.2, 00:14:35
B        172.20.10.0 [20/0] via 10.100.21.2, 00:14:35
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.100.21.2, 00:15:18
B        172.39.39.16 [20/0] via 10.100.21.2, 00:15:18
B        172.39.39.32 [20/0] via 10.11.11.253, 00:15:21
rt-a-02#


DC2

rt-b-01#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.33/32 is directly connected, GigabitEthernet2
B        10.22.22.0/24 [20/0] via 10.11.11.253, 00:16:51
S        10.100.19.0/24 [1/0] via 10.11.11.253
S        10.100.21.0/24 [1/0] via 10.11.11.253
C        10.200.1.0/24 is directly connected, GigabitEthernet1
L        10.200.1.101/32 is directly connected, GigabitEthernet1
C        10.200.19.0/24 is directly connected, GigabitEthernet3
L        10.200.19.1/32 is directly connected, GigabitEthernet3
B        10.200.21.0/24 [20/0] via 10.200.19.2, 00:16:14
      172.20.0.0/24 is subnetted, 6 subnets
B        172.20.1.0 [20/0] via 10.200.19.2, 00:15:05
B        172.20.2.0 [20/0] via 10.200.19.2, 00:15:05
B        172.20.3.0 [20/0] via 10.200.19.2, 00:15:05
B        172.20.8.0 [20/0] via 10.200.19.2, 00:15:05
B        172.20.9.0 [20/0] via 10.200.19.2, 00:15:05
B        172.20.10.0 [20/0] via 10.200.19.2, 00:15:05
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.200.19.2, 00:16:14
B        172.39.39.16 [20/0] via 10.11.11.253, 00:16:51
B        172.39.39.32 [20/0] via 10.200.19.2, 00:16:14
rt-b-01#


rt-b-02#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.34/32 is directly connected, GigabitEthernet2
B        10.22.22.0/24 [20/0] via 10.11.11.253, 00:16:25
S        10.100.19.0/24 [1/0] via 10.11.11.253
S        10.100.21.0/24 [1/0] via 10.11.11.253
C        10.200.1.0/24 is directly connected, GigabitEthernet1
L        10.200.1.102/32 is directly connected, GigabitEthernet1
B        10.200.19.0/24 [20/0] via 10.200.21.2, 00:16:25
C        10.200.21.0/24 is directly connected, GigabitEthernet3
L        10.200.21.1/32 is directly connected, GigabitEthernet3
      172.20.0.0/24 is subnetted, 6 subnets
B        172.20.1.0 [20/0] via 10.200.21.2, 00:15:40
B        172.20.2.0 [20/0] via 10.200.21.2, 00:15:40
B        172.20.3.0 [20/0] via 10.200.21.2, 00:15:40
B        172.20.8.0 [20/0] via 10.200.21.2, 00:15:40
B        172.20.9.0 [20/0] via 10.200.21.2, 00:15:40
B        172.20.10.0 [20/0] via 10.200.21.2, 00:15:40
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.200.21.2, 00:16:25
B        172.39.39.16 [20/0] via 10.11.11.253, 00:16:25
B        172.39.39.32 [20/0] via 10.200.21.2, 00:16:25
rt-b-02#


Core routing table

cs-01#show ip route bgp 
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 10.11.11.254 to network 0.0.0.0

      172.20.0.0/24 is subnetted, 6 subnets
B        172.20.1.0 [20/0] via 10.11.11.31, 00:16:19
B        172.20.2.0 [20/0] via 10.11.11.31, 00:16:19
B        172.20.3.0 [20/0] via 10.11.11.31, 00:16:19
B        172.20.8.0 [20/0] via 10.11.11.31, 00:16:19
B        172.20.9.0 [20/0] via 10.11.11.31, 00:16:19
B        172.20.10.0 [20/0] via 10.11.11.31, 00:16:19
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.11.11.32, 00:16:50
B        172.39.39.16 [20/0] via 10.11.11.32, 00:16:50
B        172.39.39.32 [20/0] via 10.11.11.34, 00:16:51
cs-01#


UDLR BGP tables

DC1

REGX-UDLR01-0> show ip bgp

Status codes: s - suppressed, d - damped, > - best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop       Metric  LocPrf  Weight AS Path
  > 10.11.11.0/24      10.100.19.1       0     100     200   65511 65510 i
    10.11.11.0/24      10.200.19.1       0     100      30   65512 65510 i
  > 10.22.22.0/24      10.100.19.1       0     100     200   65511 65510 i
    10.22.22.0/24      10.200.19.1       0     100      30   65512 65510 i
  > 10.100.19.0/24     172.39.39.1       0     100     200   ?
  > 10.100.21.0/24     172.39.39.1       0     100     200   ?
  > 10.200.19.0/24     172.39.39.2       0     100      30   ?
  > 10.200.21.0/24     172.39.39.2       0     100      30   ?
  > 172.20.1.0/24      0.0.0.0           0     100   32768   ?
  > 172.20.2.0/24      0.0.0.0           0     100   32768   ?
  > 172.20.3.0/24      0.0.0.0           0     100   32768   ?
  > 172.20.8.0/24      0.0.0.0           0     100   32768   ?
  > 172.20.9.0/24      0.0.0.0           0     100   32768   ?
  > 172.20.10.0/24     0.0.0.0           0     100   32768   ?
    172.39.39.0/28     172.39.39.1       0     100     200   ?
    172.39.39.0/28     172.39.39.2       0     100      30   ?
  > 172.39.39.0/28     0.0.0.0           0     100   32768   ?
  > 172.39.39.16/28    172.39.39.1       0     100     200   ?
  > 172.39.39.32/28    172.39.39.2       0     100      30   ?
REGX-UDLR01-0>


DC2

The UDLR control VM does not exist in DC2.

ESG BGP tables

DC1

REGA-ESG01-0> show ip bgp

Status codes: s - suppressed, d - damped, > - best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop       Metric  LocPrf  Weight AS Path
  > 10.11.11.0/24      10.100.19.1       0     100      60   65511 65510 i
    10.11.11.0/24      10.100.21.1       0     100      60   65511 65510 i
  > 10.22.22.0/24      10.100.19.1       0     100      60   65511 65510 i
    10.22.22.0/24      10.100.21.1       0     100      60   65511 65510 i
  > 10.100.19.0/24     0.0.0.0           0     100   32768   ?
  > 10.100.21.0/24     0.0.0.0           0     100   32768   ?
  > 172.20.1.0/24      172.39.39.14      0     100      60   ?
  > 172.20.2.0/24      172.39.39.14      0     100      60   ?
  > 172.20.3.0/24      172.39.39.14      0     100      60   ?
  > 172.20.8.0/24      172.39.39.14      0     100      60   ?
  > 172.20.9.0/24      172.39.39.14      0     100      60   ?
  > 172.20.10.0/24     172.39.39.14      0     100      60   ?
    172.39.39.0/28     172.39.39.14      0     100      60   ?
  > 172.39.39.0/28     0.0.0.0           0     100   32768   ?
  > 172.39.39.16/28    0.0.0.0           0     100   32768   ?
REGA-ESG01-0>


DC2

REGB-ESG01-0> show ip bgp

Status codes: s - suppressed, d - damped, > - best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop       Metric  LocPrf  Weight AS Path
  > 10.11.11.0/24      10.200.19.1       0     100      60   65512 65510 i
    10.11.11.0/24      10.200.21.1       0     100      60   65512 65510 i
  > 10.22.22.0/24      10.200.19.1       0     100      60   65512 65510 i
    10.22.22.0/24      10.200.21.1       0     100      60   65512 65510 i
  > 10.200.19.0/24     0.0.0.0           0     100   32768   ?
  > 10.200.21.0/24     0.0.0.0           0     100   32768   ?
  > 172.20.1.0/24      172.39.39.14      0     100      60   ?
  > 172.20.2.0/24      172.39.39.14      0     100      60   ?
  > 172.20.3.0/24      172.39.39.14      0     100      60   ?
  > 172.20.8.0/24      172.39.39.14      0     100      60   ?
  > 172.20.9.0/24      172.39.39.14      0     100      60   ?
  > 172.20.10.0/24     172.39.39.14      0     100      60   ?
    172.39.39.0/28     172.39.39.14      0     100      60   ?
  > 172.39.39.0/28     0.0.0.0           0     100   32768   ?
  > 172.39.39.32/28    0.0.0.0           0     100   32768   ?
REGB-ESG01-0>


Cisco1000V BGP tables

DC1

rt-a-01#show ip bgp
BGP table version is 54, local router ID is 10.11.11.31
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>  10.11.11.0/24    10.11.11.253             0             0 65510 i
 *>  10.22.22.0/24    10.11.11.253             0             0 65510 i
 r>  10.100.19.0/24   10.100.19.2                            0 65530 ?
 *>  10.100.21.0/24   10.100.19.2                            0 65530 ?
 r>  10.200.19.0/24   10.11.11.253                           0 65510 65512 65530 ?
 r>  10.200.21.0/24   10.11.11.253                           0 65510 65512 65530 ?
 *>  172.20.1.0/24    10.100.19.2                            0 65530 ?
 *>  172.20.2.0/24    10.100.19.2                            0 65530 ?
 *>  172.20.3.0/24    10.100.19.2                            0 65530 ?
 *>  172.20.8.0/24    10.100.19.2                            0 65530 ?
 *>  172.20.9.0/24    10.100.19.2                            0 65530 ?
 *>  172.20.10.0/24   10.100.19.2                            0 65530 ?
 *>  172.39.39.0/28   10.100.19.2                            0 65530 ?
     Network          Next Hop            Metric LocPrf Weight Path
 *>  172.39.39.16/28  10.100.19.2                            0 65530 ?
 *>  172.39.39.32/28  10.11.11.253                           0 65510 65512 65530 ?
rt-a-01#


rt-a-02#show ip bgp
BGP table version is 54, local router ID is 10.11.11.32
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>  10.11.11.0/24    10.11.11.253             0             0 65510 i
 *>  10.22.22.0/24    10.11.11.253             0             0 65510 i
 *>  10.100.19.0/24   10.100.21.2                            0 65530 ?
 r>  10.100.21.0/24   10.100.21.2                            0 65530 ?
 r>  10.200.19.0/24   10.11.11.253                           0 65510 65512 65530 ?
 r>  10.200.21.0/24   10.11.11.253                           0 65510 65512 65530 ?
 *>  172.20.1.0/24    10.100.21.2                            0 65530 ?
 *>  172.20.2.0/24    10.100.21.2                            0 65530 ?
 *>  172.20.3.0/24    10.100.21.2                            0 65530 ?
 *>  172.20.8.0/24    10.100.21.2                            0 65530 ?
 *>  172.20.9.0/24    10.100.21.2                            0 65530 ?
 *>  172.20.10.0/24   10.100.21.2                            0 65530 ?
 *>  172.39.39.0/28   10.100.21.2                            0 65530 ?
     Network          Next Hop            Metric LocPrf Weight Path
 *>  172.39.39.16/28  10.100.21.2                            0 65530 ?
 *>  172.39.39.32/28  10.11.11.253                           0 65510 65512 65530 ?
rt-a-02#


DC2

rt-b-01#show ip bgp
BGP table version is 48, local router ID is 10.11.11.33
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>  10.11.11.0/24    10.11.11.253             0             0 65510 i
 *>  10.22.22.0/24    10.11.11.253             0             0 65510 i
 r>  10.100.19.0/24   10.11.11.253                           0 65510 65511 65530 ?
 r>  10.100.21.0/24   10.11.11.253                           0 65510 65511 65530 ?
 r>  10.200.19.0/24   10.200.19.2                            0 65530 ?
 *>  10.200.21.0/24   10.200.19.2                            0 65530 ?
 *>  172.20.1.0/24    10.200.19.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.20.2.0/24    10.200.19.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
     Network          Next Hop            Metric LocPrf Weight Path
 *>  172.20.3.0/24    10.200.19.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.20.8.0/24    10.200.19.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.20.9.0/24    10.200.19.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.20.10.0/24   10.200.19.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.39.39.0/28   10.200.19.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.39.39.16/28  10.11.11.253                           0 65510 65511 65530 ?
 *>  172.39.39.32/28  10.200.19.2                            0 65530 ?
rt-b-01#


rt-b-02#show ip bgp
rt-b-02#show ip bgp
BGP table version is 28, local router ID is 10.11.11.34
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>  10.11.11.0/24    10.11.11.253             0             0 65510 i
 *>  10.22.22.0/24    10.11.11.253             0             0 65510 i
 r>  10.100.19.0/24   10.11.11.253                           0 65510 65511 65530 ?
 r>  10.100.21.0/24   10.11.11.253                           0 65510 65511 65530 ?
 *>  10.200.19.0/24   10.200.21.2                            0 65530 ?
 r>  10.200.21.0/24   10.200.21.2                            0 65530 ?
 *>  172.20.1.0/24    10.200.21.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.20.2.0/24    10.200.21.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
     Network          Next Hop            Metric LocPrf Weight Path
 *>  172.20.3.0/24    10.200.21.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.20.8.0/24    10.200.21.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.20.9.0/24    10.200.21.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.20.10.0/24   10.200.21.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.39.39.0/28   10.200.21.2                            0 65530 ?
 *                    10.11.11.253                           0 65510 65511 65530 ?
 *>  172.39.39.16/28  10.11.11.253                           0 65510 65511 65530 ?
 *>  172.39.39.32/28  10.200.21.2                            0 65530 ?
rt-b-02#


Core BGP tables

cs-01#show ip bgp
BGP table version is 402, local router ID is 10.11.11.253
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.11.11.0/24    0.0.0.0                  0         32768 i
*> 10.22.22.0/24    0.0.0.0                  0         32768 i
r  10.100.19.0/24   10.11.11.31                            0 65511 65530 ?
r>                  10.11.11.32                            0 65511 65530 ?
r  10.100.21.0/24   10.11.11.31                            0 65511 65530 ?
r>                  10.11.11.32                            0 65511 65530 ?
r  10.200.19.0/24   10.11.11.33                            0 65512 65530 ?
r>                  10.11.11.34                            0 65512 65530 ?
r  10.200.21.0/24   10.11.11.33                            0 65512 65530 ?
r>                  10.11.11.34                            0 65512 65530 ?
*  172.20.1.0/24    10.11.11.32                            0 65511 65530 ?
*                   10.11.11.33                            0 65512 65530 ?
*                   10.11.11.34                            0 65512 65530 ?
*>                  10.11.11.31                            0 65511 65530 ?
*  172.20.2.0/24    10.11.11.32                            0 65511 65530 ?
*                   10.11.11.33                            0 65512 65530 ?
*                   10.11.11.34                            0 65512 65530 ?
*>                  10.11.11.31                            0 65511 65530 ?
*  172.20.3.0/24    10.11.11.32                            0 65511 65530 ?
*                   10.11.11.33                            0 65512 65530 ?
*                   10.11.11.34                            0 65512 65530 ?
*>                  10.11.11.31                            0 65511 65530 ?
*  172.20.8.0/24    10.11.11.32                            0 65511 65530 ?
*                   10.11.11.33                            0 65512 65530 ?
*                   10.11.11.34                            0 65512 65530 ?
*>                  10.11.11.31                            0 65511 65530 ?
*  172.20.9.0/24    10.11.11.32                            0 65511 65530 ?
*                   10.11.11.33                            0 65512 65530 ?
*                   10.11.11.34                            0 65512 65530 ?
*>                  10.11.11.31                            0 65511 65530 ?
*  172.20.10.0/24   10.11.11.32                            0 65511 65530 ?
*                   10.11.11.33                            0 65512 65530 ?
*                   10.11.11.34                            0 65512 65530 ?
*>                  10.11.11.31                            0 65511 65530 ?
*  172.39.39.0/28   10.11.11.31                            0 65511 65530 ?
*                   10.11.11.33                            0 65512 65530 ?
*                   10.11.11.34                            0 65512 65530 ?
*>                  10.11.11.32                            0 65511 65530 ?
*  172.39.39.16/28  10.11.11.31                            0 65511 65530 ?
*>                  10.11.11.32                            0 65511 65530 ?
*  172.39.39.32/28  10.11.11.33                            0 65512 65530 ?
*>                  10.11.11.34                            0 65512 65530 ?
cs-01#


Routing path verifications

Routing path verification from the external client VM to T1-WEB-1 and T1-WEB-2

The expectation is that the traffic will route through DC1.

C:\Users\Administrator>tracert 172.20.1.1

Tracing route to 172.20.1.1 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
  3    <1 ms    <1 ms    <1 ms  10.100.19.2
  4     1 ms    <1 ms    <1 ms  172.39.39.14
  5     2 ms    <1 ms    <1 ms  172.20.1.1

Trace complete.


C:\Users\Administrator>tracert 172.20.1.2

Tracing route to 172.20.1.2 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
  3    <1 ms    <1 ms    <1 ms  10.100.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     6 ms    <1 ms    <1 ms  172.20.1.2

Trace complete.


Routing path verification from the T1-WEB-1 and T1-WEB-2 to the external client VM

root@Web01:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.184 ms  0.060 ms  0.149 ms
 2  172.39.39.1 (172.39.39.1)  0.454 ms  0.408 ms  0.438 ms
 3  10.100.19.1 (10.100.19.1)  1.412 ms  1.450 ms  1.434 ms
 4  10.11.11.50 (10.11.11.50)  1.454 ms * *
root@Web01:~#


root@Web02:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.132 ms  0.105 ms  0.082 ms
 2  172.39.39.1 (172.39.39.1)  0.922 ms  0.880 ms  0.771 ms
 3  10.100.19.1 (10.100.19.1)  1.110 ms  1.716 ms  1.704 ms
 4  10.11.11.50 (10.11.11.50)  2.738 ms * *


We are seeing that the ingress AND egress paths that are taken is the path trough DC1. And this was expected...

Routing path verification from the external client VM to T1-WEB-3 and T1-WEB-4

The expectation is that the traffic will route through DC1.

C:\Users\Administrator>tracert 172.20.1.3

Tracing route to 172.20.1.3 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     3 ms     1 ms    <1 ms  172.20.1.3

Trace complete.


C:\Users\Administrator>tracert 172.20.1.4

Tracing route to 172.20.1.4 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
  3    <1 ms    <1 ms    <1 ms  10.100.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     6 ms     1 ms    <1 ms  172.20.1.4

Trace complete.


Routing path verification from the T1-WEB-3 and T1-WEB-4 to the external client VM

The expectation is that the traffic will route through DC1.

root@Web03:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.133 ms  0.257 ms  0.137 ms
 2  172.39.39.1 (172.39.39.1)  0.510 ms  0.460 ms  0.601 ms
 3  10.100.21.1 (10.100.21.1)  1.298 ms  1.319 ms  1.266 ms
 4  10.11.11.50 (10.11.11.50)  1.380 ms * *


root@Web04:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.145 ms  0.101 ms  0.094 ms
 2  172.39.39.1 (172.39.39.1)  0.557 ms  0.631 ms  0.818 ms
 3  10.100.19.1 (10.100.19.1)  1.287 ms  1.290 ms  1.315 ms
 4  10.11.11.50 (10.11.11.50)  1.393 ms * *


The routing components in DC1 may fail or in case of a disaster the full site may go down. In that case, the routing should flow through DC2.

Let’s test this…

Turn RT-A-01 off. Verify paths again. The expectation is that the traffic will route through RT-A-02.

Routing path verification from the external client VM to T1-WEB-1, T1-WEB-2, T1-WEB-3 and T1-WEB-4

C:\Users\Administrator>tracert 172.20.1.1

Tracing route to 172.20.1.1 over a maximum of 30 hops

  1     3 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.32
  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     3 ms    <1 ms    <1 ms  172.20.1.1

Trace complete.


C:\Users\Administrator>tracert 172.20.1.2

Tracing route to 172.20.1.2 over a maximum of 30 hops

  1     1 ms     1 ms     4 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.32
  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms     1 ms    <1 ms  172.39.39.14
  5     3 ms    <1 ms    <1 ms  172.20.1.2

Trace complete.


C:\Users\Administrator>tracert 172.20.1.3

Tracing route to 172.20.1.3 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4     1 ms    <1 ms    <1 ms  172.39.39.14
  5     2 ms    <1 ms    <1 ms  172.20.1.3


C:\Users\Administrator>tracert 172.20.1.4

Tracing route to 172.20.1.4 over a maximum of 30 hops

  1     1 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.32
  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4     1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms     1 ms    <1 ms  172.20.1.4

Trace complete.


Routing path verification from the T1-WEB-1, T1-WEB-2, T1-WEB-3 and T1-WEB-4 to the external client VM

root@Web01:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.273 ms  0.157 ms  0.323 ms
 2  172.39.39.1 (172.39.39.1)  0.314 ms  0.248 ms  0.575 ms
 3  10.100.21.1 (10.100.21.1)  0.608 ms  0.594 ms  0.550 ms
 4  10.11.11.50 (10.11.11.50)  0.596 ms * *


root@Web02:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.189 ms  0.090 ms  0.097 ms
 2  172.39.39.1 (172.39.39.1)  0.408 ms  0.361 ms  0.441 ms
 3  10.100.21.1 (10.100.21.1)  0.668 ms  0.636 ms  0.774 ms
 4  10.11.11.50 (10.11.11.50)  0.733 ms * *


root@Web03:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.160 ms  0.144 ms  0.124 ms
 2  172.39.39.1 (172.39.39.1)  0.512 ms  0.500 ms  0.478 ms
 3  10.100.21.1 (10.100.21.1)  1.095 ms  1.115 ms  1.153 ms
 4  10.11.11.50 (10.11.11.50)  1.907 ms * *


root@Web04:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.155 ms  0.144 ms  0.109 ms
 2  172.39.39.1 (172.39.39.1)  4.136 ms  4.021 ms  3.921 ms
 3  10.100.21.1 (10.100.21.1)  6.588 ms  6.605 ms  6.631 ms
 4  10.11.11.50 (10.11.11.50)  6.878 ms * *


Turn RT-A-02 off. Verify paths again. The expectation is that the traffic will route through DC2 as both upstream routers in DC1 are down.

Routing path verification from the external client VM to T1-WEB-1 , T1-WEB-2, T1-WEB-3 and T1-WEB-4

C:\Users\Administrator>tracert 172.20.1.1

Tracing route to 172.20.1.1 over a maximum of 30 hops

  1    <1 ms     1 ms     2 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4     1 ms    <1 ms    <1 ms  172.39.39.14
  5    75 ms     1 ms     1 ms  172.20.1.1

Trace complete.


C:\Users\Administrator>tracert 172.20.1.2

Tracing route to 172.20.1.2 over a maximum of 30 hops

  1     3 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3     1 ms    <1 ms    <1 ms  10.200.19.2
  4     1 ms    <1 ms    <1 ms  172.39.39.14
  5     4 ms     1 ms    <1 ms  172.20.1.2

Trace complete.


C:\Users\Administrator>tracert 172.20.1.3

Tracing route to 172.20.1.3 over a maximum of 30 hops

  1     4 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms    <1 ms    <1 ms  172.20.1.3

Trace complete.


C:\Users\Administrator>tracert 172.20.1.4

Tracing route to 172.20.1.4 over a maximum of 30 hops

  1     2 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3     1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     3 ms     1 ms    <1 ms  172.20.1.4

Trace complete.


Routing path verification from the T1-WEB-1, T1-WEB-2, T1-WEB-3 and T1-WEB-4 to the external client VM

root@Web01:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.280 ms  0.217 ms  0.209 ms
 2  172.39.39.2 (172.39.39.2)  0.646 ms  0.787 ms  0.767 ms
 3  10.200.19.1 (10.200.19.1)  1.025 ms  1.678 ms  1.604 ms
 4  10.11.11.50 (10.11.11.50)  1.577 ms * *


root@Web02:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.135 ms  0.137 ms  0.056 ms
 2  172.39.39.2 (172.39.39.2)  0.669 ms  0.651 ms  0.727 ms
 3  10.200.19.1 (10.200.19.1)  2.106 ms  2.067 ms  2.061 ms
 4  10.11.11.50 (10.11.11.50)  2.195 ms * *


root@Web03:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.185 ms  0.150 ms  0.179 ms
 2  172.39.39.2 (172.39.39.2)  0.267 ms  0.285 ms  0.209 ms
 3  10.200.19.1 (10.200.19.1)  0.871 ms  0.707 ms  0.746 ms
 4  10.11.11.50 (10.11.11.50)  0.850 ms * *


root@Web04:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.119 ms  0.077 ms  0.079 ms
 2  172.39.39.2 (172.39.39.2)  0.322 ms  0.382 ms  0.447 ms
 3  10.200.19.1 (10.200.19.1)  1.584 ms  1.540 ms  1.495 ms
 4  10.11.11.50 (10.11.11.50)  1.479 ms * *


Turn RT-A-01 and RT-A-02 back on and turn ESG-A off. Verify paths again. The expectation is that the traffic will route through DC2 the only ESG in DC1 is down.

Before I turned off ESG-A I have verified if the traffic was flowing through DC1 again as an extra verification step. This is not shown below. For some reason the traffic kept flowing through DC1 so I had to turn BGP off/on again on the ESG in DC2 and the this caused the traffic to flow back trough DC1.

Routing path verification from the external client VM to T1-WEB-1 , T1-WEB-2, T1-WEB-3 and T1-WEB-4

With the ESG-A turned off...

C:\Users\Administrator>tracert 172.20.1.1

Tracing route to 172.20.1.1 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5   254 ms     1 ms     1 ms  172.20.1.1

Trace complete.


C:\Users\Administrator>tracert 172.20.1.2

Tracing route to 172.20.1.2 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms     1 ms    <1 ms  172.20.1.2

Trace complete.


C:\Users\Administrator>tracert 172.20.1.3

Tracing route to 172.20.1.3 over a maximum of 30 hops

  1     3 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms    <1 ms    <1 ms  172.20.1.3

Trace complete.


C:\Users\Administrator>tracert 172.20.1.4

Tracing route to 172.20.1.4 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     5 ms    <1 ms    <1 ms  172.20.1.4

Trace complete.


Routing path verification from the T1-WEB-1, T1-WEB-2, T1-WEB-3 and T1-WEB-4 to the external client VM

root@Web01:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.154 ms  0.115 ms  0.102 ms
 2  172.39.39.2 (172.39.39.2)  0.665 ms  0.660 ms  0.637 ms
 3  10.200.19.1 (10.200.19.1)  0.969 ms  0.998 ms  0.977 ms
 4  10.11.11.50 (10.11.11.50)  1.196 ms * *


root@Web02:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  1.620 ms  1.536 ms  1.488 ms
 2  172.39.39.2 (172.39.39.2)  0.452 ms  0.502 ms  0.510 ms
 3  10.200.19.1 (10.200.19.1)  0.763 ms  0.718 ms  0.686 ms
 4  10.11.11.50 (10.11.11.50)  0.993 ms * *


root@Web03:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.171 ms  0.131 ms  0.143 ms
 2  172.39.39.2 (172.39.39.2)  0.326 ms  0.362 ms  0.483 ms
 3  10.200.19.1 (10.200.19.1)  1.043 ms  1.063 ms  0.974 ms
 4  10.11.11.50 (10.11.11.50)  1.055 ms * *


root@Web04:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.131 ms  3.804 ms  3.799 ms
 2  172.39.39.2 (172.39.39.2)  0.443 ms  0.319 ms  0.333 ms
 3  10.200.19.1 (10.200.19.1)  0.575 ms  0.508 ms  0.772 ms
 4  10.11.11.50 (10.11.11.50)  0.782 ms * *


Turn ESG-A back on. Verify paths again. The expectation is that everything is back to normal now.

For some reason after turning on the ESG on DC1 the routes kept flowing through DC2. The same “issue” we saw before. So, I turned BGP off/on again on the ESG in DC2. This is not shown below. Below you will see the outputs after the BGP “reset” on DC2.

Routing path verification from the external client VM to T1-WEB-1 , T1-WEB-2, T1-WEB-3 and T1-WEB-4

C:\Users\Administrator>tracert 172.20.1.1

Tracing route to 172.20.1.1 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms    <1 ms    <1 ms  172.20.1.1

Trace complete.


C:\Users\Administrator>tracert 172.20.1.2

Tracing route to 172.20.1.2 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     4 ms    <1 ms    <1 ms  172.20.1.2

Trace complete.


C:\Users\Administrator>tracert 172.20.1.3

Tracing route to 172.20.1.3 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     2 ms     1 ms    <1 ms  172.20.1.3

Trace complete.


C:\Users\Administrator>tracert 172.20.1.4

Tracing route to 172.20.1.4 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
  3    <1 ms    <1 ms    <1 ms  10.100.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     2 ms    <1 ms    <1 ms  172.20.1.4

Trace complete.


Routing path verification from the T1-WEB-1, T1-WEB-2, T1-WEB-3 and T1-WEB-4 to the external client VM

root@Web01:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.163 ms  0.203 ms  0.123 ms
 2  172.39.39.1 (172.39.39.1)  0.214 ms  0.255 ms  0.266 ms
 3  10.100.21.1 (10.100.21.1)  0.647 ms  0.558 ms  0.401 ms
 4  10.11.11.50 (10.11.11.50)  0.798 ms * *


root@Web02:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.183 ms  0.117 ms  0.119 ms
 2  172.39.39.1 (172.39.39.1)  0.290 ms  0.241 ms  0.319 ms
 3  10.100.21.1 (10.100.21.1)  0.609 ms  0.555 ms  0.602 ms
 4  10.11.11.50 (10.11.11.50)  0.737 ms * *


root@Web03:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.246 ms  0.102 ms  0.065 ms
 2  172.39.39.1 (172.39.39.1)  0.465 ms  0.521 ms  0.568 ms
 3  10.100.21.1 (10.100.21.1)  0.686 ms  0.840 ms  0.812 ms
 4  10.11.11.50 (10.11.11.50)  0.983 ms * *


root@Web02:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.165 ms  0.177 ms  0.211 ms
 2  172.39.39.1 (172.39.39.1)  0.433 ms  0.500 ms  0.549 ms
 3  10.100.19.1 (10.100.19.1)  0.860 ms  0.956 ms  0.949 ms
 4  10.11.11.50 (10.11.11.50)  1.073 ms * *


Use-Cases

The end conclusion is that routing with this topology (with iBGP between the DLR and the ESG) works fine. When DC1 becomes unavailable DC2 will take over. For some reason when DC1 is available again the switch-over does not happen automatically, and we need to "kick" it by disabling BGP on the ESG-B.