I recently had chance to interact with a customer network I designed several years ago that reminded me of their multicast problem and the creative solution we implemented.
The customer has a Cisco IP phone system with centralized CallManager servers running Music on Hold (“MoH”) services at their Headquarters location. Branch offices were connected by T1 lines (often bundles of them) terminating on 2800 series routers. The Headquarters router is a 3800 series router. I’m simplifying things somewhat, but this is the gist of it.
Each branch location used L3 switches in the core of the branch network to allow faster connectivity between multiple local VLANs than would be possible if we’d configured the WAN router as the L3 gateway for the networks (either using physical ports or router-on-a-stick).
This was all well and good until it came time to deal with MoH. Unfortunately, given the limited bandwidth and several other constraints, the customer MoH source was multicast and the L3 switches at each site were 3560 switches (IP Base) which have NO PIM SUPPORT. The customer also deployed broadcast paging software which relied on multicast, so we had to find a solution that didn’t involve upgrading all of their branch core switches.
What to do?
We could have just put the WAN router on each branch subnet where we needed the multicast to flow, but that would require the router have a trunk to the branch core and that those interfaces have a “presence” on the subnet. We could control the flow of traffic for devices on those subnets going to Headquarters (or other branches) so it went via the L3 switch, but not for traffic coming from remote sites to the branch – that traffic would flow directly out of the router onto the subnets in question since the router would have a route with AD of 0 (connected) to those subnets. That’s no good.
After consideration, we deployed a combination of special VRFs (dedicated to multicast traffic) on the WAN routers combined with GRE tunnels to transport multicast between the sites, landing the traffic in the correct VRFs. We then placed WAN router interfaces on the subnets that required multicast, but did it in the multicast VRFs. This segregated the VRF interfaces from the main forwarding instance and prevents the asymmetric flows we mention above.
It’s important to recognize that we’re using VRF-lite and not full-blown MPLS/VRF functionality, which could have also been used here – at the expense of SIGNIFICANTLY more complex configs and routing. VRF-lite combined with the GRE tunnels accomplishes everything we need with a fraction of the complexity.
Why is a VRF so important here? Let’s first take a look at how flows would travel without the VRF in play:
As depicted above, without the VRFs applied to the sub interface, the outgoing traffic (the blue dashed line) goes from the client device to its configured default gateway (the L3 Branch Switch, in this case), which then forwards packets to the Branch Router and so on. Incoming traffic destined for the Voice Subnet, however, reaches the Branch Router and then, because without the sub-interface being in a VRF the router has an AD 0 (connected) router to the voice router, the Branch Router sends the traffic directly out the sub-interface onto the VLAN.
This is not only tacky (asymmetry of this type in a network this small is ugly), but it also causes numerous problems. Troubleshooting is more difficult, monitoring and graphing traffic per VLAN can’t be done from a single device, topology can’t be easily bi-directionally determined on the fly, security policies and QoS policies don’t have a single enforceable boundary, etc.
So let’s now look at the flows with the addition of the VRF:
This configuration results in flows for “regular” traffic following the default forwarding table path from the WAN to the Branch Router and then to the Branch Switch for inbound and the exact reverse for outbound. Separately, the multicast traffic flows from Headquarters, to the Headquarters Core Router, through the tunnel to the Branch Router and then out the VRF-bound sub-interface to the Voice Subnet (or whatever subnet for which the multicast is destined).
This is a much “tidier” flow overall and results in only marginally more complex configuration. Flows are symmetric and all of the issues I note as problems in the asymmetric model above are mitigated.
Now I suppose I should take a moment and note that I’m by no means against traffic asymmetry – I frankly can’t stand it when people get upset over asymmetric traffic when the asymmetry has no detrimental impact on their transport. Pleading with them to understand that IP was designed this way seems to fall on deaf ears at times. However, when legitimate problems are created by asymmetry, it’s worth addressing – even at the cost of complexity.
It’s also worth noting (in case it’s not obvious) that under the current configuration ONLY multicast traffic is captured on the tunnel. Even traffic flowing between devices on the multicast “landing network” and the multicast source network will flow over the “normal” (non VRF-tunnel) path in both directions, since the router will only “capture” and route traffic on its MoH VRF sub-interfaces that gets routed there by PIM. All other traffic follows the normal routing path.
I should reinforce that the configurations are simplified – the actual design has a “cloud” at the core and provides RP functionality that’s not addressed herein (I had to draw the line somewhere, after all).
Here’s what the Headquarters core router looked like:
! This is the VRF for the MoH traffic
!
ip vrf moh
description vrf for MoH multicast
rd 100:0
!
ip multicast-routing
ip multicast-routing vrf moh
!
! This tunnel endpoint lives “in” the MoH VRF, but the
! source and destination are in the default forwarding VRF.
!
interface Tunnel0
description MOH multicast network tunnel MOH Core
ip vrf forwarding moh
ip address 172.16.1.42 255.255.255.252
ip pim dense-mode
tunnel source 172.16.0.42
tunnel destination 172.16.0.41
!
! This is the « normal » transport interface over which
! the tunnel rides
!
interface Multilink1
description Multilink bundle to Branch
ip address 172.16.0.42 255.255.255.252
ip pim sparse-dense-mode
ppp multilink
ppp multilink group 1
!
interface GigabitEthernet0/0
no ip address
ip pim sparse-dense-mode
!
! This is the “normal traffic interface
!
interface GigabitEthernet0/0.10
encapsulation dot1Q 10
ip address 10.10.12.1 255.255.255.252
!
! This is the VRF-bound sub-interface that picks up
! the multicast traffic.
!
interface GigabitEthernet0/0.100
encapsulation dot1Q 100
ip vrf forwarding moh
ip address 192.168.198.6 255.255.255.0
ip pim dense-mode
!
! Remote voice subnets – they’re static for simplicity,
! since they don’t change very often and there’s no! alternate (backup) path for MoH.
!
ip route vrf moh 192.168.103.0 255.255.255.0 Tunnel0
ip route vrf moh 192.168.106.0 255.255.255.0 Tunnel0
And at a branch location:
! This is the VRF for the MoH traffic
!
ip vrf moh
description vrf for MOH multicast
rd 100:0
!
ip multicast-routing
ip multicast-routing vrf moh
!
! Again, here is the tunnel that lives “in” the VRF, with
! the source and destination addresses in the default forwarding
! instance.
!
interface Tunnel0
description MOH multicast network tunnel in MOH vrf
ip vrf forwarding moh
ip address 172.16.1.2 255.255.255.252
ip pim dense-mode
tunnel source 172.16.0.50
tunnel destination 172.16.0.49
!
! Transport interface for “normal” traffic
interface Multilink1
ip address 172.16.0.50 255.255.255.252
ip pim sparse-dense-mode
ppp multilink
ppp multilink group 1
!
!
interface GigabitEthernet0/0
no ip address
ip pim sparse-dense-mode
!
! This is the “normal traffic interface
!
interface GigabitEthernet0/0.10
encapsulation dot1Q 10
ip address 10.10.10.81 255.255.255.240
!
! One of the voice subnets where we need MoH
!
interface GigabitEthernet0/0.100
encapsulation dot1Q 100
ip vrf forwarding moh
ip address 192.168.103.5 255.255.255.0
ip pim sparse-dense-mode
!
! Another one of the voice subnets where we need MoH
!
interface GigabitEthernet0/0.106
encapsulation dot1Q 106
ip vrf forwarding moh
ip address 192.168.106.5 255.255.255.0
ip pim sparse-dense-mode
!
! “Normal” routing in the default table
ip route 0.0.0.0 0.0.0.0 172.16.0.49
So, looking at the configurations, you can see that while the Tunnel interface is bound to the VRF, it’s source and destination addresses are not – and are in fact in the default forwarding table. Sub-interfaces are created on each VLAN where the multicast traffic needs to go and bound to the VRF. The VLANs come to the router on a trunk directly from the L3 switch at the branch core, which is totally unaware of the VRF – it doesn’t participate in the “magic” except to be the subnet gateway for “normal” traffic and to trunk each VLAN needing MoH back to the core router at L2.
You do need to be careful to make sure that your voice traffic flows over the “normal” network and that only multicast is transported over the VRF-tunnel network.
We operate PIM sparse-dense on the LAN facing interfaces, and dense on the Tunnel interfaces. There are other possibilities, but this worked well for us.
Don’t forget that you need to enable PIM for each VRF where you want multicast routed.
This configuration has been running without issue for several years now in production. It saved the customer a good deal of capital expense at the cost of a little increased configuration complexity. A worthwhile trade!