Part4 – OPENVSWITCH – Playing with Bonding on Openvswitch

Bonding is aggregation multiple links to single link in order to increase throughput and achieve redundancy in case one of links fails. At least, officials say it so it must be true. Now, let me ask the question. How can we configure bonding?

 From what I know about bonding, they are two possibilities to configure it.

The first option is to load bonding module to kernel and use ifenslave utility for its configuration. Configuration steps for Microcore are described here.

http://brezular.wordpress.com/2011/01/20/how-to-setup-linux-microcore-3-x-router-qemu-image-in-fedora-linux-part2/

If you want to have fun as I had with bonding configuration, check this lab for reference.

 http://brezular.wordpress.com/2011/03/10/etherchannelvrrp-dhcp-ospf-configuration-cisco-vyatta-microcore/

 As the tutorial should be dedicated to bonding on Openvswitch I should focus on this topic. This is the second option how can we configure bonding without loading bonding module to Linux kernel.

 Let’s say we have two Microcore Linux boxes connected with two links. As usual, our topology is created using GNS3 and Qemu images of Microcore with installed Openvswitch 1.2.2 and Quagga 0.99.20 on the top of Microcore.

There is nothing in the world what could prevent you to download my Openvswitch Microcore Qemu image available here.

 http://sourceforge.net/projects/gns-3/files/Qemu%20Appliances/linux-microcore-4.0-openvswitch-1.2.2-quagga-0.99.20.img/download

 As for Ethernet card I suggest you to use emulated i82557b Ethernet card. Hint – check GNS3 Qemu setting. According to Bill, 640 kB is enough for anyone but it is better to assign at least 128 MB RAM for Microcore. Here is a  snapshot of Qemu setting.

 

Part1 – Basic Microcore, Openvswitch and Quagga Configuration

Before we start to configure Openvswitch we should be aware of initial configuration requirement for Openvswitch and Quagga. Frankly to say,  Openvswitch is not Cisco IOS which you can configure right after device is booted. First you need to create database, configuration files, start services etc.

My Qemu Microcore image with Openvswitch and Quaggae is preconfigured for such things so feel free to use it.

1.  Switch 1 configuration – hostname, trunks, vlan interfaces

tc@box:~$ echo "hostname Switch1" >> /opt/bootlocal.sh
tc@box:~$ sudo hostname Switch1

tc@Switch1:~$ sudo su

root@Switch1:~# ovs-vsctl add-br br0
root@Switch1:~# ovs-vsctl add-port br0 vlan10 tag=10 -- set interface vlan10 type=internal
root@Switch1:~# ovs-vsctl add-port br0 vlan20 tag=20 -- set interface vlan20 type=internal

Note: I usually specify list of VLANs to be allowed on trunk link

root@Switch1:~# ovs-vsctl add-port br0 eth0 trunks=10,20
root@Switch1:~# ovs-vsctl add-port br0 eth1 trunks=10,20

Unfortunately if we do it now, we cannot create bond0 interface, later in Part2. In this case a warning message informs us that eth0 and eth1 already exists on bridge br0. Therefore we skip a trunk definition here and we will configure the trunk later together with bonding configuration.

Start Quagga vtysh shell.

root@Switch1:~# vtysh

Hello, this is Quagga (version 0.99.20).

Copyright 1996-2005 Kunihiro Ishiguro, et al.

Switch1# conf t
Switch1(config)# interface vlan10
Switch1(config-if)# ip address 192.168.10.1/24
Switch1(config-if)# no shutdown
Switch1(config-if)#
Switch1(config-if)# interface vlan20
Switch1(config-if)# ip address 192.168.20.1/24
Switch1(config-if)# no shutdown
Switch1(config-if)# do wr
Switch1(config-if)# exit
Switch1(config)# exit
Switch1# exit.

Save Microcore configuration.

root@Switch1:~# /usr/bin/filetool.sh -b

Ctrl+S

2.  Switch 2 configuration – hostname, trunks, vlan interfaces

tc@box:~$ echo "hostname Switch2" >> /opt/bootlocal.sh
tc@box:~$ sudo hostname Switch2

tc@Switch1:~$ sudo su

root@Switch2:~# ovs-vsctl add-br br0
root@Switch2:~# ovs-vsctl add-port br0 vlan10 tag=10 -- set interface vlan10 type=internal
root@Switch2:~# ovs-vsctl add-port br0 vlan20 tag=20 -- set interface vlan20 type=internal

Start Quagga vtysh shell.

root@Switch2:~# vtysh

Hello, this is Quagga (version 0.99.20).

Copyright 1996-2005 Kunihiro Ishiguro, et al.

Switch2# conf t
Switch2(config)# interface vlan10
Switch2(config-if)# ip address 192.168.10.2/24
Switch2(config-if)# no shutdown  
Switch2(config-if)#
Switch2(config-if)# interface vlan20
Switch2(config-if)# ip address 192.168.20.2/24
Switch2(config-if)# no shutdown
Switch2(config-if)# do wr
Switch2(config-if)# exit
Switch2(config)# exit
Switch2# exit
root@Switch2:~#

Save Microcore configuration.

root@Switch2:~# /usr/bin/filetool.sh -b

Ctrl+S

Part2 – Bonding

Now, it is time to start playing  game called bonding.  They are not any rules in our game but it is not bad idea to read documentation first. Page number ten could answer many future question.

http://openvswitch.org/ovs-vswitchd.conf.db.5.pdf

As you can see, they are four bond modes with balance-slb as default mode. Modes are described on page 11.

1.  Switch1 and Switch2 configuration for static bonding

root@Switch1:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 trunks=10,20
root@Switch2:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 trunks=10,20

2.  Static Bonding Testing

Check bond interface bond0 on Switch1.

root@Switch1:~# ovs-appctl bond/show bond0

bond_mode: balance-slb
bond-hash-algorithm: balance-slb
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 9476 ms
lacp_negotiated: false

slave eth1: enabled
may_enable: true

slave eth0: enabled
active slave
may_enable: true

Now do the same for bond0 on Switch2.

root@Switch2:~# ovs-appctl bond/show bond0

bond_mode: balance-slb
bond-hash-algorithm: balance-slb
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 7420 ms
lacp_negotiated: false

slave eth1: enabled
may_enable: true

slave eth0: enabled
active slave
may_enable: true

Now, start pinging IP address 192.168.10.2 from Switch1. Ping is supposed to be working. While pinging, start tcpdump to listen on interface ethernet1 of Switch2

root@Switch2:~# tcpdump -i eth1

tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 68 bytes

11:47:07.214753 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 15, length 64
11:47:08.215483 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 16, length 64
11:47:09.215823 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 17, length 64
11:47:10.216846 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 18, length 64
11:47:11.217868 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 19, length 64

<output truncated>

15 packets captured
15 packets received by filter
0 packets dropped by kernel

Interface Ethernet1 sends icmp echo reply messages back to Switch1. As they are not any ICMP echo requests listed in the output, only the interface Ethernet0 receives these ICMP requests coming from Switch1.

My question is, what happens if we shutdown Ethernet0 on Switch2? Will be ICMP requests interrupted and possibly re-forwarded via interface Ethernet0 of Switch1? If yes, how long does it take?

root@Switch2:~# ifconfig eth0 down

Check the bond interface bond0 on Switch2 again. As soon as Switch2 detects failure of its interface, Openvswitch puts slave eth0 do disabled state

root@Switch2:~# ovs-appctl bond/show bond0

bond_mode: balance-slb
bond-hash-algorithm: balance-slb
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 7612 ms
lacp_negotiated: false

slave eth1: enabled
active slave
may_enable: true
hash 52: 0 kB load

slave eth0: disabled
may_enable: false

Switch1 cannot detect failure of far end interface Ethernet0. It continues to send ICMP requests to Switch2 via its Ethernet0 interface. Stop ping command and check bond interface bond0 on Switch1. Obviously, Ethernet0 is still enabled in bundle

root@Switch1:~# ovs-appctl bond/show bond0

bond_mode: balance-slb
bond-hash-algorithm: balance-slb
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 4007 ms
lacp_negotiated: false

slave eth1: enabled
may_enable: true

slave eth0: enabled
active slave
may_enable: true
hash 133: 0 kB load

This behaviour  we cannot tolerate. We need such as mechanism which helps to detect far end failure. Therefore LACP must be deployed in your configuration.

According to wiki,  LACP works by sending frames (LACPDUs) down all links that have the protocol enabled. These are two main advantages comparing to static bonding.

1.  Failover when a link fails when the peer will not see the link down. With static link aggregation the peer would continue sending traffic down the link causing it to be lost.

2.  The device can confirm that the configuration at the other end can handle link aggregation. With Static link aggregation a cabling or configuration mistake could go undetected and cause undesirable network behaviour.

Uhm, it looks like exactly what we want. So do not hesitate and let’s go configure bonding with LACP.

3.  Switch1 and Switch2 configuration for bonding with LACP

Delete existing bond0 port.

root@Switch1:~# ovs-vsctl del-port br0 bond0
root@Switch2:~# ovs-vsctl del-port br0 bond0

Create a new bond interface.

root@Switch1:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 lacp=active trunks=10,20
root@Switch2:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 lacp=active trunks=10,20

Check bonding on Switch1.

root@Switch1:~# ovs-appctl bond/show bond0

bond_mode: balance-slb
bond-hash-algorithm: balance-slb
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 7763 ms
lacp_negotiated: true

slave eth0: disabled
may_enable: false

slave eth1: enabled
active slave
may_enable: true

Notice state of Ethernet0 interface. Remember, we shut-downed Ethernet0 on Switch2. Thanks to LACP, Ethernet0 is disabled in bundle on Switch1. It is awesome!

Note  Even an interface Ethernet0 of Switch1 is disabled in bundle, it remains in up state

root@Switch1:~# ifconfig eth0

eth0      Link encap:Ethernet  HWaddr 00:AA:00:D8:E6:00
inet6 addr: fe80::2aa:ff:fed8:e600/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:112 errors:0 dropped:0 overruns:0 frame:0
TX packets:2030 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:36156 (35.3 KiB)  TX bytes:336741 (328.8 KiB)

4.  LACP bonding testing

The last point we want to test is how much time is required for transition for interface in bundle to disabled state, if far end fails. Logically it should depend how often are LACP messages exchanged between peers.

Start tcpdump listening on eth0 of Switch1. Then bring interface eth0 up on Switch2. Right after LACP messages are exchanged via eth0 between Switch1 and Switch2, Ethernet0 interface is brought to enabled state in bundle. See the output below

root@Switch1:~# tcpdump -i eth0 -v

tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes

< DHCP requests>

14:00:14.555758 IP (tos 0×0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 317) 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request [|bootp]

<IPV6 messages coming from Switch1>

14:00:18.457096 00:aa:00:85:34:00 (oui Unknown) > 33:33:00:00:00:16 (oui Unknown), ethertype IPv6 (0x86dd), length 110:
0×0000:  6000 0000 0038 0001 0000 0000 0000 0000  `….8……….
0×0010:  0000 0000 0000 0000 ff02 0000 0000 0000  …………….
0×0020:  0000 0000 0000 0016 3a00 0502            ……..:…
14:00:18.957535 00:aa:00:85:34:00 (oui Unknown) > 33:33:ff:85:34:00 (oui Unknown), ethertype IPv6 (0x86dd), length 78:
0×0000:  6000 0000 0018 3aff 0000 0000 0000 0000  `…..:………
0×0010:  0000 0000 0000 0000 ff02 0000 0000 0000  …………….
0×0020:  0000 0001 ff85 3400 8700 1173            ……4….s

<LACP message coming from Switch1. Notice System 00:00:00:00:00:00. It means that Switch2 has not known about partner’s system yet>

14:00:19.217584 LACPv1, length: 110
Actor Information TLV (0×01), length: 20
System 00:aa:00:85:34:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535
State Flags [Activity, Aggregation, Default]
Partner Information TLV (0×02), length: 20
System 00:00:00:00:00:00 (oui Ethernet), System Priority 0, Key 0, Port 0, Port Priority 0
State Flags [none]
Collector Information TLV (0×03), length: 16
packet exceeded snapshot

<LACP message sent to Switch2. As Switch1 knows partner ID 00:aa:00:85:34:00 – MAC address of interface eth0 of Switch2>

14:00:19.217914 LACPv1, length: 110
Actor Information TLV (0×01), length: 20
System 00:aa:00:d8:e6:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535
State Flags [Activity, Aggregation, Synchronization, Collecting, Distributing]
Partner Information TLV (0×02), length: 20
System 00:aa:00:85:34:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535
State Flags [Activity, Aggregation, Default]
Collector Information TLV (0×03), length: 16
packet exceeded snapshot

<LACP message sent to Switch1. Now, Switch1 knows partner’s ID 00:aa:00:d8:e6:00 – MAC address of interface eth0 of Switch1>

14:00:19.220189 LACPv1, length: 110
Actor Information TLV (0×01), length: 20
System 00:aa:00:85:34:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535
State Flags [Activity, Aggregation, Synchronization, Collecting, Distributing]
Partner Information TLV (0×02), length: 20
System 00:aa:00:d8:e6:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535
State Flags [Activity, Aggregation, Synchronization, Collecting, Distributing]
Collector Information TLV (0×03), length: 16
packet exceeded snapshot

Explanation

00:aa:00:85:34:00 – MAC address of eth0 interface of Switch2
00:aa:00:d8:e6:00 – MAC address of eth0 interface of Switch1:

Nice! We can see structure of LACP messages but still do not know anything about time required for transition. It is time to check it.

Start pinging 192.168.10.2 from Switch1 and with the help of tcpdump find an interface where packets enter Switch2. In my case it is Ethernet1. Shutdown this interface and cheek the output of ping command. You see, ping is interrupted and it took approximately 50 seconds for ping to be recovered. It means that for 50 seconds interface Ethernet1 of Switch1 had been in enabled state. It is too much. It must be a way to short this time to acceptable level.

Issue the command below on Switch1.  LACP statistic can be found there.

root@Switch1:~# ovs-appctl lacp/show bond0

lacp: bond0
status: active negotiated
sys_id: 00:aa:00:d8:e6:00
sys_priority: 65534
aggregation key: 5
lacp_time: slow

slave: eth0: current attached
port_id: 6
port_priority: 65535

actor sys_id: 00:aa:00:d8:e6:00
actor sys_priority: 65534
actor port_id: 6
actor port_priority: 65535
actor key: 5
actor state: activity aggregation synchronized collecting distributing

partner sys_id: 00:aa:00:85:34:00
partner sys_priority: 65534
partner port_id: 6
partner port_priority: 65535
partner key: 5
partner state: activity aggregation synchronized collecting distributing

slave: eth1: current attached
port_id: 5
port_priority: 65535

actor sys_id: 00:aa:00:d8:e6:00
actor sys_priority: 65534
actor port_id: 5
actor port_priority: 65535
actor key: 5
actor state: activity aggregation synchronized collecting distributing

partner sys_id: 00:aa:00:85:34:00
partner sys_priority: 65534
partner port_id: 5
partner port_priority: 65535
partner key: 5
partner state: activity aggregation synchronized collecting distributing

Focus on lacp-time parameter which is set to slow. According to documentation it is:

The LACP timing used on this Port. Possible values are fast, slow and a positive number of milliseconds. By default slow is used. When configured to be fast LACP heartbeats are requested at a rate of once per second causing connectivity problems to be detected more quickly. In slow mode, heartbeats are requested at a rate of once every 30 seconds.

That is exactly what we need. Changing lacp_time from slow to fast should help us to detect failure of peer’s interface more quickly.

5.  Switch1 and Switch2 configuration for bonding with LACP and lacp time set to fast

Delete existing bond interface bond0.

root@Switch1:~# ovs-vsctl del-port br0 bond0
root@Switch2:~# ovs-vsctl del-port br0 bond0

Create a new bond interface.

root@Switch1:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 lacp=active other_config:lacp-time=fast trunks=10,20
root@Switch2:~# ovs-vsctl add-bond br0 bond0 eth0 eth1 lacp=active other_config:lacp_time=fast trunks=10,20

Save configuration.

root@Switch1:~# /usr/bin/filetool.sh -b
root@Switch2:~# /usr/bin/filetool.sh -b

Ctrl+S

If you want to check interval of sending LACP messages, start tcpdump on Switch2 while pinging IP address of VLAN interface of Switch2 from Switch1. Now, LACP message are sent every 1 second

root@Switch2:~# tcpdump -i eth0

tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes
21:42:25.525007 LACPv1, length: 110
21:42:26.529846 LACPv1, length: 110
21:42:27.532092 LACPv1, length: 110
21:42:28.534532 LACPv1, length: 110
21:42:29.543080 LACPv1, length: 110
21:42:30.545620 LACPv1, length: 110
21:42:31.547848 LACPv1, length: 110
21:42:32.550717 LACPv1, length: 110
21:42:33.551271 LACPv1, length: 110
21:42:34.553382 LACPv1, length: 110

10 packets captured
10 packets received by filter
0 packets dropped by kernel

6.  LACP bonding testing with enabled lacp_time=fast option

Start pinging 192.168.10.2 from Switch1 and find interface receiving ICMP echo requests on  Switch2. In my case it is interface eth0. While pinging,  shutdown this interface and check ping statistic

root@Switch1:~# ping 192.168.10.2

PING 192.168.10.2 (192.168.10.2): 56 data bytes
64 bytes from 192.168.10.2: seq=0 ttl=64 time=1.810 ms
64 bytes from 192.168.10.2: seq=1 ttl=64 time=0.940 ms
64 bytes from 192.168.10.2: seq=2 ttl=64 time=1.610 ms
64 bytes from 192.168.10.2: seq=3 ttl=64 time=0.755 ms

<output truncated>

— 192.168.10.2 ping statistics —
27 packets transmitted, 24 packets received, 11% packet loss
round-trip min/avg/max = 0.755/3.378/48.484 ms

You see, three packet are lost. That is what we can call link redundancy if one of links fails.

Following  commands helpful during troubleshooting.

Show MAC address table.

root@Switch1:~# ovs-appctl fdb/show br0

List ports.

root@Switch1:~# ovs-vsctl list port

END.

Part3 – OPENVSWICH – Campus Model with Layer2 Access, Built with Open-Source Applications

In part one we showed how to create Openvswitch extension and submit it to Microcore repository. There were also presented after-install steps for Openvswitch,  adapted for specific Microcore needs.

http://brezular.wordpress.com/2011/09/03/part1-openvswich-creating-and-submitting-openvswitch-extension-to-microcore-upstream/

In part two we did several tests in order to test feature such as vlans, 8021q trunks and VLAN interfaces widely used in typical multilayer switches.

http://brezular.wordpress.com/2011/06/25/part2-openvswich-vlans-trunks-l3-vlan-interface-intervlan-routing-configuration-and-testing/

In part three we are going to create Campus Model using Linux Microcore with installed Openvswitch extension. As we need to enable routing between Distribution and Core layer to follow reccomendations of design guide, we have to install Quagga open-source routing suite.

Keepalived extension helps us to achieve default gateway’s redundancy using VRRP protocol, in case of failure of default gateway.  Other  extensions such as iproute2, tcpdump are not neccessary but useful and they are installed in our Qemu image.

Here is my own linux-microcore-3.0.3 Qemu image with installed Openvswitch 1.2.2, Quagga 0.99.20 and Keepalived 1.2.2 as extensions.

http://sourceforge.net/projects/gns-3/files/Qemu%20Appliances/linux-microcore-4.0-openvswitch-1.2.2-quagga-0.99.20.img/download

http://www.4shared.com/file/CAi_zNuO/linux-microcore-40-openvswitch.html

I had to recompile the kernel because a default Microcore kernel is not compiled with Multipath option. Read more here.

http://brezular.wordpress.com/2011/11/13/part1-recompiling-linux-microcore-4-0-2-for-equal-cost-routes/

The image also contains tcpdump and iproute2 command. I recommend you to use my Qemu image otherwise additional after install configuration for Quagga and Openvswitch is required. At least you should put following commands to /opt/bootlocal.sh

# put other system startup commands here
#Load modules to kernel
modprobe openvswitch_mod
modprobe ipv6
#Start ovsdb-server
ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,manager_options --private-key=db:SSL,private_key --certificate=db:SSL,certificate --bootstrap-ca-cert=db:SSL,ca_cert --pidfile --detach

#Initialize database
ovs-vsctl –no-wait init

#Start vswitchd daemon
ovs-vswitchd –pidfile –detach

#Enable forwarding between interfaces
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv6.conf.all.forwarding=1

#Start Quagga routing daemons
/usr/local/sbin/zebra -u root -d -f /usr/local/etc/quagga/zebra.conf
/usr/local/sbin/ripd -u root -d -f /usr/local/etc/quagga/ripd.conf
/usr/local/sbin/ripngd -u root -d -f /usr/local/etc/quagga/ripngd.conf
/usr/local/sbin/ospfd -u root -d -f /usr/local/etc/quagga/ospfd.conf
/usr/local/sbin/ospf6d -u root -d -f /usr/local/etc/quagga/ospf6d.conf
/usr/local/sbin/bgpd -u root -d -f /usr/local/etc/quagga/bgpd.conf
/usr/local/sbin/isisd -u root -d -f /usr/local/etc/quagga/isisd.conf

Note: 8021q module does not have to be loaded to get trunk working with Openvswitch. It has to be loaded when creating sub-interfaces using vconfig command is needed. On the other hand, choosing the right emulated network card in GNS3 Qemu settings is crucial to get trunk working. You cannot miss with i82557b ethernet card.

Note: Iproute2 extension is needed to show multiple equal-cost routes presented in Linux kernel. The topic is described more in detail here.

http://brezular.wordpress.com/2011/11/14/part2-testing-equal-cost-routes-in-linux-microcore-4-0-2/

Part1 – Core Layer

Core layer consists from two Multilayer Core switches – Core1 nad Core2. They are connected with point-to-point layer3 links. The full-mesh topology between Core and Distribution layer provides path even  in case of failure of two links. For example, if both interfaces eth4 and eth1 fail on switch Distrib1 there is still path to Core Layer via interface eth0.

Quagga routing daemon is running on all Core and Distribution switches. It offers routing capabilities using OSPF routing protocol.

Use vtysh – Quagga shell to configure Core switches as following.

1.  Command for accessing Quagga CLI

tc@box:~$ sudo /usr/local/bin/vtysh

Hello, this is Quagga (version 0.99.20).

Copyright 1996-2005 Kunihiro Ishiguro, et al.

2.  Configure hostname, interfaces and OSPF routing protocol for Core1

box# configure terminal

box(config)# hostname Core1

Core1(config)# interface eth0

Core1(config-if)# description  Link to Core2

Core1(config-if)# ip address 10.10.10.5/30

Core1(config-if)# no shutdown

Core1(config-if)# exit

Core1(config)# interface eth2

Core1(config-if)# description Link to Distrib2

Core1(config-if)# ip address 10.10.10.18/30

Core1(config-if)# no shutdown

Core1(config-if)# exit

Core1(config)# interface  eth1

Core1(config-if)# description Link to Distrib1

Core1(config-if)# ip address 10.10.10.10/30

Core1(config-if)# no shutdown

Core1(config-if)# exit

Core1(config)# router ospf

Core1(config-router)# network 10.10.10.4/30 area 0

Core1(config-router)# network 10.10.10.16/30 area 0

Core1(config-router)# network 10.10.10.8/30 area 0

Core1(config-router)# do write

Building Configuration…
Configuration saved to /usr/local/etc/quagga/zebra.conf
Configuration saved to /usr/local/etc/quagga/ripd.conf
Configuration saved to /usr/local/etc/quagga/ripngd.conf
Configuration saved to /usr/local/etc/quagga/ospfd.conf
Configuration saved to /usr/local/etc/quagga/ospf6d.conf
Configuration saved to /usr/local/etc/quagga/bgpd.conf
Configuration saved to /usr/local/etc/quagga/isisd.conf
[OK]

Now, exit from vtysh. As Quagga had not saved hostname “Core1″ to /usr/local/etc/quagga/zebra.conf, configure it from Microcore CLI..

tc@box:~$ echo "hostname Core1" >> /opt/bootlocal.sh

tc@box:~$ sudo hostname Core1

Force Microcore to save Quagga’s configuration files in /usr/locac/etc/quagga/ and other files in /opt/.filetool.lst.

tc@Core1:~$ /usr/bin/filetool.sh -b

Save GNS3. Go to File-> Save or use Ctrl + s.

3.  Configure hostname, interfaces and OSPF routing protocol for Core2

Start Core2 and configure router according to topology.

Check if Core switches can see themselves as OSPF neighbours.

Core2# show ip ospf neighbor

Neighbor ID      Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
10.10.10.18       1 Full/DR           35.785s 10.10.10.5      eth0:10.10.10.6          0     0     0

Check if routes are properly propagated in the routing table.

Core2# show ip route ospf

Codes: K – kernel route, C – connected, S – static, R – RIP, O – OSPF,
I – ISIS, B – BGP, > – selected route, * – FIB route

O   10.10.10.4/30 [110/10] is directly connected, eth0, 00:08:20
O>* 10.10.10.8/30 [110/20] via 10.10.10.5, eth0, 00:08:10
O   10.10.10.12/30 [110/10] is directly connected, eth1, 00:00:27
O>* 10.10.10.16/30 [110/20] via 10.10.10.5, eth0, 00:08:10
O   10.10.10.20/30 [110/10] is directly connected, eth2, 00:08:20

Part2 – Distribution Layer

Distribution layer consists from two Multilayer Distribution switches – Distrib1 and Distrib2. The main job for Distribution switches is routing between different vlan’s subnet that are terminated here.  Any traffic filtering rules should be configured on Distribution swiches.

Uplink interfaces connecting Distribution switches to Core switches are layer3 interfaces. They participate in OSPF messages forwarding.  Downlink interfaces connecting Distribution switches to Access switches are Layer2 interfaces. They are trunks capable of carrying traffic belongig to  multiple VLANs.

The next configuration’s steps are shown for Distrib1 switch only. Continue and make similiar configuration for Distrib2 switch.

1.  Configure Layer3 interfaces and OSPF routing protocol on Distrib1 switch

tc@box:~$ echo "hostname Distrib1" >> /opt/bootlocal.sh

tc@box:~$ /usr/bin/filetool.sh -b

tc@box:~$ sudo hostname Distrib1

tc@Distrib1:~$ sudo /usr/local/bin/vtysh

Distrib1# conf t

Distrib1(config)# hostname Distrib1

Distrib1(config)# interface eth1

Distrib1(config-if)# description Link to Core1

Distrib1(config-if)# ip address 10.10.10.9/30

Distrib1(config-if)# no shutdown

Distrib1(config-if)# exit

Distrib1(config)# int eth2

Distrib1(config-if)# description Link to Core2

Distrib1(config-if)# ip address 10.10.10.21/30

Distrib1(config-if)# no shutdown

Distrib1(config-if)# exit

Distrib1(config)# int eth0

Distrib1(config-if)# description Link to Distrib2

Distrib1(config-if)# ip address 10.10.10.1/30

Distrib1(config-if)# no shutdown

Distrib1(config-if)# exit

Distrib1(config)# router ospf

Distrib1(config-router)# network 10.10.10.0/30 area 0

Distrib1(config-router)# network 10.10.10.20/30 area 0

Distrib1(config-router)# network 10.10.10.8/30 area 0

Distrib1(config-router)# exit

Distrib1(config)# do write

Exit from vtysh and save content of /usr/local/etc/quagga/ directory.

tc@box:~$ /usr/bin/filetool.sh -b

Ctrl + s

2.  Configure Layer3 interfaces and OSPF routing protocol on Distrib2 switch

Configure Distrib1 switch according to the topology. Check if Distrib2 can see all three OSPF neighbours.

Distrib2# show ip ospf neighbor

Neighbor ID Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
10.10.10.21       1 Full/DR           38.389s 10.10.10.1      eth0:10.10.10.2          0     0     0
10.10.10.22       1 Full/DR           38.927s 10.10.10.14     eth1:10.10.10.13         0     0     0
10.10.10.18       1 Full/DR           33.153s 10.10.10.18     eth2:10.10.10.17         0     0     0

Check if routes are properly propagated in the routing table of Quagga.

Distrib2# show ip route ospf

Codes: K – kernel route, C – connected, S – static, R – RIP, O – OSPF,
I – ISIS, B – BGP, > – selected route, * – FIB route

O   10.10.10.0/30 [110/10] is directly connected, eth0, 00:01:47
O>* 10.10.10.4/30 [110/20] via 10.10.10.14, eth1, 00:01:17
*                        via 10.10.10.18, eth2, 00:01:17
O>* 10.10.10.8/30 [110/20] via 10.10.10.1, eth0, 00:01:17
*                        via 10.10.10.18, eth2, 00:01:17
O   10.10.10.12/30 [110/10] is directly connected, eth1, 00:01:37
O   10.10.10.16/30 [110/10] is directly connected, eth2, 00:01:24
O>* 10.10.10.20/30 [110/20] via 10.10.10.1, eth0, 00:01:31
*                         via 10.10.10.14, eth1, 00:01:31

Exit from Quagga vtysh shell and check routing table of Linux Microcore.

tc@Distrib2:~$ ip route show

10.10.10.0/30 dev eth0  proto kernel  scope link  src 10.10.10.2
10.10.10.4/30  proto zebra  metric 20
nexthop via 10.10.10.14  dev eth1 weight 1
nexthop via 10.10.10.18  dev eth2 weight 1
10.10.10.8/30  proto zebra  metric 20
nexthop via 10.10.10.1  dev eth0 weight 1
nexthop via 10.10.10.18  dev eth2 weight 1
10.10.10.12/30 dev eth1  proto kernel  scope link  src 10.10.10.13
10.10.10.16/30 dev eth2  proto kernel  scope link  src 10.10.10.17
10.10.10.20/30  proto zebra  metric 20
nexthop via 10.10.10.1  dev eth0 weight 1
nexthop via 10.10.10.14  dev eth1 weight 1
127.0.0.1 dev lo  scope link

They are two euqal-cost paths for each of network 10.10.10.4/30, 10.10.10.8/30, 10.10.10.20/30 presented in kernel routing table of Distrib2.

3.  Openvswitch Configuration – configure Layer2 trunk ports, Layer3 VLAN interfaces on Distrib1 switch

Openvswitch does not have separate CLI for its configuration thus all the configuration must be done from Microcore CLI.

a)  Configure eth3 and to become trunk port to allow carry traffic from VLAN 10 and VLAN 20

tc@Distrib1:~$ sudo ovs-vsctl add-br br0

tc@Distrib1:~$ sudo ovs-vsctl add-port br0 eth3 trunks=10,20

b)  Configure eth4 to become trunk port to allow carry traffic from VLAN 30 and 40

tc@Distrib1:~$ sudo ovs-vsctl add-port br0 eth4 trunks=30,40

c)  Create VLAN interfaces

tc@Distrib1:~$ sudo ovs-vsctl add-port br0 vlan10 tag=10 -- set interface vlan10 type=internal

tc@Distrib1:~$ sudo ovs-vsctl add-port br0 vlan20 tag=20 -- set interface vlan20 type=internal

tc@Distrib1:~$ sudo ovs-vsctl add-port br0 vlan30 tag=30 -- set interface vlan30 type=internal

tc@Distrib1:~$ sudo ovs-vsctl add-port br0 vlan40 tag=40 -- set interface vlan40 type=internal

d)  Check Openvswitch configuration

tc@Distrib1:~$ sudo ovs-vsctl show

a66779ff-0224-40ef-89f1-0deb21b939dBridge “br0″
Port “eth3″
trunks: [10, 20]
Interface “eth3″
Port “eth4″
trunks: [30, 40]
Interface “eth4″
Port “vlan20″
tag: 20
Interface “vlan20″
type: internal
Port “vlan10″
tag: 10
Interface “vlan10″
type: internal
Port “br0″
Interface “br0″
type: internal
Port “vlan30″
tag: 30
Interface “vlan30″
type: internal
Port “vlan40″
tag: 40
Interface “vlan40″
type: internal

e)  Configure IP addresses of VLAN interfaces and OSPF routing protocol on Distrib1 switch

We have two options here – either to use Linux kernel command – ifconfig or Quagga vtysh shell. I chose vtysh – no need to put commands to /opt/bootlocal.sh

tc@Distrib1:~$ sudo /usr/local/bin/vtysh

Distrib1# conf  t

Distrib1(config)# interface vlan10

Distrib1(config-if)# ip address  192.168.10.2/24

Distrib1(config-if)# no shutdown

Distrib1(config-if)# interface vlan20

Distrib1(config-if)# ip address  192.168.20.2/24

Distrib1(config-if)# no shutdown

Distrib1(config-if)# interface vlan30

Distrib1(config-if)# ip address  192.168.30.2/24

Distrib1(config-if)# no shutdown

Distrib1(config-if)# interface vlan40

Distrib1(config-if)# ip address  192.168.40.2/24

Distrib1(config-if)# no shutdown

Distrib1(config-if)# router ospf

Distrib1(config-router)# network 192.168.10.0/24 area 0

Distrib1(config-router)# network 192.168.20.0/24 area 0

Distrib1(config-router)# network 192.168.30.0/24 area 0

Distrib1(config-router)# network 192.168.40.0/24 area 0

Distrib1(config)# do write

Exit from vtysh shell and save configuration.

tc@Distrib1:~$ /usr/bin/filetool.sh -b

Ctrl + s.

Do similar configuration for Distrib2 switch.

Part3 – Access Layer

Access Layer consists of two Layer2 switches – Access1 and Access2. VLANs are created here and switchports are assigned to VLANS. As each VLAN is restricted to the local Access switch we call them – local VLANs.  Primary task of Access layer is to provide switchports for end users and forward traffic.  Since Layer2 Access switches do not route between different VLANs, traffic is sent to Distribution layer via 8021q trunks when routing is required. The design guides recommend a campus model with local VLANs when 20 percent of user’s traffic stays in Campus and 80 percent of traffic is forwarded outside campus.

1.  Configure hostname, access ports for VLAN 10 and 20 and 8021q trunk ports on Access1 switch

tc@box:~$ echo "hostname Access1" >> /opt/bootlocal.sh

tc@box:~$ sudo hostname Access1

tc@Access1:~$ sudo ovs-vsctl add-br br0

tc@Access1:~$ sudo ovs-vsctl add-port br0 eth0 tag=10

tc@Access1:~$ sudo ovs-vsctl add-port br0 eth1 tag=20

tc@Access1:~$ sudo ovs-vsctl add-port br0 eth3 trunks=10,20

tc@Access1:~$ sudo ovs-vsctl add-port br0 eth4 trunks=10,20

2.  Check Openvswitch configuration

tc@Access1:~$ sudo ovs-vsctl show

a66779ff-0224-40ef-89f1-0deb21b939db
Bridge “br0″
Port “br0″
Interface “br0″
type: internal
Port “eth1″
tag: 20
Interface “eth1″
Port “eth4″
trunks: [10, 20]
Interface “eth4″
Port “eth0″
tag: 10
Interface “eth0″
Port “eth3″
trunks: [10, 20]
Interface “eth3″

Save configuration.

tc@Access1:~$ /usr/bin/filetool.sh -b

Ctrl + s.

3.  Configure access ports for VLAN 30 and 40 and 8021q trunk ports on Access2 switch

Configure Access2 switch according to topology.

4.  Configure hostname and IP seetings for all hosts, PC1 is an example

tc@box:~$ echo "hostname PC1" >> /opt/bootlocal.sh

tc@box:~$ sudo hostname PC1

tc@PC1:~$ echo "ifconfig eth0 192.168.10.10 netmask 255.255.255.0 up" >> /opt/bootlocal.sh

tc@PC1:~$ sudo ifconfig eth0 192.168.10.10 netmask 255.255.255.0 up

tc@PC1:~$ echo "route add default gw 192.168.10.1" >> /opt/bootlocal.sh

tc@PC1:~$ sudo route add default gw 192.168.10.1

tc@PC1:~$ /usr/bin/filetool.sh -b

5.  Test connectivity inside the same VLAN, PC1 is an example

PC1 should ping IP address of vlan10 interface on both Distribution switches.

tc@PC1:~$ ping 192.168.10.2

PING 192.168.10.2 (192.168.10.2): 56 data bytes
64 bytes from 192.168.10.2: seq=0 ttl=64 time=1.949 ms
64 bytes from 192.168.10.2: seq=1 ttl=64 time=2.342 ms
64 bytes from 192.168.10.2: seq=2 ttl=64 time=3.016 ms
^C
— 192.168.10.2 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.949/2.435/3.016 ms

tc@PC1:~$ ping 192.168.10.3

PING 192.168.10.3 (192.168.10.3): 56 data bytes
64 bytes from 192.168.10.3: seq=0 ttl=64 time=2.297 ms
64 bytes from 192.168.10.3: seq=1 ttl=64 time=2.470 ms
64 bytes from 192.168.10.3: seq=2 ttl=64 time=2.815 ms
^C
— 192.168.10.3 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 2.297/2.527/2.815 ms2

Part4 – Virtual Router Redundancy Protocol /VRRP/ Configuration

Things look good now. We can ping a virtual interfaces on Distribution switches from host residing in the same VLAN. But we still  cannot ping any IP address residing in different subnet than host. It is because both Distribution switches have not been configured to act as default gateway yet.

You might have noticed that all the hosts were configured with IP address of default gateway 192.168.x.1. This is a virtual IP address created by Keepalived extension.

Keepalived offers default gateway’s redundancy using VRRP protocol. In case of failure one of the Distribution switches, the other Distrib switch takes responsibility and continues to forward packets. Switch with higher priority is called Master and forwards packet. At least it is always one Master switch in one VRRP group. For example, in VRRP group 10  it is Distrib1 with priority 150.  Switch with lower priority is called Backup and as it was said, it forwards packets only if Master switch fails. The Backup switch in VRRP group 10 is Distrib2 with priority 100. The higher is priority, more likely switch becomes Master switch.

Obviously, it must be communication between switches to let known each other about its existence. By default every one second Advertisiment is sent from Master switch to each memember of VRRP group to multicast IP address 224.0.0.18. After three missing Advertisiment plus screw time, Backup switch knowns that Master is down and transition  from Backup to Master state occurs.  It forwards packets now.

Note:

Virtual IP address is  tied with virtual MAC address – 00-00-5E-00-01-XX. The last byte of the address (XX) is the Virtual Router Identifier (VRID), which is different for each VRRP instance in the network. This address is used by only one physical router at a time, and it will reply with this MAC address when an ARP request is sent for the virtual router’s IP address. If  Master fails, the new Master will broadcast Gratious ARP containing the virtual router MAC address for the associated IP address. If I understand it correctly, nothing has been change in host configuration but Access switches had changed their CAM table – frames with destination MAC address 00-00-5E-00-01-XX will be send via new path to the new MASTER router.

Keepalived works slightly differently – it uses real MAC address of interface instead of  Virtual MAC address.  Check ARP cache of hosts in VRRP testing section – it seems that the current Keepalived/VRRP does not support Virtual MAC addresses.

1.  VRRP configuration on switch Distrib1

tc@Distrib1:~$ sudo su

root@Distrib1:~# mkdir /usr/local/etc/keepalived/

root@Distrib1:~# echo "/usr/local/etc/keepalived/" >> /opt/.filetool.lst

root@Distrib1:~# vi /usr/local/etc/keepalived/keepalived.conf

vrrp_instance 10  {                  # VRRP instance declaration
state MASTER                     # Start-up default state
interface vlan10                 # Binding interface
virtual_router_id 10        # VRRP VRID
priority 150                          # VRRP PRI

authentication {
auth_type PASS                       # Simple Passwd or IPSEC AH
auth_pass Campus123          # Password string
}

virtual_ipaddress {                  # VRRP IP address block
192.168.10.1/24 brd 192.168.10.255 dev vlan10
}
}

vrrp_instance 20  {
state BACKUP
interface vlan20
virtual_router_id 20
priority 100

authentication {
auth_type PASS
auth_pass Campus123
}

virtual_ipaddress {
192.168.20.1/24 brd 192.168.20.255 dev vlan20
}
}

vrrp_instance 30  {
state BACKUP
interface vlan30
virtual_router_id 30
priority 100

authentication {
auth_type PASS
auth_pass Campus123
}

virtual_ipaddress {
192.168.30.1/24 brd 192.168.30.255 dev vlan30
}
}

vrrp_instance 40  {
state MASTER
interface vlan40
virtual_router_id 40
priority 150

authentication {
auth_type PASS
auth_pass Campus123
}

virtual_ipaddress {
192.168.40.1/24 brd 192.168.40.255 dev vlan40
}
}

Start Keepalived daemon. As Distrib2 is not configured for VRRP, all the VRRP instances should transit to MASTER state.

root@Distrib1:~# /usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf

Make Keepalived daemon to be started after boot of Microcore.

root@Distrib1:~# echo "/usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf" >> /opt/bootlocal.sh

Save Keepalived configuration file.

root@Distrib1:~# /usr/bin/filetool.sh -b

Note:  After each change in keepalived.conf file you have to restart keepalived daemon to accept changes.

2.  VRRP configuration on switch Distrib2

tc@Distrib2:~$ sudo su

root@Distrib2:~# mkdir /usr/local/etc/keepalived/

root@Distrib2:~# echo "/usr/local/etc/keepalived/" >> /opt/.filetool.lst

root@Distrib2:~# vi /usr/local/etc/keepalived/keepalived.conf

vrrp_instance 10  {                  # VRRP instance declaration
state BACKUP                             # Start-up default state
interface vlan10                        # Binding interface
virtual_router_id 10               # VRRP VRID
priority 100                                 # VRRP PRI

authentication {
auth_type PASS                       # Simple Passwd or IPSEC AH
auth_pass Campus123          # Password string
}

virtual_ipaddress {             # VRRP IP address block
192.168.10.1/24 brd 192.168.10.255 dev vlan10
}
}

vrrp_instance 20  {
state MASTER
interface vlan20
virtual_router_id 20
priority 150

authentication {
auth_type PASS
auth_pass Campus123
}

virtual_ipaddress {
192.168.20.1/24 brd 192.168.20.255 dev vlan20
}
}

vrrp_instance 30  {
state MASTER
interface vlan30
virtual_router_id 30
priority 150

authentication {
auth_type PASS
auth_pass Campus123
}

virtual_ipaddress {
192.168.30.1/24 brd 192.168.30.255 dev vlan30
}
}

vrrp_instance 40  {
state BACKUP
interface vlan40
virtual_router_id 40
priority 100

authentication {
auth_type PASS
auth_pass Campus123
}

virtual_ipaddress {
192.168.40.1/24 brd 192.168.40.255 dev vlan40
}
}

Start keepalived daeomon.

root@Distrib2:~# /usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf

Distrib1 should become BACKUP router in VRRP group 20 and 30 and stays in MASTER state in VRRP 10 and 40. Similarly, DOSTRIB2 is BACKUP router in VRRP group 10 and 40 and MASTER in VRRP group 20 and 30.

Make keepalived daeomon to be started after boot of Microcore.

root@Distrib2:~# echo "/usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf" >> /opt/bootlocal.sh

Save keepalived configuration file.

root@Distrib2:~# /usr/bin/filetool.sh -b

Part5 – Testing

Connectivity should be established between any two nodes in our Campus network and we are going to test it.

1.  Test routing between VLANs

Issue ping from host PC1 in VLAN10 to host P2, PC3, PC4 and check if intervlan routing is working.

tc@PC1:~$ ping 192.168.20.1

PING 192.168.20.10 (192.168.20.10): 56 data bytes
64 bytes from 192.168.20.10: seq=0 ttl=63 time=14.443 ms
64 bytes from 192.168.20.10: seq=1 ttl=63 time=4.257 ms
64 bytes from 192.168.20.10: seq=2 ttl=63 time=4.962 ms
^C
— 192.168.20.10 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 4.257/7.887/14.443 ms

tc@PC1:~$ ping 192.168.30.10

PING 192.168.30.10 (192.168.30.10): 56 data bytes
64 bytes from 192.168.30.10: seq=0 ttl=63 time=9.720 ms
64 bytes from 192.168.30.10: seq=1 ttl=63 time=4.125 ms
64 bytes from 192.168.30.10: seq=2 ttl=63 time=4.920 ms
^C
— 192.168.30.10 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 4.125/6.255/9.720 ms

tc@PC1:~$ ping 192.168.40.10

PING 192.168.40.10 (192.168.40.10): 56 data bytes
64 bytes from 192.168.40.10: seq=0 ttl=63 time=8.404 ms
64 bytes from 192.168.40.10: seq=1 ttl=63 time=7.604 ms
64 bytes from 192.168.40.10: seq=2 ttl=63 time=4.798 ms
^C
— 192.168.40.10 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 4.798/6.935/8.404 ms0

2.  Check a routing table of  Core switch

Check if redundant routes are presented in kernel routing table of Core1. Non-redundant routes such as 10.10.10.4/30, 10.10.10.8/30, 10.10.10.16/30 and 127.0.0.1 are directly connected networks. Two equal-cost path should exist to other network learned by OSPF routing protocol.

tc@Core1:~$ ip route show

10.10.10.0/30  proto zebra  metric 20
nexthop via 10.10.10.9  dev eth1 weight 1
nexthop via 10.10.10.17  dev eth2 weight 1
10.10.10.4/30 dev eth0  proto kernel  scope link  src 10.10.10.5
10.10.10.8/30 dev eth1  proto kernel  scope link  src 10.10.10.10
10.10.10.12/30  proto zebra  metric 20
nexthop via 10.10.10.6  dev eth0 weight 1
nexthop via 10.10.10.17  dev eth2 weight 1
10.10.10.16/30 dev eth2  proto kernel  scope link  src 10.10.10.18
10.10.10.20/30  proto zebra  metric 20
nexthop via 10.10.10.6  dev eth0 weight 1
nexthop via 10.10.10.9  dev eth1 weight 1
127.0.0.1 dev lo  scope link
192.168.10.0/24  proto zebra  metric 20
nexthop via 10.10.10.9  dev eth1 weight 1
nexthop via 10.10.10.17  dev eth2 weight 1
192.168.20.0/24  proto zebra  metric 20
nexthop via 10.10.10.9  dev eth1 weight 1
nexthop via 10.10.10.17  dev eth2 weight 1
192.168.30.0/24  proto zebra  metric 20
nexthop via 10.10.10.9  dev eth1 weight 1
nexthop via 10.10.10.17  dev eth2 weight 1
192.168.40.0/24  proto zebra  metric 20
nexthop via 10.10.10.9  dev eth1 weight 1
nexthop via 10.10.10.17  dev eth2 weight 1

3.  Check connectivity between Access and Core layer

Issue ping from host PC1 in VLAN10 to eth0 interface of Core2.

tc@PC1:~$ ping 10.10.10.6

PING 10.10.10.6 (10.10.10.6): 56 data bytes
64 bytes from 10.10.10.6: seq=0 ttl=63 time=7.052 ms
64 bytes from 10.10.10.6: seq=1 ttl=63 time=4.383 ms
64 bytes from 10.10.10.6: seq=2 ttl=63 time=5.246 ms
^C
— 10.10.10.6 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 4.383/5.560/7.052 ms6

4.  VRRP testing

Issue ping from PC1 to IP address of VLAN10 interfaces – 192.168.10.2 and 192.168.10.3 on both Distribution switches. Stop ping sequence and ping VRRP Virtual IP address 192.168.10.1.

Now check ARP cache of PC1.

tc@PC1:~$ arp

? (192.168.10.2) at 00:23:20:bc:57:67 [ether]  on eth0
? (192.168.10.1) at 00:23:20:bc:57:67 [ether]  on eth0
? (192.168.10.3) at 00:23:20:8c:1d:cf [ether]  on eth

Virtual IP address 192.168.10.1 has assigned MAC address of VLAN10 interface – 00:23:20:bc:57:67 on Distrib1 switch. Packets destined outside VLAN10 will leave subnet 192.168.10.0/24 via vlan10 interface on Distrib1. We have expected it because Distrib1 is MASTER router for VRRP group 10.

Now start to ping IP address of eth0 interface of Core2 – 10.10.10.6/30 from PC1. We are going to kill keepalived proccess on Distrib1 switch to check if Disrib2  will transit to MASTER state for VRRP group 10 and 40.

List the keepalived proccess on Distrib1 switch

root@Distrib1:~# ps -ef | grep keepalived

2549 root     /usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf
2550 root     /usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf
4937 root     grep keepalived

Now, kill Keepalived process.

root@Distrib1:~# kill 2549

 Keepalived: Terminating on signal
Keepalived: Stopping Keepalived v1.2.2 (06/28,2011)
Keepalived_vrrp: Terminating VRRP child process on signal

Check the output of console of Distrib2 switch. Information messages tell us about transition of Distrib2 switch to MASTER state for VRRP group 10 and 40.

Keepalived_vrrp: VRRP_Instance(10) Transition to MASTER STATE
Keepalived_vrrp: VRRP_Instance(40) Transition to MASTER STATE
Keepalived_vrrp: VRRP_Instance(10) Entering MASTER STATE
Keepalived_vrrp: VRRP_Instance(40) Entering MASTER STATE

Check ARP cache PC1 again. It is not big surprise that Virtual IP address 192.168.10.1 has assigned MAC address 00:23:20:8c:1d:cf of VLAN10 interface of Distrib2 switch.

tc@PC1:~$ arp

? (192.168.10.2) at 00:23:20:bc:57:67 [ether]  on eth0
? (192.168.10.1) at 00:23:20:8c:1d:cf [ether]  on eth0
? (192.168.10.3) at 00:23:20:8c:1d:cf [ether]  on eth0p

It seems that series of ping requests from PC1 to 10.10.10.6 has not been broken by transition.

— 10.10.10.6 ping statistics —
206 packets transmitted, 206 packets received, 0% packet loss
round-trip min/avg/max = 1.664/4.343/15.437 ms.

Start keepalived daemon on Distrib1 switch again. Distrib1 is going immediately to state MASTER for VRRP group 10 and 40 and to BACKUP for VRRP group 20 and 30.

root@Distrib1:~# /usr/local/sbin/keepalived -P -f /usr/local/etc/keepalived/keepalived.conf

Keepalived: Starting VRRP child process, pid=5134
Keepalived_vrrp: Registering Kernel netlink reflector
Keepalived_vrrp: Registering Kernel netlink command channel
Keepalived_vrrp: Registering gratutious ARP shared channel
Keepalived_vrrp: Opening file ‘/usr/local/etc/keepalived/keepalived.conf’.

Keepalived_vrrp: Configuration is using : 45592 Bytes
Keepalived_vrrp: Using LinkWatch kernel netlink reflector…
Keepalived_vrrp: VRRP_Instance(20) Entering BACKUP STATE
Keepalived_vrrp: VRRP_Instance(30) Entering BACKUP STATE
Keepalived_vrrp: VRRP_Instance(10) Transition to MASTER STATE
Keepalived_vrrp: VRRP_Instance(40) Transition to MASTER STATE
Keepalived_vrrp: VRRP_Instance(10) Entering MASTER STATE
Keepalived_vrrp: VRRP_Instance(40) Entering MASTER STATE

Distrib2 received VRRP Advertisiment with priority higher than its own and transit to BACKUP state for VRRP instance 10 and 40.

Keepalived_vrrp: VRRP_Instance(10) Received higher prio advert
Keepalived_vrrp: VRRP_Instance(10) Entering BACKUP STATE
Keepalived_vrrp: VRRP_Instance(40) Received higher prio advert
Keepalived_vrrp: VRRP_Instance(40) Entering BACKUP STATE.

Also in this case ping started on PC1 was not interrupted by transition back to MASTER state on Distrib1 for VRRP group 10.

END.

http://en.wikipedia.org/wiki/Virtual_Router_Redundancy_Protocol
http://www.estoile.com/links/vrrp.htm
http://osdir.com/ml/linux.keepalived.devel/2005-01/msg00000.html

Part2 – Testing Equal-Cost Routes in Linux Microcore 4.0

Assuming you have your Linux Micorocore compiled with enabled option MULTIPATH, we are going to create simple lab to demonstrate existence of multiple equal-cost routes in the kernel routing table.

They are two Microcore boxes presented in our testing topology. Boxes are connected with point-to-point Layer 3 links, residing on separate subnets.

They both have Quagga routing suite installed and OSPF protocol enabled and configured. On router R1, Openvswitch is configured and interface vlan10 is created.

If kernel was successfully compiled with Multipath option in part2, a network 192.168.10.0/24 should be accessible via two possible routes from R2. Two next hop interfaces – 192.168.1.1/30 and 192.168.2.1/30 should be presented in both – Quagga and kernel routing table, pointing to the network 192.168.10.0/24.

Note: Kernel IP routing table is immediately updated by Quagga if change occurs.

The next few steps will help you to configure both routers. There is not need to configure both Quagga and Microcore form scratch, basic configuration

such as starting daemon, creating and backup of configuration files had been done for you. You only need to configure R1 and R2 according to topology.

Microcore Qemu image with Quagga 0.99.20, Openvswitch 1.2.2 and other networking utilities is available here:

http://sourceforge.net/projects/gns-3/files/Qemu%20Appliances/linux-microcore-4.0-openvswitch-1.2.2-quagga-0.99.20.img/download
http://www.4shared.com/file/CAi_zNuO/linux-microcore-40-openvswitch.html

1. R2 configuration

a) Configure hostname and backup /opt/bootlocal.sh file

echo "hostname R2" >> /opt/bootlocal.sh

sudo hostname R2

/usr/bin/filetool.sh -b

b) Start Quagga vtysh shell and configure router

sudo vtysh

Hello, this is Quagga (version 0.99.20).

Copyright 1996-2005 Kunihiro Ishiguro, et al.

Router2#

Router2# conf t

Router2(config)# interface eth0

Router2(config-if)# description Link to R1

Router2(config-if)# ip address 192.168.1.2/30

Router2(config-if)# no shutdown

Router2(config-if)#

Router2(config-if)# interface eth1

Router2(config-if)# description Link to R1

Router2(config-if)# ip address 192.168.2.2/30

Router2(config-if)# no shutdown

Router2(config-if)#

Router2(config-if)# router ospf

Router2(config-router)# network 192.168.1.0/30 area 0

Router2(config-router)# network 192.168.2.0/30 area 0

Router2(config-router)#

Router2(config-router)# do write

Building Configuration…

Configuration saved to /usr/local/etc/quagga/zebra.conf

Configuration saved to /usr/local/etc/quagga/ripd.conf

Configuration saved to /usr/local/etc/quagga/ripngd.conf

Configuration saved to /usr/local/etc/quagga/ospfd.conf

Configuration saved to /usr/local/etc/quagga/ospf6d.conf

Configuration saved to /usr/local/etc/quagga/bgpd.conf

Configuration saved to /usr/local/etc/quagga/isisd.conf

[OK]

Exit from vtysh shell and save Quagga configuration files from MIcrocore shell.

/usr/bin/filetool.sh -b

1. R1 configuration

a) Configure hostname and backup /opt/bootlocal.sh file

echo "hostname R1" >> /opt/bootlocal.sh

sudo hostname R1

/usr/bin/filetool.sh -b

b) Configure Openvswitch – create interface vlan10

sudo ovs-vsctl add-br br0

sudo ovs-vsctl add-port br0 vlan10 tag=10 — set interface vlan10 type=internal

Save Openvswitch database.

/usr/bin/filetool.sh -b

c) Start Quagga vtysh shell and configure router

sudo vtysh

Hello, this is Quagga (version 0.99.20).

Copyright 1996-2005 Kunihiro Ishiguro, et al.

Router1#

Router1# conf t

Router1(config)# interface eth0

Router1(config-if)# description Link to R2

Router2(config-if)# ip address 192.168.1.1/30

Router1(config-if)# no shutdown

Router1(config-if)#

Router1(config-if)# interface eth1

Router1(config-if)# description Link to R2

Router1(config-if)# ip address 192.168.2.1/30

Router1(config-if)# no shutdown

Router1(config-if)#

Router1(config-if)# interface vlan10

Router1(config-if)# description Virtual interface for vlan10

Router1(config-if)# ip address 192.168.10.1/24

Router1(config-if)# no shutdown

Router1(config-if)#

Router1(config-if)# router ospf

Router1(config-router)# network 192.168.1.0/30 area 0

Router1(config-router)# network 192.168.2.0/30 area 0

Router1(config-router)# network 192.168.10.0/24 area 0

Router1(config-if)#

Router1(config-router)# do write

Building Configuration…

Configuration saved to /usr/local/etc/quagga/zebra.conf

Configuration saved to /usr/local/etc/quagga/ripd.conf

Configuration saved to /usr/local/etc/quagga/ripngd.conf

Configuration saved to /usr/local/etc/quagga/ospfd.conf

Configuration saved to /usr/local/etc/quagga/ospf6d.conf

Configuration saved to /usr/local/etc/quagga/bgpd.conf

Configuration saved to /usr/local/etc/quagga/isisd.conf

[OK]

Exit from vtysh shell and save Quagga configuration files from MIcrocore shell.

/usr/bin/filetool.sh -b

3. Testing

a) Check if 192.168.10.0/24 is established in Quagga routing table on R2

sudo vtysh

Hello, this is Quagga (version 0.99.20).

Copyright 1996-2005 Kunihiro Ishiguro, et al.

Router2# show ip route

Codes: K – kernel route, C – connected, S – static, R – RIP, O – OSPF,

I – ISIS, B – BGP, > – selected route, * – FIB route

C>* 127.0.0.0/8 is directly connected, lo

K>* 127.0.0.1/32 is directly connected, lo

O 192.168.1.0/30 [110/10] is directly connected, eth0, 01:54:15

C>* 192.168.1.0/30 is directly connected, eth0

O 192.168.2.0/30 [110/10] is directly connected, eth1, 01:54:15

C>* 192.168.2.0/30 is directly connected, eth1

O>* 192.168.10.0/24 [110/20] via 192.168.1.1, eth0, 01:53:18

* via 192.168.2.1, eth1, 01:53:18

As we can see from Quagga IP routing table, two next hop interfaces are shown to network 192.168.10.0/24.

b) Check if 192.168.10.0/24 is established in kernel IP routing table on R2

Exit from vtysh shell and check kernel routing table.

tc@Router2:~$ route

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

127.0.0.1 * 255.255.255.255 UH 0 0 0 lo

192.168.1.0 * 255.255.255.252 U 0 0 0 eth0

192.168.2.0 * 255.255.255.252 U 0 0 0 eth1

192.168.10.0 192.168.1.1 255.255.255.0 UG 20 0 0 eth0

Command “route” lists only one path to the network 192.168.10.0/24 via next hop interface 192.168.1.1. The newer “ip” command should be used to see multiple equal-cost routes.

tc@Router2:~$ ip route show

127.0.0.1 dev lo scope link

192.168.1.0/30 dev eth0 proto kernel scope link src 192.168.1.2

192.168.2.0/30 dev eth1 proto kernel scope link src 192.168.2.2

192.168.10.0/24 proto zebra metric 20

nexthop via 192.168.1.1 dev eth0 weight 1

nexthop via 192.168.2.1 dev eth1 weight 1

Two next hop interfaces are listed in the output of ip command for network 192.168.10.0/24.

c) Test connectivity with ping command. Use Ctrl-c to break sequence

tc@Router2:~$ ping 192.168.10.1

PING 192.168.10.1 (192.168.10.1): 56 data bytes

64 bytes from 192.168.10.1: seq=0 ttl=64 time=1.581 ms

64 bytes from 192.168.10.1: seq=1 ttl=64 time=1.208 ms

64 bytes from 192.168.10.1: seq=2 ttl=64 time=2.324 ms

— 192.168.10.1 ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 1.208/1.704/2.324 ms

d) Testing if packet loss occurs when primary interface fails

Studying output of “route” command on R2, we notice that packets sent to the network 192.168.10.0/24 are forwarded to next hop IP address 192.168.1.1 via local interface eth0.

Now, start pinging IP address 192.168.10.1 from router R2. During ping, shutdown eth0 interface on R1. Check if series of pings is interrupted or not.

tc@Router2:~$ ping 192.168.10.1

PING 192.168.10.1 (192.168.10.1): 56 data bytes

64 bytes from 192.168.10.1: seq=0 ttl=64 time=1.042 ms

64 bytes from 192.168.10.1: seq=1 ttl=64 time=2.380 ms

64 bytes from 192.168.10.1: seq=2 ttl=64 time=1.216 ms

<output truncated>

— 192.168.10.1 ping statistics —

37 packets transmitted, 37 packets received, 0% packet loss

round-trip min/avg/max = 0.587/1.599/4.758 ms

Regarding statistics of ping command, we cannot see any packet loss. The benefit of multiple equal-cost paths is that in case of primary route failure, packets are immediately forwarded via backup available path.

Check the kernel routing table on R2 for changes.

tc@Router2:~$ ip route show

127.0.0.1 dev lo scope link

192.168.1.0/30 dev eth0 proto kernel scope link src 192.168.1.2

192.168.2.0/30 dev eth1 proto kernel scope link src 192.168.2.2

192.168.10.0/24 via 192.168.2.1 dev eth1 proto zebra metric 20

Not surprisingly, only one path exists to the network 192.168.10.0/24.

END.

Part1 – Recompiling Linux Microcore 4.0.2 for Equal-cost Routes

The article describes steps to recompile Microcore Linux 4.0.2 with kernel 3.0.3 in order to achieve redundant multiple equal-cost routes presented in a kernel routing table.

The following two points must be accomplished to get it working.

1. If you use Quagga routing suite, Quagga must to be compiled with parameter –enable-multipath=ARG to allow to copy multiple equal-cost routes to the kernel routing table. Parameter ARG determines maximum number of equal routes. When ARG equals 0, the number of routes is unlimited.

2. Kernel have to be compiled with following options enabled.

CONFIG_IP_ADVANCED_ROUTER=y

CONFIG_IP_MULTIPLE_TABLES=y

CONFIG_IP_ROUTE_MULTIPATH=y

CONFIG_IP_ROUTE_VERBOSE=y

These parameters are set in default Microcore kernel config file.

CONFIG_IP_ADVANCED_ROUTER=y

CONFIG_IP_MULTIPLE_TABLES=y

# CONFIG_IP_ROUTE_MULTIPATH is not set

# CONFIG_IP_ROUTE_VERBOSE is not set

As you can see MIcrocore kernel had not been compiled with MULTIPATH option enabled. For this reason we have to enable it and recompile kernel. If you do not do it, although multiple cost-equal routes will be presented in running-config of Quagga but not presented in the kernel routing table.

1. Start Microcore Linux

Assuming you have your Microcore already installed, start it.

qemu-kvm -no-acpi -boot c linux-microcore-4.0.2-clean.img -m 1G

2. Download and untar Linux kernel 3.0.3 patched for Microcore Linux

sudo su

mkdir /usr/src/

cd /home/tc/

wget http://distro.ibiblio.org/tinycorelinux/4.x/x86/release/src/kernel/linux-3.0.3-patched.tar.xz

tar xvf linux-3.0.3-patched.tar.xz

rm linux-3.0.3-patched.tar.xz

mv ./linux-3.0.3/ /usr/src/

Note: If you want to keep kernel persistently in /usr/src/, you must edit /opt/.filetool.lst and add /usr/src/ to the list of after-restart kept files.

/usr/bin/filetool -b

3. Install tools for compiling and download an original .config kernel file

tce-load -wi compiletc.tcz ncurses-dev perl5

Note: Perl is needed by make command.

cd /usr/src/

Download the kernel config file.

http://distro.ibiblio.org/tinycorelinux/4.x/x86/release/src/kernel/config-3.0.3-tinycore

Create a symbolic link /usr/src/linux pointing to /usr/src/linux-3.0.3 and copy config file to /usr/src/linux.

ln -sf ln -sf linux-3.0.3/ linux

cp ./config-3.0.3-tinycore ./linux/.config

4. Enable MULTIPATH support in kernel config file, and build the new kernel

cd /usr/src/linux/

make oldconfig

make menuconfig

Navigate to:

Networking support-> Networking Options

Enter Y to include both options – IP: equal cost multipath and IP: verbose route monitoring. Exit and select Yes. Configuration will be saved to /usr/src/linux/.config file.

Build the kernel.

make bzImage

In the end, your new kernel bzImage is saved to /usr/src/linux/arch/x86/boot/. Replace old kernel with the new one and reboot.

mv /usr/src/linux/arch/x86/boot/bzImage /mnt/hda1/boot/vmlinuz

After boot, check if the new kernel is loaded. Use uname -ra command.

Note: In case, you do not want to bother yourself with compiling, download my own bzImage. It is the kernel 3.0.3, patched for Microcore 4.x. Just rename bzImage to vmlinuz and replace existing /mnt/hda1/boot/vmlinuz with this file.

http://www.4shared.com/file/8uersVSM/bzImage.html

Used links

http://pontus.ullgren.com/view/multiple_interfaces_on_the_same_subnet

http://www.linfo.org/vmlinuz.html

Part1 – OPENVSWICH – Creating and Submitting Openvswitch Extension To Microcore Upstream

In May 2011, I read a request for installation Openvswitch on Qemu image. I started to play with Openvswitch and finally became addictive to this project. I realized how powerful  can be Openvswitch, offering many features listed here.  Spending my free time playing with Microcore and Openvswitch after several days I was able to create Microcore Qemu image with 8021q support  and install Openvswitch on top of it.

In this tutorial I would like to show how to create Openvswitch extension and make it ready for submit to the Microcore/Tinycore upstream.  I chose Microcore Linux, because I am familiar with this minimal Linux distribution and the Microcore is incredibly  small. Unless there is not the newer version than Openvswitch 1.2.2 you don’t have to create an Openvswitch extension since my extension has been added to  Microcore repository.

The following steps describe installation Openvswitch on Qemu image with pre-installed Microcore Linux.

If you don’t want to install openvswitch by yourself, you can download my own Openvswitch 1.2.2 /Microcore 4.0 Qemu mage.

http://www.4shared.com/file/6m9U-Gzd/linux-microcore-40-openvswitch.html

I also created three labs using Openvswitch. I tested how can Openvswitch works with VLANs, 802.1q trunk ports,  if it was capable of creating L3 VLAN interfaces and Inter VLAN routing was working.  Labs are available here:

1. Start Qemu Microcore image

Assuming your Microcore Qemu image with console support has been created and supports 8021q VLAN tagging, start the image:

qemu-kvm /home/brezular/linux-microcore-4.0.img -m 512

2. Download Openvswitch and Extract it

wget http://openvswitch.org/releases/openvswitch-1.2.2.tar.gz

tar zxvf ./openvswitch-1.2.2.tar.gz

cd ./openvswitch-1.2.2

3. Install Necessary Dependences for Openvswitch Compilation

tce-load -w -i compiletc.tcz python.tcz perl5.tcz openssh.tcz openssl-0.9.8.tcz openssl-0.9.8-dev.tcz tunctl.tcz linux-headers-3.0.3-tinycore.tcz bridge-utils.tcz

To run the ovsdmonitor tool, the machine must also have the following installed software:

tce-load -w -i python_twisted-2.7.tcz python-simplejson.tcz python_zope.interface-2.7 pyqt.tcz

Note:  The kernel source will be saved in /lib/modules/`uname -r`/build

It is recommended to check the list of necessary applications here:

4. BusyBox Story :-)

BusyBox 1.18.3 has problem with the cksum application. It doesn’t compute a checksum properly and the checksum failed message prevents the building of Openvswitch . Due to this reason, upgrade to BusyBox  1.18.4 is neccessary.

wget http://busybox.net/downloads/busybox-1.19.0.tar.bz2

tar jxvf ./busybox-1.19.0.tar.bz2

cd ./busybox-1.19.0/

make defconfig

make

sudo su;

cd /home/tc/busybox-1.19.0/

make install

Replace cksum version 1.18.3 with 1.19.0. Cksum version 1.19.0 is stored in ./_install/usr/bin/

sudo cp ./_install/usr/bin/cksum /usr/bin/

5. Openvswitch Installation

It is recommended to start here and continue here :-) After that you can type following:

export CFLAGS=”-march=i486 -mtune=i686 -Os -pipe”
export CXXFLAGS=”-march=i486 -mtune=i686 -Os -pipe”
export LDFLAGS=”-Wl,-O1″

cd ./openvswitch-1.2.2

./configure –prefix=/usr/local –with-linux=/lib/modules/`uname -r`/build

make -j2

touch /tmp/mark

sudo su

cd /home/tc/openvswitch-1.2.2/

make  DESTDIR=/tmp/openvswitch install

6. BACKUP and LOAD Module openvswitch_mod_ko

a) BACKUP module openvswitch_mod_ko 

After building, a kernel module “openvswitch_mod_ko” will be saved in ./datapath/linux/openvswitch_mod.ko. We’ve built both userspace and kernel module as well.  The performance is better when a kernel module is used.

mkdir -p /tmp/openvswitch/usr/local/lib/modules/3.0.3-tinycore/kernel/openvswitch/

cp /home/tc/openvswitch-1.2.2/datapath/linux/openvswitch_mod.ko /tmp/openvswitch/usr/local/lib/modules/3.0.3-tinycore/kernel/openvswitch/

b) Check if module can be loaded correctly

mkdir -p /usr/local/lib/modules/3.0.3-tinycore/kernel/openvswitch/

cp /home/tc/openvswitch-1.2.2/datapath/linux/openvswitch_mod.ko /usr/local/lib/modules/3.0.3-tinycore/kernel/openvswitch/

depmod -a

modprobe openvswitch_mod

Check if module openvswitch_mod.ko is loaded to the kernel with lsmod command.

7. Backup /home/tc/openvswitch-1.1.1/vswitchd/vswitch.ovsschema

In order to initialize the configuration database using ovsdb-tool , file /home/tc/openvswitch-1.2.0/vswitchd/vswitch.ovsschema is needed. You need to copy it to /tmp/openvswitch/usr/local/etc/openvswitch/vswitchd/ as we want to have it as a part of submitted Openvswitch extension.

mkdir -p /tmp/openvswitch/usr/local/etc/openvswitch/vswitchd/

cp /home/tc/openvswitch-1.2.2/vswitchd/vswitch.ovsschema /tmp/openvswitch/usr/local/etc/openvswitch/vswitchd/

8. Create Openvswitch Extension

a) Remove unneccessary files

rm -rf /tmp/openvswitch/usr/local/share/man/

b) Install squashfs and create openvswitch.tcz extension

tce-load -w -i squashfs-tools-4.x

sudo su

cd /tmp/

mksquashfs openvswitch/ openvswitch.tcz

c) Create a list of files presented in extension

cd /tmp/openvswitch/

sudo su

find usr -not -type d > ../openvswitch.tcz.list

d) Create md5 check sum of openvswitch.tcz

cd /tmp/ ls

md5sum openvswitch.tcz > openvswitch.tcz.md5.txt

e) Create openvswitch.tcz.info

an info file describing its contents (.tcz.info) – this content is standardized. Check repository for examples.

f) Create build-dep file

additional build instructions in a plain text file for future reference, mentioning such things as which extensions are required to build the package and what compile flags were used.

g) Create the dependency list openvswitch.tcz.dep

List of the extensions that have to be presented to run openvswitch extension correctly.

9. Test and Submit Openvswitch Extension

a) Create the new  clean Microcore Qemu image without any extensions installed

Assuming an image is ready to start, run it:

qemu-kvm /home/brezular/linux-microcore-4.0-clean.img -m 512

b) Install submitqc.tcz

tce-load -wi submitqc4.tcz

c) Copy the openvswitch files you have created in steps 1 – 8

(.tcz, .list, .md5.txt, .info, .dep, build-dep) from old Qemu image to the clean Qemu image, to the directory  /tmp/.

cd /tmp/

Run script:

sudo submitqc4

Script checks all openvswitch files in /tmp/ and create the directory /tmp/submitqc/ with log files.

d) Send extension

Note: If they are another files in /tmp/ , create directory /tmp/All and move openvswitch files, submitqc directory and openvswitch source to /tmp/All/

mv /tmp/openvswitch* /tmp/All/

sudo mv /tmp/submitqc/ /tmp/build-dep /tmp/All/

cd /tmp/All

sudo tar zcf openvswitch-1.2.2-extension.tar.gz *

sudo bcrypt openvswitch-1.2.2-extension.tar.gz

Password for bcrypt operation is tinycore Send openvswitch.tar.gz.bfe to  tcesubmit@gmail.com

10. Openvswitch After-Install Configuration

a) Make openvswitch_mod_ko, 8021q, ipv6 modules to be loaded to the kernel during boot of Microcore

echo "modprobe openvswitch_mod" >> /opt/bootlocal.sh>

echo "modprobe 8021q" >> /opt/bootlocal.sh

echo "modprobe ipv6" >> /opt/bootlocal.sh

sudo modprobe openvswitch_mod

sudo modprobe 8021q

sudo modprobe ipv6

b) Initialize the configuration database using ovsdb-tool

Check if a directory /usr/local/etc/openvswitch/ exists, if not create it.

sudo mkdir -p /usr/local/etc/openvswitch/

Create conf.db configuration file.

sudo ovsdb-tool create /usr/local/etc/openvswitch/conf.db /usr/local/etc/openvswitch/vswitchd/vswitch.ovsschema

Add /usr/local/etc/openvswitch/ to the list of after restart kept files.

echo "/usr/local/etc/openvswitch/" >> /opt/.filetool.lst

c) Make ovsdb-server to be started after start of the Microcore

vi /opt/bootlocal.sh

/usr/local/sbin/ovsdb-server –remote=punix:/usr/local/var/run/openvswitch/db.sock –remote=db:Open_vSwitch,manager_options –private-key=db:SSL,private_key –certificate=db:SSL,certificate –bootstrap-ca-cert=db:SSL,ca_cert –pidfile –detach

:wq!

Note: If you are not familiar with vi editor, use the reference below, please:

http://nemesis.lonestar.org/reference/docs/vi.html

d) Make the database initialialization using ovs-vsctl

This is only necessary the first time after you create the database with ovsdb-tool (but running it at any time is harmless).

echo "/usr/local/bin/ovs-vsctl --no-wait init" >> /opt/bootlocal.sh

e) Make the main Open vSwitch daemon being started, telling it to connect to the same Unix domain socket

echo "/usr/local/sbin/ovs-vswitchd --pidfile --detach" >> /opt/bootlocal.sh

f) Enable IPv4 and IPV6 packets forwarding between interfaces

Although not directly connected with Openvswitch configuration we need to enable ipv4 and ipv6 packets forwarding between interfaces for Microcore.  This option  is disabled in kernel by default.

sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv6.conf.all.forwarding=1

echo "sysctl -w net.ipv4.ip_forward=1" >> /opt/bootlocal.sh
echo "sysctl -w net.ipv6.conf.all.forwarding=1" >> /opt/bootlocal.sh

g) Run commands you have entered into  /opt/bootlocal.sh and make them persistent after restart of the Microcore

sudo /opt/bootlocal.sh

Optionally: It is recommended to delete the history of used commands.

echo "" > /home/tc/.ash_history

Save changes made for files and directories which are listed in  /opt/.filetool.lst

/usr/bin/filetool.sh -b

11. Example of configuration

Now you may use ovs-vsctl to set up bridges and other Open vSwitch features.  For example, to create a bridge named br0 and add ports eth0, eth1  and eth2 to it:

sudo ovs-vsctl add-br br0

sudo ovs-vsctl add-port br0 eth0
sudo ovs-vsctl add-port br0 eth1
sudo ovs-vsctl add-port br0 eth2

Before shutdown you always force Microcore to save configuration changes in the openvswitch database file -  /usr/local/etc/openvswitch/conf.db

/usr/bin/filetool.sh -b

Used Links

http://openvswitch.org/ http://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL.userspace;hb=HEAD http://openvswitch.org/?page_id=14 http://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL.Linux;hb=HEAD

End.

VirtualBox 4.1.0 Installation and Configuration for GNS3 0.8.0 on Fedora 15 Linux

GNS3 0.8.0 has been out for while and for the first time it has been released by someone else other than main developer – Jeremy Grossman.

Alexey Eromenko “Technologov”, the new GNS3 contributor brought some new fresh wind to GNS3 in the form of the integration of VirtualBox in to GNS3 project.

The goal of this tutorial is to show installation VirtualBox 4.1.1 on Fedora 15 Linux and GNS3 configuration for VirtualBox support. After that we are going to convert existing Qemu image and create the new VirtualBox Machine. Finally, a brief conclusion sums up benefits of Virtualbox integration in the new GNS3 0.8.0 release.

VirtualBox Installation

1. Install VirtualBox repo

sudo wget http://download.virtualbox.org/virtualbox/rpm/fedora/virtualbox.repo -P /etc/yum.repos.d/

2. Install dependency packages for VirtualBox

yum install gcc kernel-devel kernel-headers dkms.noarch xdotool

or for PAE kernel

yum install gcc kernel-PAE-devel kernel-headers dkms.noarch xdotool

3. Install the latest version of VirtualBox (current 4.1.0)

yum install VirtualBox-4.1

Note:

This command create automatically vboxusers group and VirtualBox user must be member of that group. It also builds VirtualBox  kernel module.

4. Add your user account to vboxusers group

What a surprise, mine is brezular ;-)

usermod -a -G vboxusers brezular

If you plan to run GNS3 as root you also need to add user root to the vboxusers group.

usermod -a -G vboxusers root

5. Convert existing Qemu disk image to normal VDI disk image

qemu-img convert -O raw linux-microcore-3.4.img linux-microcore-3.4.raw

VBoxManage convertfromraw -format VDI linux-microcore-3.4.raw linux-microcore-3.4.vdi

6. Convert existing normal type VDI hard disk image to multiattach  image

We need to convert normal VDI image to the multi-attach image which can be used by many VBox virtual machines simultaneously.  In order to do it, you have to create a new VirtualBox machine with attached  linux-microcore -3.4.vdi image. The steps are detailed described  in the following part -  Creating the New Virtual Box Machine.  Then you must deattach linux-microcore-3.4.vdi image from VirtualBox machine using this way.

a.  Right click on Virtual Box machine and select option  Settings
b.  Navigate to Storage and make left-click on linux-microcore-3.4.vdi (under option SATA controller)
c.  Right click on linux-microcore-3.4.vdi and select Remove Attachement. Close virtualbox GUI.

Now you can convert normal VDI image type to multiattach VDI image type with the command.

VBoxManage modifyhd linux-microcore-3.4.vdi --type multiattach

You can show info about hard disk image with following command. Notice, image is now multiattach type.

VBoxManage showhdinfo linux-microcore-3.4.vdi
UUID:                 d3412b7a-916f-46b5-976f-26fb54ad7206
Accessible:           yes
Logical size:         100 MBytes
Current size on disk: 25 MBytes
Type:                 multiattach
Storage format:       VDI
Format variant:       dynamic default
Location:             /home/brezular/linux-microcore-3.4.vdi

Creating The New Virtual Box Machine

You need to create the new Vbox machine for each VirtalBox network device you want to use. All the VBoxes machines will share the existing multi-attach linux-microcore-3.4.vdi image.

The default store for Vbox machines is echo `cd ~; pwd`/VirtualBox VMs. If you wish to store VirtualBoxes in another place, you can change that path just running virtualbox command. Invoke the command under the user whose account is typically used to start GNS3 application. If it is a root, start the VirtualBox as the root.

sudo su

virtualbox

Go to File-> Preferences-> General and change parameter Default Machine folder.

Press Ctrl-N – Create New Vbox machine.

Click Next.

Choose the Name of VM, Operating System and Version. Click Next.

Choose RAM: For Linux Microcore, 48 MB should be enough.

Click Next and Select option “Use Existing Hard disk“. Navigate to your new created linux-microcore-3.4.vdi disk.

Click Next and Create.

Your Vbox machine is created and we can continue with configuring GNS3 for VirtualBox support.

GNS3 configuration for VirtualBox support

Assuming that your GNS3 0.8.0 is correctly installed and running we are going to configure GNS3 to support VirtualBox virtualization.

Note: If you want to setup GNS3 for emulation of Cisco hardware with Dynamips, check GNS3 forum for help. If you want to compile and install Dynamips from source, check this link.

GNS3 settings

Go to Edit-> Preferences-> VirtualBox

Set path to the Vboxwrapper and click on the Test button. The green colour mesage “VBoxwrapper and VirtualBox 4.1.0 have succesfully started” should appear .

Now, switch to the VirtualBox Guest window. In this window we need to define VirtualBox machine we were previously created.

VM Name/UUID must match the name of VM. In our case it is Microcore_3.4. For Identifier name field choose the name whatever you want to have.

Cloning VirtualBox machine

Now we are in situation when the first VirtualBox machine – Microcore_3.4 has been created and defined in GNS3 VirtualBox Guest setting. If we want to have another VirtualBox Guest, we must create the new VirtualBox machine. The fastest method is to clone original Mircorcore_3.4 Vbox machine.

Press Ctrl-O – Clone and click on Clone button.

The new cloned VirtualBox machine – Microcore_3.4 Clone is created.

We are going to attach linux-microcore-3.4.vdi hardisk which was created in section 5 of this tutorial, to the Microcore-3.4 Clone VirtualBox machine. Right click on Microcore_3.4 Cloned and select Settings (Ctrl-S). Navigate to the Storage section.

Under SATA Controller, make a right click on Microcore_3.4 Clone-disk.vdi. Select option Remove Attachment.

Click on Add HardDisk icon. A new dialog window should appear.

Click on Choose existing disk button and navigate to exisiting multi-attached linux-microcore-3.4.vdi hard disk.

You have successfully attached linux-microcore-3.4.vdi hard disk to Microcore_3.4 Clone VirtualBox machine.

Conclusion

Although I have not done any deep performance testing, at the first look it seems that x86 emulation with VirtualBox is much faster comparing to pure Qemu running without any speed up accelerators. It makes Virtualbox virtualization very efficient on legacy computers whose CPUs do not support hardware virtualization with kvm. It is mainly true because of Kqemu – a Qemu software accelerator is available only up to Qemu version 0.11.x.

Other benefit which is offered by VirtualBox integration in GNS3 is support for UDP tunnels. It enables VirtualBox Guest to talk to Dynamips through UDP. Thanks to Technlogov effort, the patch is officialy part of VirtualBox since version 4.1.0.

Regardless of indisputable benefit of VirtualBox integration there are some issues that should be mentioned. First, it is the strange NIC labeling mismatch, originally mentioned here. As a result of this, if the Ethernet interface shown by GNS3 connection is e.g. eth1, it must considered as eth2 interface when IP address is being configured for the interface.

Another issue is related to the VirtualBox implementation. Comparing to GNS3 Qemu part, VirtualBox does not offer concept of base images. Every time is new the VitualBox Guest device needed, it must be manually created first in VirtualBox GUI and sequentially defined in GNS3 VirtualBox Guest settings window. If another GNS3 project is created, also the new VirtualBox machine has be created (or cloned) and defined. If they are five GNS3 projects and there is requirement of six VirtualBox Guests for each project, we have to create and define thirty Virtualbox machines.

Qemu clones implementation is much clever. Every time the new Qemu device is moved to GNS3 desktop, clone of base image is created behind the scene, without user’s intervention. Thanks to this concept, user only define path to Qemu base image in GNS3 setting and there is no need to define every single Qemu image.

Links

http://www.gns3.net/phpBB/topic3262.html?sid=b0d3fe94bfd162f87cee505a988e8517

http://www.if-not-true-then-false.com/2010/install-virtualbox-with-yum-on-fedora-centos-red-hat-rhel/

Part2 – OPENVSWICH – VLANs, Trunks, L3 VLAN interface, InterVLAN Routing – Configuration And Testing

In a previous tutorial we showed how to install Openvswitch on Qemu image with Microcore Linux. At the end of tutorial we created Openvswitch extension and submitted it to Microcore upstream. Assuming that Openvswitch is configured and functional, we are ready to make three labs which  helps us to test features such as VLANs, trunks and interVLAN routing.

LAB 1  Access VLAN Configuration And Testing

In this lab we are going to test connectivity between PCs which reside in same VLAN. PC1 and PC2 are assigned to VLAN 10 and PC3 and PC4 reside in VLAN 20. According to theory, each VLAN should have its own separate subnet. In our case, we are going to test not only connectivity between computers in the same VLANs but  also to test connectivity between PCs placed in different VLAN.

For this reason I chose one subnet 192.168.1.0/24 for all the computers so L3 connectivity exists between e.g. PC1 (VLAN 10) and PC3 (VLAN20). Of course, the computers assigned to different VLANs  are separated on Layer 2 so we do not expect any connectivity between those computers.

Figure 1 Openvswitch with configured VLANs on its access ports – click  image to enlarge

1) Openvswitch configuration

Login is tc without password set.

a) Let’s create bridge br0

sudo ovs-vsctl add-br br0

b) Add access port eth0 to the bridge br0 and assign it  to VLAN 10

sudo ovs-vsctl add-port br0 br0 eth0 tag=10

c) Assign remaining ports to the br0

sudo ovs-vsctl add-port br0 eth2 tag=10
sudo ovs-vsctl add-port br0 eth1 tag=20
sudo ovs-vsctl add-port br0 eth3 tag=20

d) Assign hostname to Microcore

sudo hostname openvswitch
echo "hostname openvswitch" >> /opt/bootlocal.sh

e) Save conf.db file to keep it persistent after the next reboot of Microcore Linux

/usr/bin/filetool.sh -b

2) PC1 configuration

Login is tc without password set.

Assign IP address 192.168.1.1/24 to eth0 and make it persistent after next reboot of Microcore

sudo hostname PC1
sudo ifconfig eth0 192.168.1.1 netmask 255.255.255.0

echo "hostname PC1" >> /opt/bootlocal.sh
echo "ifconfig eth0 192.168.1.1 netmask 255.255.255.0" >> /opt/bootlocal.sh

/usr/bin/filetool.sh -b

3) PC2 to PC4 configuration

Configure all remaining computers with correct IP paramters, similary as it was done in point 2)..

4) Access VLAN connectivity test

a) Connectivity test between PC which are residing in the same VLAN

Issue ping from PC1 (VLAN10) to PC2 (VLAN10).

tc@bPC1:~$ ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2): 56 data bytes
64 bytes from 192.168.1.2: seq=0 ttl=64 time=26.666 ms
64 bytes from 192.168.1.2: seq=1 ttl=64 time=0.000 ms
64 bytes from 192.168.1.2: seq=2 ttl=64 time=3.334 ms
64 bytes from 192.168.1.2: seq=3 ttl=64 time=0.000 ms

— 192.168.1.2 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.000/6.000/26.666 ms
tc@PC1:~$

Issue ping from PC3 (VLAN20) to PC4 (VLAN20).

tc@PC3:~$ ping 192.168.1.4
PING 192.168.1.4 (192.168.1.4): 56 data bytes
64 bytes from 192.168.1.4: seq=0 ttl=64 time=3.334 ms
64 bytes from 192.168.1.4: seq=1 ttl=64 time=3.333 ms
64 bytes from 192.168.1.4: seq=2 ttl=64 time=3.333 ms
64 bytes from 192.168.1.4: seq=3 ttl=64 time=0.000 ms

— 192.168.1.4 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.000/1.666/3.334 ms
tc@PC3:~$

As we have expected ping should have worked in both cases.

b) Test connectivity between computers placed to different VLANs

Issue ping from PC1 (VLAN10) to PC3 (VLAN20).

tc@PC1:~$ ping 192.168.1.4
PING 192.168.1.4 (192.168.1.4): 56 data bytes

— 192.168.1.4 ping statistics —
11 packets transmitted, 0 packets received, 100% packet loss

We can see that ping is not working between PC1 and PC3. We confirm that there is not connectivity between computers placed in to different VLANs and openvswitch is working correctly.

LAB 2  VLAN Trunk Configuration And Testing

In previous LAB we proved, that openvswitch only forwards traffic between computers which reside in the same VLAN.  In this LAB we are going to add additional switch to the topology – openvswitch2 and connect  switches with  link configured as 8021.1q trunk. The access VLAN 10, 20, 30 is added to openvswitch2 and VLAN 30 is added to  openvswitch. There is only  VLAN 10 and 30 allowed on both sides of the trunk.

Our goal is to show that only traffic from VLAN 10 and VLAN 30 is  forwarded between switches.

Figure 2 Two openvswitches connected with a link configured as 8021.q trunk – click  image to enlarge

1) Openvswitch configuration

Assuming that the previous configuration of openvswitch has been stored, we only add following commands.

a) Add access port eth4 to the bridge br0 and assign it to VLAN 30

sudo ovs-vsctl add-port br0 eth4 tag=30

b) Add trunk port eth5 to bridge br0 and allow VLAN 10 and 30 on the trunk

sudo ovs-vsctl add-port br0 eth5 trunks=10,30

c) Save configuration of Microcore

/usr/bin/filetool.sh -b

2) Openvswitch2 configuration

As it was explained every single step of configuration of openvswitch in previous examples I am going to skip it now.

sudo hostname openvswitch2
echo "hostname openvswitch2" >> /opt/bootlocal.sh

sudo ovs-vsctl add-br br0
sudo ovs-vsctl add-port br0 eth5 trunks=10,30
sudo ovs-vsctl add-port br0 eth0 tag=30
sudo ovs-vsctl add-port br0 eth1 tag=20
sudo ovs-vsctl add-port br0 eth2 tag=10

/usr/bin/filetool.sh -b

Similarity, configure IP parameters for the PC5, PC6 to PC8 according to the topology.

3) Trunk VLAN test

a) Issue the ping from PC1 (VLAN10) to PC8 (VLAN10)

tc@PC1:~$ ping 192.168.1.8
PING 192.168.1.8 (192.168.1.8): 56 data bytes
64 bytes from 192.168.1.8: seq=0 ttl=64 time=6.667 ms
64 bytes from 192.168.1.8: seq=1 ttl=64 time=3.333 ms
64 bytes from 192.168.1.8: seq=2 ttl=64 time=0.000 ms
64 bytes from 192.168.1.8: seq=3 ttl=64 time=3.334 ms
^C
— 192.168.1.8 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.000/3.333/6.667 ms
tc@PC1:~$

We can see that traffic in VLAN 10 is succesfully transferred through trunk. If the same VLAN exists on two switches we call it end-to-end VLAN.

b) Issue the ping from PC5 (VLAN30) to PC6 (VLAN30)

tc@PC5:~$ ping 192.168.1.6
PING 192.168.1.6 (192.168.1.6): 56 data bytes
64 bytes from 192.168.1.6: seq=0 ttl=64 time=6.667 ms
64 bytes from 192.168.1.6: seq=1 ttl=64 time=3.334 ms
64 bytes from 192.168.1.6: seq=2 ttl=64 time=3.333 ms
64 bytes from 192.168.1.6: seq=3 ttl=64 time=3.333 ms
^C
— 192.168.1.6 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 3.333/4.166/6.667 ms
tc@PC5:~$

From what we see, trunk  is successfully carrying the traffic marked with tag 30 between switches.

c) Issue the ping from PC3 (VLAN20) to PC7 (VLAN20)

tc@PC3:~$ ping 192.168.1.7
PING 192.168.1.7 (192.168.1.7): 56 data bytes

— 192.168.1.7 ping statistics —
6 packets transmitted, 0 packets received, 100% packet loss
tc@PC3:~$

Traffic from vlan 20 is not forwarded between switches. It is because VLAN 20 is not allowed on the trunk. If there would be need to allow all the VLANs on the trunk, only parameter trunks (without specific VLAN) should be presented.

LAB 3  L3 VLAN interfaces, InterVLAN routing – Configuration And Testing

In this last LAB we are going to create L3 VLAN interface for VLAN 10, 20, 30. After that we will check if Microcore can forward traffic between different VLANs.  We assume that particular ports were added to  VLANs in previous LABs and only additional configuration will be shown in this LAB.  I have also changed a topology to make it more simple and create the new IP plan for the devices.

Figure 3 Topology for InterVLAN routing testing – click  image to enlarge

1) Openvswitch configuration

a) Add an internal port vlan10 to bridge br0 as a VLAN access port for VLAN 10

sudo ovs-vsctl add-port br0 vlan10 tag=10 -- set interface vlan10 type=internal

b) Create vlan20 and vlan30 interfaces

sudo ovs-vsctl add-port br0 vlan20 tag=20 -- set interface vlan20 type=internal
sudo ovs-vsctl add-port br0 vlan30 tag=30 -- set interface vlan30 type=internal

c) Add access port eth4 and eth5 to the bridge br0 and assigned them to VLAN 30

First, We need to delete configuration for eth4 and eth5 that was done for LAB1
and LAB2.

sudo ovs-vsctl del-port br0 eth4
sudo ovs-vsctl del-port br0 eth5

Now we can assign ports to the VLAN 30.

sudo ovs-vsctl add-port br0 eth4 tag=30
sudo ovs-vsctl add-port br0 eth5 tag=30

d) Assign IP adresses to the VLAN interfaces

sudo ifconfig vlan10 192.168.10.254 netmask 255.255.255.0
sudo ifconfig vlan20 192.168.20.254 netmask 255.255.255.0
sudo ifconfig vlan30 192.168.30.254 netmask 255.255.255.0

echo "ifconfig vlan10 192.168.10.254 netmask 255.255.255.0" >> /opt/bootlocal.sh
echo "ifconfig vlan20 192.168.20.254 netmask 255.255.255.0" >> /opt/bootlocal.sh
echo "ifconfig vlan30 192.168.30.254 netmask 255.255.255.0" >> /opt/bootlocal.sh

/usr/bin/filetool.sh -b

e) Enable IPv4 and IPV6 packets forwarding between interfaces

Forwarding between network interfaces is disabled by default. To activate ipv4
and ipv6 forwarding for Microcore you need to enable it:

sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv6.conf.all.forwarding=1

echo "sysctl -w net.ipv4.ip_forward=1" >> /opt/bootlocal.sh
echo "sysctl -w net.ipv6.conf.all.forwarding=1" >> /opt/bootlocal.sh

/usr/bin/filetool.sh -b

2) PC1 to PC6 – IP address, netmask, default gateway – configuration

a) Delete old IP addresses from the computers

We should split IP address space in such way when each VLAN would have assigned separate subnet. First, for each computer delete a row with old IP address from /opt/bootlocal.sh which was previously assigned in LAB1 (use dd command for delete of particular row).

b) Assign IP address and default GW to PC1 to PC6

Example for PC1:

sudo ifconfig eth0 192.168.10.1 netmask 255.255.255.0
sudo route add default gw 192.168.10.254

echo "ifconfig eth0 192.168.10.1 netmask 255.255.255.0" >> /opt/bootlocal.sh
echo "route add default gw 192.168.10.254" >> /opt/bootlocal.sh
/usr/bin/filetool.sh -b

Similiary, do it for all the PCs, according to topology and IP address plan.

3) InterVLAN routing testing

As long as PCs have had correct IP address, netmask and default gateway assigned, we can either ping PC in the same VLAN or any other vlan interface.

a) Issue ping from PC1 (VLAN10) to PC2 (VLAN10)

tc@PC1:~$ ping 192.168.10.2
PING 192.168.10.2 (192.168.10.2): 56 data bytes
64 bytes from 192.168.10.2: seq=0 ttl=64 time=3.333 ms
64 bytes from 192.168.10.2: seq=1 ttl=64 time=3.333 ms
64 bytes from 192.168.10.2: seq=2 ttl=64 time=3.334 ms
64 bytes from 192.168.10.2: seq=3 ttl=64 time=3.333 ms

— 192.168.10.2 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 3.333/3.333/3.334 ms
tc@PC1:~$

b) Issue ping from PC1 (VLAN10) to interface vlan20

tc@PC1:~$ ping 192.168.20.254
PING 192.168.20.254 (192.168.20.254): 56 data bytes
64 bytes from 192.168.20.254: seq=0 ttl=64 time=3.333 ms
64 bytes from 192.168.20.254: seq=1 ttl=64 time=0.000 ms
64 bytes from 192.168.20.254: seq=2 ttl=64 time=0.000 ms
64 bytes from 192.168.20.254: seq=3 ttl=64 time=3.333 ms

— 192.168.20.254 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.000/1.666/3.333 ms
tc@PC1:~$

We can see that in both cases, ping was succesfull.

c) Issue the ping from PC1 (VLAN10) to PC3 (VLAN20)

tc@PC1:~$ ping 192.168.20.1
PING 192.168.20.1 (192.168.20.1): 56 data bytes
64 bytes from 192.168.20.1: seq=0 ttl=63 time=0.000 ms
64 bytes from 192.168.20.1: seq=1 ttl=63 time=3.333 ms
64 bytes from 192.168.20.1: seq=2 ttl=63 time=3.333 ms
64 bytes from 192.168.20.1: seq=3 ttl=63 time=3.334 ms

— 192.168.20.1 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.000/2.500/3.334 ms
tc@PC1:~$

Thanks to enabled forwarding between interfaces in kernel, ping is working between different VLANS. Microcore with Openvswitch is acting as the Layer3 switch.

End.

Building the Linux Microcore Kernel for 802.1Q Support

UPDATE This tutorial is valid for Microcore Linux up to version 3.8.x, kernel 2.6.33.3.  The newest Microcore 4.0, kernel  3.0.3 loads 8021q module without any difficulties.

Recently I have decided to add support for 802.1Q to Microcore Linux. First I  tried to install  8021q-2.6.33.3-tinycore.tcz extension which is available for download  in the Tinycore Linux 3.x repository. This extension is 8021q module that we can simply load with command modprobe 8021q to to the kernel.   Unfortunately, after I tried to load the module I always kept this error message:

modprobe: can't load module 8021q (kernel/net/8021q/8021q.ko.gz): unknown symbol in module, or unknown parameter

The output from dmesg  said:

8021q: Unknown symbol vlan_dev_vlan_id

This was the same error which  was originally presented in this topic  but there was not solution mentioned there except of simply saying – it is working now. Unfortunately, I deal with the error even when I had compiled 8021q module by my own. For this reason I decided to compile my own linux kernel with direct support for 8021q Vlan tagging.

The kernel version 2.6.33.3 is the kernel which is used by Linux Microcore 3.x. These are the steps you need to follow to compile and install the kernel:

1. Install Microcore Linux

I assume you have your Microcore already installed.  I mainly use  Linux Microcore installed on Qemu image  in GNS3 environment  to  simulate PC hosts on LAN. In order to start your Microcore Qemu image invoke this command:

qemu -no-acpi -boot c linux-microcore-3.6.img -m 1500M

I have noticed we need to have enough RAM assigned for the image thus to use of   tmpfs in which are files stored.   Also the content saved in this filesystem  will be lost after reboot.  To avoid lost of new kernel we need later to do some steps to have kernel  kept in /usr/src/ after reboot.

2. Download and untar Linux kernel 2.6.33.3 which is patched for Microcore Linux

sudo su
mkdir /usr/src/
cd /home/tc/

wget http://distro.ibiblio.org/tinycorelinux/3.x/release/src/kernel/linux-2.6.33.3-patched.tbz2

tar jxvf linux-2.6.33.3.tar.bz2 -C /usr/src

Note: if you want to keep kernel persistent in /usr/src/ you have to edit /opt/.filetool.lst and add /usr/src/ to the list of after-restart kept directories.

After that you have to call /usr/bin/filetool -b

3. Install tools for compiling, download an original .config kernel file

tce-load -wi compiletc.tcz ncurses-dev perl5

Note: perl is needed by make command.

cd /usr/src/

Download kernel config file.

wget http://distro.ibiblio.org/tinycorelinux/3.x/release/src/kernel/config-2.6.33.3-tinycore

Create symbolic link /usr/src/linux pointed to /usr/src/linux-2.6.33.3 and copy config file to /usr/src/linux.

ln -sf linux-2.6.33.3 linux
cp ./config-2.6.33.3-tinycore ./linux/.config

4. Enable 8021q support in kernel config file,  build the kernel with 8021q support

a) Enable 8021q support – change kernel config file

make oldconfig
make menuconfig

Navigate to:

Network support-> Networking Options-> 8021q Support

Enter Y to include 8021q VLAN support. Exit and Save configuration to .config file.

b/ Build the  kernel

make bzImage

After finishing, your new kernel bzImage is saved to /usr/src/linux/arch/x86/boot/. Replace old kernel with the new one and reboot. After the boot,  check the version of the built kernel with uname -ra command.

mv /usr/src/linux/arch/x86/boot/bzImage /mnt/hda1/boot/

Note: In case you don’t want to bother yourself with building your kernel, you can download my bzImage. It is  the kernel 2.6.33.3, dedicated for Microcore 3.x.   Just replace /mnt/hda1/boot/bzImage with this file.

http://www.4shared.com/file/VWMmD6BC/bzImage.html

Links:

http://wiki.tinycorelinux.net/wiki:custom_kernel
http://www.fogonacaixadagua.com.br/2009/09/how-to-compile-a-new-kernel-in-linux-centos-red-hat/
http://www.cyberciti.biz/tips/compiling-linux-kernel-26.html
http://www.digitalhermit.com/linux/Kernel-Build-HOWTO.html

GNS3 Qemu Troubleshooting

Many GNS3 users use Qemu emulator to run various network OS installed on Qemu images – for example Olive, ASA, Vyatta and many others. Some users complain about not having a network connections between these devices or a console window appears and disappears  after the short time. In both cases Qemu hosts are not started properly and Qemu images don’t work.

This issues are typically caused by entering a wrong parameter in GNS3 setting.  For instance  when user checks a box “Use kvm” in GNS3 settings and CPU doesn’t support kvm or kvm is not enabled in BIOS, it ends with warning message “unknown parameter kvm” and Qemu image is not started. Please remember that  kvm is not supported by Windows so you shouldn’t have  “Use kvm” checked if GNS3 is installed on Windows.

Another issue mainly occurs on Linux and Mac OS and it is caused by using unpatched Qemu binary. GNS3 create UDP tunnels in order to make connection between Qemu devices  and that is why  Qemu source have to be patched for UDP tunnels, compiled and installed by user. Although many Linux distributions already have Qemu installed in /usr/bin/ directory, this version of Qemu is unlikely to be patched for UDP tunnels. For this reason the Qemu doesn’t understand parameter -net type udp and it is halted after its start.

As long as patched Qemu binary is included in GNS3 all-in-one package, the Windows users  don’t need to patch, compile and install Qemu by themselves.

Except of the wrong parameters I have already mentioned there can be another problems which  prevent Qemu  starting successfully.  If you find Qemu difficult to start you should consider to do following troubleshooting to find out the reasons why Qemu doesn’t work.

1/ Start qemuwrapper.py script from Linux terminal console.  The script is located in a in  the GNS3 root directory, in the directory ./qemuwrapper

sudo /home/brezular/Download/GNS3-files/GNS3/Stable/GNS3-0.7.3-src/qemuwrapper/qemuwrapper.py

Qemu Emulator Wrapper (version 0.7.3)
Copyright (c) 2007-2010 Thomas Pani & Jeremy Grossmann

Unpacking pemu binary.
Qemu TCP control server started (port 10525).
Listenning on all network interfaces

The script has started Qemuwrapper server on TCP port 10525.

2/ Start GNS3 application

Create a new blank project and save it. Drug two Qemu hosts or Junipers from left panel and drop them in to the GNS3 Desktop (Figure 1). You should see a warning message – Qemu is already running on port 10525, it will not be shutdown after you quit GNS3.  It simply means that GNS3 has tried to start Qemu wrapper server on TCP port 10525 but another Qemuwrapper server is running on this port. We started it by ourselves with qemuwrapper script.

It is time to make connection between nodes . Connect them with Ethernet link and  start both Qemu hosts.

Figure 1. Qemu hosts connected through e0 interfaces

3/ Check the output of qemuwrapper script in console window. It should look like this:

Qemu path is now /usr/bin/qemu
Qemu-img path is now /usr/bin/qemu-img
!! QEMU1.console = 3000
!! QEMU1.netcard = e1000
!! QEMU1.image = /home/brezular/Download/GNS3-files/PC/Microcore/Stable/Host/3.4/linux-microcore-3.4.img
!! QEMU1.ram = 48
!! QEMU1.options = -no-acpi -nographic
!! QEMU2.console = 3001
!! QEMU2.netcard = e1000
!! QEMU2.image = /home/brezular/Download/GNS3-files/PC/Microcore/Stable/Host/3.4/linux-microcore-3.4.img
!! QEMU2.ram = 48
!! QEMU2.options = -no-acpi -nographic
command: ['/usr/bin/qemu', '-name', 'QEMU1', '-m', '48', '/tmp/QEMU1/FLASH', '-hdb', '/tmp/QEMU1/SWAP', '-net', 'nic,vlan=0,macaddr=00:aa:00:18:6c:00,model=e1000', '-net', 'udp,vlan=0,sport=20003,dport=20002,daddr=127.0.0.1','-net', 'nic,vlan=1,macaddr=00:00:ab:da:3d:01,model=e1000', '-net', 'nic,vlan=2,macaddr=00:00:ab:c3:46:02,model=e1000', '-net', 'nic,vlan=3,macaddr=00:00:ab:e9:c9:03,model=e1000', '-net', 'nic,vlan=4,macaddr=00:00:ab:77:f1:04,model=e1000', '-net', 'nic,vlan=5,macaddr=00:00:ab:f2:3c:05,model=e1000', '-serial', 'telnet::3000,server,nowait', '-no-acpi', '-nographic']
Invalid -net type ‘udp’
pid: 30268
Renicing to 19
30268: old priority 0, new priority 19
command: ['/usr/bin/qemu', '-name', 'QEMU2', '-m', '48', '/tmp/QEMU2/FLASH', '-hdb', '/tmp/QEMU2/SWAP', '-net', 'nic,vlan=0,macaddr=00:aa:00:98:21:00,model=e1000', '-net', 'udp,vlan=0,sport=20002,dport=20003,daddr=127.0.0.1', '-net', 'nic,vlan=1,macaddr=00:00:ab:09:1d:01,model=e1000', '-net', 'nic,vlan=2,macaddr=00:00:ab:26:50:02,model=e1000', '-net', 'nic,vlan=3,macaddr=00:00:ab:25:86:03,model=e1000', '-net', 'nic,vlan=4,macaddr=00:00:ab:6d:a3:04,model=e1000', '-net', 'nic,vlan=5,macaddr=00:00:ab:53:34:05,model=e1000', '-serial', 'telnet::3001,server,nowait', '-no-acpi', '-nographic']
Invalid -net type ‘udp’
pid: 30367
Renicing to 19
30367: old priority 0, new priority 19

The green lines refer to Qemu1 device. They are several parameters there such as RAM , console port, netcard type and path to  the base image. Please, notice QEMU1.options  parameters – no-acpi and -nographic.  If you have any problems with Qemu it is probably a good idea to leave this Qemu option box blank in GNS3 setting. Likewise blue lines refer to QEMU2 device.

As you can see from Figure 1 QEMU1 and QEMU2 are connected together via their eth0 interfaces.  They both use UDP tunnels for traffic transferred between them – QEMU1 sends all the traffic destined for QEMU2 to IP address 127.0.0.1,  destination UDP port 20002. QEMU2 is listening to traffic from QEMU1 on its source UDP port 20002. Likewise QEMU2 sends traffic destined for QEMU1 to IP address 127.0.0.1, destination UDP port 20003 and QEMU1 is listening to traffic from QEMU2 on its source port 20003.

If the Qemu had been properly patched for UDP tunnels a network connection would have been working between Qemu hosts.  For this demonstration  unpatched qemu binary in /usr/bin/ directory is used thus Qemu is stopped with  an error message Invalid -net type ‘udp’. If I edit .net file and set qemu path to /usr/local/bin/qemu ,  hosts will be started correctly because version of Qemu located in /usr/local/bin/ directory  is patched for UDP tunnels.

4/ Conclusion

You should have now a brief  idea of Qemu  troubleshooting for GNS3.  At least you know how to get an output of qemuwrapper and post it on GNS3 forum.  More about Qemu patching is on GNS3 blog:

http://blog.gns3.net/2009/10/olive-juniper/2/

In addition to stable UDP and multicast patch which is available for download in GNS3 Download section there is continuously  updated  patch for almost each new Qemu version in the  GNS3 Development section.  As I wrote Windows users don’t need to patch source by themselves due to the fact that GNS3 all-in-one package contains patched Qemu version.

Dynamips-0.2.8-RC3 Installation from Source on Linux Fedora

Hi,

these are the steps for compiling and installing Dynamips 0.2.8-RC3 (community version) under Fedora 13 Linux, 32 bit.  Windows users can download binary version here:

1. http://sourceforge.net/projects/gns-3/files/Dynamips/0.2.8-RC3-community/dynamips-0.2.8-RC3-community.tar.gz/download

2. tar zxvf ./dynamips-0.2.8-RC3-community.tar.gz

3. cd ./dynamips-0.2.8-RC3-community

4. sudo yum install libpcap elfutils-libelf.i686 elfutils-libelf-devel.i686 libuuid libuuid-devel gcc libpcap libpcap-devel

5. cd /usr/lib/;  sudo ln -s ./libelf.so ./libelf.a

6. cd –

7. make

Binary dynamips.stable is created in the current directory. Copy it to /usr/local/bin/.

sudo cp ./dynamips.stable /usr/local/bin/dynamips-0.2.8-RC3-x86.bin

You can download binary here:

http://www.4shared.com/file/R2bqbQhy/dynamips-028-RC3-x86.html

Used links:

http://www.gns3.net/phpBB/topic2995.html?sid=0ae9521e778db3d71d9fb2e804309e9a

http://www.davidsudjiman.info/2007/06/12/compiling-dynamips-on-intel-mac/