Friday, March 30, 2012

OSPF Type 1 - 5 LSA

In this video Keith Barker walks through the 5 most common OSPF LSA types, with examples of each as seen on a live network.

OSPF LSA Types, part 1 

I also recommend you get the CBT nuggets CCNP route series. I have the series and watched OSPF Routing: Foundation Concepts, Part 1 and OSPF Routing: Foundation Concepts, Part 2 for a second time.  I highly recommend the series and pretty much anything from CBT..

CBT NUGGETS

Thursday, March 29, 2012

Changing the MAC Address Aging Time

Depending on how your network is built, its sometimes is best to increase the aging time of mac-addresses(CAM) to reduce the unnecessary flooding that is caused by the default time out value of 5 minutes. You also have to take caution not to fill up the cam table by having to many stale entries.

CAM—All Catalyst switch models use a CAM table for Layer 2 switching. As frames arrive on switch ports, the source MAC addresses are learned and recorded in the CAM table. The port of arrival and the VLAN are both recorded in the table, along with a timestamp. If a MAC address learned on one switch port has moved to a different port, the MAC address and timestamp are recorded for the most recent arrival port. Then, the previous entry is deleted. If a MAC address is found already present in the table for the correct arrival port, only its timestamp is updated.


Command:

mac address-table aging-time [0 | 10-1000000] [vlan vlan-id]
   
Verification:
show mac address-table aging-time

Recommended Read:
http://www.techexams.net/forums/ccnp/41520-cam-table-aging.html

Wednesday, March 28, 2012

Monday, March 26, 2012

Adding Interfaces to an Existing PAgP Port-channel


I always like to default the interfaces if there is an existing configuration or you can nonnegate what ever is still configured.  Additionally, etherchannel is a Cisco term, port-channel is an industry term but they are the same and you will hear people use them interchangeably.

Default interface X/X
!
Default Interface X/X

Next we view what the current configuration is on the existing interfaces that are already members of the port-channel and we copy and paste that same configuration into the new interfaces. If you don't, they wont be added to the port-channel as it wont match the port-channels current configuration. Going forward, any configurations that need to be made, will need to be done under the port-channel.

Verification.
show etherchannel summary

Extra information from Wikipedia.

EtherChannel aggregates the traffic across all the available active ports in the channel. The port is selected using a Cisco-proprietary hash algorithm, based on source or destination MAC addressesIP addresses or TCP and UDP port numbers. The hash function gives a number between 0 and 7, and the following table shows how the 8 numbers are distributed among the 2 to 8 physical ports. In the hypothesis of real random hash algorithm, 2, 4 or 8 ports configurations lead to fair load-balancing, whereas other configurations lead to unfair load-balancing.

Hashing Algorithm


Number of Physical Links Link #8765432 11222234 21122234 3111222 411112 51111 6111 711 81

OSPF Feature Summary & EIGRP

Transport
IP, protocol type 89(Does not use UDP or TCP. EIGRP uses Type 88.

Metric
Based on cumulative cost of all outing interfaces in a route. The interface cost defaults to a function of interface bandwidth but can be set explicitly. (IP OSPF Bandwidth value). EIGRP uses, Load, Reliability, bandwidth and delay. (BW and DW are default).

Hello Interval
interval at which a router send OSPF Hello messages on an interface.

Dead Interval
Timer used to determine when a neighboring router has failed, based on a router not receiving any OSPF messages, including Hellos in this timer period.

Update destination address
Normally sent to 224.0.0.5(All SPF routers( and 224.0.0.6(All Designated routers)

Full or Partial Updates
Full updates are used when new neighbors are discovered, otherwise partial updates are used.

Authentication:
supports MD5 and clear-test authentication.(IPV4)

VLSM/ClasslessOSPF includes the mask with each route, also allowing it to support discontiguous networks and VLSM.

Route Tags
allows OSPF to tag routes as they are redistributed into OSPF

next-hop field
supports the advertisement of routes with a different next-hop router than the advertising router.

manual route summarization
Allows route summarization at ABR routers only. Eigrp you can summarize on any device per interface.









Sunday, March 25, 2012

Most Commonly Used OSPF Terms & Commands

The terms and commands come from the Cisco press CCNP Route Book. A Great read and a must for the CCNP. This is not a copy a paste, I typed it up to help me memorize them and I added a little extra to some of them.


Source:

Most Commonly Used OSPF terms:

Link state database
The data structure held by an ospf router for the purpose of storing topology data. 

Shortest path first(SPF)
The name of the algorithm OSPF uses to analyze the LSDB. The analysis determines the best(Lowest cost) route for each prefix/length. You might also hear people refer to this as Dijkstra's algorithm.

Link state Update
The name of the OSPF packet that holds the detailed topology information, specifically LSAs. LSA's are sent within the LSU.

Link state advertisement(LSA)
The name of a class of OSPF data structures that hold topology information. LSAs are helf in memory in the LSDB(Topology Table) and communicated over the network in LSU(Link State Updates) messages.

Area
A Contiguous grouping of routers and router interfaces. Routers in an area strive to learn all topology information about the area, but they do not learn topology information about areas to which they do not connect. An ABR will send summarize information about other areas to devices in other areas. 

Area border router (ABR)
A router that has interfaces connected to at least two different OSPF areas, including the backbone area. ABRs hold topology data for each area, and calculate routes for each area, and advertise about those routes between areas.

Backbone router
Any router that has at least one interface connected to the backone area. Most of the time the backbone area is AREA 0. In order for one area to speak to another area they most go through the backbone area.

Internal routers 
A router that has interface connected to only one area, making the router completely internal to that one area. 

Designated Router(DR).
On multiaccess data links like LANS, an OSPF router elected by the routers on that data link to perform special functions. These functions include the generation of LSAs representing the subnet, and playing a key roll in the database exchange process. 

Backup Designated Router(BDR) 
A router on a multiaccess data link that monitors the DR and becomes prepared to take over for the DR, should the DR fail. All devices within the same VLAN/Subnet establish a full adjacency with the DR and BDR.



Most Commonly Used OSPF commands:

show ip ospf interface brief 
lists the interfaces on which OSPF is enabled (based on the network commands and area command in interface mode); it omits passive interfaces.

show ip protocols
Lists the contents of the network configuration commands for each routing process, and list of enabled but passive interfaces.

show ip ospf neighbors
lists known neighbors, including neighbor state; does not list neighbors for which some mismatched parameter is preventing a valid OSPF neighbor relationship. You could have two devices listed under this command that have a mismatched mtu size.

show ip ospf database
Lists all LSAs for all connected areas. (Type 1-7)

show ip route
Lists the contents of the IP routing table, listing OSPF-learned routes with a code of O on the left side of the output.










Wednesday, March 21, 2012

OSPF MTU

I was going to write something up for the MTU size in OSPF and then I came across a good article. Why not share it and save me some time.

OSPF MTU - Cisco

OSPF - Router ID

In OSPF you can not have devices with duplicate router ID's. In the hello message there is a field that indicated the router ID, two devices with the same router ID with now form a neighborship. However, it does not mean that two devices with the same router ID cannot form a neighborship with another device. When you have two devices with the same ID it confuses the other device because of the way the topology database flooding works.

We will use diagram 1 for this write up.

On R3, you can see it's happy and has a neighborship with its peers. Routes looks good as well.

R3#show ip ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1          10   FULL/DR         00:00:39    192.168.10.1    FastEthernet0/0
2.2.2.2           1   FULL/DROTHER    00:00:30    192.168.10.2    FastEthernet0/0
R3#
R3#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
     100.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
O       100.100.100.1/32 [110/2] via 192.168.10.1, 00:00:04, FastEthernet0/0
C       100.100.103.0/24 is directly connected, Loopback103
O       100.100.102.1/32 [110/2] via 192.168.10.2, 00:00:04, FastEthernet0/0
C    192.168.10.0/24 is directly connected, FastEthernet0/0
R1#show ip ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2           1   FULL/DROTHER    00:00:33    192.168.10.2    FastEthernet0/0
3.3.3.3           1   FULL/BDR        00:00:32    192.168.10.3    FastEthernet0/0
R1#

Now we are going to change the router id on R2 to be 1.1.1.1 like R1 and see what happens. We are also going to turn on debugging R1#debug ip ospf adj


R1 Start complaining about a duplicate router ID.
R1#
*Mar  1 00:36:42.675: OSPF: Rcv LS UPD from 2.2.2.2 on FastEthernet0/0 length 76 LSA count 1
*Mar  1 00:36:42.731: %OSPF-4-DUP_RTRID_NBR: OSPF detected duplicate router-id 1.1.1.1 from 192.168.10.2 on interface FastEthernet0/0
Even though the routers have duplicate ID's R3 still forms a neighborship.

R3#show ip ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1          10   FULL/DR         00:00:39    192.168.10.1    FastEthernet0/0
1.1.1.1           1   FULL/DROTHER    00:00:38    192.168.10.2    FastEthernet0/0
R3#
The routing table is only displaying routes from R1 and not R2. This could happen in reverse, it depends which neighbor comes up first. Eventually that route will disappear, its not a good situation.

R3#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
     100.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
O       100.100.100.1/32 [110/2] via 192.168.10.1, 00:00:07, FastEthernet0/0
C       100.100.103.0/24 is directly connected, Loopback103
C    192.168.10.0/24 is directly connected, FastEthernet0/0
Checking the OSPF database for router ID 1.1.1.1. This further confirms that you need to have unique router id's because of how the OSPF database flooding works.

R3#show ip ospf database router 1.1.1.1
            OSPF Router with ID (3.3.3.3) (Process ID 1)
                Router Link States (Area 0)
  Adv Router is not-reachable
  LS age: 5
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 1.1.1.1
  Advertising Router: 1.1.1.1
  LS Seq Number: 80000056
  Checksum: 0x8429
  Length: 48
  Number of Links: 2
    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 100.100.100.1
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metrics: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.10.1
     (Link Data) Router Interface address: 192.168.10.2
      Number of TOS metrics: 0
       TOS 0 Metrics: 1

OSPF Authentication - IPV4

In ospf you have three different authentication types and two different ways to configure them. The three types are in table 1 and very brief  description.

Table 1

Type #Description
Type 0 no authentication(Default)
Type 1Clear Text
Type 2MD5


We are going to use Diagram 1 for this write up.

Diagram 1
 Configuring Authentication:
You can enable MD5 authentication globally for the entire area via router configuration mode or you can enable it on a per interface and the per interface overrides the global configuration method. You also still need to define the key under the interface, I think it's just easier and more formal to configure it under the interface and not globally.

Enable MD5 Auth on all interface in Area 0

R1(config)#router ospf 1
R1(config-router)#area 0 authentication message-digest

Clear Text - Type 1

R1(config-if)#ip ospf authentication
R1(config-if)#ip ospf authentication-key joel

MD5 - Type 2

R1(config-if)#ip ospf authentication message-digest
R1(config-if)#ip ospf message-digest-key 1 md5 joel

You can also configure more then one key but note that you will send duplicate hello's out, one for each key. In eigrp you have the luxury of using a key-chain setup that allows you to configure expiration dates on keys. When you configure expiration dates, make sure your devices are using NTP for time sync configuration or PTP - 1588.

Verifying Authentiion Settings:

You can see all the way at the bottom of this output that we are using MD5. The only way to truly confirm all your configurations are correct is to verify you have established a neighborship.

R1#show ip ospf int f0/0
FastEthernet0/0 is up, line protocol is up
  Internet Address 192.168.10.1/24, Area 0
  Process ID 1, Router ID 1.1.1.1, Network Type BROADCAST, Cost: 1
  Enabled by interface config, including secondary ip addresses
  Transmit Delay is 1 sec, State DR, Priority 10
  Designated Router (ID) 1.1.1.1, Interface address 192.168.10.1
  Backup Designated router (ID) 2.2.2.2, Interface address 192.168.10.2
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    Hello due in 00:00:01
  Supports Link-local Signaling (LLS)
  Index 1/1, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 1, maximum is 3
  Last flood scan time is 0 msec, maximum is 0 msec
  Neighbor Count is 1, Adjacent neighbor count is 1
    Adjacent with neighbor 2.2.2.2  (Backup Designated Router)
  Suppress hello for 0 neighbor(s)
  Message digest authentication enabled
    Youngest key id is 1


R1#show ip ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2           1   FULL/BDR        00:00:39    192.168.10.2    FastEthernet0/0
R1#

Monday, March 19, 2012

Useful Linux/Ubuntu Commands

Lately I have been doing a lot more Linux troubleshooting as I have been tasked to setup some new sniffer appliances. Below are some of the commands I have been using to assist me with the process. If you want to transfer files via ssh you can use Winscp if you are going from a Windows pc.

su = Become super user or another user.

cd = Changes the directory.

ls - l = shows you huge amounts of information (permissions, owners, size, and when last modified.)

cat = Allows you to look, modify or combine a file.

nano= Text Editor.(Easier to use then VI)

VI = Text Editor(Not for new users)

ifconfig = view information about the configured network interfaces

netstat -rn = Displays routing table for all ips bound to the server. Useful for seeing the default gateway


rm= Deletes a file without confirmation (by default).


Lets see some of the commands in action:



ubuntu@ubuntu:~$ ls -l
total 0
drwxr-xr-x 2 ubuntu ubuntu 80 2012-03-19 23:33 Desktop
drwxr-xr-x 2 ubuntu ubuntu 40 2012-03-19 19:19 Documents
drwxr-xr-x 2 ubuntu ubuntu 40 2012-03-19 19:19 Downloads
drwxr-xr-x 2 ubuntu ubuntu 40 2012-03-19 19:19 Music
drwxr-xr-x 2 ubuntu ubuntu 40 2012-03-19 19:19 Pictures
drwxr-xr-x 2 ubuntu ubuntu 40 2012-03-19 19:19 Public
drwxr-xr-x 2 ubuntu ubuntu 40 2012-03-19 19:19 Templates
drwxr-xr-x 2 ubuntu ubuntu 40 2012-03-19 19:19 Videos






ubuntu@ubuntu:~$ cd Desktop
ubuntu@ubuntu:~/Desktop$


ubuntu@ubuntu:~/Desktop$ ifconfig
eth0      Link encap:Ethernet  HWaddr 5c:26:0a:45:2e:36
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:f6900000-f6920000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:60 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4512 (4.5 KB)  TX bytes:4512 (4.5 KB)

wlan0     Link encap:Ethernet  HWaddr 00:24:d7:99:fc:f0
          inet addr:192.168.0.102  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::224:d7ff:fe99:fcf0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8129 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5385 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7261637 (7.2 MB)  TX bytes:1267545 (1.2 MB)

ubuntu@ubuntu:~/Desktop$







ubuntu@ubuntu:~/Desktop$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.0.0     0.0.0.0         255.255.255.0   U         0 0          0 wlan0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 wlan0
0.0.0.0         192.168.0.1     0.0.0.0         UG        0 0          0 wlan0
ubuntu@ubuntu:~/Desktop$



ubuntu@ubuntu:~/Desktop$ ls -l
total 24
-rwxr-xr-x 1 ubuntu ubuntu  203 2012-03-19 19:19 examples.desktop
-rw-r--r-- 1 ubuntu ubuntu 4131 2012-03-19 23:42 readme.txt
-rwxr-xr-x 1 ubuntu ubuntu 8329 2012-03-19 19:19 ubiquity-gtkui.desktop

ubuntu@ubuntu:~/Desktop$ cat readme.txt
=== Plugin Name ===
Contributors: markjaquith, mdawaffe (this should be a list of wordpress.org userid's)
Donate link: http://example.com/
Tags: comments, spam

ubuntu@ubuntu:~/Desktop$ ls -l 

total 24
-rwxr-xr-x 1 ubuntu ubuntu  203 2012-03-19 19:19 examples.desktop
-rw-r--r-- 1 ubuntu ubuntu 4131 2012-03-19 23:42 readme.txt
-rwxr-xr-x 1 ubuntu ubuntu 8329 2012-03-19 19:19 ubiquity-gtkui.desktop
ubuntu@ubuntu:~/Desktop$ rm readme.txt


ubuntu@ubuntu:~/Desktop$ ls -l
total 16
-rwxr-xr-x 1 ubuntu ubuntu  203 2012-03-19 19:19 examples.desktop
-rwxr-xr-x 1 ubuntu ubuntu 8329 2012-03-19 19:19 ubiquity-gtkui.desktop
ubuntu@ubuntu:~/Desktop$ 


Recommended links:


Unix, Linux, and variants

WinScp

OSPF - Router ID & Influencing DR Election

In ospf, every neighbor needs to have a unique router ID. This is crucial, if not you will have an unstable ospf database.Once a unique router ID is selected, routers will decide who becomes the DR/BDR in the broadcast network.

I will explain how ospf selects a router ID and how to influence the DR election process. The purpose of the DR is to form a full adjancy with its nighbors and those neighbors advertise there updates to it and then in turn the DR advertises those updates to its fully adjacent neighhors. Its very efficient since this is all done via multicast address 224.0.0.5 (ALL OSPF ROUTERS) and 224.0.0.6(ALL OSPF DR ROUTERS). The DR and BDR always maintan the same database incase the DR was to go down.

How OSPF chooses the router ID:

1. You can manually configure it under ospf by using the the router-id command.
2. Highest IP address on any loopback interface that is UP/UP.
3. Highest IP address on any UP/UP interface.

DR/BDR election:

The router with highest router ID is DR and next highest ip addresses on the neighboring devices become the BDR. You can influence this via the interface priorty number. We will use Diagram 1 to perform this.

Diagram 1

Currently R2 is the DR because of loopback 2 having the higher IP address. I don't want R2 to be the DR as it’s a 2600 devices and R1 is a 2851 which is more powerful and better device. Your DR should be the better router when possible.

Verifying who the DR/BDR is currently:

R1#show IP ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2           1   FULL/DR         00:00:37    192.168.1.2     FastEthernet0/0
R2#show IP ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1           1   FULL/BDR        00:00:36    192.168.1.1     FastEthernet0/0

The output from R1 confirms that R2 is the DR. We are going to modify the interface priority on R2 so that it does become the DR.

Verifying interface priorities:

R1#show IP ospf interface fastEthernet 0/0
FastEthernet0/0 is up, line protocol is up
  Internet Address 192.168.1.1/30, Area 0
  Process ID 1, Router ID 1.1.1.1, Network Type BROADCAST, Cost: 1
  Enabled by interface config, including secondary IP addresses
  Transmit Delay is 1 sec, State BDR, Priority 1  Designated Router (ID) 2.2.2.2, Interface address 192.168.1.2
  Backup Designated router (ID) 1.1.1.1, Interface address 192.168.1.1
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    Hello due in 00:00:07
  Supports Link-local Signaling (LLS)
  Index 1/1, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 1, maximum is 1
  Last flood scan time is 0 msec, maximum is 4 msec
  Neighbor Count is 1, Adjacent neighbor count is 1
    Adjacent with neighbor 2.2.2.2  (Designated Router)
  Suppress hello for 0 neighbor(s)
R2#show IP ospf interface fastEthernet 0/0
FastEthernet0/0 is up, line protocol is up
  Internet Address 192.168.1.2/30, Area 0
  Process ID 1, Router ID 2.2.2.2, Network Type BROADCAST, Cost: 1
  Enabled by interface config, including secondary IP addresses
  Transmit Delay is 1 sec, State DR, Priority 1  Designated Router (ID) 2.2.2.2, Interface address 192.168.1.2
  Backup Designated router (ID) 1.1.1.1, Interface address 192.168.1.1
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    Hello due in 00:00:02
  Supports Link-local Signaling (LLS)
  Index 1/1, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 0, maximum is 2
  Last flood scan time is 0 msec, maximum is 0 msec
  Neighbor Count is 1, Adjacent neighbor count is 1
    Adjacent with neighbor 1.1.1.1  (Backup Designated Router)
  Suppress hello for 0 neighbor(s)

Modifying the interface priority:

We will modify the priority on R2 so that it never becomes the DR again.

R2#config t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int f0/0
R2(config-if)#IP ospf priority 0
R2(config-if)#end

Now we need to clear the ospf process for this to take place.

R2#clear IP ospf process
Reset ALL OSPF processes? [no]: y

*Mar  1 00:34:15.799: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached
Verifying our changes:


R1#show IP ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2           0   FULL/DROTHER    00:00:37    192.168.1.2     FastEthernet0/0
R1#

R2#show IP ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1           1   FULL/DR         00:00:30    192.168.1.1     FastEthernet0/0
R2#

R1#show IP ospf int f0/0
FastEthernet0/0 is up, line protocol is up
  Internet Address 192.168.1.1/30, Area 0
  Process ID 1, Router ID 1.1.1.1, Network Type BROADCAST, Cost: 1
  Enabled by interface config, including secondary IP addresses
  Transmit Delay is 1 sec, State DR, Priority 1  Designated Router (ID) 1.1.1.1, Interface address 192.168.1.1
  No backup designated router on this network
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    Hello due in 00:00:06
  Supports Link-local Signaling (LLS)
  Index 1/1, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 2, maximum is 3
  Last flood scan time is 0 msec, maximum is 4 msec
  Neighbor Count is 1, Adjacent neighbor count is 1
    Adjacent with neighbor 2.2.2.2
  Suppress hello for 0 neighbor(s)
R2#show IP ospf interface fastEthernet 0/0
FastEthernet0/0 is up, line protocol is up
  Internet Address 192.168.1.2/30, Area 0
  Process ID 1, Router ID 2.2.2.2, Network Type BROADCAST, Cost: 1
  Enabled by interface config, including secondary IP addresses
  Transmit Delay is 1 sec, State DROTHER, Priority 0  Designated Router (ID) 1.1.1.1, Interface address 192.168.1.1
  No backup designated router on this network
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    Hello due in 00:00:06
  Supports Link-local Signaling (LLS)
  Index 1/1, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 1, maximum is 2
  Last flood scan time is 0 msec, maximum is 0 msec
  Neighbor Count is 1, Adjacent neighbor count is 1
    Adjacent with neighbor 1.1.1.1  (Designated Router)
  Suppress hello for 0 neighbor(s)


This confirms our changes were successful. Now you see how it displays the neighbor as DROTHER, it’s because it can no longer become a DR or BDR.

Recommended Reads:

Saturday, March 17, 2012

FTP ACL Issue on a cisco router.


I highly recommend you read Understanding How FTP Works before going any further in this write up. The ACL issue will make sense.

I had a customer that was able to establish what people refer to the command connection (Port 21) and enter there username and password but not transfer files, thats refered to the data connection (Port 20) portion.  The root cause for this not working was a misconfigured ACL.

Throughout this writeup, we will use Diagram 1 and telnet connections as they are tcp based just like FTP to explain the concept.
Diagram 1
 On Diagram 1, Core_NJ_WAN1_1  interface F0/0 has the following ACL applied (IN)

CORE_NJ_WAN_1#show ip access-lists test
Extended IP access list test
    10 permit tcp any any established (207 matches)
    30 permit eigrp any any (4083 matches)
    40 permit icmp any any echo-reply (15 matches)
    50 permit icmp any any echo (60 matches)

Now pay close attention to seq#10, what does it mean? It means that any connection that is not initiated outbound first will not be allowed inbound. If you were to try to telnet from Dist_Data_Center_NJ to CORE_VA_WAN_1 it would NOT WORK because of seq#10.

If you try this in reverse, telnet from CORE_VA_WAN_1 to Dist_Data_Center_NJ  it would work because you initiated an outbound connection first.

Now think about how FTP works, would this allow an ftp connection to work properly from users connected to CORE_VA_WAN_1? No. It will only allow you to use port21(connection portion) but not the data portion since the FTP server is going to initiate a connection back to CORE_VA_WAN_1.

So how do you fix this? You will need to update your ACL with the ip address of your ftp server.

CORE_NJ_WAN_1#show ip access-lists test
Extended IP access list test
    10 permit tcp any any established (330 matches)
    30 permit eigrp any any (4716 matches)
    40 permit icmp any any echo-reply (20 matches)
    50 permit icmp any any echo (60 matches)
    60 permit tcp host X.X.X.X any (4 matches) <--------- What ever the ip address is of your ftp server. You can also get more granular and permit just port 20 and 21. In our cause this server does more then ftp.

Let test this all out now.

Before the ACL change:

Dist_Data_Center_NJ#telnet 200.200.200.200 /source-interface loopback 100
Trying 200.200.200.200 ...
% Destination unreachable; gateway or host down
CORE_VA_WAN_1#telnet 100.100.100.100 /source-interface loopback 200
Trying 100.100.100.100 ... Open

After the ACL change:

Dist_Data_Center_NJ#telnet 200.200.200.200 /source-interface loopback 100
Trying 200.200.200.200 ... Open

CORE_VA_WAN_1#telnet 100.100.100.100 /source-interface loopback 200
Trying 100.100.100.100 ... Open


How did I discover this was the issue? If you put a sniffer up you will see FTP response codes that guide you in certain directions. check the following link, FTP Response Codes. In my case I was seeing a code of 425 Can't open data connection. I would love to put up the capture for refrence but it require alot of editing because the real ip's and username and passwords are displayed.




Thursday, March 15, 2012

Cisco ASA Object-Group

What is an object group?
It allows you to group ports and protocols within a group and simplifies your ACL and how many ACLs you require.

How to view an object group.

CISCOASA# show run object-group id OBJECT_GROUP_NAME
object-group service OBJECT_GROUP_NAME tcp
 description testing ports
 port-object range 4000 4999
 port-object range 8000 8999
 port-object eq 25570
 port-object range 5000 5999
 port-object range 21000 21999
 port-object eq 30101
 port-object range 30005 30006
 port-object eq 19420
 port-object eq 19720
 port-object eq 19920
CISCOASA#

Some examples of an object group in use.

access-list from-switchch extended permit udp any 172.200.18.0 255.255.255.0 object-group OBJECT_GROUP_NAME

access-list from-switchch extended permit tcp any 172.200.18.0 255.255.255.0 object-group OBJECT_GROUP_NAME

Wednesday, March 14, 2012

Load Balancing & CEF

How to verify the load balancing method you’re using. Very basic and simple, CEF is a huge topic.

Load balancing describes functionality in a router that distributes packets across multiple links based on layer 3 routing information. If a router discovers multiple paths to a destination, the routing table is updated with multiple entries for that destination.

1. Check the routing table to make sure your dynamic routing protocol has the same metric for the destination network via the different paths. In our case, it does.


Cisco-4900M#show ip route 192.168.179.193
Routing entry for 192.168.179.193/32
  Known via "ospf 4567", distance 110, metric 3, type inter area
  Last update from 192.168.222.93 on TenGigabitEthernet1/3, 2w5d ago
  Routing Descriptor Blocks:
 
* 192.168.222.109, from 192.168.179.112, 2w5d ago, via TenGigabitEthernet1/4
      Route metric is 3, traffic share count is 1
    192.168.222.93, from 192.168.179.111, 2w5d ago, via TenGigabitEthernet1/3
      Route metric is 3, traffic share count is 1

There is also an asterisk (*) next to one of the block entries. This corresponds to the active route that is used for new traffic. The term 'new traffic' corresponds to a single packet or an entire flow to a destination, depending on the type of switching configured.

Source: http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094820.shtml

2. Verify If CEF is enabled on the interfaces. It's enabled by default on all newer Cisco devices. In our case it’s enabled  and its indicated below in red.

Cisco-4900M#show cef interface tenGigabitEthernet 1/3
TenGigabitEthernet1/3 is up (if_number 64)
  Corresponding hwidb fast_if_number 64
  Corresponding hwidb firstsw->if_number 64
  Internet address is 192.168.222.92/31
  ICMP redirects are never sent
  IP unicast RPF check is disabled
  Inbound access list is not set
  Outbound access list is not set
  IP policy routing is disabled
  BGP based policy accounting on input is disabled
  BGP based policy accounting on output is disabled
  Hardware idb is TenGigabitEthernet1/3
  Fast switching type 1, interface type 155
  IP CEF switching enabled  IP CEF switching turbo vector
  IP Null turbo vector
  IP prefix lookup IPv4 mtrie 8-8-8-8 optimized
  Input fast flags 0x0, Output fast flags 0x0
  ifindex 63(63)
  Slot 1 Slot unit 3 VC -1
  Transmit limit accumulator 0x0 (0x0)
  IP MTU 9198

Cisco-4900M#show cef interface tenGigabitEthernet 1/4
TenGigabitEthernet1/4 is up (if_number 65)
  Corresponding hwidb fast_if_number 65
  Corresponding hwidb firstsw->if_number 65
  Internet address is 192.168.222.108/31
  ICMP redirects are never sent
  IP unicast RPF check is disabled
  Inbound access list is not set
  Outbound access list is not set
  IP policy routing is disabled
  BGP based policy accounting on input is disabled
  BGP based policy accounting on output is disabled
  Hardware idb is TenGigabitEthernet1/4
  Fast switching type 1, interface type 155
  IP CEF switching enabled  IP CEF switching turbo vector
  IP Null turbo vector
  IP prefix lookup IPv4 mtrie 8-8-8-8 optimized
  Input fast flags 0x0, Output fast flags 0x0
  ifindex 64(64)
  Slot 1 Slot unit 4 VC -1
  Transmit limit accumulator 0x0 (0x0)
  IP MTU 9198

Cisco-4900M#show ip cef 192.168.179.193
192.168.179.193/32
  nexthop 192.168.222.93 TenGigabitEthernet1/3
  nexthop 192.168.222.109 TenGigabitEthernet1/4

3. Verifying the load balancing method. In our scenario, we are using per-destination load balancing, indicated below in red.

Cisco-4900M#show ip cef 192.168.179.193 internal
192.168.179.193/32, epoch 1, RIB[I], refcount 6, per-destination sharing  sources: RIB
  feature space:
   Broker: linked
  ifnums:
   TenGigabitEthernet1/3(64): 192.168.222.93
   TenGigabitEthernet1/4(65): 192.168.222.109
  path 2031A37C, path list 20311034, share 1/1, type attached nexthop, for IPv4
  nexthop 192.168.222.93 TenGigabitEthernet1/3, adjacency IP adj out of TenGigabitEthernet1/3, addr 192.168.222.93 206DE9E0
  path 2031A3F0, path list 20311034, share 1/1, type attached nexthop, for IPv4
  nexthop 192.168.222.109 TenGigabitEthernet1/4, adjacency IP adj out of TenGigabitEthernet1/4, addr 192.168.222.109 206DE860
  output chain:
    loadinfo 202FD5C0, per-session, 2 choices, flags 0003, 34 locks
    flags: Per-session, for-rx-IPv4
    16 hash buckets (hardware has 8 hash buckets)
      < 0 > IP adj out of TenGigabitEthernet1/3, addr 192.168.222.93 206DE9E0
      < 1 > IP adj out of TenGigabitEthernet1/4, addr 192.168.222.109 206DE860
      < 2 > IP adj out of TenGigabitEthernet1/3, addr 192.168.222.93 206DE9E0
      < 3 > IP adj out of TenGigabitEthernet1/4, addr 192.168.222.109 206DE860
      < 4 > IP adj out of TenGigabitEthernet1/3, addr 192.168.222.93 206DE9E0
      < 5 > IP adj out of TenGigabitEthernet1/4, addr 192.168.222.109 206DE860
      < 6 > IP adj out of TenGigabitEthernet1/3, addr 192.168.222.93 206DE9E0
      < 7 > IP adj out of TenGigabitEthernet1/4, addr 192.168.222.109 206DE860
      < 8 > IP adj out of TenGigabitEthernet1/3, addr 192.168.222.93 206DE9E0
      < 9 > IP adj out of TenGigabitEthernet1/4, addr 192.168.222.109 206DE860
      <10 > IP adj out of TenGigabitEthernet1/3, addr 192.168.222.93 206DE9E0
      <11 > IP adj out of TenGigabitEthernet1/4, addr 192.168.222.109 206DE860
      <12 > IP adj out of TenGigabitEthernet1/3, addr 192.168.222.93 206DE9E0
      <13 > IP adj out of TenGigabitEthernet1/4, addr 192.168.222.109 206DE860
      <14 > IP adj out of TenGigabitEthernet1/3, addr 192.168.222.93 206DE9E0
      <15 > IP adj out of TenGigabitEthernet1/4, addr 192.168.222.109 206DE860
    Subblocks:
     None

You can also issue the below command if you don’t care about seeing the hashing algorithm currently in use and other misc details.

Cisco-4900M#show ip cef 192.168.179.193 detail
192.168.179.193/32, epoch 1, per-destination sharing
  nexthop 192.168.222.93 TenGigabitEthernet1/3
  nexthop 192.168.222.109 TenGigabitEthernet1/4

You can set load-balancing to work per-destination or per-packet. Per-destination load balancing means the router distributes the packets based on the destination address. Given two paths to the same network, all packets for destination1 on that network go over the first path, all packets for destination2 on that network go over the second path, and so on. This preserves packet order, with potential unequal usage of the links. If one host receives the majority of the traffic all packets use one link, which leaves bandwidth on other links unused. A larger number of destination addresses leads to more equally used links. To achieve more equally used links use IOS software to build a route-cache entry for every destination address, instead of every destination network, as is the case when only a single path exists. Therefore traffic for different hosts on the same destination network can use different paths. The downside of this approach is that for core backbone routers carrying traffic for thousands of destination hosts, memory and processing requirements for maintaining the cache become very demanding.


You want to avoid per-packet if you are using VOIP in your network as packets can arrive out of order. Per packet works good with there is only one server ont he other side of the link.

To change the load balancing method, under the interface issue the following commands.

ip load-sharing per-packet
ip load-sharing per-destination

Note, per packet is not always an option on some devices.


 If you want to learn more about load-balancing and cef, visit the below links.


Switchport nonegotiate VS Speed nonegotiate.

Today I was doing a port-channel turn up and the command Speed nonegotiate populated its self into the port-channel we thought maybe it was causing some DTP issues. It was not but there is another command with the nonegotiate syntax that does disable DTP.

Switchport nonegotiate:

When using the nonegotiate keyword, Dynamic Inter-Switch Link Protocol and Dynamic Trunking Protocol (DISL/DTP)-negotiation packets are not sent on the interface. The device trunks or does not trunk according to the mode parameter given: access or trunk. This command returns an error if you attempt to execute it in dynamic (auto or desirable) mode.

Speed nonegotiate:

The nonegotiate keyword is available only for SFP module ports. SFP module ports operate only at 1000 Mbps but can be configured to not negotiate if connected to a device that does not support autonegotiation.

Monday, March 12, 2012

How the Router makes routing decisions:

How the Router makes routing decisions:

1) The Route with longest prefix length will be selected.

2) If there are multiple routes with the same prefix length, the route with the lowest AD will be used.

3) If there are multiple routes with the same prefix length & same AD, the route with the lowest metric will be preferred.

4) Finally, if the preceding 3 values are all equal, equal-cost sharing load sharing will be put into action.

Saturday, March 10, 2012

MD5 Checksum

I came across a neat program that calculates MD5 Sums and you can also use it to verify programs that you downloaded. They must provide you with a pre-calculated MD5 Key. The program is called MD5sums 1.2 and you can download on the following website pc-tools.net - MD5sums 1.2.

Every IOS/NXOS image that you download from the Cisco website has an MD5 key associated with it. It is crucial that you validate the key before you upload the image onto a device and after you upload the image onto the device.  I will explain how to do this.

You will need the MD5 Key from the Cisco website before you download the image, below you can see how the Cisco website clearly displays the MD5 key. In our case the key -is eb7d5b53ca697b82ea68e33ddc0462a7.(See figure 1)  Once you download the MD5 program and the Cisco image, you need to drag and drop the image onto the executable file and it will display the MD5 key that should match the one from the Cisco website. In our case it does (See figure 2)



Figure 1



Figure 2
 
Now we can upload this image onto our cisco router or switch. One the file is uploaded, you can perform the same check within the device.

Verifying the image on the switch/router:

3750Device#verify /md5 flash:c3750-ipservicesk9-mz.122-55.SE5.bin eb7d5b53ca697b82ea68e33ddc0462a7
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done!
verify /md5 (flash:c3750-ipservicesk9-mz.122-55.SE5.bin) = eb7d5b53ca697b82ea68e33ddc0462a7



The = sign confirms the MD5 Hash matches.



Runts and CRC errors


I had one user complain about their HTTP traffic to one specific Google appliance really being slow. Turns out the interface had a ton of CRC and Runt errors. At first I thought this was cable related because of the CRC errors but the Runts took away from that notion. I still had them replace the cable but Runts are Ethernet packets that are less than 64 bytes. Most operating systems and devices don’t process Runts. The issue turned out to be duplex mismatch on the server side as the switch was already set to 100Mb/Full. Good Cisco article about this is below. 

https://supportforums.cisco.com/docs/DOC-2809


6509E#show int fa4/36
FastEthernet4/36 is up, line protocol is up (connected)
  Hardware is C6k 100Mb 802.3, address is 0011.bb28.0d23 (bia 0011.bb28.0d23)
 Description: Google Appliance
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 221/255, txload 22/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 04:37:32
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 435000 bits/sec, 736 packets/sec
  5 minute output rate 8837000 bits/sec, 796 packets/sec
     12677713 packets input, 894017675 bytes, 0 no buffer
     Received 362263 broadcasts (362262 multicasts)
     451875 runts, 0 giants, 0 throttles
     3591374 input errors, 20316 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     13181204 packets output, 18277208769 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Server Side

eth0      Link encap:Ethernet  HWaddr 78:2B:CB:19:3A:79 
          inet addr:192.168.232.49  Bcast:192.168.232.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8606244989 errors:3 dropped:24215 overruns:0 frame:3
          TX packets:5810977239 errors:15811579 dropped:0 overruns:0 carrier:15811579
          collisions:2206833034 txqueuelen:1000
          RX bytes:11961560341542 (10.8 TiB)  TX bytes:510772576280 (475.6 GiB)
          Interrupt:36 Memory:d6000000-d6012700


 Duplex settings were changed on the server side since the switch was already set to 100Mb/Full. Issue disappears.


6509E#show int fa4/36
FastEthernet4/36 is up, line protocol is up (connected)
  Hardware is C6k 100Mb 802.3, address is 0011.bb28.0d23 (bia 0011.bb28.0d23)
  Description: Google Appliance
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 12/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 03:43:09
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 305000 bits/sec, 375 packets/sec
  5 minute output rate 4863000 bits/sec, 459 packets/sec
     7917902 packets input, 741546238 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     11120155 packets output, 15401101799 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Configuring route summary addresses for EIGRP.


Configuring route summary addresses for EIGRP.

Route summaries reduce the size of the routing table, improves convergence and cpu load along with memory usage.  We will use the diagram below for our configurations. The data center router has many subnets and we need to aggregate those routes onto the core routers. In order to have a sustainable route summary plan, you need to carefully plan out your ip addressing scheme/subnet assignments.


In order for an EIGRP route summary to be advertised to the neighboring device, you must have at least one interface using one of the subnets in the aggregate and the interface must be up/up. In our case we will use loopback interfaces to simulate this. You can configure multiple aggregate addresses under the interface as we are doing. Unlike OSPF you can summarize on any router, in OSPF you can only perform summarization an ABR or ASBR.

LoopBacks were created before hand and ip's have were assigned. These interfaces need to be associated with the EIGRP process using the network command.

Dist_Data_Center_NJ#show ip int brief
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            192.168.5.1     YES manual up                    up 
FastEthernet0/1            unassigned      YES unset  administratively down down
Loopback10                 206.200.0.1     YES manual up                    up 
Loopback11                 206.200.5.1     YES manual up                    up 

Dist_Data_Center_NJ#


Associating the new loopback interfaces and future loopbacks.

Dist_Data_Center_NJ(config)#router eigrp 10
Dist_Data_Center_NJ(config-router)# network 206.200.0.0 0.0.3.255
Dist_Data_Center_NJ(config-router)# network 206.200.4.0 0.0.3.255


Confirms which interfaces are park of the EIGRP process.
 
Dist_Data_Center_NJ#show ip eigrp interfaces
IP-EIGRP interfaces for process 10
                        Xmit Queue   Mean   Pacing Time   Multicast    Pending
Interface        Peers  Un/Reliable  SRTT   Un/Reliable   Flow Timer   Routes
Fa0/0              1        0/0        28       0/2          116           0
Lo11               0        0/0         0       0/1            0           0
Lo10               0        0/0         0       0/1            0           0
Dist_Data_Center_NJ#
 


I updated the interface with the summary command, highlighted in red. This will cause your neighborship to re-sync.

Dist_Data_Center_NJ#show run int f0/0
Building configuration...

Current configuration : 212 bytes
!
interface FastEthernet0/0
 ip address 192.168.5.1 255.255.255.252
 ip summary-address eigrp 10 206.200.4.0 255.255.252.0 5
 ip summary-address eigrp 10 206.200.0.0 255.255.252.0 5

 duplex auto
 speed auto
end


CORE_NJ_WAN_1#
*Mar  1 00:09:06.307: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 10: Neighbor 192.168.5.1 (FastEthernet0/0) is resync: peer graceful-restart




CORE_NJ_WAN_1#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     192.168.5.0/30 is subnetted, 1 subnets
C       192.168.5.0 is directly connected, FastEthernet0/0
D    206.200.0.0/22 [90/409600] via 192.168.5.1, 00:18:06, FastEthernet0/0
D    206.200.4.0/22 [90/409600] via 192.168.5.1, 00:16:26, FastEthernet0/0