Categorie: Netwerk (C-NSP)

Cisco 7201 (G2) Traffic Performance (High CPU Utilization)

> If the CPU load is high, then check to see what’s causing it.
His sh proc c output snippet indicates that almost all his CPU usage is interrupts – on a software-based platform, that’s indicative of pps, whether directed to the box or through the box.

Is this router on the Internet, or on a private WAN?

Can you enabled NetFlow on the router and take a look at the contents of the NetFlow cache?  No, the additional CPU is not a big deal, it’s single-digit.

Take a look at this preso:

<https://app.box.com/s/mnshn99c13uekrggy99b>

sh proc c | e 0.00 would be helpful.

Here’s the older post:
https://puck.nether.net/pipermail/cisco-nsp/2007-April/039999.html

> Ultimately I want to know am I simply hitting a practical
> limit of the box already?
> I’m very scared to enable more WAN links on these routers
> as I am affraid it will max out the available resources.

It’s been a while since I ran any decent traffic through the
NPE-G2, but if memory serves, CPU utilization was not always
linear with traffic. But at some point, it levels out and
climbs slower (given that you’re not running any features
that could cause this).

That said, this was back in the days of SRC, and the NPE-
G2’s I have now are looking glasses, so no major drama
there.

We saw higher CPU utilization at low traffic levels compared
to the NPE-G1, but saw a slower climb as traffic climbed. In
the role you describe, we got to 950Mbps at ~93% CPU
utilization.

Cisco, Netwerk (C-NSP)

Cisco 6500 QOS

By enabling ‘mls qos’ you’re splitting your already small buffer space on this platform into 4 pieces and using just one of them for normal traffic “mls qos” activates QoS, and that includes _ALL_ of the default QoS configs including automatic classify and remark, and all the default queues.

This lets the interfaces use (almost) as much as possible from the common buffers on the switch.

mls qos queue-set output 1 threshold 1 3100 3100 100 3200 
mls qos queue-set output 1 threshold 2 3100 3100 100 3200 
mls qos queue-set output 1 threshold 3 3100 3100 100 3200 
mls qos queue-set output 1 threshold 4 3100 3100 100 3200 
mls qos
Voorbeeld:
http://ampere.rathlev.dk/3560-3750-QoS-basis-template.txt

The “QoS Overview” chapter of the “Cisco Catalyst 3750 QoS  Configuration Examples” document explains the architecture:

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/91862-cat3750-qos-config.html#topic1

Any one ever worked on Cisco 6500 QOS specifically 6503 or 6524(help) needed
The 6500 series switch has unique, complex and restrictive hardware QOS compared to a software based router/switch.http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12-2SX/configuration/guide/book/qos.html6748 is a switch based line card. It has limited qos features supported as
compared to a routed platform such as ISR router.
On 12 Oct 2014 10:45, “Ahsan Rasheed” <ahsanrasheed9@gmail.com> wrote:> Hi All,
>
> I am having issue specifically doing QOS configuration on 6503 or 6524 or
> 6509 switches. I am unable to match any EF(voice) traffic for eompls(vlan
> based) on 6503 cisco switch. If i use any other router as 2811 or 2821 my
> QOS configuration works perfect but if i put 6503 as PE2 it does not work.i
> am using vlan based eompls.
>
> Below is the scenario & configuration which i am having issue.
>
>
> CE1(2821 router)(dot1Q)———>PE1(2821 router)——->P(6524
> switch)——–>PE2(6503 switch)——->(dot1Q)(2821 switch)CE2.
>
> On CE1 i can match ip-precedence 5 traffic and mark that traffic to cos5 on
> outbound port.On PE1 i can match cos5 packet and mark with mpls exp top5 on
> inbound port, on outbound port i can match mpls exp 5.
>
> On PE2(6503) i am unable to match that mpls exp5 packet on inbound port.
> none of the configuration worked on 6500 series switches with mls qos, ,mls
> qos trust dscp,mls qos trust cos etc. Although i can match cos5 traffic on
> CE2 on inbound interface.i can not match mpls exp 5 traffic on 6503 and all
> i can see traffic as default-class on 6503 switch. I tried many things and
> many configurations on 6503 but nothing worked.If i put 2821 router as PE2
> instead of 6503 my qos configuration works. but why if i put 6503 my same
> qos configuration does not work?
>
> —match means=classification or classify
>
> Can anyone tell me how qos works on 6500 series switches or where i am
> having issue in my scenario.
> i am using this ios on 6503: s72033-advipservicesk9_wan-mz.122-33.SXI3.bin.
>
> below r my questions for 6503 qos:
>
> 1.do i need to use some other map tables,am i  using correct map tables on
> 6503 as cos-dscp,dscp-cos,exp-dscp etc.
> 2.any other configuration of qos needed on 6503?
> 3.i am unable to match anything on outbound port of 6503.
> 4.on 6503 i am using sup720 and PFC3BXL.any specific configuration needed
> for PFC3bxl.
> 5. 6503 not allowing me to match qos-group on inbound interface, not
> allowing me to set cos5 on outbound interface. not allowing me to set cos5
> as an inbound interface.
>
>
> CE1(2821) config:
> ————————
> !
> class-map match-any EF
>  match ip precedence 5
> class-map match-any data
>  match ip precedence 3
> !
> !
> policy-map ip2mpls
>  class EF
>   set cos 5
>  class data
>   set cos 3
> !
> interface FastEthernet0/0
>  no ip address
>  duplex auto
>  speed auto
> !
> interface FastEthernet0/0.455
>  encapsulation dot1Q 455
>  ip address 172.16.15.1 255.255.255.252
>  service-policy output EF
> !
>
> PE1(2821) config:
> ————————-
> ————————-
> mls qos map cos-dscp 0 8 16 24 32 40 48 56
> !
> class-map match-all exp_3
>  match mpls experimental topmost 3
> class-map match-all mpls_exp
>  match mpls experimental topmost 5
> class-map match-any cos3
>  match cos  3
> class-map match-any LOO1
>  match cos  5
> !
> !
> policy-map EF
>  class LOO1
>   set mpls experimental imposition 5
>  class cos3
>   set mpls experimental imposition 3
> policy-map QOS_G_5
>  class mpls_exp
>   priority
>  class exp_3
>   bandwidth 500
> !
> interface Loopback0
>  ip address 3.3.3.3 255.255.255.255
> !
> interface FastEthernet0/0
>  ip address 192.168.23.2 255.255.255.0
>  ip ospf network point-to-point
>  duplex auto
>  speed auto
>  mpls ip
>  service-policy output QOS_G_5
> !
> interface FastEthernet0/1.455
>  encapsulation dot1Q 455
>  xconnect 5.5.5.5 455 encapsulation mpls
>  service-policy input EF
> !
> ——————————–
> ——————————–
> PE2(6503 qos):
> R1#show module
> Mod Ports Card Type                              Model              Serial
> No.
> — —– ————————————– ——————
> ———–
>   1    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE
>  SAL09401U2L
>   2   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX
> SAL114247YN
>   3   16  16 port 1000mb GBIC ethernet           WS-X6416-GBIC
>  SAL0712AM69
>   4   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP
> SAL10019J4N
>   5    2  Supervisor Engine 720 (Hot)            WS-SUP720-3BXL
> SAD102805VM
>   6    2  Supervisor Engine 720 (Active)         WS-SUP720-BASE
> SAD0846060F
>
> Mod  Sub-Module                  Model              Serial       Hw
> Status
> —- ————————— —————— ———– ——-
> ——-
>   1  Distributed Forwarding Card WS-F6700-DFC3BXL   SAD102504EF  5.3    Ok
>   2  Centralized Forwarding Card WS-F6700-CFC       SAD111300PD  3.1    Ok
>   4  Centralized Forwarding Card WS-F6700-CFC       SAL1004BQ2A  2.0    Ok
>   5  Policy Feature Card 3       WS-F6K-PFC3BXL     SAD10270189  1.8    Ok
>   5  MSFC3 Daughterboard         WS-SUP720          SAD102801G5  2.5    Ok
>   6  Policy Feature Card 3       WS-F6K-PFC3BXL     SAL1415FE95  1.11   Ok
>   6  MSFC3 Daughterboard         WS-SUP720          SAD08440794  2.4    Ok
>
> R1#show mls qos maps
>    Normal Burst Policed-dscp map:                                  (dscp=
> d1d2)
>      d1 :  d2 0  1  2  3  4  5  6  7  8  9
>      ————————————-
>       0 :    01 01 02 03 04 05 06 07 08 09
>       1 :    10 11 12 13 14 15 16 17 18 19
>       2 :    20 21 22 23 24 25 26 27 28 29
>       3 :    30 31 32 33 34 35 36 37 38 39
>       4 :    40 41 42 43 44 45 01 47 48 49
>       5 :    50 51 52 53 54 55 56 57 58 59
>       6 :    60 61 62 63
>
>    Maximum Burst Policed-dscp map:                                  (dscp=
> d1d2)
>      d1 :  d2 0  1  2  3  4  5  6  7  8  9
>      ————————————-
>       0 :    00 01 02 03 04 05 06 07 08 09
>       1 :    10 11 12 13 14 15 16 17 18 19
>       2 :    20 21 22 23 24 25 26 27 28 29
>       3 :    30 31 32 33 34 35 36 37 38 39
>       4 :    40 41 42 43 44 45 46 47 48 49
>       5 :    50 51 52 53 54 55 56 57 58 59
>       6 :    60 61 62 63
>
>    Dscp-cos map:                                  (dscp= d1d2)
>      d1 :  d2 0  1  2  3  4  5  6  7  8  9
>      ————————————-
>       0 :    00 00 00 00 00 00 00 00 01 01
>       1 :    01 01 01 01 01 01 02 02 02 02
>       2 :    02 02 02 02 03 03 03 03 03 03
>       3 :    03 03 04 04 04 04 04 04 04 04
>       4 :    05 05 05 05 05 05 05 05 06 06
>       5 :    06 06 06 06 06 06 07 07 07 07
>       6 :    07 07 07 07
>
>    Dscp-exp map:                                  (dscp= d1d2)
>      d1 :  d2 0  1  2  3  4  5  6  7  8  9
>      ————————————-
>       0 :    00 00 00 00 00 00 00 00 01 01
>       1 :    01 01 01 01 01 01 02 02 02 02
>       2 :    02 02 02 02 03 03 03 03 03 03
>       3 :    03 03 04 04 04 04 04 04 04 04
>       4 :    05 05 05 05 05 05 05 05 06 06
>       5 :    06 06 06 06 06 06 07 07 07 07
>       6 :    07 07 07 07
>
> Cos-dscp map:
>          cos:   0  1  2  3  4  5  6  7
>      ————————————
>         dscp:   0 10 18 24 34 46 48 56
>
>    IpPrecedence-dscp map:
>       ipprec:   0  1  2  3  4  5  6  7
>      ————————————
>         dscp:   0  8 16 24 32 40 48 56
>
>    Exp-dscp map:
>          exp:   0  1  2  3  4  5  6  7
>      ————————————
>         dscp:   0  8 16 24 32 40 48 56
>
>
> mls netflow interface
> mls qos map cos-dscp 0 10 18 24 34 46 48 56
> mls qos
> !
> class-map match-all exp_3
>  match mpls experimental topmost 3
> class-map match-all EXP_5
>  match mpls experimental topmost 5
> class-map match-all QOS_GROUP_5
>  match qos-group 5
> class-map match-all prec5
>  match ip precedence 5
> class-map match-all cos5
>  match cos  5
> !
> policy-map mpls2ip
> class QOS_GROUP_5
>  set cos 5
> !
> policy-map IN_FROM_R3
>  class EXP_5
>   set qos-group 5
> !
> interface Loopback0
>  ip address 5.5.5.5 255.255.255.255
> !
> interface GigabitEthernet2/2
>  mls qos trust cos
> or <———— (tried both individually but none worked)
>  mls qos trust dscp
> !
> interface GigabitEthernet2/2.455
>  encapsulation dot1Q 455
>  xconnect 3.3.3.3 455 encapsulation mpls
>  service-policy output mpls2ip
> !
> interface GigabitEthernet2/1
>  ip address 192.168.34.4 255.255.255.0
>  ip ospf network point-to-point
>  mls qos trust cos
> or <———— (tried both individually but none worked)
>  mls qos trust dscp
>  mpls ip
>  service-policy input IN_FROM_R4
> !
> Thanks & regards,
> Ahsan Rasheed

 

Cisco, Netwerk (C-NSP)

Cisco ASA return traffic with explicit deny on outside interfac

I have an ASA running 8.4 in a pretty simple setup with 2 interfaces (inside/outside). I have to 2 ACLs where one is applied inbound on the inside, and one ACL applied inbound on the outside interface. The outside ACL has an explicit deny ip any any statement for logging purposes.

I am wondering, does return traffic (for connections originated on the inside network) get through  the ASA with the explicit deny ip any any statement in the outside ACL?  I know it works without an ACL applied to the outside interface, but the explicit deny got me thinking. I haven’t a device with me to test it unfortunately
Return traffic will be permitted.

Any traffic originating on a network connected to a higher security interface will not need an ACL to ingress.  When the traffic egresses to a lower security interface it will automatically be let back in.

Any traffic originating on a network connected to a lower security interface will need an ACL to allow ingress.  When the traffic egresses to a higher security interface it will also be let back in.

That’s how I remember it anyway.. 🙂

Point 3. In the below link seems to back me up.

http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113396-asa-packet-flow-00.html

Cisco, Netwerk (C-NSP), Security

How to troubleshoot input queue drop continuously on Cisco7600?

> k1#sho int te1/4
>
>   Last clearing of “show interface” counters 00:12:35
>
>   Input queue: 0/200/69739/69739 (size/max/drops/flushes); Total
> output
> drops: 0

On this platform those drops could be for traffic forwarded to the CPU.
What does the interface configuration look like? Is it a L3 interface (“no switchport”), or is it a L2 interface? If the latter, and you have SVIs (VLAN interfaces) for some of the VLANs it carries, look for drops on those SVIs.

If they are “microburst” drops on L3-interfaces then you can possibly mitigate it using “hold-queue <N> in” on the L3-interfaces (SVIs or the “no switchport” interface itself). This is a trade-off; don’t use a too high setting for input-queue or you will get other problems.

You could use a SPAN session to look at what traffic might cause this.
The SPAN replication is before traffic is forwarded to the CPU so you should be able to see floods.

>   1    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE

This card has rather small queues so it might simply be microbursts. In that case you might have to find a port on another card.

For general info about input queue drops this is a good read:

http://www.cisco.com/c/en/us/support/docs/routers/10000-series-routers/6343-queue-drops.html

Oversubscribing the card would cause output drops, not input queue drops.
As someone else mentioned, this is punted traffic.

I’d suggest using NetDR to find out.
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/116475-technote-product-00.html

On Mon, Oct 20, 2014 at 9:42 PM, PlaWanSai RMUTT CPE IX <
pws_admin@thaicpe.com> wrote:

Cisco, Netwerk (C-NSP)

Traffic Generators

opensource solutions, scapy will probably be able

to do the job.
> IMIX and other standard load tests.I’m using IPERF now, but this is
> simply a load gen.  No IMIX etc Suggestions ?A little script with bwping (TOS, MTU) is not enough to try to make a ~~pseudo~~ RFC2544 test ?
Sorry if it is not what you are searching for.

So I have started playing with this, I think it should do we you want (I have the same requirement);http://traffic.comics.unina.it/software/ITG/

> When it comes to minimum size packets, I’m not aware of open source
> software that can congest 1GE port. Operating systems are not really
> tuned to do 1Gbps UDP streams on small packets. You can achieve that,
> even more, but you need to go quite low level, UDP socket you must
> forget immediately event with sendmmsg/recvmmsg. Raw sockets and
> modern CPU and you’ll probably be able to reach 1Gbps per core, but I
> don’t know software available that would be productized even to iperf level here, would love to hear about one.

Some open source software I am writing can do 1Gbps (and likely beyond, I don’t have any 10G NICs to test on, yet!), although it’s for testing at the Ethernet layer so not really applicable here;

https://github.com/jwbensley/Etherate

One of the main features I am working on now (as I’m still writing the initial version) is loading the frame payload from file so that payload data *could* be UDP but it’s not really ment for testing higher than Ethernet and/or MPLS level.

Cheers,
James.

>
> When it comes to minimum size packets, I’m not aware of open source
> software that can congest 1GE port. Operating systems are not really
> tuned to do 1Gbps UDP streams on small packets. You can achieve that,
> even more, but you need to go quite low level, UDP socket you must
> forget immediately event with sendmmsg/recvmmsg. Raw sockets and
> modern CPU and you’ll probably be able to reach 1Gbps per core, but I
> don’t know software available that would be productized even to iperf level here, would love to hear about one. 

Scapy [1] should be able to do that easily, so should mausezahn [2]. Of course you don’t open use regular TCP/UDP socket API for this, but some raw form of it and generate the whole packet in userspace.[1] http://www.secdev.org/projects/scapy/
[2] http://www.perihel.at/sec/mz/

Netwerk (C-NSP)

Cisco CoPP

Cisco CoPP Best practices

Cisco CoPP Cat6k

http://events.cisco.hu/2014/techtorial/doc/lavarga_v1.0.pdf

Hi All,

Thanks for all the valuable input!

I wrote up a CoPPs policy, and deployed it in a non-limiting fasion

and monitored for a while. Once happy we enabled the policers and its

working fine, however the software counters are going up, and it’s not

clear to me why that is.

Further down is the config, immediately below is partial the output

from an example 7600 (as the CoPPs policy is quite long):

abr1#show policy-map control-plane input

Control Plane

  Service-policy input: Control-Plane-Filter-In

  Hardware Counters:

    class-map: CoPP-Limit-and-Permit-Critical (match-any)

      Match: access-group name CoPP-Limit-and-Permit-BGP

      Match: access-group name CoPP-Limit-and-Permit-BGPv6

      Match: access-group name CoPP-Limit-and-Permit-RSVP

      Match: access-group name CoPP-Limit-and-Permit-LDP

      Match: access-group name CoPP-Limit-and-Permit-OSPF

      Match: access-group name CoPP-Limit-and-Permit-OSPFv3

      Match: access-group name CoPP-Limit-and-Permit-HSRP

      Match: access-group name CoPP-Limit-and-Permit-BFD

      police :

        10000000 bps 312000 limit 312000 extended limit

      Earl in slot 6 :

        631028621 bytes

        5 minute offered rate 86968 bps

        aggregate-forwarded 631028621 bytes action: transmit

        exceeded 0 bytes action: transmit

        aggregate-forward 79648 bps exceed 0 bps

  Software Counters:

    Class-map: CoPP-Limit-and-Permit-Critical (match-any)

      4646556 packets, 411683229 bytes

      5 minute offered rate 54000 bps, drop rate 0000 bps

      Match: access-group name CoPP-Limit-and-Permit-BGP

        4035626 packets, 367873184 bytes

        5 minute rate 48000 bps

      Match: access-group name CoPP-Limit-and-Permit-BGPv6

        2101 packets, 174550 bytes

        5 minute rate 0 bps

      Match: access-group name CoPP-Limit-and-Permit-RSVP

        0 packets, 0 bytes

        5 minute rate 0 bps

      Match: access-group name CoPP-Limit-and-Permit-LDP

        173745 packets, 13108073 bytes

        5 minute rate 1000 bps

      Match: access-group name CoPP-Limit-and-Permit-OSPF

        77045 packets, 8382206 bytes

        5 minute rate 1000 bps

      Match: access-group name CoPP-Limit-and-Permit-OSPFv3

        0 packets, 0 bytes

        5 minute rate 0 bps

      Match: access-group name CoPP-Limit-and-Permit-HSRP

        358039 packets, 22145216 bytes

        5 minute rate 2000 bps

      Match: access-group name CoPP-Limit-and-Permit-BFD

        0 packets, 0 bytes

        5 minute rate 0 bps

      police:

          cir 10000000 bps, bc 312500 bytes, be 312500 bytes

        conformed 4646556 packets, 411683229 bytes; actions:

          transmit

        exceeded 0 packets, 0 bytes; actions:

          transmit

        violated 0 packets, 0 bytes; actions:

          drop

        conformed 54000 bps, exceeded 0000 bps, violated 0000 bps

I’m not sure why traffic like BGP would match into both the hardware

and software policiers, when its such a simple match statement (I am

assuming that because the packet count under the software counters is

much lower than the ACL match, so the rest were policied by

hardware?):

I am trying to write a CoPP template for some 7600s running as PEs. It

 

> would be handy if they were running a similar CoPP configuration to

> that on our Juniper PEs we are going to be connecting these 7600’s too

> so we have consistent CoPP across that domain of equally exposed

 

It’s not going to be consistent, CoPP does work, but can’t really compete with over decade newer Trio kit.

 

> CoPP on 7600’s can’t police ARP but one can use the MLS HWRL for that.

> The HWRLs can also handle other protocols like HSRP and CoPP can’t

> police multicast in hardware, so do people usually police ARP and HSRP

> using the MLS HWRLs instead of CoPP?

 

Pretty sure HSRP works just fine. For multicast, you just need to allow all multicast in CoPP (otherwise it’s processed by software CoPPP, which you don’t

want) and then limit in mls rate-limit.

 

> The HWRLs support other protocols too that ARE supported in CoPP in

> hardware, so are there any other protocols that people prefer to

> police using the HWRLs?

 

Example please, I cannot recall overlap.

 

> With regards to ACLs, do people really have giant access lists of

> peers they allow BGP to/from? The 7600 I am piloting this on has over

 

Yes. Two, one low rate for SYN and one for other BGP, alas. However if your master configuration data is not network, but database, then it does not really matter how complex or simple the network configuration is, as it’s generated automatically from database.

If it’s pure CLI jockey network, it can be a challenge.

 

But even this is isn’t as good as it should be, as each neighbour would need to have their own policer, like Cisco LPTS or Juniper ddos-protection can do.

But this is one of those things where you’ll rather carry a risk than overcomplicate the configuration.

 

> any any eq 179″ as above then I feel I need to police, unless I can

> guarantee at the edge I have filtered out traffic on TCP 179 from

> everywhere it shouldn’t be coming from. What approach to others take

> here? Why?

 

I have opted for simplicity, I have CoPP class for internal signalling stuff, which is critical, another for important customer/peer stuff, like BGP, and another for unimportant stuff (ping, udp traceroute…)

 

> What do people do with unusual traffic like IP fragments? I am

> discarding them. Thoughts?

 

If you can get away discarding fragments hitting control-plane, do it. If not, police it.

 

> What about packets with IP options set, I am allowing record-route

> only and dropping the rest. Thoughts?

 

No on expects IP options to work in the Internet, there is mls command to police them, I would do that, even for transit

 

> ICMP, I’m just proposing to allow the follow:

> ip access-list extended CoPP-Limit-and-Permit-ICMP  permit icmp any

> any echo  permit icmp any any echo-request  permit icmp any any

> unreachable  permit icmp any any ttl-exceeded  permit icmp any any

> packet-too-big  deny icmp any any

>

> Again, any thoughts there?

 

Never use ‘deny’ in PFC3 CoPP ACLs. It’s not needed, and it may not be supported and may cause negative match and stop of evaluation (i.e. won’t fall to next classs).

Cisco, Netwerk (C-NSP), Security
1 … 6 7 8