Troubleshooting Firewalls (2012 San Diego)

2. Agenda Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Online Resources Best PracticesBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 3

3. Packet Flow

4. Understanding the Packet Flow To effectively troubleshoot a problem, one must first understand the packet path through the network Attempt to isolate the problem down to a single device Then perform a systematic walk of the packet path through the device to determine where the problem could be For problems relating to the Cisco ASA/FWSM, always ‒ Determine the flow: SRC IP, DST IP, SRC port, DST port, and protocol ‒ Determine the interfaces through which the flow passes Note: All Firewall Issues Can Be Simplified to Two Interfaces (Ingress and Egress) and the Rules Tied to BothBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 5

5. Example Flow Flow SRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCP DST IP: 198.133.219.25 DST Port: 80 Interfaces Source: Inside Destination: Outside Client: 10.1.1.9 With the Flow Servers Packet Flow Defined, Examination of Configuration Eng Accounting Issues Boils Down to Just the Two Outside Interfaces: Inside and Outside Server: 198.133.219.25BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 6

6. Packet Processing: Ingress Interface CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 YesRX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route AddrPkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP Packet arrives on ingress interface Input counters incremented Software input queue is an indicator of load No buffers and overruns indicates packet drops (usually traffic bursts) ASA# show interface GigabitEthernet0/1 Interface GigabitEthernet0/1 "", is up, line protocol is up Hardware is bcm56800 rev 01, BW 1000 Mbps, DLY 10 usec Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps) Input flow control is unsupported, output flow control is off Active member of Port-channel10 MAC address 4055.3980.0e67, MTU 1500 IP address unassigned 410179 packets input, 42620992 bytes, 0 no buffer Received 138749 broadcasts, 0 runts, 0 giants …BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 7

7. Packet Processing: Locate Connection CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 YesRX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route AddrPkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP Check first for existing connection If connection exists, flow is matched: bypass ACL check If no existing connection ‒ TCP non-SYN packet, drop and log ‒ TCP SYN or UDP packet, pass to ACL checks Established Connection: ASA-5540# show conn TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO Syslog Because of No existing connection, and non-SYN Packet: ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface insideBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 8

8. Packet Processing: NAT Un-Translate CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 YesRX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route AddrPkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP Incoming packet is checked against NAT rules Starting version 8.3+, packet is un-translated first, before ACL check ‒ In version 8.2, incoming packet was subjected to ACL check prior to un-translation More on this in the NAT section of this presentation…BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 9

9. Packet Processing: ACL Check CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 YesRX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route AddrPkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP First packet in flow is processed through interface ACLs ACLs are first match First packet in flow matches ACE, incrementing hit count by one Denied packets are dropped and logged Packet Permitted by ACL: ASA-5540B# show access-list inside access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1) Syslog when Packet Is Denied by ACL: ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access- group "inside"BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 10

10. Packet Processing: Inspections/Sec Checks CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Inspections are applied to ensure protocol compliance  (Optional) customized AIC inspections  NAT-embedded IPs in payload  Additional security checks are applied to the packet  (Optional) packets passed to Content Security and Control (CSC) module Syslog from Packets Denied by Security Check: ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 11

11. Packet Processing: NAT IP Header CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Translate the IP address in the IP header  Translate the port if performing PAT  Update checksums  (Optional) Following the above, pass packet to IPS (AIP) module BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 12

12. Packet Processing: Egress Interface CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet)  Egress interface is determined first by translation rules NOT the routing table  If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface  Example: Inbound Packets to Inside Outside 192.168.12.4 Get Routed to 172.16.0.0/16 Inside Based on Order of Statics DMZ 172.16.12.0/24 172.16.12.4 nat (inside,outside) source static 172.16.0.0-net 192.168.0.0-net nat (dmz,outside) source static 172.16.12.0-net 192.168.12.0-net BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 13

13. Packet Processing: L3 Route Lookup CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Once on egress interface, an interface route lookup is performed  Only routes pointing out the egress interface are eligible  Remember: NAT rule can forward the packet to the egress interface, even though the routing table may point to a different interface Syslog from Packet on Egress Interface with No Route Pointing Out Interface: %ASA-6-110003: Routing failed to locate next hop for TCP from inside:192.168.103.220/59138 to dmz:172.18.124.76/23 BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 14

14. Packet Processing: L2 Address Lookup CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed  Layer 2 rewrite of MAC header  If Layer 2 resolution fails—no syslog  show arp will not display an entry for the L3 next hop  debug arp will indicate if we are not receiving an ARP reply arp-req: generating request for 10.1.2.33 at interface outside arp-req: request for 10.1.2.33 still pending BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 15

15. Packet Processing: Transmit Packet CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Packet is transmitted on wire  Interface counters will increment on interface ASA# show interface GigabitEthernet 0/2 Interface GigabitEthernet0/2 "", is up, line protocol is up Hardware is bcm56800 rev 01, BW 1000 Mbps, DLY 10 usec Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps) Input flow control is unsupported, output flow control is off Active member of Port-channel11 MAC address 4055.3980.0e68, MTU 1500 IP address unassigned … 101315 packets output, 13086040 bytes, 0 underruns 0 pause/resume output 0 output errors, 0 collisions, 0 interface resets 0 late collisions, 0 deferred 0 rate limit drops 0 switch egress policy drops 0 input reset drops, 0 output reset drops BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17. Cisco ASA — Understanding the Architecture ASA processes all packets in software (via the central CPU) All packets are processed first in… usually also first out ASA platforms have software imposed connection limits Multi-CPU / Multi-Core systems hash packets in the same flow to the same CPU/core. 10 Gig interfaces hash flow to same RX ring. Architecture optimized for multi-flow traffic patterns ASASM packet processing is also done in software, unlike FWSMBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 18

18. Cisco ASA — Understanding the Architecture Global Shared Memory Pool ‒ Global Shared Pool of Memory used by ASA processes Show free memory with CLI, ASDM, or ‗show memory‘ and ‗show memory detail‘ commands ASA# show memory Free memory: 250170904 bytes (47%) Used memory: 286700008 bytes (53%) ------------- ------------------ Total memory: 536870912 bytes (100%) ASA# Use ASDM to graph free memory available • If available memory trends down over time, call Cisco TAC %ASA-3-211001: Memory allocation Error • For 64-bit software (8.4+) use CISCO-ENHANCED-MEMPOOL-MIB.my for accurate countersBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 19

19. Cisco ASA — Understanding the Architecture Memory Blocks ‒ Fixed-size blocks of memory allocated at startup, used for packet processing, VPN, etc Current number of free ASA# show blocks blocks available SIZE MAX LOW CNT 0 400 397 400 4 100 99 99 80 403 379 401 256 1200 1190 1195 1550 6511 803 903 2048 1200 1197 1200 2560 264 264 264 4096 100 100 100 8192 100 100 100 Use ASDM to graph free blocks available 16384 102 102 102 65536 16 16 16 ASA# 1550 and 2048 byte blocks are used for processing ethernet frames %ASA-3-321007: System is low on free memory blocks of size 1550 (10 CNT out of 7196 MAX)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 20

20. Maximum ACL Limits  No hard-coded limit on the number of elements (ACEs) in an ACL. Bound only by Memory.  Each ACE uses a minimum of 212 bytes of RAM  However, maximum performance may decrease (typically 10- 15%) as you reach or exceed the Max Recommended ACEs. 5585 ASA 5505 5510 5512-X 5515-X 5520 5525-X 5540 5545-X 5550 5555-X 5580 10/20/40/6 SM 0 Max 500k / 750kRecommended 25k 80k 100k 100k 200k 200k 500k 300k 700k 500k 750k 2 million 1 / 2 million ACEs 1 500k / 750k Tested ACEs 80k 300k 700k 700k 2 million million+ 1 / 2 millionMax Observed 15.959 3.5 million 2.74 million(from customers) million (SSP-40) Note: Issue show access-list | include elements to see how many ACEs you have BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 21

21. Warning - ACE Explosion Object-groups: Single line ACL • Sources (10 addresses) explodes to • Destinations (21 addresses) • Ports (33 ports) • Result: 10x21x33 = 6,930 rules Nested object-groups: • Assume you add a SRC object-group to the above, which contains 25 additional sources • Result: (10+25)x21x33 = 24,255 rules (ACEs)• New command to reduce ACL memory impact for large ACLs. Available starting in 8.3(1) ASA-5585(config)# object-group-search access-controlBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 22

22. Global ACLs ASAInterface Independent Policies Only Global ACLs introduced in version 8.3 Best used for new installations, or config migration from other vendors ASA(config)# access-group <access_list> global Policy Ordering Interface Specific access-list Global access-list Default (implicit) deny ip any anyBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 23

23. Object-NAT (Auto-NAT) (version 8.3+) Object NAT is the simplest form of NAT, and is defined within an object Host NAT object network obj-WebServer host 10.3.19.50 nat (inside,outside) static 198.51.100.50 Network NAT object network Servers subnet 10.0.54.0 255.255.255.0 nat (inside,outside) static 203.0.113.0 Dynamic PAT (interface overload) object network InternalUsers subnet 192.168.2.0 255.255.255.0 nat (inside,outside) dynamic interface BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 24

24. Manual-NAT (Twice-NAT) (version 8.3+) Manual NAT can specify the source and the destination translation Network Objects object network 10.10.10.0-net subnet 10.10.10.0 255.255.255.0 ! object network 192.168.1.0-net subnet 192.168.1.0 255.255.255.0 Manual NAT Config nat (inside,outside) source static 10.10.10.0-net 10.10.10.0-net destination static 192.168.1.0-net 192.168.1.0-net BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 25

25. NAT Order of Operation version 8.3+ For your reference The ASA configuration is built into the NAT table The NAT Table is based on First Match (top to bottom) The show nat command will display the NAT table in order NAT Table Static NAT Manual NAT Policies First Match Longest Prefix (Section 1) (in config) Shortest Prefix Auto NAT Policies Dynamic NAT (Section 2) Longest Prefix Manual NAT [after auto] Policies First Match Shortest Prefix (Section 3) (in config) BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 26

26. Network Address Translation in ASA 8.3+Optional NAT Parameters in Manual NAT For your reference no-proxy-arp 8.4.(2)+/8.5(1) ‒ For static NAT, disables proxy ARP for incoming packets to the mapped IP addresses. route-lookup 8.4.(2)+/8.5(1) ‒ For identity NAT in routed mode, determines the egress interface using a route lookup instead of using the interface specified in the NAT command. If you do not specify interfaces in the NAT command, a route lookup is used by default. pat-pool <obj> 8.4.(2)+/8.5(1) ‒ Enables a PAT pool of addresses; all addresses in the object are used as PAT addresses and not dynamic 1- to-1 round-robin 8.4.(2)+/8.5(1) ‒ Enables round-robin address allocation for a PAT pool. By default, all ports for a PAT address will be allocated before the next PAT address is used. Round Robin will pick one address from each PAT address extended 8.4.(3) ‒ Enables extended PAT for a PAT pool. Extended PAT uses 65535 ports per service, as opposed to per IP address, by including the destination address and port in the translation information.BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 27

27. Real-IP (version 8.3+) Finally, a reminder that with 8.3+ Real-IPs are used in ACLs ACL contains REAL object network obj-WebServer (local) IP of server host 10.3.19.50 nat (inside,outside) static 198.51.100.50 ! access-list allowIn permit tcp any host 10.3.19.50 eq 80 ! access-group allowIn in interface outside BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 28

28. FWSM—Understanding the Architecture FWSM Processes most packets in hardware, with some packets needing to be processed in software—via the Control Point (CP) Packets processed in hardware have zero impact on CPU Similarly, if the CPU is pegged at 100%, this has zero impact on packets processed in hardware Note that FWSM packet processing is different from ASABRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 29

29. FWSM Architectural Overview Control Point (CP) Control Point Central CPU ACL Compilation, Software Fixups, Syslog, AAA, IPv6 in Hardware Software Session Manager Session Manager NP 3 Session Establishment and Teardown, AAA Cache, ACLs Fast Path Fast Path NP 1 Fast Path NP 2 Flow Identification Security Checks and FWSM C6K Backplane Interface NAT in hardwareBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 30

30. FWSM—Hardware Limits See Appendix FWSM has several hardware limits that should be considered in your network design Limits are hard set, but vary based on single or multimode Some limits include: Increase over 2.3 Increase over 3.2 3.2 / 4.0 /4.1 2.3 (Multimode) 3.1/3.2 (Multimode) 4.0/4.1 (Multimode) Configurable ACEs 56,627 (9,704) 72,806 (11,200) 100,567 (14,801) X AAA Rules 3,942 (606) 6,451 (992) 8,744 (1,345) X Global Statements 1K (1K) 4K (4K) 4K (4K) Static NAT Statements 2K (2K) 2K (2K) 2K (2K) Policy NAT ACEs 3,942 (606) 1,843 (283) 2,498 (384) X NAT Translations 256K (256K) 256K (256K) 256K (256K) Connections 999,990 (999,990) 999,990 (999,990) 999,990 (999,990) Route Table Entries 32K (32K) 32K (32K) 32K (32K) Fixup/Inspect Rules 32 (32 per) 4147 (1,417) 5621 (1,537) X Filter Statements 3942 (606) 2764 (425) 3747 (576) X *Complete List in FWSM Docs, Appendix A (Specifications)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 31

31. show np blocks (FWSM Only) The show np blocks command is used to see if the FWSM is over subscribed Data and Control Data packets Warning packets dropped dropped FWSM# show np blocks MAX FREE THRESH_0 THRESH_1 THRESH_2 NP1 (ingress) 32768 32768 0 0 550 (egress) 521206 521206 0 0 0 NP2 (ingress) 32768 32768 0 0 92 (egress) 521206 521206 0 0 0 NP3 (ingress) 32768 32768 13 460417 4427509 (egress) 521206 521206 0 0 0 BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33. Failover Basics Internet  Active/Standby vs. Primary/Secondary  Stateful failover (optional)  A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall  Both firewalls swap MAC Stateful and IP addresses when a failover occurs LAN Link  Level 1 syslogs will give reason of failover Secondary Primary (Standby) (Active) Corp BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 34

34. Verifying Failover Configuration ASA# show failover Failover On Failover unit Primary Failover LAN Interface: failover Redundant5 (up) Unit Poll frequency 200 milliseconds, holdtime 1 seconds Interface Poll frequency 500 milliseconds, holdtime 5 seconds Interface Policy 1 Monitored Interfaces 2 of 250 maximum Version: Ours 8.2(2), Mate 8.2(1) Last Failover at: 10:37:11 UTC May 14 2010 This host: Primary - Active Interface Monitoring Active time: 1366024 (sec) slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys) Interface outside (10.8.20.241): Normal Interface inside (10.89.8.29): Normal Other host: Secondary - Standby Ready Active time: 0 (sec) slot 0: ASA5580 hw/sw rev (1.0/8.1(2)24) status (Up Sys) Interface outside (10.8.20.242): Normal Interface inside (10.89.8.30): Normal Stateful Failover Logical Update Statistics Link : stateful Redundant6 (up) Stateful Obj xmit xerr rcv rerr General 424525 0 424688 0 sys cmd 423182 0 423182 0BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 35

35. What Triggers a Failover? Power loss/reload (this includes crashes) on the Active firewall SSM interface/module failure The Standby becoming healthier than the Active firewall In LAN based Failover, what happens if the LAN interface communication is severed?BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 36

36. What Triggers a Failover? (Con’t) Two consecutive hello messages missed on any monitored interface forces the interface into testing mode Both units first verify the link status on the interface Next, both units execute the following tests Network activity test ARP test Broadcast ping test The first test passed causes the interface on that unit to be marked healthy; only if all tests fail will the interface be marked failedBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 37

37. How Well do you Understand Failover?What Happens When… You disable failover? (By issuing no failover) You RMA/Replace the Primary unit? You don‘t define Standby IP addresses on interfaces?BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 38

38. What to Do After a Failover Always check the syslogs to determine root cause ‒ Example: switch port failed on inside interface of active firewall Syslogs from Primary (Active) ASA ASA-4-411002: Line protocol on Interface inside, changed state to down ASA-1-105007: (Primary) Link status ‗Down‘ on interface 1 ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier Syslogs from Secondary (Standby) ASA ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me ActiveBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 39

41. Uses of Syslogs Primary mechanism to record traffic to and through the firewall The best troubleshooting tool available Archival Purposes Debugging Purposes Console Syslog/FTP Server ASDM flash Local Trap Syslog. Buffer SNMP Server Monitor (ssh, telnet)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 42

42. ASA Syslog Level vs. Number of Messages Number of Messages (SUM) Log Description Level Ver. 7.0 Ver. 7.2 Ver. 8.0 Ver. 8.1 Ver. 8.2 Ver. 8.3 Ver. 8.4 0 Emergencies 0 0 0 0 0 0 0 1 Alerts 62 (62) 77 (77) 78 (78) 87 (87) 87 (87) 95 (95) 109 (109) 2 Critical 29 (91) 35 (112) 49 (127) 50 (137) 56 (143) 57 (152) 63 (172) More messages 3 Errors 274 (365) 334 (446) 361 (488) 363 (500) 384 (527) 408 (560) 448 (620) 4 Warnings 179 (544) 267 (713) 280 (768) 281 (781) 315 (842) 324 (884) 357 (997) 5 Notifications 161 (705) 206 (919) 216 (984) 218 (999) 237 (1079) 246 (1130) 265 (1242) 6 Informational 234 (939) 302 (1221) 335 (1319) 337 (1336) 368 (1447) 377 (1507) 395 (1637) 217 7 Debugging 258 (1479) 266 (1585) 267 (1603) 269 (1716) 269 (1776) 276 (1913) (1156)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 43

43. What Are Modifiable Syslog Levels? [no] logging message <syslog_id> level <level>  Modifiable syslog levels Levels Allows one to move any syslog message to any level 0—Emergency 1—Alert  Problem 2—Critical You want to record what exec commands are being executed on the firewall; syslog ID 111009 3—Errors records this information, but by default it is at 4—Warnings level seven (debug) 5—Notifications %ASA-7-111009: User ‗johndoe‘ 6—Informational executed cmd: show run 7—Debugging The problem is we don‘t want to log all 1775 other syslogs that are generated at debug level BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 44

44. How to Create Modifiable Syslog Levels  Lower syslog message 111009 to level 3 (error) ASA(config)# logging message 111009 level 3 Ifyou were syslog looks as follows Now our only interested in logging one syslog message, how could you do it? ASA-3-111009: User ‗johndoe‘ executed cmd: show run  To restore the default syslog level ASA(config)# no logging message 111009 level 3 Tip: Use show logging message all to see the default level for any message BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 45

45. Logging – Common Issues  logging flash-bufferwrap – should only be used when logging to buffer at Level 1  logging history – should only be used when you really have an SNMP server that you want to receive all syslogs  logging console – should only be enabled while actively troubleshooting on the Console  logging standby – should only be used if you want to receive double the syslogs  logging permit-hostdown – should always be used with TCP syslogging BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 46

46. Debug Commands1. Debugs should not be the first choice to troubleshoot a problem2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution3. Debugs are not conditional*4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debugBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 47

47. Show Output Filters See Appendix show <cmd> | begin|include|exclude|grep [-v] <regular_exp>  Use output filters to filter the output of show command to only the information you want to see  To use them, at the end of show <Command>, use the pipe character ―|‖ followed by ‒begin Start displaying the output beginning at the first match of the RegEx, and continue to display the remaining output ‒include Display any line that matches the RegEx ‒exclude Display any line that does not match the RegEx ‒grep Same as include ‒grep –v Same as exclude BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 48

48. show cpu usage Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near 100% until ACL is compiled The show cpu usage command displays the CPU over time as a running average ASA# show cpu usage CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 49

49. show processes cpu-usage The show processes cpu-usage command displays the amount of CPU used on a per-process basis for the last 5sec, 1min, and 5min ASA# show process cpu-usage sorted non-zero PC Thread 5Sec 1Min 5Min Process 0x08dc4f6c 0xc81abd38 14.4% 8.2% 8.0% SNMP Notify Thread 0x087798cc 0xc81b0658 6.8% 5.0% 4.9% esw_stats 0x081daca1 0xc81bcf70 1.3% 1.1% 1.0% Dispatch Unit 0x08e7b225 0xc81a28f0 1.2% 0.1% 0.0% ssh 0x08ebd76c 0xc81b5db0 0.6% 0.3% 0.3% Logger 0x087b4c65 0xc81aaaf0 0.1% 0.1% 0.1% MFIB 0x086a677e 0xc81ab928 0.1% 0.1% 0.1% ARP Thread ASA# *First Introduced in Cisco ASA Version 7.2(4.11), 8.0(4.5), 8.1(1.100), 8.2(1)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 50

50. Show traffic The show traffic command displays the traffic received and transmitted out each interface of the firewall since last ‗clear traffic ‘ ASA# show traffic outside: received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec inside: received (in 124.650 secs): 261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/secBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 51

51. show xlate and show xlate debug The show xlate command displays information about the translations through the firewall You can limit the output to just the local or global IP ASA-5585# show xlate 5014 in use, 5772 most used TCP PAT from inside:192.168.103.220/57762 to outside:10.2.1.2/43756 flags ri idle 0:00:00 timeout 0:00:30 TCP PAT from inside:192.168.103.220/57761 to outside:10.2.1.2/54464 flags ri idle 0:00:00 timeout 0:00:30 ASA-5585# show nat pool TCP PAT pool outside, address 10.2.1.2, range 1-511, allocated 1 TCP PAT pool outside, address 10.2.1.2, range 512-1023, allocated 0 TCP PAT pool outside, address 10.2.1.2, range 1024-65535, allocated 2321 ASA-5585# Added in version 8.3BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 52

52. show nat detail The show nat command displays information about the nat table of the firewall The detail keyword will display object definitions ASA-5585# show nat detail Manual NAT Policies (Section 1) 1 (inside) to (outside) source static science-obj science-obj destination static vpn-obj vpn-obj translate_hits = 0, untranslate_hits = 0 Source - Origin: 192.168.0.0/16, Translated: 192.168.0.0/16 Destination - Origin: 172.16.1.0/24, Translated: 172.16.1.0/24 Auto NAT Policies (Section 2) 1 (dmz) to (outside) source static webserver-obj 14.36.103.83 translate_hits = 0, untranslate_hits = 3232 Source - Origin: 192.168.22.32/32, Translated: 14.36.103.83/32 2 (inside) to (outside) source dynamic science-obj interface translate_hits = 37723, untranslate_hits = 0 Source - Origin: 192.168.0.0/16, Translated: 14.36.103.96/16 ASA-5585/admin#BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 53

53. Show Conn and Show Conn Detail Real Interface Names Added in Idle Time, Connection 7.2(4), 8.0(4) Bytes Transferred Flags ASA# show conn 2 in use, 64511 most used TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIO UDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags – detail Adds Uptime ASA# show conn detail and Timeout in 7.2(4), 2 in use, 64511 most used 8.0(4) Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN, B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump, E - outside back connection, F - outside FIN, f - inside FIN, G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data, i - incomplete, J - GTP, j - GTP data, K - GTP t3-response k - Skinny media, M - SMTP data, m - SIP media, n - GUP O - outbound data, P - inside back connection, q - SQL*Net data, R - outside acknowledged FIN, R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN, s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS, X - inspected by service module TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101, flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127 UDP outside:172.18.124.1/123 dmz:10.1.1.9/123, flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 54

54. Example—Connection Build Up  Firewall receives an initial SYN packet from the inside; the SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags saA  The outside device responds to the SYN packet with a SYN+ACK; the connection flags are updated to reflect this, and now show A  The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake, and the connection is now considered up (U flag)  The outside device sends the first data packet; the connection is updated and an I is added to the flags to indicate the firewall received Inbound data on that connection  Finally, the inside device has sent a data packet and the connection is updated to include the O flag Connection Flags SYN+ACK SYN Data ACK 1 5 3 42 UI UIO s A UaA Inside Outside Client ServerBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 55

55. Example—Connection Teardown  Firewall receives a FIN packet from the inside; as the FIN passes through the firewall, it updates the connection flags by adding an f to indicate that the FIN was received on the Inside interface  The outside device immediately responds to the FIN packet with a FIN+ACK; the connection flags are updated to reflect this, and now show UfFR  The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection; thus, there are no more connection flags, because the connection no longer exists Connection Flags FIN+ACK ACK FIN 3 1 2 UfFR Uf r UfFR Inside Outside Client ServerBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 56

57. TCP Connection Termination Reasons  If a TCP connection is built through the firewall, it will always have a teardown reason  The TCP teardown syslog is logged at level six  If you are having problems with connections abnormally What does the Reset-O Termination reason mean in the closing, temporally increase your logging level (or move Teardown TCP connection syslog? the syslog down), and check the teardown reason ASA-6-302014: Teardown TCP connection number for intf_name:real_IP/real_port to intf_name:real_IP/real_port duration time bytes number [reason] [(user)]BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 58

58. TCP Connection Termination Reasons—Quick Reference For your reference Reason Description Connection Ended Because It Was Idle Longer Than the Conn-Timeout Configured Idle Timeout Deny Terminate Flow Was Terminated by Application Inspection The Standby Unit in a Failover Pair Deleted a Connection Failover Primary Closed Because of a Message Received from the Active Unit Force Termination After Ten Minutes Awaiting the Last ACK or FIN Timeout After Half-Closed Timeout Flow Closed by Inspection Flow Was Terminated by Inspection Feature Flow Terminated by IPS Flow Was Terminated by IPS Flow Reset by IPS Flow Was Reset by IPS Flow Terminated by Flow Was Terminated by TCP Intercept TCP Intercept Invalid SYN SYN Packet Not Valid Connection Timed Out Because It Was Idle Longer than the Idle Timeout Timeout Value IPS Fail-Close Flow Was Terminated Due to IPS Card Down SYN Control Back Channel Initiation from Wrong Side BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 59

59. TCP Connection Termination Reasons—Quick Reference(Cont.) For your Reason Description reference Force Termination After Two Minutes Awaiting SYN Timeout Three-Way Handshake Completion TCP Bad Retransmission Connection Terminated Because of Bad TCP Retransmission TCP Fins Normal Close Down Sequence TCP Invalid SYN Invalid TCP SYN Packet TCP Reset-I TCP Reset Was Sent From the Inside Host TCP Reset-O TCP Reset Was Sent From the Outside Host TCP Segment Partial Overlap Detected a Partially Overlapping Segment TCP Unexpected Window Connection Terminated Due to a Variation in the Size Variation TCP Window Size Tunnel Has Been Torn Down Flow Terminated Because Tunnel Is Down Unauth Deny Connection Denied by URL Filtering Server Unknown Catch-All Error Xlate Clear User Executed the ‗Clear Xlate‘ Command BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 60

60. show local-host command A local-host entry is created for any IP tracked through the firewall It groups the xlates, connections, and AAA information Very useful for seeing the connections terminating on servers ASA# show local-host detail connection tcp 50 Interface dmz: 0 active, 0 maximum active, 0 denied Interface inside: 1 active, 1 maximum active, 0 denied local host: <192.168.103.220>, TCP flow count/limit = 798/unlimited TCP embryonic count to host = 0 TCP intercept watermark = unlimited UDP flow count/limit = 0/unlimited Conn: TCP outside:172.18.124.76/80 inside:192.168.103.220/34078, flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0 TCP outside:172.18.124.76/80 inside:192.168.103.220/34077, flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0 (output truncated)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 61

61. show service-policy The show service-policy command is used to quickly see what inspection policies are applied and the packets matching them ASA-5585/admin# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default Inspect: ftp, packet 0, lock fail 0, drop 0, reset-drop 0 … Inspect: sqlnet, packet 0, lock fail 0, drop 0, reset-drop 0 Inspect: http, packet 1215927, lock fail 0, drop 0, reset-drop 0 Inspect: icmp, packet 57, lock fail 0, drop 0, reset-drop 0 ASA-5585/admin# …BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 62

62. show service-policy flow Use to determine what policies a given flow will match in the Modular Policy Framework (MPF) ASA# show service-policy flow tcp host 10.1.9.6 host 10.8.9.3 eq 1521 Global policy: Service-policy: global_policy Interface outside: Service-policy: outside Class-map: oracle-dcd Match: access-list oracle-traffic Access rule: permit tcp host 10.1.9.6 host 10.8.9.3 eq sqlnet Action: Input flow: set connection timeout dcdBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 63

63. show asp drop Packets dropped in the Accelerated Security Path (ASP) will increment a counter FWSM – applies only to traffic sent to the control-point Frame drop counters are per packet, flow drops are per flow Some counters have corresponding syslogs ASA# show asp drop Frame drop: Invalid encapsulation (invalid-encap) 10897 Invalid tcp length (invalid-tcp-hdr-length) 9382 Invalid udp length (invalid-udp-length) 10 No valid adjacency (no-adjacency) 5594 No route to host (no-route) 1009 Reverse-path verify failed (rpf-violated) 15 Flow is denied by access rule (acl-drop) 25247101 First TCP packet not SYN (tcp-not-syn) 36888 Bad TCP flags (bad-tcp-flags) 67148 TCP option list invalid (tcp-bad-option-list) 731 TCP MSS was too large (tcp-mss-exceeded) 10942 Bad TCP Checksum (bad-tcp-cksum) 893 *Drop Counters Are Documented in the CMD Ref, Under show asp dropBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 64

64. Packet Capture  Capture command first introduced in Cisco 7.0; FWSM need to use 3.1.5 or later  ASA 7.2(3) and 8.0(3) added a real-time option  ASDM 6.0 adds a capture wizard  Capture sniffs packets on an interface that match an ACL, or match line  Key steps ‒ Use the ‗match‘ keyword to specify what traffic to capture (implicitly bi-directional) ‒ Define the capture and bind it to an access-list and interface ‒ View the capture on the firewall, or copy it off in .pcap format ASA# capture outcap interface outside match ip any host 10.1.2.3 ASA# capture incap interface inside match ip any host 10.1.2.3 ASA# ASA# show cap outcap 2 packets captured 1: 16:04:28.023741 802.1Q vlan#10 P0 172.18.254.139 > 10.1.2.3: icmp: echo request 2: 16:04:29.023741 802.1Q vlan#10 P0 172.18.254.139 > 10.1.2.3: icmp: echo request BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 65

65. Packet Capture (Cont.) See Appendix  Traffic can be captured both before and after it passes through the firewall; one capture on the inside interface, one capture on the outside interface  Capture buffer saved in RAM (default size 512 KB)  Default is to stop capturing when buffer is full  Default packet length is 1518 bytes  Copy captures off via TFTP or HTTPS Inside Capture Outside Capture Inside Outside Capture In Capture OutBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 66

66. Where Packets Are Captured in Packet Flow CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 YesRX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route AddrPkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP Ingress Packets Egress Packets Captured Captured  Packets are captured at the first and last points they can be in the flow  Ingress packets are captured before any packet processing has been done on them  Egress packets are captured after all processing (including L2 source MAC rewrite) BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 67

67. Capturing Packets Dropped by the ASP ASA Only  Capture all packets dropped by the ASP ASA# capture drops type asp-drop all  Capture on a specific drop reason ASA# capture drops type asp-drop tcp-not-syn  Applies to both ASA and FWSM ASA# capture drop type asp-drop ? acl-drop Flow is denied by configured rule all All packet drop reasons bad-crypto Bad crypto return in packet bad-ipsec-natt Bad IPSEC NATT packet bad-ipsec-prot IPSEC not AH or ESP bad-ipsec-udp Bad IPSEC UDP packet bad-tcp-cksum Bad TCP checksum bad-tcp-flags Bad TCP flagsBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 68

68. Packet Tracer - Overview ASA Only Introduced in ASA version 7.2 A packet tagged with the trace option is injected into the interface, and processed in the data-plane Each action taken on the packet is recorded in the packet itself When the packet reaches the egress interface, or is dropped, it is punted to the control-plane The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configurationBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 69

69. Packet Tracer: ASDM (Located off Tools Menu) Define Packet Action Matching Link Back Config to Edit Rule Final ResultBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 70

70. Packet Tracer: Example Output ASA# packet-tracer input outside tcp 172.18.124.66 1234 172.18.254.139 3389 Phase: 1 Type: CAPTURE Subtype: Result: ALLOW Config: Additional Information: MAC Access list Phase: 2 Type: ACCESS-LIST Subtype: Result: ALLOW Config: Implicit Rule Additional Information: MAC Access list Phase: 3 Type: UN-NAT Subtype: static Result: ALLOW Config: nat (outside,dmz) source dynamic any interface destination static interface Win7-vm service rdp-outside rdp-outside Additional Information: NAT divert to egress interface dmz Untranslate 172.18.254.139/3389 to 192.168.103.221/3389 …….BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 71

71. Packet Tracer: Example Output Phase: 4 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group outside_in in interface outside access-list outside_in extended permit tcp any any eq 3389 Additional Information: …… Phase: 8 Type: NAT Subtype: Result: ALLOW Config: nat (outside,dmz) source dynamic any interface destination static interface jajohnstWin7-vm service rdp-outside rdp- outside Additional Information: Dynamic translate 172.18.124.66/1234 to 192.168.103.221/1234 …… Phase: 12 Type: FLOW-CREATION Subtype: Result: ALLOW Config: Additional Information: New flow created with id 16538274, packet dispatched to next moduleBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 72

72. Packet Tracer: Tracing Captured Packet Important!  Create a capture using the trace option ASA# capture inside access-list web interface inside trace  Find the packet in the capture you want traced ASA# show capture inside 68 packets captured 1: 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: S 2: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: S ack 3: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: . ack 4: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: P ack 5: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: . Ack ...  Then select that packet to be traced ASA# show capture inside trace packet-number 4 .BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 73

73. TCP Ping  New troubleshooting tool added in ASA ver 8.4.1  Why is it needed??? Consider the following… www server 10.1.1.7 (209.165.200.225)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 74

74. TCP Ping  Previously – limited reachability tools: Ping and Traceroute  Access to client machine? Attempts to about the path What validate NAT and/or PAT? …but with ICMP ICMP Echo Request ICMP Echo Request ICMP Echo Reply ICMP Echo Reply www server 10.1.1.7 (209.165.200.225)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 75

75. TCP Ping  Sources TCP SYN packet with Client’s IP and injects it into Client’s interface of the ASA Packet with SRC of 10.1.1.7 injected TCP SYN sent on Inside interface to server Internal hosts are PATed to 198.51.100.2 inside outside www server 10.1.1.7 (209.165.200.225) TCP SYN+ACK ASA Datapath Packet PATed to sent from server Validated 198.51.100.2 (NAT, ACLs, etc) on EgressBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 76

76. TCP Ping – The Big Picture  Validates 2 of the 3 legs of the connection from client to server TCP path from client side of ASA to Server through the cloud -Validated- inside outside www server 10.1.1.7 (209.165.200.225) 1st Leg 2nd Leg 3rd LegBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 77

77. TCP Ping - Example Specify Client‘s asa# ping tcp Interface: inside source Interface Target IP address: 209.165.200.225 Target IP port: 80 Specify Client‘s Specify source? [n]: y Source IP address: 10.1.1.7 real IP Address Source IP port: [0] Repeat count: [5] Timeout in seconds: [2] Type escape sequence to abort. Sending 5 TCP SYN requests to 209.165.200.225 port 80 from 10.1.1.7 starting port 3465, timeout is 5 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms inside outside www server 10.1.1.7 (209.165.200.225)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 78

79. Case StudyLeveraging Smart Call Home

80. Case Study: Smart Call Home ASA Only Email CMD Output to You  Objective – Send the output of a command directly to your e-mail.  This is easily accomplished with SCH. Use the command: call-home send <―cmd‖> email <email_addr> Example: call-home send ―show run‖ email [email protected]  This will send a plain-text e-mail with the output of the command to the e- mail address specified, with the command in the subject line. ‒ Example: Subject: CLI ‗show run‘ outputBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 81

81. Case Study: Smart Call Home Collecting Memory Diagnostics over Time  Objective – Memory appears to be depleting over time on your ASA. Use SCH to collect the detailed memory output hourly, for further investigation.  This is easily accomplished with SCH. Setting a ‖snapshot‖ alert- group to e-mail commands at a specified interval  Snapshot will contain the following command: show conn count show memory detailBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 82

82. Case Study: Smart Call Home Example Config service call-home call-home alert-group-config snapshot add-command ―show conn count‖ add-command "show memory detail― contact-email-addr [email protected] sender from [email protected] sender reply-to [email protected] mail-server smtp-server.cisco.com priority 1 profile SENDCMD active destination address email [email protected] destination preferred-msg-format long-text destination transport-method email subscribe-to-alert-group snapshot periodic hourlyBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 83

83. Case StudyIntermittent Access to Web Server

84. Case Study: Intermittent Access to Web Server Problem  Most external clients are not able to load company‘s web page NATed to 10.1.1.50 HTTP Requests to 192.168.1.50 Internet Web Server ASA-5510 10.1.1.50 ClientsBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 85

86. Case Study: Intermittent Access to Web Server  show perfmon indicates high number of embryonic connections ASA-5510# show perfmon PERFMON STATS: Current Average Xlates 0/s 0/s Connections 2059/s 299/s TCP Conns 2059/s 299/s UDP Conns 0/s 0/s URL Access 0/s 0/s URL Server Req 0/s 0/s TCP Fixup 0/s 0/s TCP Intercept Established Conns 0/s 0/s TCP Intercept Attempts 0/s 0/s TCP Embryonic Conns Timeout 1092/s 4/s HTTP Fixup 0/s 0/s FTP Fixup 0/s 0/s AAA Authen 0/s 0/s AAA Author 0/s 0/s AAA Account 0/s 0/s VALID CONNS RATE in TCP INTERCEPT: Current Average N/A 95.00%BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 87

87. Case Study: Intermittent Access to Web Server  Issue show conn to see ‗who‘ is creating the connections Random Sources Embryonic Conns ASA-5510# show conn 54764 in use, 54764 most used TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aBBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 88

89. Case Study: Intermittent Access to Web Server  Apply TCP Intercept to stop the SYN flood attack access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 ! service-policy interface_policy interface outsideBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 90

91. Case Study: Intermittent Access to Web Server  Apply per-client-max option to limit the number of connections any single client can establish access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 per-client-max 25 ! service-policy interface_policy interface outsideBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 92

95. Online Resources  Support Communities - Supportforums.cisco.com  TAC Security Show Podcast  Online learning modules (VoD Training)  Security RSS FeedsBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 96

96. Supportforums.cisco.com  Public wiki – anyone can author articles  Combines supportwiki and Netpro forums  Sections for: ASA, FWSM and PIX  Hundreds of Sample Configs  Troubleshooting Docs  FAQs http://supportforums.cisco.com/BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 97

97. IPSec VPNs Using Certificates on ASA and IOS Interesting TAC CasesHTTP filteringon the ASA  Great way to obtain valuable troubleshooting insights. ASA Services  Conversational shows, which Module focus on providing in-depth information on a given feature.  New episodes posted Monthly Monitoring Firewall Using Packet Captures Performance on ASA and FWSM Useful ASA and ASA AnyConnect VPN IPS Commands Cisco Identity Services Engine TCP Connections through ASA and FWSM Cisco Identity Services Engine Multiple Context Mode BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 98

98. Online Learning Modules – VoD Training  Great way to learn about new features in the ASA  From www.cisco.com select: Products and Services Security Network Security (expand) Cisco ASA 5500 Series Training resources Online learning modules  Search cisco.com for ASA Online Learning Modules  Direct link ‒ http://www.cisco.com/en/US/partner/products/ps6120/tsd_ products_support_online_learning_modules_list.htmlBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 99

99. Security Hot Issues – RSS Feeds  Subscribe with an RSS reader  Receive weekly updates on the Hot Issues customers are facing  Separate feeds for: ASA, FWSM, ASDM https://supportforums.cisco.com/docs/DOC-5727BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 100

101. Cisco ASA/FWSM Best Practices  Enable ip verify reverse-path on all interfaces  Set embryonic and maximum connection counts for traffic destined to your servers and destined to the internet  Configure logging to syslog server  Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you don‘t want to see  Enable authentication for management access (console/ SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL as the fallbackBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 102

102. Cisco ASA/FWSM Best Practices  Baseline CPU load, connection counts, xlate counts, and traffic (per interface)  Monitor stats using MRTG or other snmp graphing tools  Keep config archives and show tech outputs (use smart call home)  Run the latest maintenance release in your train  Upgrade major feature trains only when you need new features, or after train has maturedBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 103

103. Complete Your OnlineSession Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our Don‘t forget to activate your portal) or visit one of the Internet Cisco Live Virtual account for access to stations throughout the Convention all session material, communities, and on-demand and live activities throughout Center. the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 104

104. Final Thoughts Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042 Come see demos of many key solutions and products in the main Cisco booth 2924 Visit www.ciscoLive365.com after the event for updated PDFs, on- demand session videos, networking, and more! Follow Cisco Live! using social media: ‒ Facebook: https://www.facebook.com/ciscoliveus ‒ Twitter: https://twitter.com/#!/CiscoLive ‒ LinkedIn Group: http://linkd.in/CiscoLIBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 105

106. Q&A

107. Appendix  Lucky You  This appendix contains extra information which you may find useful, but I just didn‘t have enough time to cover in the lecture – or which was covered in previous years.  Enjoy… :-)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 108

108. Appendix  ASA 8.3 Memory Requirements  SNMP OIDs to Monitor  Example: Show Output Filters  Code Base History  Case studies Poor Voice Quality Out-of-order packet buffering TCP MSS issue Out of memory High CPU Capture Example  FWSM Additional Architecture Slides  Failover Extras  Packet Capture Example  Online Tools  ASDM  Information to include when opening a TAC caseBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 109

109. Redirecting Debugs to Syslog  Problem ‒ Log only debug output to syslog  Solution ‒ Create a logging list with only syslog ID 711001 ‒ ASA(config)# logging list Networkers message 711001 . Enable debug output to syslogs ‒ ASA(config)# logging debug-trace INFO: logging debug-trace is enabled. All debug messages are currently being redirected to syslog:711001 and will not appear in any monitor session ‒ Log on the logging list ASA(config)# logging trap Networkers .BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 110

110. ASA 8.3 Memory Requirements  ASA Models 5505 – 5540 Require Memory Upgrades before upgrading to ASA version 8.3  New ASAs ship with the upgraded RAM installed ASA Model Original Required RAM Upgrade Kit Part Default RAM for version 8.3 Number 5505 * 256 MB 512 MB ASA5505-MEM-512= 5510 256 MB 1024 MB ASA5510-MEM-1GB= 5520 512 MB 2048 MB ASA5520-MEM-2GB= 5540 1024 MB 2048 MB ASA5540-MEM-2GB= * For the 5505, only the Security Plus or Unlimited licenses require the memory upgradeBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 111

111. ASA Software Trains Bug Fixes Waterfall Down 7.2.1 7.2.2 7.2.3 7.2.4 7.2.57.2 8.0.2 8.0.3 8.0.4 8.0.58.0 EOL 8.1.1 8.1.28.1 ASA-5580 only EOL 8.2.1 8.2.2 8.2.3 8.2.4 8.2.58.2 8.3.1 8.3.28.3 8.4.1 8.4.2 8.4.3 8.4.48.4 8.5.18.5 ASA-SM only 8.6.18.6 BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 112

112. High Availability – Zero Downtime Upgrades State State Start Act Stb Primary Secondary Issue ―failover active‖ Act Stb Copy new image over and reboot Wait for failover to finish syncing, and to ―normalize‖ – approx 2 min Verify config; conns replicated Act Issue ―failover active‖ Copy new image over and reboot Stb Wait for failover to finish syncing, and to ―normalize‖ – approx 2 min Verify config; conns replicated Upgrade Complete BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 113

113. SNMP OIDs For your reference  CPU usage • 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 (5 sec) • 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 (1 min) • 1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 (5 min)  Connections • 1.3.6.1.4.1.9.9.147.1.2.2.2.1.5.40.6 (Current total) • 1.3.6.1.4.1.9.9.147.1.2.2.2.1.5.40.7 (Max total)  Traffic • 1.3.6.1.2.1.2.2.1.{10|16}.n (in/out octets) • Use SNMPwalk to verify the interfaces!BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 114

114. Example: Show Output Filters show <cmd> | begin|include|exclude|grep [-v] <regular_exp> Examples  Display the interface stats starting with the ‗inside‘ interface ‒show interface | begin inside  Display the access-list entries that contain address 10.1.1.5 ‒show access-list | grep 10.1.1.5  Display the config, except for the access-lists ‒show run | exclude access-list  Display only access-list entries that have non-zero hitcounts ‒show access-list | grep –v hitcnt=0  Display a count of the number of connections each host has ‒show local-host | include host|count/limit Note: You must Include a Space on Either Side of the Pipe for the Command to Be Accepted; Also, Trailing Spaces Are CountedBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 115

115. Cisco PIX/ASA/FWSM Code Base History In Sync PIX/ASA PIX Feature Releases 6.0(1) 6.1(1) 6.2(1) 6.3(1) 7.0(1) 7.1(1) 7.2(1) 8.0(2) 8.3(1) Port Features Bug Fixes FWSM 1.1(1) 2.2(1) 2.3(1) 3.1(1) 3.2(1) 4.0(1) 4.1(1) Feature Releases SafeHarbor 2.3(2) 4.0(2) 4.0(4) 4.0(11) 1.1(2) 1.1(3) Maintenance SafeHarbor Releases 3.2(2) 3.2(4) 3.2(17) SafeHarbor GD 3.1(2) 3.1(6) 3.1(10) 3.1(17) TimeBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 116

116. Debug ICMP Trace Internet http://www.cisco.com  Valuable tool used to troubleshoot connectivity issues  Provides interface and translation information to quickly determine flow  Echo-replies must be explicitly permitted through ACL, or ICMP inspection must be enabled Example debug icmp trace output ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80 ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22 ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2 BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 117

117. Case StudyPoor Voice Quality

118. Case Study: Poor Voice Quality Problem  Poor outbound voice quality at SOHO sites Outbound RTP Stream Cable WAN 100 Mbps 100 Mbps 2 Mbps Modem ASA-5505BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 119

119. Case Study: Poor Voice Quality Solution: Traffic Shaping  What is traffic shaping, and why is it needed here?  Why won‘t policing work?  Why won‘t priority queuing alone work? Shape to 2 Mbps Cable WAN Modem 2 Mbps 100 Mbps ASA-5505 100 MbpsBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 120

120. Case Study: Poor Voice Quality – Configuration Example (Traffic Shaping)Solution Prioritize voice traffic and shape all traffic down to 2 Mbps on the outside interface. class-map voice-traffic match dscp af13 ef ! policy-map qos_class_policy class voice-traffic priority ! policy-map qos_outside_policy class class-default shape average 2000000 service-policy qos_class_policy ! service-policy qos_outside_policy interface outside  To view statistics on the operation of the shaper, use the command show service-policy shapeBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 121

121. Case Study: Poor Voice Quality Things to Keep in Mind:  Shaping can only be applied to the class class-default  Shaping only works in the outbound direction on an interface  The shaping value is in bits per second, and must be a multiple of 8000  The shaping policy is applied to all sub-interfaces on a physical interface  Not supported on the ASA-5580 platform  Not supported in Transparent or Multi-context modeBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 122

122. Case Study Out-of-Order Packet Buffering

123. Case Study: Out-of-Order Packets Inspections require ordered packets Packets sent to the SSM (AIP and CSC) require ordered packets Cisco ASA/PIX will buffer up to three packets by default Buffering can be increased on ASA by using the queue- limit option under the tcp-mapBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 124

124. Case Study: Out-of-Order Packets Problem  Some networks have high numbers of out-of-order packets; often caused by asymmetric traffic flows  If the out-of-order packet buffer isn‘t large enough, traffic is dropped and packets must be retransmitted Inside Outside 192.168.1.30 10.16.9.2 Client Server Packet 10 Dropped on Network Packet 11 Packet 12 Buffer Packet 13 Packet 14 Dropped by Firewall Packet 15BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 125

125. Case Study: Out-of-Order Packet Buffering Example  How to detect? ASA# show asp drop Frame drop: ... TCP packet SEQ past window 46331 TCP packet buffer full 90943 ...  How to fix? access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0 ! tcp-map OOO-Buffer queue-limit 6 ! class-map tcp-options match access-list OOB-nets ! policy-map global_policy class tcp-options set connection advanced-options OOO-Buffer ! service-policy global_policy globalBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 126

126. Case Study: Out-of-Order PacketBuffering Example  How to verify? ASA# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default ... Class-map: tcp-options Set connection policy: Set connection advanced-options: OOB-Buffer Retransmission drops: 0 TCP checksum drops : 0 Exceeded MSS drops : 0 SYN with data drops: 0 Out-of-order packets: 2340 No buffer drops : 0BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 127

127. Case Study TCP MSS (Maximum Segment Size)

128. Case Study: TCP MSS  MSS is the Maximum Segment Size—or the maximum amount of data that can be sent in a single packet  The MSS is set in the SYN packets  The device that receives the MSS advertisement cannot send more data in a single packet to the peer than specified by the MSSBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 129

129. Case Study: TCP MSS Problem  Some servers have broken TCP stacks and ignore the MSS advertised by the Client  The firewall will drop packets that exceed the advertised MSS Inside Outside 192.168.1.30 10.16.9.2 Client Server SYN MSS=1380 SYN+ACK MSS=1400 DATA=1390BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 130

130. Case Study: TCP MSS Example  How to detect? ASA# show asp drop Frame drop: TCP MSS was too large 943 %ASA-4-419001: Dropping TCP packet from outside:10.16.9.2/80 to inside:192.168.1.30/1025, reason: MSS exceeded, MSS 1380, data 1390  How to fix? access-list MSS-hosts permit tcp any host 10.16.9.2 ! tcp-map mss-map exceed-mss allow ! class-map mss match access-list MSS-hosts ! policy-map global_policy class mss set connection advanced-options mss-map ! service-policy global_policy globalBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 131

131. Case Study: TCP MSS Example How to verify? ASA# capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518 ASA# show capture mss-capture 0 packets captured 0 packets shown  How else could you verify?BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 132

132. Case Study Out Of Memory

133. Case Study: Out of Memory Problem  Users are unable to access the Internet  No new connections are working  All old (long lived) connections continue to work Step 1: Check the Syslogs %PIX-3-211001: Memory allocation Error %PIX-3-211001: Memory allocation Error Step 2: Check the Amount of Free Memory Available Hardware: PIX-515E, 64 MB RAM pixfirewall# show memory Free memory: 714696 bytes Used memory: 66394168 bytes ------------- ---------------- Total memory: 67108864 bytesBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 134

134. Case Study: Out of Memory Step 3: What Eats Up Memory (RAM) on the Cisco PIX?  Cisco PIX image (run from RAM)  Configuration What Can Eat Up  IPSec database 64 MB on a Cisco  Xlates (translations) PIX-515E?  Connections A Small Global Pool Is Step 4: Let’s Check the Translations Used, Overloading to a pixfirewall# show xlate PAT Address 251 in use, 258 most used PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716) PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276) Global 209.165.201.25 Local 10.1.1.102 PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783) PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197) PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543) Varied Source IPsBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 135

135. Case Study: Out of Memory Step 5: Check the Connections pixfirewall# show conn 147456 in use, 147456 most used TCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags OIU TCP out 64.101.22.236:80 in 10.1.1.74:32209 idle 0:00:14 Bytes 239 flags OIU TCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saA TCP out 64.103.31.215:80 in 10.1.1.136:18664 idle 0:00:46 Bytes 934 flags OIU TCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags OIU TCP out 64.101.205.10:135 in 10.1.1.139:62296 idle 0:00:15 Bytes 0 flags saA TCP out 64.101.200.200:80 in 10.1.1.83:51864 idle 0:00:32 Bytes 902 flags OIU TCP out 64.102.80.27:80 in 10.1.1.66:52301 idle 0:00:03 Bytes 7813 flags OIU TCP out 64.103.95.35:80 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags OIU TCP out 64.102.206.172:80 in 10.1.1.223:28585 idle 0:00:28 Bytes 239 flags OIU TCP out 64.102.57.106:80 in 10.1.1.135:44945 idle 0:00:48 Bytes 9717 flags OIU TCP out 64.102.21.85:80 in 10.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIU TCP out 64.101.25.203:80 in 10.1.1.170:28149 idle 0:00:47 Bytes 419 flags OIU TCP out 64.101.86.97:135 in 10.1.1.54:43703 idle 0:00:12 Bytes 0 flags saA . . .BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 136

136. Case Study: Out of Memory Take a Look at the Traffic Load pixfirewall# show traffic outside: received (in 25.000 secs): Outside 1475 packets 469050 bytes 59 pkts/sec 18762 bytes/sec transmitted (in 25.000 secs): 167619 packets 9654480 bytes 6704 pkts/sec 386179 bytes/sec inside: Traffic Flow received (in 25.000 secs): 180224 packets 10410480 bytes 7208 pkts/sec 416419 bytes/sec Inside transmitted (in 25.000 secs): 1050 packets 118650 bytes 42 pkts/sec 4746 bytes/sec  Vast majority of traffic is coming in the inside interface and going out the outside interfaceBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 137

137. Case Study: Out of Memory Step 6: Review What We Know and Take Action pixfirewall# show conn count 147456 in use, 147456 most used pixfirewall# show xlate count 251 in use, 258 most used Conn Count Is Very High, but xlate Count Is Low  Many connections per xlate  Probably one, or a few hosts, are generating the vast majority of connections  Most likely due to a virus on the host(s)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 138

138. Case Study: Out of Memory Step 7: Find the Host(s) Generating All the Connections pixfirewall# show local-host | include host|count/limit local host: <10.1.1.131>, TCP connection count/limit = 0/unlimited UDP connection count/limit = 0/unlimited Only Show Lines That Have local host: <10.1.1.51>, the Word host or TCP connection count/limit = 2/unlimited count/limit in Them UDP connection count/limit = 0/unlimited local host: <10.1.1.236>, TCP connection count/limit = 0/unlimited UDP connection count/limit = 0/unlimited . . . local host: <10.1.1.99>, TCP connection count/limit = 146608/unlimited UDP connection count/limit = 0/unlimited  Host 10.1.1.99 is eating up all the connections, and they are TCP-based connectionsBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 139

139. Case Study: Out of Memory Step 8: Now that We Found the Host, Let‘s Look at the Connections It Is Generating pixfirewall# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, TCP connection count/limit = 146608/unlimited TCP embryonic count = 146606 Note: All Connections Are UDP connection count/limit = 0/unlimited Embryonic Xlate(s): Global 209.165.201.21 Local 10.1.1.99 Conn(s): TCP out 64.101.32.157:135 in 10.1.1.99:34580 idle 0:01:43 Bytes 0 flags saA TCP out 64.103.108.191:135 in 10.1.1.99:8688 idle 0:01:43 Bytes 0 flags saA TCP out 64.100.205.160:135 in 10.1.1.99:7774 idle 0:01:43 Bytes 0 flags saA TCP out 64.101.182.19:135 in 10.1.1.99:39193 idle 0:01:43 Bytes 0 flags Connections to Random saA TCP out 64.102.218.45:135 in 10.1.1.99:16462 idle 0:01:43 Bytes 0 flags Destinations on TCP/135– saA TCP out 64.100.21.120:135 in 10.1.1.99:30322 idle 0:01:43 Bytes 0 flags MS Blaster saA TCP out 64.101.25.195:135 in 10.1.1.99:41116 idle 0:01:43 Bytes 0 flags saA TCP out 64.103.17.219:135 in 10.1.1.99:59163 idle 0:01:43 Bytes 0 flags saA TCP out 64.102.201.141:135 in 10.1.1.99:2978 idle 0:01:43 Bytes 0 flags saA TCP out 64.103.176.75:135 in 10.1.1.99:41589 idle 0:01:43 Bytes 0 flags saA . . .BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 140

140. Case Study: Out of Memory  Cisco PIX provides two methods to limit the number of connections per host ‒ TCP intercept ‒ Max connections Question: Which One Can Be Used Here?  TCP intercept won‘t help because the source address is valid  Limiting the maximum number of connections each internal host can have is the only optionBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 141

141. Case Study: Out of MemoryStep 9: Limit Infected Host(s) Impact on Network Configure the MAX TCP connections for NATed hosts to be 50 pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0 Note: the local-host must be cleared before the new connection limits are applied pixfirewall(config)# clear local-host 10.1.1.99 pixfirewall(config)# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, TCP connection count/limit = 50/50 The Infected Host TCP embryonic count = 50 Is Limited to 50 TCP intercept watermark = unlimited TCP Connections UDP connection count/limit = 0/unlimited . . .BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 142

142. Case Study: Out of Memory Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit pixfirewall# show conn count 126 in use, 147456 most used pixfirewall# show memory Free memory: 47716152 bytes Used memory: 19392712 bytes ------------- ---------------- Total memory: 67108864 bytes  Things look much better now  Question: How could we configure the Cisco PIX so the connection limit was only applied to the one host (10.1.1.99) which was infected with the virus? nat (inside) 1 10.1.1.99 255.255.255.255 50 0 nat (inside) 1 0.0.0.0 0.0.0.0 0 0BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 143

143. Classifier in Multimode When the firewall receives a packet, it must classify it to determine where to send the packet (which context) Packets are classified based on the following ‒ Unique ingress interface/VLAN ‒ Packet‘s destination IP matches a global IP FWSM has a single MAC address for all interfaces ASA has single MAC for shared interfaces (physical interfaces have unique MACs) ‒ ASA Ver 7.2 introduces mac-address auto option to change thisBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 144

144. Classifier in Multimode on FWSMExample  Inbound traffic is classified to context CTX3, based on the global IP in the NAT translation FWSM Inside DST IP SRC IP CTX1 .1 10.14.3.89 192.168.5.4 VLAN 3—10.14.3.x VLAN 4 10.1.1.2 Inbound Packet Inside Outside CTX2 .2 MSFC VLAN 5 10.1.2.2 Inside CTX3 .3 Shared interface VLAN 6 10.1.3.2 static (inside, outside) 10.14.3.89 10.1.3.2 BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 145

145. Multi-Context - Common Issues on FWSM Overlapping statics (globals) across contexts Missing statics (globals), and unable to classify packets (shared inside interface) – check Admin context log %FWSM-6-106025: Failed to determine security context for packet: vlan3 tcp src 192.168.5.4/1025 dest 198.51.100.50/80 Forgetting to ‗monitor-interface‘ for Failover Forgetting to assign unique IP for each Transparent mode context Transparent mode, multi-BVI, one routing tableBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 146

146. FWSM—ACL Rule Limits ACL rules are about the only hardware limit users encounter In multimode, ACL resources are divided in 13 equal partitions (12 active, one backup) If you have less than 12 contexts, wasted reserved space Single Context Multi-Context Tree 0 : Active Tree 0 : active = 14,801 ACEs  100,567 ACEs Tree 1 : active = 14,801 ACEs Tree 2 : active = 14,801 ACEs Tree 3 : active = 14,801 ACEs Tree 4 : active = 14,801 ACEs Tree 5 : active = 14,801 ACEs 177612 Tree 6 : active = 14,801 ACEs combined total Backup Tree:  100,567 Tree 7 : active = 14,801 ACEs ACEs (mirror of active tree) Tree 8 : active = 14,801 ACEs Tree 9 : active = 14,801 ACEs Tree 10 : active = 14,801 ACEs Tree 11 : active = 14,801 ACEs Tree 12 : backupBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 147

147. FWSM and ACLs (Multimode)  Use the command resource acl-partition <num-of- partitions> to reduce the number of active partitions created; default is 12  Use the command allocate-acl-partition <num> to assign a context to a specific ACL tree FWSM(config)# context Accounting FWSM(config-context)# allocate-acl-partition 0 FWSM(config-context)# show np 3 acl tree -------------------------------------------- ACL Tree Instance <-> Context Name (ID) Map --------------------------------------------Use Tree 0 Both Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Tree Instance 0 Context (004) Accounting --------------------------------------------BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 148

148. Case Study High CPU Usage

149. High CPU Usage on the Cisco PIX Problem: Cisco PIX CPU Running Very High  A quick overview of the show processes command Number of msec This Process Has Been on The Name of the CPU the Process pixfirewall(config)# show processes PC SP STATE Runtime SBASE Stack Process Hsi 001eab19 008a5a74 00557910 0 008a4aec 3628/4096 arp_timer Lsi 001f00bd 00a28dbc 00557910 0 00a27e44 3832/4096 FragDBGC Lwe 00119abf 02d280dc 0055b070 0 02d27274 3688/4096 dbgtrace Lwe 003e4425 02d2a26c 00557dd8 74440 02d28324 6936/8192 Logger Crd 001e26fb 0533940c 00557d88 6070290 05338484 3684/4096 557poll Lsi 00300a29 04c0f504 00557910 0 04c0e57c 3944/4096 xlate clean For more Information on the Output of the show processes Command, See http://www.cisco.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtmlBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 150

150. High CPU Usage on the Cisco PIX Step 1: Determine What Process Is Eating the CPU  Take the difference in output of two show processes over a period of time  The following output was a diff of the processes taken one minute apart In One Minute, These Process_Name Runtime (msec)Processes Account for 44 Logger 25940 Seconds of CPU Time ~ 73% pix/intf3 18410 557poll 9250 The Interface Polling i82543_timer 4180 Processes Always Run, and Are not Counted in the CPU i82542_timer 2230 UsageBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 151

151. High CPU Usage on the Cisco PIX Step 2: Focus on the Processes with High CPU Time  Logging is taking up much of the CPU; let‘s review what we have configured to log pixfirewall(config)# show log This Is Cumulative Since Syslog logging: enabled the Cisco PIX Was Last Standby logging: disabled Rebooted Console logging: disabled Monitor logging: disabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, 5919412 messages logged Logging to lab 172.18.173.123 History logging: disabled Notice the Change Over a . . . pixfirewall(config)# show log Few Minutes Syslog logging: enabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, 6172472 messages logged Logging to lab 172.18.173.123BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 152

152. High CPU Usage on the Cisco PIX Syslog Server Is Controlled by a Different Group  Enable buffered logging to same level as syslog server, and examine the buffered messages pixfirewall(config)# show log Buffer logging: level warnings, 31527 messages logged Trap logging: level warnings, 6453127 messages logged Logging to lab 172.18.173.123 . . . 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface labBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 153

153. High CPU Usage on the Cisco PIX Examine IDS Configuration pixfirewall(config)# show run | grep audit ip audit name IDS info action alarm ip audit interface lab IDS  Syslog service was down on the syslog server  ICMP unreachable was generated by syslog server for each syslog message the Cisco PIX sent it  Cisco PIX‘s IDS configuration also logged every ICMP unreachable message, creating the exponentially increasing problem Syslog Lab Outside Server Syslog Message ICMP Unreachable IDS Syslog MessageBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 154

154. High CPU Usage on the Cisco PIX  Bring back up syslog service on server  Take server offline  Configure Cisco PIX to not log IDS ICMP unreachable messages ip audit signature 2001 disable or no logging message 400011 pixfirewall# show run | grep signature ip audit signature 2001 disable pixfirewall# show cpu usage CPU utilization for 5 seconds = 2%; 1 minute: 50%; 5 minutes: 99%BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 155

155. High CPU Usage on the Cisco PIX Summary  Examine the DIFF of two show processes taken over a one minute interval  Find the process taking up the highest amount of CPU (excluding the polling processes)  Take actions to lower that process‘s CPU time  Reexamine the CPU output, and repeat as necessaryBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 156

156. Show process cpu-hog The show processes cpu-hog command displays a list of processes, and the function stack (Traceback) which executed, and lead to a process running on the CPU longer than the minimum platform threshold ASA# show processes cpu-hog Process: ssh_init, NUMHOG: 18, MAXHOG: 15, LASTHOG: 10 LASTHOG At: 14:18:47 EDT May 29 2009 PC: 8b9ac8c (suspend) Traceback: 8b9ac8c 8ba77ed 8ba573e 8ba58e8 8ba6971 8ba02b4 8062413 CPU hog threshold (msec): 10.240 Last cleared: None  A corresponding syslog message is also generated Note: The Traceback syslog below does not signify a crash May 29 2009 14:18:47: %ASA-7-711002: Task ran for 10 msec, Process = ssh_init, PC = 8b9ac8c, Traceback = 0x08B9AC8C 0x08BA77ED 0x08BA573E 0x08BA58E8 0x08BA6971 0x08BA02B4 0x08062413BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 157

158. FWSM Syslog Level vs. Number of Messages Number of Messages (SUM) Log Description Level Ver. 2.3 Ver. 3.1 Ver. 3.2 Ver. 4.0 Ver. 4.1 0 Emergencies 0 0 0 0 0 1 Alerts 58 (58) 67 (67) 67 (67) 67 (67) 67 (67) 2 Critical 21 (79) 29 (96) 29 (96) 29 (96) 29 (96) 3 Errors 94 (173) 305 (401) 306 (402) 318 (414) 318 (414) 4 Warnings 131 (304) 194 (595) 196 (598) 199 (613) 199 (613) 5 Notifications 26 (330) 167 (762) 169 (767) 178 (791) 178 (791) 6 Informational 116 (446) 245 (1007) 248 (1015) 255 (1046) 259 (1050) 7 Debugging 23 (469) 225 (1232) 225 (1240) 226 (1272) 231 (1281)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 159

159. FWSM and ACLs  ACLs on the FWSM are compiled on the control point and pushed down into hardware (NP 3)  During compile time, CPU should stay at ~ 99% ‒ ACL compile uses all free CPU cycles ‒ Allows compile to complete in shortest time possible  Once compile is complete, rules are attempted to be pushed into hardware ‒ Successful download Access Rules Download Complete: Memory Utilization: 49% ‒ Failed download (exceeded HW memory) ERROR: Unable to add, access-list config limit reachedBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 160

160. FWSM and ACLs (Multimode)  Use show np 3 acl stats to see the current ACL resource utilization in that context FWSM/admin(config)# show np 3 acl stats ---------------------------- ACL Tree Statistics ---------------------------- Rule count : 9584 Total Number of ACEs Bit nodes (PSCBs): 8760 Leaf nodes : 8761 Total nodes : 17521 (max 24260) This Is the Hardware Limit Leaf chains : 6912 Total stored rules: 15673 Max rules in leaf : 3 Node depth : 32 ----------------------------Note: One ACE Does not Equal One Node BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 161

161. FWSM and ACLs (Multimode) Use show np 3 acl tree to see which ACL tree a context is mapped to FWSM# show np 3 acl tree ACL Tree Number -------------------------------------------- ACL Tree Instance <-> Context Name (ID) Map -------------------------------------------- Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Context Name Tree Instance 3 Context (004) Accounting --------------------------------------------BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 162

162. FWSM—ACL Rule Limits  FWSM 2.3 introduced ‒ resource acl-partition—set the number of ACL partitions allocate-acl-partition—assigns a context to a specific partition  FWSM 3.2 introduced ‒ resource-rule—allows further customization of a partition  FWSM 4.0 introduced ‒ resource partition—customize the size of individual partitions access-list optimization enable—merges and/or deletes redundant and conflicting ACEs without affecting the policyBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 163

163. FWSM and ACLs (Multimode)  Use the command resource acl-partition <num-of- partitions> to reduce the number of active partitions created; default is 12  Use the command allocate-acl-partition <num> to assign a context to a specific ACL tree FWSM(config)# context Accounting FWSM(config-context)# allocate-acl-partition 0 FWSM(config-context)# show np 3 acl tree -------------------------------------------- ACL Tree Instance <-> Context Name (ID) Map -------------------------------------------- Tree Both Use 0 Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Tree Instance 0 Context (004) Accounting --------------------------------------------BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 164

164. FWSM—Resource Rule  FWSM 3.2 introduced ‒resource-rule—allows further customization of a partition resource rule nat 10000 acl 2200 filter 400 fixup 595 est 70 aaa 555 console 283 show resource-rule—displays information about the current rule allocation FWSM# show resource rule Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------- Policy NAT 1843 1843 10000 ACL 74188 74188 74188 Filter 2764 2764 5528 Fixup 4147 4147 10000 Est Ctl 460 460 460 Est Data 460 460 460 AAA 6451 6451 10000 Console 1843 1843 3686 -----------+---------+----------+--------- Total 92156 92156 Partition Limit - Configured Limit = Available to allocate 92156 - 92156 = 0BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 165

165. FWSM—Resource Partition  FWSM 4.0 introduced ‒ resource partition—allows customization of the size of individual partitions (multi-context mode) FWSM(config)# resource partition 10 FWSM(config-partition)# size 1000 WARNING: The rule max has been reset based on partition size 1000. The <size> command leads to re-partitioning of ACL Memory. It will not take effect until you save the configuration and reboot. Before After FWSM# show resource rule partition 10 FWSM# show resource rule partition 10 Default Configured Absolute Default Configured Absolute CLS Rule Limit Limit Max CLS Rule Limit Limit Max -----------+---------+----------+--------- -----------+---------+----------+--------- Policy NAT 384 384 833 Policy NAT 20 20 43 ACL 14801 14801 14801 ACL 770 770 770 Filter 576 576 1152 Filter 30 30 60 Fixup 1537 1537 3074 Fixup 80 80 160 Est Ctl 96 96 96 Est Ctl 5 5 5 Est Data 96 96 96 Est Data 5 5 5 AAA 1345 1345 2690 AAA 70 70 140 Console 384 384 768 Console 20 20 40 -----------+---------+----------+--------- -----------+---------+----------+--------- Total 19219 19219 Total 1000 1000 Partition Limit - Configured Limit = Available to Partition Limit - Configured Limit = Available to allocate allocate 1000 - 1000 = 0 19219 - 19219 = 0BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 166

166. FWSM and Control Point  The traffic that makes it to the control point is traffic that requires Layer 7 fixup (embedded NAT, or cmd inspection) Control Point (CP) Central CPU ‒FTP ‒VoIP (SIP/SKINNY/H.323/RTSP) ‒DNS ‒XDMCP, etc. Session Manager NP 3  Traffic sourced from, or destined to, the FWSM also goes through the control point ‒Syslogs Fast Path Fast Path NP 1 NP 2 ‒AAA (RADIUS/TACACS+) ‒URL filtering (WebSense/N2H2) FWSM ‒Management traffic C6K Backplane Interface (telnet/SSH/HTTPS/SNMP) ‒Failover communications ‒Routing protocols (OSPF/ RIP) ‒etc.BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 167

167. FWSM and Network Processors Control Point (CP) Central CPU  The session manager—NP 3 ‒Processes first packet in a flow Session Manager ‒ACL checks NP 3 ‒Translation creation ‒Embryonic/established connection counts Fast Path Fast Path NP 1 NP 2 ‒TCP/UDP checksums ‒Sequence number randomization FWSM C6K Backplane Interface ‒TCP intercept ‒etc. Control Point (CP) Central CPU  The fast path—NP 1 and 2 ‒Performs per packet session lookup Session Manager ‒Maintains connection table NP 3 ‒Performs NAT/PAT Fast Path Fast Path ‒TCP checks NP 1 NP 2 ‒Fragmentation reassembly ‒etc. FWSM C6K Backplane InterfaceBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 168

168. FWSM—Enabling the Completion Unit  Due to the FWSM‘s NP architecture, there exists a possibility that packets arriving with a low inter-packet gap might be re-ordered by the firewall 4 3 2 1 4 2 3 1  This issue might be encountered when performing TCP throughput testing, or passing high speed TCP flows through the FWSM  Examples: CIFS, FTP, AFP, backups  FWSM version 3.1(10) and 3.2(5) introduce a new command sysopt np completion-unit to ensure the firewall maintains the packet order (by enabling a hardware knob on the NPs called the completion unit)  In multiple mode enter this command in the admin context configuration; It will then be enabled for all contexts on the firewallBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 169

169. Case Study Advanced Syslog Analysis

170. Case Study: Advanced Syslog Analysis Problem – Find Services which are permitted through the firewall, yet the servers no longer exist  Get a fast Linux/Solaris machine with a decent amount of memory  Learn to use the following commands: • cat • grep, egrep, fgrep • cut • awk (basic) • sort • uniq • Perl (advanced manipulation)  Pipe the commands to construct the necessary outputs!BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 171

171. Case Study: Advanced Syslog Analysis  Interesting syslogs appear as follows: Syslog ID Destination May 24 2010 23:19:53: %ASA-6-302014: Teardown TCP connection 1019934 for outside:203.0.113.126/6243 to inside:10.100.19.190/21 duration 0:00:30 bytes 0 SYN Timeout ReasonBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 172

172. Case Study: Advanced Syslog Analysis Results:syslogserver-sun% grep 302014 syslog.txt | grep "SYN Timeout" | awk {print $13} |uniq -c | sort -r -n 673 inside:10.100.19.190/21 451 dmz:192.168.5.13/80 392 dmz:192.168.5.11/443 358 inside:10.0.0.67/1521 119 inside:10.0.1.142/80  grep – used to find the syslogs we want  awk – used to print the destination column (IP/port)  uniq – used to print only unique entries, with a count  sort – used to display ordered list, highest count firstBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 173

174. What to Do After a Failover  Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the reason for failover is saved in the failover history  This information is not saved across reboots ASA# show failover history ========================================================================== From State To State Reason ========================================================================== Disabled Negotiation Set by the CI config cmd Negotiation Just Active No Active unit found Just Active Active Drain No Active unit found Active Drain Active Applying Config No Active unit found Active Applying Config Active Config Applied No Active unit found Active Config Applied Active No Active unit found Active Failed Interface check ==========================================================================BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 175

175. Other Useful Failover Commands  failover exec mate – allows you to execute commands on the peer and receive the response back.  failover reload-standby – only valid on Active unit  prompt – changes the prompt to display failover priority and state. ASA(config)# prompt hostname priority state ASA/sec/act(config)#BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 176

176. Failover Prompt Display Configuration  The firewall‘s prompt maybe changed to display certain keyword  Usage ‒ prompt <keyword> [<keyword> ...]  Syntax ‒ keywords: ‒ Hostname Configures the prompt to display the hostname ‒ Domain Configures the prompt to display the domain ‒ Context Configures the prompt to display the current context (multi-mode only) ‒ Priority Configures the prompt to display the failover lan unit setting ‒ State Configures the prompt to display the current traffic handling state ‒ Slot Configures the prompt to display the slot location (when applicable)  Example ‒ FWSM(config)# prompt hostname domain priority state slot ‒ FWSM/cisco.com/sec/actNoFailover/4(config)#BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 177

177. Capture Command: Example  Problem: user on the inside with an IP of 10.1.3.2 is having a problem accessing www.cisco.com (198.133.219.25); the user is getting PATed to 192.168.2.2 Capture In Capture Out www.cisco.com Inside Outside Internet 10.1.3.2 192.168.2.2 198.133.219.25 10.1.3.2 Step 1: Create ACL for Both Inside and Outside Interface Step 2: Create Captures on Both Inside and Outside Interface Step 3: Have Inside User Access www.cisco.com Step 4: Copy the Captures Off to a TFTP Server Step 5: Analyze Captures with Sniffer ProgramBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 178

178. Capture Feature Example

179. Capture Command: Example  Step 1: create ACL for both inside and outside interface ! Outside Capture ACL Access-list 100 permit tcp host 192.168.2.2 host 198.133.219.25 eq 80 Access-list 100 permit tcp host 198.133.219.25 eq 80 host 192.168.2.2 ! Inside Capture ACL Access-list 101 permit tcp host 10.1.3.2 host 198.133.219.25 eq 80 Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.3.2  Step 2: create captures on both inside and outside interface capture out access-list 100 interface outside packet-length 1518 capture in access-list 101 interface inside packet-length 1518  Step 3: have inside user access www.cisco.com  Step 4: copy the captures off to a TFTP server ! ASA ver 7.0+ / FWSM 3.0+ copy capture copy /pcap capture:out tftp://10.1.3.5/out.pcap copy /pcap capture:in tftp://10.1.3.5/in.pcap ! PIX ver 6.x / FWSM 2.3 copy capture copy capture:out tftp://10.1.3.5/out.pcap pcap copy capture:in tftp://10.1.3.5/in.pcap pcap Or copy using https: https://<FW_IP>/capture/out/pcapBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 180

180. Packet Capture: Example  Step 5: analyze captures with sniffer program Outside CAP Outbound SYN, Inside CAP No SYN+ACKBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 181

181. Packet Capture: Limitations on FWSM  Capture functionality is available on the FWSM starting in 2.3 Control Point (CP) Central CPU ‒However, only packets processed by the control point could be captured  FWSM 3.1(1) added support to capture packets in hardware Session Manager NP 3 ‒Only ingress packets were captured  FWSM 3.1(5) both ingress and egress transient packets can Fast Path Fast Path be captured which flow NP 1 NP 1 through hardware FWSM ‒Capture requires an ACL to be applied C6K Backplane Interface ‒Capture copies the matched packets in hardware to the control point where they are captured; be careful not to flood the control point with too much trafficBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 182

182. Online Tools

188. Output Interpreter Linked off the Technical Support and Documentation— Tools and Resources Section on CCO Great Tool for Catching Configuration Errors Paste in the show run Output and Hit submitBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 189

189. Output Interpreter: Example Output Warning: Unused Statics Warning: Unapplied Crypto Map Warning: Invalid Crypto Map https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.plBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 190

190. ASDM

193. ASDM: Editing Rules from the Log Viewer Select Log Entry from Viewer Right-Click on Message to View or Edit Associated RuleBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 194

195. Opening a TAC Case  If after using all your troubleshooting tools you still cannot resolve the problem, please open a TAC case ‒ http://www.cisco.com/techsupport/servicerequest/  At a minimum include: ‒ Detailed problem description ‒ Output from show tech  Optionally include: ‒ Syslogs captured during time of problem ‒ Sniffer traces from both interfaces using the capture command (capturing only the relevant packets, and saved in pcap format)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 196

196. Complete Your OnlineSession Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our Don‘t forget to activate your portal) or visit one of the Internet Cisco Live Virtual account for access to stations throughout the Convention all session material, communities, and on-demand and live activities throughout Center. the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 197

197. Final Thoughts Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042 Come see demos of many key solutions and products in the main Cisco booth 2924 Visit www.ciscoLive365.com after the event for updated PDFs, on- demand session videos, networking, and more! Follow Cisco Live! using social media: ‒ Facebook: https://www.facebook.com/ciscoliveus ‒ Twitter: https://twitter.com/#!/CiscoLive ‒ LinkedIn Group: http://linkd.in/CiscoLIBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 198

Troubleshooting Firewalls (2012 San Diego)

Cisco Security

Views

Actions

Embeds 0

Report content

Transcript

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Troubleshooting Firewalls (2012 San Diego)

Cisco Security

Views

Actions

Embeds 0

Report content

Transcript