2013年6月24日星期一

7606 Router cpu utilization is hitting 99%!


Question:

7606-IGW-1#more bootdisk:crashinfo_WS-C3750X-48P-L Price 20100824-055939-UTC

Writing crashinfo to bootdisk:crashinfo_20100824-055939-UTCYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4921508  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B065610 B065798 8B91528 8B91BC4 8B92338 8B94CFC 8B8F8A8 8B73080

004120: *Aug 23 13:38:22.446 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004121: *Aug 23 13:38:32.450 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004122: *Aug 23 13:38:42.466 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004123: *Aug 23 13:38:44.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4683836  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004124: *Aug 23 13:38:52.470 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004125: *Aug 23 13:39:02.482 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004126: *Aug 23 13:39:12.490 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004127: *Aug 23 13:39:14.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5064576  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94CFC 8B8F8A8 8B73080

004128: *Aug 23 13:39:22.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004129: *Aug 23 13:39:32.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004130: *Aug 23 13:39:42.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004131: *Aug 23 13:39:44.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5399612  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94CFC 8B8F8A8 8B73080

004132: *Aug 23 13:39:52.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004133: *Aug 23 13:40:02.522 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004134: *Aug 23 13:40:12.522 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004135: *Aug 23 13:40:14.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5001240  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94040 8B8F7B4 8B73080

004136: *Aug 23 13:40:22.530 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004137: *Aug 23 13:40:32.554 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004138: *Aug 23 13:40:42.578 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004139: *Aug 23 13:40:44.430 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4847268  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004140: *Aug 23 13:40:52.606 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004141: *Aug 23 13:41:02.614 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004142: *Aug 23 13:41:12.626 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004143: *Aug 23 13:41:14.446 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4596772  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94040 8B8F7B4 8B73080

004144: *Aug 23 13:41:22.630 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004145: *Aug 23 13:41:32.638 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004146: *Aug 23 13:41:42.638 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004147: *Aug 23 13:41:44.462 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5020716  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004148: *Aug 23 13:41:52.646 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004149: *Aug 23 13:42:02.658 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004150: *Aug 23 13:42:12.674 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004151: *Aug 23 13:42:14.478 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5479568  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004152: *Aug 23 13:42:22.690 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004153: *Aug 23 13:42:32.694 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004154: *Aug 23 13:42:42.706 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004155: *Aug 23 13:42:44.478 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5044544  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004156: *Aug 23 13:42:52.710 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004157: *Aug 23 13:43:02.730 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004158: *Aug 23 13:43:12.730 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004159: *Aug 23 13:43:14.502 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4698184  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004160: *Aug 23 13:43:22.746 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004161: *Aug 23 13:43:32.754 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004162: *Aug 23 13:43:42.778 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

Answer:

According to these messages, your router was not able to allocate more memory for internal IOS processes including NAT. There are various possible reasons for that:

Your router is low on memory with respect to the amount of services and clients it is serving
Your router has been running for an extended period of time without restart and the memory allocation became so fragmented that the IOS is unable to find an unfragmented section of free RAM for the requesting process
Your NAT configuration results in too many translation entries being created and maintained
You have experienced an IOS bug

Does this problem steadily reappear after restarting the device? If yes, I suggest considering upgrading the IOS and verifying the configuration for obvious memory-hungry processes WS-C3750X-24S-S that could be optimized. Extending the RAM may also be recommendable.

没有评论:

发表评论