Foreword  
Prefa e.  
About the Authors  
A knowledgments  
PART ONE Observability Methods  
Chapter 1 Introdu tion to Observability Tools  
1.1 Observability Tools  
1.2 Drill-Down Analysis  
1.3 About Part One  
Chapter 2 CPUs  
2.1 Tools for CPU Analysis  
2.2 vmstat Tool  
2.3 CPU Utilization  
2.4 CPU Saturation  
2.5 psrinfo Command  
2.6 uptime Command  
2.7 sar Command  
2.8 Clock Tick Woes  
2.9 mpstat Command  
2.10 Who Is Using the CPU?  
2.11 CPU Run Queue Latency  
2.12 CPU Statisti s Internals  
2.13 Using DTrace to Explain Events from Performance Tools  
2.14 DTrace Versions of runq- sz, %runo  
2.15 DTrace Probes for CPU States  
Chapter 3 Pro esses  
3.1 Tools for Pro ess Analysis  
3.2 Process Statistics Summary: prstat  
3.3 Process Status: ps  
3.4 Tools for Listing and Controlling Pro esses  
3.5 Process Introspection Commands  
3.6 Examining User-Level Locks in a Process  
3.7 Tracing Processes  
3.8 Java Processes  
Chapter 4 Disk Behavior and Analysis  
4.1 Terms for Disk Analysis  
4.2 Randomcvs. Sequential I/O  
4.3 Storage Arrays  
4.4 Sector Zoning  
4.5 Max I/O Size  
4.6 iostat Utility  
4.7 Disk Utilization  
4.8 Disk Saturation  
4.9 Disk Throughput  
4.10 iostat Referen e  
4.11 Reading iostat  
4.12 iostat Internals  
4.13 sar -d  
4.14 Trace Normal Form TNF) Tracing for I/O  
4.15 DTrace for I/O  
4.16 Disk I/O Time  
4.17 DTraceToolkit Commands  
4.18 DTraceTazTool  
Chapter 5 File Systems  
5.1 Layers of File System and I/O  
5.2 Observing Physi al I/O  
5.3 File System Latency  
5.4 Causes of Read/Write File System Latency  
5.5 Observing File System “Top End” Activity  
5.6 File System Caches  
5.7 NFS Statistics  
Chapter 6 Memory  
6.1 Tools for Memory Analysis  
6.2 vmstat(1M) Command  
6.3 Types of Paging  
6.4 Physical Memory Allocation  
6.5 Relieving Memory Pressure  
6.6 Scan Rate asca Memory Health Indicator  
6.7 Process Virtual and Resident Set Size  
6.8 Using pmap to Inspect Process Memory Usage  
6.9 Calculating Pro ess Memory Usage with ps and pmap  
6.10 Displaying Page-Size Information with pmap  
6.11 Using DTrace for Memory Analysis  
6.12 Obtaining Memory Kstats  
6.13 Using the Perl Kstat API to Look at Memory Statistics  
6.14 System Memory Allo ation Kstats  
6.15 Kernel Memow with kstat  
6.16 System Paging Kstats  
6.17 Observing MMU Performance Impact with trapstat  
6.18 Swap Space  
Chapter 7 Networks  
7.1 Terms for Network Analysis  
7.2 Packets Are Not Bytes  
7.3 Network Utilization  
7.4 Network Saturation  
7.5 Network Errors  
7.6 Misconfigurations  
7.7 Systemwide Statistics  
7.8 Per-Process Network Statistics  
7.9 TCP Statistics  
7.10 IP Statistics  
7.11 ICMP Statistics  
Chapter 8 Performance Counters  
8.1 Introducing CPU Caches  
8.2  pustat Command
8.3  putrack Command  
8.4 busstat Command  
Chapter 9 Kernel Monitoring  
9.1 Tools for Kernel Monitoring  
9.2 Profiling the Kernel and Drivers  
9.3 Analyzing Kernel Locks  
9.4 DTrace lockstat Provider  
9.5 DTrace Kernel Profiling  
9.6 Interrupt Statistics: vmstat -i  
9.7 Interrupt Analysis: intrstat  
PART TWO Observability Infrastructure  
Chapter 10 Dynami Tracing  
10.1 Introduction to DTrace  
10.2 The Basics  
10.3 Inspecting Java Appli ations with DTrace  
10.4 DTrace Architecture  
10.5 Summary  
10.6 Probe Reference  
10.7 MDB Reference  
Chapter 11 Kernel Statistics  
11.1 C-Level Kstat Interface  
11.2 Command-Line Interface  
11.3 Using Peri to A ess kstats  
11.4 Snooping a Program's kstat Use with DTrace  
11.5 Adding Statistics to the Solaris Kernel  
11.6 Additional Information  
PART THREE Debugging  
Chapter 12 The Modular Debugger  
12.1 Introduction to the Modular Debugger  
12.2 MDB Concepts  
Chapter 13 An MDB Tutorial  
13.1 Invoking MDB  
13.2 MDB Command Syntax  
13.3 Working with Debugging Targets  
13.4 GDB-to-MDB Referen e  
13.5 dcmd and Walker Reference  
Chapter 14 Debugging Kernels  
14.1 Working with Kernel Cores  
14.2 Examining User Process Stacks within a Kernel Image  
14.3 Swit hing MDB to Debugca Specifi Process  
14.4 kmdb, the Kernel Modular Debugger  
14.5 Kernel Built-In MDB dcmds  
APPENDICES  
Appendix A Tunables and Settings  
Appendix B DTrace One-Liners  
Appendix C Java DTrace Scripts  
Appendix D Sample Peri Kstat Utilities  
Bibliography  
Index