Part I: Technologies
1 Introduction 1
1.1 What Are BPF and eBPF? 1
1.2 What Are Tracing, Snooping, Sampling, Profiling, and
Observability? 2
1.3 What Are BCC, bpftrace, and IO Visor? 3
1.4 A First Look at BCC: Quick Wins 4
1.5 BPF Tracing Visibility 6
1.6 Dynamic Instrumentation: kprobes and uprobes 8
1.7 Static Instrumentation: Tracepoints and USDT 9
1.8 A First Look at bpftrace: Tracing open() 10
1.9 Back to BCC: Tracing open() 12
1.10 Summary 14
2 Technology Background 15
2.1 BPF Illustrated 15
2.2 BPF 16
2.3 Extended BPF (eBPF) 17
2.3.1 Why Performance Tools Need BPF 19
2.3.2 BPF Versus Kernel Modules 21
2.3.3 Writing BPF Programs 22
2.3.4 Viewing BPF Instructions: bpftool 23
2.3.5 Viewing BPF Instructions: bpftrace 30
2.3.6 BPF API 31
2.3.7 BPF Concurrency Controls 35
2.3.8 BPF sysfs Interface 36
2.3.9 BPF Type Format (BTF) 37
2.3.10 BPF CO-RE 37
2.3.11 BPF Limitations 38
2.3.12 BPF Additional Reading 38
2.4 Stack Trace Walking 39
2.4.1 Frame Pointer–Based Stacks 39
2.4.2 debuginfo 40
2.4.3 Last Branch Record (LBR) 40
2.4.4 ORC 40
2.4.5 Symbols 41
2.4.6 More Reading 41
2.5 Flame Graphs 41
2.5.1 Stack Trace 41
2.5.2 Profiling Stack Traces 41
2.5.3 Flame Graph 42
2.5.4 Flame Graph Features 44
2.5.5 Variations 44
2.6 Event Sources 45
2.7 kprobes 46
2.7.1 How kprobes Work 46
2.7.2 kprobes Interfaces 47
2.7.3 BPF and kprobes 48
2.7.4 kprobes Additional Reading 49
2.8 uprobes 49
2.8.1 How uprobes Work 49
2.8.2 Uprobes Interfaces 51
2.8.3 BPF and uprobes 51
2.8.4 uprobes Overhead and Future Work 52
2.8.5 uprobes Additional Reading 52
2.9 Tracepoints 53
2.9.1 Adding Tracepoint Instrumentation 53
2.9.2 How Tracepoints Work 55
2.9.3 Tracepoint Interfaces 56
2.9.4 Tracepoints and BPF 56
2.9.5 BPF Raw Tracepoints 57
2.9.6 Additional Reading 58
2.10 USDT 58
2.10.1 Adding USDT Instrumentation 58
2.10.2 How USDT Works 60
2.10.3 BPF and USDT 61
2.10.4 USDT Additional Reading 61
2.11 Dynamic USDT 61
2.12 PMCs 63
2.12.1 PMC Modes 63
2.12.2 PEBS 64
2.12.3 Cloud Computing 64
2.13 perf_events 64
2.14 Summary 65
3 Performance Analysis 67
3.1 Overview 67
3.1.1 Goals 68
3.1.2 Activities 68
3.1.3 Mulitple Performance Issues 69
3.2 Performance Methodologies 69
3.2.1 Workload Characterization 70
3.2.2 Drill-Down Analysis 71
3.2.3 USE Method 72
3.2.4 Checklists 72
3.3 Linux 60-Second Analysis 73
3.3.1 uptime 73
3.3.2 dmesg | tail 74
3.3.3 vmstat 1 74
3.3.4 mpstat -P ALL 1 75
3.3.5 pidstat 1 75
3.3.6 iostat -xz 1 76
3.3.7 free -m 77
3.3.8 sar -n DEV 1 77
3.3.9 sar -n TCP,ETCP 1 78
3.3.10 top 78
3.4 BCC Tool Checklist 79
3.4.1 execsnoop 80
3.4.2 opensnoop 80
3.4.3 ext4slower 80
3.4.4 biolatency 81
3.4.5 biosnoop 81
3.4.6 cachestat 82
3.4.7 tcpconnect 82
3.4.8 tcpaccept 82
3.4.9 tcpretrans 83
3.4.10 runqlat 83
3.4.11 profile 84
3.5 Summary 84
4 BCC 85
4.1 BCC Components 86
4.2 BCC Features 86
4.2.1 Kernel-Level Features 87
4.2.2 BCC User-Level Features 87
4.3 BCC Installation 88
4.3.1 Kernel Requirements 88
4.3.2 Ubuntu 88
4.3.3 RHEL 89
4.3.4 Other Distributions 89
4.4 BCC Tools 89
4.4.1 Highlighted Tools 90
4.4.2 Tool Characteristics 91
4.4.3 Single-Purpose Tools 91
4.4.4 Multi-Purpose Tools 93
4.5 funccount 94
4.5.1 funccount Examples 94
4.5.2 funccount Syntax 97
4.5.3 funccount One-Liners 97
4.5.4 funccount Usage 98
4.6 stackcount 99
4.6.1 stackcount Example 99
4.6.2 stackcount Flame Graphs 100
4.6.3 stackcount Broken Stack Traces 101
4.6.4 stackcount Syntax 102
4.6.5 stackcount One-Liners 102
4.6.6 stackcount Usage 103
4.7 trace 104
4.7.1 trace Example 104
4.7.2 trace Syntax 105
4.7.3 trace One-Liners 106
4.7.4 trace Structs 107
4.7.5 trace Debugging File Descriptor Leaks 107
4.7.6 trace Usage 109
4.8 argdist 110
4.8.1 argdist Syntax 111
4.8.2 argdist One-Liners 112
4.8.3 argdist Usage 113
4.9 Tool Documentation 114
4.9.1 Man Page: opensnoop 115
4.9.2 Examples File: opensnoop 118
4.10 Developing BCC Tools 119
4.11 BCC Internals 120
4.12 BCC Debugging 121
4.12.1 printf() Debugging 122
4.12.2 BCC Debug Output 124
4.12.3 BCC Debug Flag 125
4.12.4 bpflist 126
4.12.5 bpftool 127
4.12.6 dmesg 127
4.12.7 Resetting Events 127
4.13 Summary 128
5 b