-
Notifications
You must be signed in to change notification settings - Fork 3
/
Copy pathnettrace.8
287 lines (254 loc) · 8.58 KB
/
nettrace.8
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
.\" Manpage for nettrace.
.\" Contact [email protected] to correct errors or typos.
.TH man 8 "11 Nov 2020" "1.0" "nettrace man page"
.SH NAME
nettrace \- A skb (network package) trace tool for kernel
.SH SYNOPSIS
nettrace [-h] [-s SADDR] [-d DADDR] [--addr ADDR] [-p PROTO]
[--dport DPORT] [--sport SPORT] [--port PORT]
[--tcp-flags TCP_FLAGS] [-t TRACER] [-v] [--detail] [--stack]
[--stack-tracer STACK_TRACER] [--timeline] [-c COUNT] [--ret]
[--skb-mode] [--force-stack] [-o] [--output]
.SH DESCRIPTION
nettrace is is a powerful tool to trace network packet and diagnose network problem inside kernel on TencentOS.
It make use of eBPF and BCC.
\'skb\' is the struct that used in kernel to store network package. By tracing kernel function and tracepoint (with the help of kprobe based on eBPF) that handle skb, nettrace is able to show the path of skb bypass inside kernel network stack. Therefor, some network issue (such as package drop) can be solved simply.
.SH OPTIONS
.TP
.B \-h
.PD 0
.TP
.B \-\-help
.PD
show this help message and exit
.TP
.B \-s saddr
.PD 0
.TP
.B \-\-saddr saddr
.PD
filter by ip source address
.TP
.B \-d daddr
.PD 0
.TP
.B \-\-daddr daddr
.PD
filter by ip dest address
.TP
.B \-\-addr addr
.PD
filter by ip source or dest address
.TP
.B \-p proto
.PD 0
.TP
.B \-\-proto proto
.PD
filter by network protocol (L3 or L4) in lower case, such ip, tcp, udp, etc.
.TP
.B \-\-dport dport
.PD
filter by TCP/UDP dest port
.TP
.B \-\-sport sport
.PD
filter by TCP/UDP source port
.TP
.B \-\-port port
.PD
filter by TCP/UDP source or dest port
.TP
.B \-\-tcp\-flags tcp_flags
.PD
filter by TCP flags, such as S(syn), A(ack), R(rst) and P(push). Multiple flags are supported, such as:
.B --tcp-flags SA
means filter tcp packet with flags syn + ack.
.TP
.B \-t tracers
.PD 0
.TP
.B \-\-tracer tracers
.PD
The network module or kernel function to trace.
kernel function and tracepoint are organized in a 'tracer' tree, and every node of the tree is a tracer. 'all' is the root of the 'tracer' tree. The tree can be displayed with the command: \`nettrace -t ?\`
.RS 9
$ nettrace -t ?
.br
available tracer:
.br
---------------------------------------------------
.br
.br
all: trace the whole kernel network stack
.br
link: link layer (L2) of the network stack
.br
link-in: link layer (L2) of packet in
.br
napi_gro_receive
.br
enqueue_to_backlog
.br
__netif_receive_skb_core
.br
do_xdp_generic
.br
xdp_do_generic_redirect
.br
generic_xdp_tx
.br
sch_handle_ingress
.br
link-out: link layer (L2) of packet out
.br
dev_queue_xmit
.br
dev_hard_start_xmit
.br
sched: TC(traffic control) module
.br
tcf_classify
.br
sch_handle_egress
.br
cls_bpf_classify
.br
tcf_bpf_act
.br
......
.br
.RE
.RS 7
This is the part of the tracer tree. 'link' just represent the link layer (L2) of the network stack, and 'link-in' represent receive package, 'link-out' represent send package. 'ip' represent ip layer, 'tcp' represent tcp layer, etc.
When trace skb, tracer can be specified by '-t' (or '--tracer'), and multiple are supported. By default, 'all' tracer is used. For example, 'nettrace -t napi_gro_receive,ip' to trace the function napi_gro_receive (function is tracer too) and ip layer.
.RE
.TP
.B \-o output_format
.PD 0
.TP
.B \-\-output output_format
.PD
print extern info. options include: pid, if, id, cpu and module. pid: process info; if: ifindex and ifname; id: memory address of skb; cpu: the cpu id that run on; module: the network module of the tracer belong to. multiple options should be splited by ","
.TP
.B --detail
.PD
show all info for trace output, which means enable all options in "--output"
.TP
.B --stack
.PD
print the function call stack for the kernel function or tracepoint that enabled.
.TP
.B --stack-tracer tracers
.PD
maybe you don't want to print stack for all kernel function that enable, you can specify the tracer (or kernel function) that you want to print stack.
.TP
.B --force-stack
.PD
by default, you can't set `-t all` and `--stack` together for performance problem. With this option you can do it.
.TP
.B --ret
.PD
trace the return value. once this enabled, both kprobe and kretprobe will be create for kernel function in the tracer that enabled, and the return value will be printed.
.TP
.B --timeline
.PD
enable timeline mode.
By default, kernel function are printed directly. It's not easy to distinguish single skb, especially multi skb are printed at the same time. To solve this problem, \'timeline mode\' is introduced. In this mode, every skb has a \'context\', which is used to mange the life of the skb. The skb won't be printed until it 'dies', and it will be print singly.
What's more, the skb cloned from one skb is thought to be in the same context, and will be printed together.
.TP
.B -c count
.PD 0
.TP
.B --count count
.PD
skb count to trace (timeline should be enabled)
.TP
.B --skb-mode
.PD
keep tracing skb once it is matched
Traditional packet filter is supported, such protocol, ip source address, ip destination address, source port, destination port, etc. However, while the data of skb changing (for example, NAT), it can't work probably.
`skb mode` is used to solve this problem. While it is enabled with `--skb-mode`, nettrace will keep tracing the skb once it is matched, even if ip address or port changed. This mode is helpful when you want to trace the skb whose data will be change by iptables.
As we can see below, we want to trace icmp package with ip address `192.168.122.8`. However, DNAT is done by iptables, and the ip source address of this skb changed to `9.135.224.89`. With this mode enabled, we still can keep tracing it.
.RS 9
$ nettrace -p icmp \-\-addr 192\.168\.122\.8 \-\-timeline \-\-skb\-mode
.br
begin tracing......
.br
<------------------- skb: ffff88818f02f900 ---------------------->
.br
463697.331957: [__netif_receive_skb_core]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.331972: [nf_hook_slow ]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.331985: [nf_hook_slow ]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.331990: [__netif_receive_skb_core]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.331994: [ip_rcv ]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.331998: [ip_rcv_core ]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.332001: [nf_hook_slow ]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.332004: [ip_rcv_finish ]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.332010: [ip_forward ]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.332014: [nf_hook_slow ]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.332024: [ip_output ]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.332027: [nf_hook_slow ]: ICMP: 192.168.122.8 -> 10.123.119.98, ping request , seq: 0
.br
463697.332037: [ip_finish_output ]: ICMP: 9.135.224.89 -> 10.123.119.98, ping request , seq: 0
.br
463697.332039: [ip_finish_output2 ]: ICMP: 9.135.224.89 -> 10.123.119.98, ping request , seq: 0
.br
463697.332042: [dev_queue_xmit ]: ICMP: 9.135.224.89 -> 10.123.119.98, ping request , seq: 0
.br
463697.332046: [dev_hard_start_xmit ]: ICMP: 9.135.224.89 -> 10.123.119.98, ping request , seq: 0
.br
463697.332060: [consume_skb ]: ICMP: 9.135.224.89 -> 10.123.119.98, ping request , seq: 0
.br
.RE
.TP
.B -v
.PD 0
.TP
.B --verbose
.PD
show more verbose info
.SH EXAMPLES
.TP
trace icmp package with source ip '192.168.1.8':
#
.B nettrace -p icmp -s 192.168.1.8
.TP
trace tcp package with source ip '192.168.1.8' and syn+ack flags:
#
.B nettrace -p tcp -s 192.168.1.8 --tcp-flags SA
.TP
trace icmp package with source ip '192.168.1.8' in ip and icmp layer:
#
.B nettrace -p icmp -s 192.168.1.8 -t ip,icmp
.TP
trace icmp package with source ip '192.168.1.8' in timeline mode:
#
.B nettrace -p icmp -s 192.168.1.8 --timeline
.TP
trace icmp package with source ip '192.168.1.8' in skb mode:
#
.B nettrace -p icmp -s 192.168.1.8 --skb-mode
.TP
trace icmp package with source ip '192.168.1.8' and print detail information
#
.B nettrace -p icmp -s 192.168.1.8 --detail
.SH REQUIREMENTS
CONFIG_BPF, CONFIG_KPROBE and bcc.
.SH OS
Linux
.SH AUTHOR
Menglong Dong
.SH SEE ALSO
bpf(2), bcc-*(8)