注意如果想使用bcc追踪python相关程序，python需要在编译的时候，启用dtrace。
参考这篇文章1
进程追踪
查看nginx的打开文件
opensnoop -u `id -u www-data`
See what files docker daemon opens
systemctl restart docker
# in another terminal
opensnoop -T -u 0 -n dockerd
查看到nginx的新建连接
tcpaccept -P 80
查看nginx往外发出的tcp连接
tcpconnect -u `id -u www-data`
the time from SYN sent to the response packet
tcpconnlat 1000
追踪某一个nginx worker的tcp会话
tcptracer -p `pgrep -f 'nginx: worker process'|head -1`
发现进程创建
execsnoop -t
案例分享
大量的"stat /etc/localtime"
statsnoop
PID COMM FD ERR PATH
12068 python 0 0 /etc/localtime
12068 python 0 0 /etc/localtime
15963 statsnoop -1 2 /usr/lib/python2.7/encodings/ascii
2172 python 0 0 /etc/localtime
2172 python 0 0 /etc/localtime
2173 python 0 0 /etc/localtime
2173 python 0 0 /etc/localtime
12068 python 0 0 /etc/localtime
12068 python 0 0 /etc/localtime
2172 python 0 0 /etc/localtime
2172 python 0 0 /etc/localtime
2173 python 0 0 /etc/localtime
2173 python 0 0 /etc/localtime
是个python进程
root@work:/usr/share/bcc/tools# ps -eLf |grep 2172
root 1470 1409 2172 0 2 2020 ? 05:25:24 python /opt/xxx.py
root 10513 4206 10513 0 1 17:18 pts/0 00:00:00 grep --color=auto 2172
只看这个进程，发现每隔100ms会有一组调用
statsnoop -p 2172 -t
TIME(s) PID COMM FD ERR PATH
0.000000000 2172 python 0 0 /etc/localtime
0.000084294 2172 python 0 0 /etc/localtime
0.100321893 2172 python 0 0 /etc/localtime
0.100397601 2172 python 0 0 /etc/localtime
0.200623126 2172 python 0 0 /etc/localtime
0.200680171 2172 python 0 0 /etc/localtime
0.300889229 2172 python 0 0 /etc/localtime
0.300954184 2172 python 0 0 /etc/localtime
0.401143016 2172 python 0 0 /etc/localtime
0.401187302 2172 python 0 0 /etc/localtime
0.501445091 2172 python 0 0 /etc/localtime
0.501501638 2172 python 0 0 /etc/localtime
0.601769073 2172 python 0 0 /etc/localtime
0.601811762 2172 python 0 0 /etc/localtime
查看代码
133 while not self.thread_stop:
134 try:
135 current_time_min = int(time.strftime('%Y%m%d%H%M', time.localtime(int(time.time()))))
136 current_time_hour = int(time.strftime('%Y%m%d%H', time.localtime(int(time.time()))))
137
138 if current_time_min <= start_time_min:
139 time.sleep(0.1)
140 continue
参考这篇文章2，这些stat操作是可以避免掉的。
频繁的"stat /root/.netrc"
statsnoop -t|grep netrc
0.397792612 28783 python -1 2 /root/.netrc
0.397837557 28783 python -1 2 /root/_netrc
1.382101607 28783 python -1 2 /root/.netrc
1.382147505 28783 python -1 2 /root/_netrc
2.396700909 28783 python -1 2 /root/.netrc
2.396748717 28783 python -1 2 /root/_netrc
3.396576766 28783 python -1 2 /root/.netrc
3.396623382 28783 python -1 2 /root/_netrc
4.402687988 28783 python -1 2 /root/.netrc
4.402734070 28783 python -1 2 /root/_netrc
5.401360047 28783 python -1 2 /root/.netrc
5.401427160 28783 python -1 2 /root/_netrc
6.392472923 28783 python -1 2 /root/.netrc
6.392518524 28783 python -1 2 /root/_netrc
查看requests库的代码，./libs/requests/utils.py
178 netrc_file = os.environ.get('NETRC')
179 if netrc_file is not None:
180 netrc_locations = (netrc_file,)
181 else:
182 netrc_locations = ('~/{}'.format(f) for f in NETRC_FILES)
183
184 try:
参考这个issue3修改调用代码：
s = requests.session()
s.trust_env = False