LISA2019 Linux Systems Performance

Linux Systems Performance Brendan Gregg Senior Performance Engineer Oct, 2019 USENIX LISA 2019, Portland, Oct 28-30 Experience: A 3x Perf Difference mpstat serverA# mpstat 10 Linux 4.4.0-130-generic (serverA) 07/18/2019 _x86_64_ (48 CPU) 10:07:55 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 10:08:05 PM all 89.72 0.00 7.84 0.00 0.00 0.04 0.00 0.00 0.00 2.40 10:08:15 PM all 88.60 0.00 9.18 0.00 0.00 0.05 0.00 0.00 0.00 2.17 10:08:25 PM all 89.71 0.00 9.01 0.00 0.00 0.05 0.00 0.00 0.00 1.23 […] Average: all 89.49 0.00 8.47 0.00 0.00 0.05 0.00 0.00 0.00 1.99 serverB# mpstat 10 Linux 4.19.26-nflx (serverB) 07/18/2019 _x86_64_ (64 CPU) 09:56:11 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 09:56:21 PM all 23.21 0.01 0.32 0.00 0.00 0.10 0.00 0.00 0.00 76.37 09:56:31 PM all 20.21 0.00 0.38 0.00 0.00 0.08 0.00 0.00 0.00 79.33 09:56:41 PM all 21.58 0.00 0.39 0.00 0.00 0.10 0.00 0.00 0.00 77.92 […] Average: all 21.50 0.00 0.36 0.00 0.00 0.09 0.00 0.00 0.00 78.04 load averages: serverA 90, serverB 17 pmcarch serverA# ./pmcarch -p 4093 10 K_CYCLES K_INSTR IPC BR_RETIRED BR_MISPRED BMR% LLCREF LLCMISS LLC 982412660 575706336 0.59 126424862460 2416880487 1.91 15724006692 10872315070 30.86 999621309 555043627 0.56 120449284756 2317302514 1.92 15378257714 11121882510 27.68 991146940 558145849 0.56 126350181501 2530383860 2.00 15965082710 11464682655 28.19 996314688 562276830 0.56 122215605985 2348638980 1.92 15558286345 10835594199 30.35 979890037 560268707 0.57 125609807909 2386085660 1.90 15828820588 11038597030 30.26 C serverB# ./pmcarch -p 1928219 10 K_CYCLES K_INSTR IPC BR_RETIRED BR_MISPRED BMR% LLCREF LLCMISS LLC% 147523816 222396364 1.51 46053921119 641813770 1.39 8880477235 968809014 89.09 156634810 229801807 1.47 48236123575 653064504 1.35 9186609260 1183858023 87.11 152783226 237001219 1.55 49344315621 692819230 1.40 9314992450 879494418 90.56 140787179 213570329 1.52 44518363978 631588112 1.42 8675999448 712318917 91.79 136822760 219706637 1.61 45129020910 651436401 1.44 8689831639 617678747 92.89 perf serverA# perf stat -e cs -a -I 1000 # time counts unit events 1.000411740 2,063,105 cs 2.000977435 2,065,354 cs 3.001537756 1,527,297 cs 4.002028407 515,509 cs 5.002538455 2,447,126 cs […] serverB# perf stat -e cs -p 1928219 -I 1000 # time counts unit events 1.001931945 1,172 cs 2.002664012 1,370 cs 3.003441563 1,034 cs 4.004140394 1,207 cs 5.004947675 1,053 cs […] bcc/BPF serverA# /usr/share/bcc/tools/cpudist -p 4093 10 1 Tracing on-CPU time…

Link: LISA2019 Linux Systems Performance
via http://www.slideshare.net

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s