Добрый день.
На sda только OS. Своп отключён. На сервере запущены ВМ с nvme0n1, nvme1n1
Периодически возникает жуткий page scan, локальный диск sda грузится по чтению. OS зависает по D+, так минут 5, потом отвисает само.
Помогите разобраться. Свободная память вроде есть, ~9Гб. В page cache ~200-300мб.
Даже если возникает page scan, своп у меня отключен. Почему грузится локальный диск?
В 12:58
12:58:02 -------------- 1m0s elapsed
PRC | sys 51.27s | user 2m57s | | | #proc 15621 | #trun 5 | #tslpi 705 | #tslpu 1 | #zombie 0 | clones 16042 | | | #exit 15276 |
CPU | sys 82% | user 300% | irq 1% | | | idle 755% | wait 63% | | steal 0% | guest 252% | | curf 4.30GHz | curscal 82% |
cpu | sys 8% | user 36% | irq 0% | | | idle 50% | cpu005 w 5% | | steal 0% | guest 34% | | curf 4.30GHz | curscal 82% |
cpu | sys 7% | user 32% | irq 0% | | | idle 55% | cpu001 w 5% | | steal 0% | guest 28% | | curf 4.30GHz | curscal 82% |
cpu | sys 9% | user 30% | irq 0% | | | idle 57% | cpu004 w 4% | | steal 0% | guest 28% | | curf 4.30GHz | curscal 82% |
cpu | sys 8% | user 30% | irq 0% | | | idle 55% | cpu003 w 6% | | steal 0% | guest 27% | | curf 4.30GHz | curscal 82% |
cpu | sys 8% | user 28% | irq 0% | | | idle 59% | cpu002 w 5% | | steal 0% | guest 24% | | curf 4.30GHz | curscal 82% |
cpu | sys 9% | user 25% | irq 0% | | | idle 61% | cpu000 w 5% | | steal 0% | guest 22% | | curf 4.30GHz | curscal 82% |
cpu | sys 5% | user 26% | irq 0% | | | idle 63% | cpu008 w 6% | | steal 0% | guest 17% | | curf 4.30GHz | curscal 82% |
cpu | sys 4% | user 24% | irq 0% | | | idle 66% | cpu006 w 6% | | steal 0% | guest 21% | | curf 4.30GHz | curscal 82% |
cpu | sys 5% | user 21% | irq 0% | | | idle 70% | cpu009 w 5% | | steal 0% | guest 17% | | curf 4.30GHz | curscal 82% |
cpu | sys 6% | user 19% | irq 0% | | | idle 71% | cpu007 w 5% | | steal 0% | guest 15% | | curf 4.30GHz | curscal 82% |
cpu | sys 5% | user 19% | irq 0% | | | idle 72% | cpu010 w 4% | | steal 0% | guest 13% | | curf 4.30GHz | curscal 82% |
cpu | sys 7% | user 9% | irq 0% | | | idle 76% | cpu011 w 6% | | steal 0% | guest 5% | | curf 4.30GHz | curscal 82% |
CPL | avg1 6.13 | | avg5 6.06 | avg15 5.73 | | | csw 1718695 | | intr 1178894 | | | numcpu 12 | |
MEM | tot 62.6G | free 9.6G | cache 285.6M | dirty 1.0M | buff 0.0M | slab 1.1G | slrec 50.3M | shmem 211.1M | shrss 0.8M | shswp 0.0M | vmbal 0.0M | hptot 0.0M | hpuse 0.0M |
SWP | tot 0.0M | free 0.0M | | | | | | | | | vmcom 65.8G | | vmlim 31.3G |
PAG | scan 13054 | steal 11869 | | stall 391 | | | | | | | swin 0 | | swout 0 |
DSK | sda | busy 45% | | read 102463 | write 558 | KiB/r 8 | KiB/w 9 | | MBr/s 14.3 | MBw/s 0.1 | avq 1.17 | | avio 0.26 ms |
DSK | nvme0n1 | busy 28% | | read 1429 | write 16738 | KiB/r 39 | KiB/w 39 | | MBr/s 0.9 | MBw/s 10.8 | avq 2.52 | | avio 0.91 ms |
DSK | nvme1n1 | busy 28% | | read 1400 | write 16857 | KiB/r 39 | KiB/w 39 | | MBr/s 0.9 | MBw/s 10.8 | avq 2.45 | | avio 0.91 ms |
DSK | sdb | busy 0% | | read 37 | write 95 | KiB/r 7 | KiB/w 6 | | MBr/s 0.0 | MBw/s 0.0 | avq 1.04 | | avio 1.23 ms |
PID TID RDDSK WRDSK WCANCL DSK CMD 1/908
22149 - 790.6M 0K 0K 34% logrotate
9230 - 320K 130.4M 0K 6% qemu-kvm
23863 - 8264K 84132K 0K 4% qemu-kvm
749 - 84255K 0K 0K 4% z_wr_int_4
748 - 81314K 0K 0K 3% z_wr_int_3
745 - 80129K 0K 0K 3% z_wr_int_0
746 - 80111K 0K 0K 3% z_wr_int_1
747 - 79480K 0K 0K 3% z_wr_int_2
750 - 79309K 0K 0K 3% z_wr_int_5
752 - 79283K 0K 0K 3% z_wr_int_7
751 - 78957K 0K 0K 3% z_wr_int_6
737 - 58453K 0K 0K 2% z_wr_iss
743 - 58065K 0K 0K 2% z_wr_iss
739 - 57923K 0K 0K 2% z_wr_iss
738 - 55525K 0K 0K 2% z_wr_iss
735 - 55319K 0K 0K 2% z_wr_iss
741 - 54982K 0K 0K 2% z_wr_iss
Через минуту начинается PAG scan:
12:59:02 -------------- 1m0s elapsed
PRC | sys 44.04s | user 2m26s | | | #proc 13924 | #trun 3 | #tslpi 678 | #tslpu 3 | #zombie 0 | clones 13649 | | | #exit 13580 |
CPU | sys 69% | user 237% | irq 1% | | | idle 811% | wait 83% | | steal 0% | guest 232% | | curf 4.30GHz | curscal 82% |
cpu | sys 8% | user 29% | irq 0% | | | idle 54% | cpu005 w 9% | | steal 0% | guest 28% | | curf 4.30GHz | curscal 82% |
cpu | sys 7% | user 28% | irq 0% | | | idle 58% | cpu000 w 6% | | steal 0% | guest 28% | | curf 4.30GHz | curscal 82% |
cpu | sys 7% | user 25% | irq 0% | | | idle 61% | cpu004 w 7% | | steal 0% | guest 24% | | curf 4.38GHz | curscal 84% |
cpu | sys 6% | user 25% | irq 0% | | | idle 61% | cpu002 w 7% | | steal 0% | guest 25% | | curf 4.30GHz | curscal 82% |
cpu | sys 7% | user 25% | irq 0% | | | idle 60% | cpu003 w 8% | | steal 0% | guest 24% | | curf 4.30GHz | curscal 82% |
cpu | sys 8% | user 23% | irq 0% | | | idle 61% | cpu001 w 8% | | steal 0% | guest 22% | | curf 4.30GHz | curscal 82% |
cpu | sys 3% | user 17% | irq 0% | | | idle 74% | cpu009 w 6% | | steal 0% | guest 16% | | curf 4.30GHz | curscal 82% |
cpu | sys 3% | user 16% | irq 0% | | | idle 74% | cpu007 w 7% | | steal 0% | guest 16% | | curf 4.30GHz | curscal 82% |
cpu | sys 2% | user 17% | irq 0% | | | idle 75% | cpu008 w 6% | | steal 0% | guest 17% | | curf 4.30GHz | curscal 82% |
cpu | sys 3% | user 16% | irq 0% | | | idle 76% | cpu006 w 5% | | steal 0% | guest 16% | | curf 4.30GHz | curscal 82% |
cpu | sys 5% | user 13% | irq 0% | | | idle 73% | cpu010 w 9% | | steal 0% | guest 13% | | curf 4.30GHz | curscal 82% |
cpu | sys 7% | user 3% | irq 1% | | | idle 84% | cpu011 w 5% | | steal 0% | guest 3% | | curf 4.30GHz | curscal 82% |
CPL | avg1 5.11 | | avg5 5.88 | avg15 5.69 | | | csw 2123017 | | intr 1221074 | | | numcpu 12 | |
MEM | tot 62.6G | free 9.6G | cache 277.9M | dirty 1.1M | buff 0.0M | slab 1.1G | slrec 50.6M | shmem 211.1M | shrss 0.8M | shswp 0.0M | vmbal 0.0M | hptot 0.0M | hpuse 0.0M |
SWP | tot 0.0M | free 0.0M | | | | | | | | | vmcom 65.6G | | vmlim 31.3G |
PAG | scan 22067e3 | steal 103970 | | stall 6716 | | | | | | | swin 0 | | swout 0 |
DSK | sda | busy 88% | | read 194668 | write 420 | KiB/r 9 | KiB/w 8 | | MBr/s 31.4 | MBw/s 0.1 | avq 3.70 | | avio 0.27 ms |
DSK | nvme1n1 | busy 27% | | read 911 | write 14127 | KiB/r 41 | KiB/w 40 | | MBr/s 0.6 | MBw/s 9.4 | avq 2.13 | | avio 1.07 ms |
DSK | nvme0n1 | busy 20% | | read 1175 | write 14615 | KiB/r 37 | KiB/w 39 | | MBr/s 0.7 | MBw/s 9.4 | avq 1.68 | | avio 0.75 ms |
DSK | sdb | busy 0% | | read 3 | write 50 | KiB/r 4 | KiB/w 5 | | MBr/s 0.0 | MBw/s 0.0 | avq 1.07 | | avio 3.04 ms |
PID TID RDDSK WRDSK WCANCL DSK CMD 1/808
22149 - 1.5G 8K 0K 49% logrotate
9230 - 7264K 114.5M 0K 4% qemu-kvm
23863 - 14880K 81816K 0K 3% qemu-kvm
746 - 75510K 0K 0K 2% z_wr_int_1
751 - 74760K 0K 0K 2% z_wr_int_6
745 - 74597K 0K 0K 2% z_wr_int_0
749 - 73683K 0K 0K 2% z_wr_int_4
750 - 73506K 0K 0K 2% z_wr_int_5
748 - 70174K 0K 0K 2% z_wr_int_3
752 - 69027K 0K 0K 2% z_wr_int_7
747 - 68774K 0K 0K 2% z_wr_int_2
8858 - 47880K 2496K 0K 2% qemu-kvm
737 - 48999K 0K 0K 2% z_wr_iss
736 - 48744K 0K 0K 2% z_wr_iss
738 - 46608K 0K 0K 1% z_wr_iss
8662 - 16640K 28676K 0K 1% qemu-kvm
743 - 45208K 0K 0K 1% z_wr_iss
В 13:01:
PRC | sys 4m19s | user 2m33s | | | #proc 4205 | #trun 22 | #tslpi 497 | #tslpu 29 | #zombie 0 | clones 4040 | | | #exit 3815 |
CPU | sys 416% | user 242% | irq 1% | | | idle 135% | wait 407% | | steal 0% | guest 241% | | curf 4.30GHz | curscal 82% |
cpu | sys 41% | user 22% | irq 0% | | | idle 12% | cpu004 w 26% | | steal 0% | guest 22% | | curf 4.30GHz | curscal 82% |
cpu | sys 40% | user 22% | irq 0% | | | idle 9% | cpu002 w 29% | | steal 0% | guest 22% | | curf 4.30GHz | curscal 82% |
cpu | sys 38% | user 21% | irq 0% | | | idle 11% | cpu003 w 30% | | steal 0% | guest 21% | | curf 4.30GHz | curscal 82% |
cpu | sys 33% | user 24% | irq 0% | | | idle 10% | cpu000 w 32% | | steal 0% | guest 24% | | curf 4.30GHz | curscal 82% |
cpu | sys 31% | user 26% | irq 0% | | | idle 12% | cpu006 w 31% | | steal 0% | guest 26% | | curf 4.30GHz | curscal 82% |
cpu | sys 39% | user 17% | irq 0% | | | idle 3% | cpu005 w 40% | | steal 0% | guest 17% | | curf 4.30GHz | curscal 82% |
cpu | sys 34% | user 20% | irq 0% | | | idle 13% | cpu001 w 33% | | steal 0% | guest 20% | | curf 4.30GHz | curscal 82% |
cpu | sys 30% | user 21% | irq 0% | | | idle 13% | cpu008 w 35% | | steal 0% | guest 21% | | curf 4.30GHz | curscal 82% |
cpu | sys 37% | user 14% | irq 0% | | | idle 21% | cpu010 w 28% | | steal 0% | guest 14% | | curf 4.30GHz | curscal 82% |
cpu | sys 33% | user 16% | irq 1% | | | idle 1% | cpu011 w 49% | | steal 0% | guest 16% | | curf 4.30GHz | curscal 82% |
cpu | sys 25% | user 25% | irq 0% | | | idle 16% | cpu009 w 34% | | steal 0% | guest 25% | | curf 4.30GHz | curscal 82% |
cpu | sys 34% | user 13% | irq 0% | | | idle 14% | cpu007 w 39% | | steal 0% | guest 13% | | curf 4.30GHz | curscal 82% |
CPL | avg1 26.51 | | avg5 11.44 | avg15 7.60 | | | csw 2045975 | | intr 1640155 | | | numcpu 12 | |
MEM | tot 62.6G | free 9.7G | cache 229.4M | dirty 0.0M | buff 0.0M | slab 1.1G | slrec 49.7M | shmem 211.2M | shrss 0.8M | shswp 0.0M | vmbal 0.0M | hptot 0.0M | hpuse 0.0M |
SWP | tot 0.0M | free 0.0M | | | | | | | | | vmcom 64.4G | | vmlim 31.3G |
PAG | scan 18111e4 | steal 1582e3 | | stall 84968 | | | | | | | swin 0 | | swout 0 |
DSK | sda | busy 102% | | read 38361 | write 85 | KiB/r 164 | KiB/w 10 | | MBr/s 102.6 | MBw/s 0.0 | avq 50.81 | | avio 1.56 ms |
DSK | nvme1n1 | busy 7% | | read 829 | write 5582 | KiB/r 37 | KiB/w 32 | | MBr/s 0.5 | MBw/s 2.9 | avq 4.20 | | avio 0.67 ms |
DSK | nvme0n1 | busy 7% | | read 980 | write 5521 | KiB/r 36 | KiB/w 32 | | MBr/s 0.6 | MBw/s 2.9 | avq 3.75 | | avio 0.66 ms |
DSK | sdb | busy 0% | | read 1 | write 21 | KiB/r 4 | KiB/w 5 | | MBr/s 0.0 | MBw/s 0.0 | avq 1.80 | | avio 1.14 ms |
NFC | rpc 1 | read 0 | | write 0 | retxmit 0 | autref 1 | | | | | | | |
NET | transport | tcpi 275 | tcpo 342 | udpi 1 | udpo 1 | tcpao 1 | tcppo 33 | | tcprs 6 | tcpie 0 | tcpor 2 | udpnp 1 | udpie 0 |
NET | network | ipi 298 | | ipo 293 | ipfrw 0 | deliv 287 | | | | | icmpi 3 | | icmpo 0 |
NET | kvm2032 17% | pcki 4070 | pcko 8167 | sp 10 Mbps | si 1772 Kbps | so 210 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | kvm2068 3% | pcki 2866 | pcko 6357 | sp 10 Mbps | si 357 Kbps | so 158 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | kvm2061 1% | pcki 335 | pcko 3886 | sp 10 Mbps | si 179 Kbps | so 34 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | kvm2229 0% | pcki 37 | pcko 3402 | sp 10 Mbps | si 0 Kbps | so 27 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | kvm2011 0% | pcki 22 | pcko 3382 | sp 10 Mbps | si 0 Kbps | so 27 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | kvm2067 0% | pcki 15 | pcko 3379 | sp 10 Mbps | si 1 Kbps | so 27 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | kvm2030 0% | pcki 17 | pcko 3374 | sp 10 Mbps | si 0 Kbps | so 27 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | kvm2549 0% | pcki 9 | pcko 3377 | sp 10 Mbps | si 0 Kbps | so 27 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | kvm2021 0% | pcki 9 | pcko 3372 | sp 10 Mbps | si 0 Kbps | so 27 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | kvm2004 0% | pcki 161 | pcko 2865 | sp 10 Mbps | si 6 Kbps | so 25 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | eno1 0% | pcki 12971 | pcko 16436 | sp 1000 Mbps | si 372 Kbps | so 2430 Kbps | | coll 0 | mlti 37 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | kvm2041 0% | pcki 56 | pcko 2849 | sp 10 Mbps | si 10 Kbps | so 23 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | br0 ---- | pcki 3650 | pcko 295 | sp 0 Mbps | si 24 Kbps | so 18 Kbps | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
PID TID RDDSK WRDSK WCANCL DSK CMD 1/236
5196 - 865.6M 0K 0K 13% php-fpm
1198 - 591.8M 0K 0K 9% qemu-img
23863 - 414.8M 19312K 0K 6% qemu-kvm
8662 - 391.2M 5080K 0K 6% qemu-kvm
22149 - 323.2M 0K 0K 5% logrotate
8454 - 319.3M 328K 0K 5% qemu-kvm
9230 - 285.6M 23596K 0K 5% qemu-kvm
6725 - 274.5M 124K 0K 4% qemu-kvm
6438 - 219.6M 16012K 0K 3% qemu-kvm
7824 - 230.0M 336K 0K 3% qemu-kvm
9007 - 227.1M 140K 0K 3% qemu-kvm
7483 - 207.1M 472K 0K 3% qemu-kvm
4432 - 207.0M 0K 0K 3% virsh
6993 - 192.5M 692K 0K 3% qemu-kvm
775 - 178.2M 4K 0K 3% bash
8858 - 175.3M 2404K 0K 3% qemu-kvm
2344 - 153.2M 0K 0K 2% sshd
