LINUX.ORG.RU
Форум — Admin  

Пионерские вопросы про mdadm и raid 5


0

0

Собрал из старых железок машинку под бэкап, начал собирать пятый рейд из 4х винтов (mdadm -C /dev/md1 -n4 -l5 /dev/sd[abcd]).

В течении 20 минут после запуска отвалились 2 винта. В выводе mdadm --detail /dev/md1 помечены так: sda - faulty-spare и sdd - spare.

Остановил рейд, обнулил заголовки (mdadm --zero-superblock /dev/sd[abcd], собрал по новой, опять отпали те же диски (sd[ad], mdadm отметил их таким же образом).

Собрал рейд из 3х дисков (mdadm -C /dev/md1 -n4 -l5 /dev/sd[bcd]), дождался окончания сборки, сделал
mdadm -G /dev/md1 -n4
mdadm --manage /dev/md1 --add /dev/sda
опять же дождался окончания перестроения. В течении примерно 12 часов после этого рейд не распался на составляющие.

Поверх рейда сделал lvm и один раздел, занимающий весь рейд. Раздел отформатировал в килвайфФС третей версии. Решил для проверки забить его весь из /dev/urandom и посмотреть, что будет происходить в процессе.

В /proc/mdstat наблюдаю сейчас такую вот картинку (процесс забивания рейда мусором в самом разгаре):
Personalities : [raid6] [raid5] [raid4]
md1 : active raid5 sda[3] sdd[2] sdc[1] sdb[0]
937713408 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[====>................] check = 20.2% (63145216/312571136) finish=173.1min speed=24001K/sec

unused devices: <none>


mdadm --detail /dev/md1
/dev/md1:
Version : 00.90
Creation Time : Fri Jul 3 16:14:53 2009
Raid Level : raid5
Array Size : 937713408 (894.27 GiB 960.22 GB)
Used Dev Size : 312571136 (298.09 GiB 320.07 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Sun Jul 5 01:47:36 2009
State : clean, recovering
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Rebuild Status : 22% complete

UUID : 441f64fd:72d6ffe1:f1e307f1:2e795dcb (local to host server2)
Events : 0.207874

Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
3 8 0 3 active sync /dev/sda


Собственно вопросы:
-чего оно вдруг стало чекаться, это нормально?
-стоит ли на таком рейде хранить что более ценное, чем фотографии любимой кошки?
-не пора ли хоронить sda?

← Расширить partition
Как забекапить не останавливая? →

Для полноты картины желательно добавить вывод smartctl --all для обоих подозрительных винтов (sda и sdd).

nnz ★★★★
()
Ответ на: комментарий от nnz 

sda

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Second Generation Serial ATA family
Device Model:     WDC WD3200AAKS-00VYA0
Serial Number:    WD-WCARW0038635
Firmware Version: 12.01B01
User Capacity:    320 072 933 376 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Sun Jul  5 14:44:53 2009 MSD
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0003   179   177   021    Pre-fail  Always       -       4041
  4 Start_Stop_Count        0x0032   099   099   000    Old_age   Always       -       1124
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000e   200   200   051    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   093   093   000    Old_age   Always       -       5744
 10 Spin_Retry_Count        0x0012   100   100   051    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0012   100   100   051    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1122
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       525
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1125
194 Temperature_Celsius     0x0022   103   091   000    Old_age   Always       -       44
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   200   200   051    Old_age   Offline      -       0

SMART Error Log Version: 1
ATA Error Count: 20 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 20 occurred at disk power-on lifetime: 5697 hours (237 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 00 06 89 e0  Error: UNC at LBA = 0x00890600 = 8979968

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 00 06 89 01 00   2d+02:56:58.034  READ DMA EXT
  27 00 00 00 00 00 00 00   2d+02:56:58.034  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 00 02   2d+02:56:58.025  IDENTIFY DEVICE
  ef 03 45 00 00 00 00 02   2d+02:56:58.018  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 00 00   2d+02:56:58.018  READ NATIVE MAX ADDRESS EXT

Error 19 occurred at disk power-on lifetime: 5697 hours (237 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 00 06 89 e0  Error: UNC at LBA = 0x00890600 = 8979968

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 00 06 89 01 00   2d+02:56:55.556  READ DMA EXT
  27 00 00 00 00 00 00 00   2d+02:56:55.556  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 00 02   2d+02:56:55.547  IDENTIFY DEVICE
  ef 03 45 00 00 00 00 02   2d+02:56:55.540  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 00 00   2d+02:56:55.540  READ NATIVE MAX ADDRESS EXT

Error 18 occurred at disk power-on lifetime: 5697 hours (237 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 00 06 89 e0  Error: UNC at LBA = 0x00890600 = 8979968

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 00 06 89 01 00   2d+02:56:53.074  READ DMA EXT
  27 00 00 00 00 00 00 00   2d+02:56:53.074  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 00 02   2d+02:56:53.065  IDENTIFY DEVICE
  ef 03 45 00 00 00 00 02   2d+02:56:53.058  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 00 00   2d+02:56:53.058  READ NATIVE MAX ADDRESS EXT

Error 17 occurred at disk power-on lifetime: 5697 hours (237 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 00 06 89 e0  Error: UNC at LBA = 0x00890600 = 8979968

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 00 06 89 01 00   2d+02:56:50.597  READ DMA EXT
  27 00 00 00 00 00 00 00   2d+02:56:50.597  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 00 02   2d+02:56:50.588  IDENTIFY DEVICE
  ef 03 45 00 00 00 00 02   2d+02:56:50.581  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 00 00   2d+02:56:50.581  READ NATIVE MAX ADDRESS EXT

Error 16 occurred at disk power-on lifetime: 5697 hours (237 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 00 06 89 e0  Error: UNC at LBA = 0x00890600 = 8979968

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 00 06 89 01 00   2d+02:56:48.119  READ DMA EXT
  27 00 00 00 00 00 00 00   2d+02:56:48.119  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 00 02   2d+02:56:48.110  IDENTIFY DEVICE
  ef 03 45 00 00 00 00 02   2d+02:56:48.103  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 00 00   2d+02:56:48.103  READ NATIVE MAX ADDRESS EXT

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Lonli-Lokli ★★
() автор топика
Ответ на: комментарий от nnz 

sdd

smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Second Generation Serial ATA family
Device Model:     WDC WD3200AAKS-00VYA0
Serial Number:    WD-WCARW0038406
Firmware Version: 12.01B01
User Capacity:    320 072 933 376 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Sun Jul  5 14:45:02 2009 MSD
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		 (8400) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 106) minutes.
Conveyance self-test routine
recommended polling time: 	 (   6) minutes.
SCT capabilities: 	       (0x303f)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0003   182   176   021    Pre-fail  Always       -       3875
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       674
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000e   200   200   051    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   094   094   000    Old_age   Always       -       4563
 10 Spin_Retry_Count        0x0012   100   100   051    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0012   100   100   051    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       672
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       372
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       676
194 Temperature_Celsius     0x0022   104   090   000    Old_age   Always       -       43
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       3
200 Multi_Zone_Error_Rate   0x0008   200   200   051    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Lonli-Lokli ★★
() автор топика
Ответ на: комментарий от Lonli-Lokli

Ну, как вы уже, наверное, догадались, sda капец. sdd вроде живой.

Правда, настораживает ненулевой Spin_Up_Time, на наших рабочих винтах он нулевой. Правда, у нас не WD, а seagate. Зато у нас Seek_Error_Rate растет с огромной скоростью и это ровно ничего не значит :)

За свою долгую и счастливую жизнь я накопил определенный опыт. Обычно существуют следующие признаки смерти по smartctl:
1. Непустой SMART Error Log
2. Ненулевой Raw_Read_Error_Rate
3. Ненулевой Reallocated_Sector_Ct
4. Никаких признаков нет, в смарте все идеально, а винт умирает. (Примерно четверть всех смертей винтов от старости не предсказывается смартом.)

Насчет рейда и лвм от рекомендаций воздержусь, ибо пока не спец в этом.

nnz ★★★★
()
Ответ на: комментарий от nnz

Всегда интересовало, когда смотрел в вывод smartctl, что означает "TYPE": "Pre-fail", "Old_age"? Первое значит, что такое значение параметра характерно для винта, пребывающего в состоянии "вот-вот на**нется", а второе - "старый, но пока еще шевелится"?

anonymous4
()
Ответ на: комментарий от anonymous4

Я понимаю это так:
Old_age - чем оно больше, тем старее винт.
Pre-fail - если не ноль - винту капец.

Но это не истина в последней инстанции, а всего лишь чьи-то соображения. Вот у нас на сигейтах, как я уже говорил, Seek_Error_Rate, который Pre-fail, крутится с бешеной скоростью годами. А винты ничего, работают. Тьфу x3 :)

nnz ★★★★
()
Вы не можете добавлять комментарии в эту тему. Тема перемещена в архив.
← Расширить partition
Admin
Как забекапить не останавливая? →