目次
FreeNAS でCurrently unreadable (pending) sectorsが発生、その対処方法は
昨日FreeNASを導入していたファイルサーバーがスタックしてしまい、電源を強制断したところWEB管理画面にCRITICALエラー発生のアラートが表示されました。
導入から5年、ついに来てしまったかと思いながら確認したところ、どうもエラーが発生したような形跡がないんです。
Currently unreadable (pending) sectorsの発見から対処までの流れをまとめてみます。
zpool statusの確認
[root@kaga] ~# zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h1m with 0 errors on Wed Apr 26 03:46:15 2017
config:
NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
gptid/145b3613-a3e8-11e4-8597-001517cff324 ONLINE 0 0 0
errors: No known data errors
pool: tank
state: ONLINE
scan: scrub repaired 0 in 4h39m with 0 errors on Sun Feb 19 04:39:18 2017
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/1bcf0dfa-3c6a-11e3-a467-38eaa7a92833 ONLINE 0 0 0
gptid/1ca4547c-3c6a-11e3-a467-38eaa7a92833 ONLINE 0 0 0
gptid/1da21f6c-3c6a-11e3-a467-38eaa7a92833 ONLINE 0 0 0
gptid/1e8c3084-3c6a-11e3-a467-38eaa7a92833 ONLINE 0 0 0
errors: No known data errors
今回発生したCRITICALエラーには、「Device: /dev/ada1, 1 Currently unreadable (pending) sectors」とあったので、/dev/sda1でエラーが起きていると判断して、zpool statusを叩いてみたのですが、結果はエラー無し。
smartctl -a /dev/ada1
CRITICALエラー発生しているのに何で・・・？と思いながら/dev/sda1のオフラインテストを行った後、SMART値を確認してみました。
オフラインテスト実施（2TBのHDDで7時間ほどかかりました）
smartctl -t long /dev/ada1
SMART値の確認
[root@kaga] ~# smartctl -a /dev/ada1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green (AF)
Device Model: WDC WD20EARS-00MVWB0
Serial Number: WD-WCAZA6097262
LU WWN Device Id: 5 0014ee 25b069974
Firmware Version: 51.0AB51
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.6, 3.0 Gb/s
Local Time is: Wed Apr 26 11:50:06 2017 JST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 121) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: (36780) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 355) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x3035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 253 165 021 Pre-fail Always - 950
4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2398
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 033 033 000 Old_age Always - 49443
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 85
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 49
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 1648955
194 Temperature_Celsius 0x0022 116 084 000 Old_age Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 49429 19506320
# 2 Extended offline Interrupted (host reset) 60% 34995 -
# 3 Extended offline Completed without error 00% 34992 -
# 4 Extended offline Completed without error 00% 34986 -
# 5 Extended offline Completed without error 00% 34979 -
# 6 Extended offline Completed without error 00% 34972 -
# 7 Extended offline Completed without error 00% 34965 -
# 8 Extended offline Completed without error 00% 34958 -
# 9 Extended offline Completed without error 00% 34951 -
#10 Extended offline Completed without error 00% 34944 -
#11 Extended offline Completed without error 00% 34937 -
#12 Extended offline Completed without error 00% 34930 -
#13 Extended offline Completed without error 00% 34923 -
#14 Extended offline Completed without error 00% 34916 -
#15 Extended offline Completed without error 00% 34909 -
#16 Extended offline Completed without error 00% 34902 -
#17 Extended offline Completed without error 00% 34895 -
#18 Extended offline Completed without error 00% 34888 -
#19 Extended offline Completed without error 00% 34881 -
#20 Extended offline Completed without error 00% 34874 -
#21 Extended offline Completed without error 00% 34867 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
結果を確認してみると、確かにCurrent_Pending_Sectorが1つあるようですね。
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1
Current Pending Sectorは不安定な状態にはなっているけど、読み込める可能性もあるのでremapせずにひとまず放置しているセクターになります。
直ちに影響が出るものでもありませんが、こちらが解決されない限りずっとアラートが出続けるので、サクッと対処します。
上記のSMART値の結果の内、
Sector Size: 512 bytes logical/physical
と
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 49429 19506320
を確認します。 セクターサイズが512バイトであり、Current Pending Sectorが19506320セクターであることが分かります。
ここのセクターに対して書き込みを行います。
[root@kaga] ~# sysctl kern.geom.debugflags=16 kern.geom.debugflags: 0 -> 16 [root@kaga] ~# dd if=/dev/ada1 of=/dev/ada1 bs=512 count=1 iseek=19506320 oseek=19506320 conv=noerror,sync dd: /dev/ada1: Input/output error 0+0 records in 0+0 records out 0 bytes transferred in 13.892026 secs (0 bytes/sec) dd: /dev/ada1: Input/output error 1+0 records in 1+0 records out 512 bytes transferred in 13.981968 secs (37 bytes/sec)
この状態で再度、SMART値を確認します。
[root@kaga] ~# smartctl -a /dev/ada1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green (AF)
Device Model: WDC WD20EARS-00MVWB0
Serial Number: WD-WCAZA6097262
LU WWN Device Id: 5 0014ee 25b069974
Firmware Version: 51.0AB51
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.6, 3.0 Gb/s
Local Time is: Wed Apr 26 12:00:53 2017 JST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 121) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: (36780) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 355) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x3035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 253 165 021 Pre-fail Always - 950
4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2398
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 033 033 000 Old_age Always - 49443
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 85
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 49
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 1648958
194 Temperature_Celsius 0x0022 116 084 000 Old_age Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 49429 19506320
# 2 Extended offline Interrupted (host reset) 60% 34995 -
# 3 Extended offline Completed without error 00% 34992 -
# 4 Extended offline Completed without error 00% 34986 -
# 5 Extended offline Completed without error 00% 34979 -
# 6 Extended offline Completed without error 00% 34972 -
# 7 Extended offline Completed without error 00% 34965 -
# 8 Extended offline Completed without error 00% 34958 -
# 9 Extended offline Completed without error 00% 34951 -
#10 Extended offline Completed without error 00% 34944 -
#11 Extended offline Completed without error 00% 34937 -
#12 Extended offline Completed without error 00% 34930 -
#13 Extended offline Completed without error 00% 34923 -
#14 Extended offline Completed without error 00% 34916 -
#15 Extended offline Completed without error 00% 34909 -
#16 Extended offline Completed without error 00% 34902 -
#17 Extended offline Completed without error 00% 34895 -
#18 Extended offline Completed without error 00% 34888 -
#19 Extended offline Completed without error 00% 34881 -
#20 Extended offline Completed without error 00% 34874 -
#21 Extended offline Completed without error 00% 34867 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Current_Pending_Sectorの値が1から0になっていることが確認できます。
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
なお、末尾にRead failureのログが表示されていますが、前回実施したテストの結果ですので、上記のCurrent_Pending_Sectorの値が1になっていれば問題ありません。
まとめ
CRITICALエラーのアラートを見てかなり焦りましたが、そこまで深刻な物ではなく安心しました。
SMARTの値をCUIで確認するなんて初めての経験でしたので、ちょうど良い障害対応でした。
