Currently unreadable (pending) sectors アーカイブ

FreeNAS でCurrently unreadable (pending) sectorsが発生、その対処方法は 昨日FreeNASを導入していたファイルサーバーがスタックしてしまい、電源を強制断したところWEB管理画面にCRITICALエラー発生のアラートが表示されました。 導入から5年、ついに来てしまったかと思いながら確認したところ、どうもエラーが発生したような形跡がないんです。 Currently unreadable (pending) sectorsの発見から対処までの流れをまとめてみます。 zpool statusの確認 [root@kaga] ~# zpool status pool: freenas-boot state: ONLINE scan: scrub repaired 0 in 0h1m with 0 errors on Wed Apr 26 03:46:15 2017 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 gptid/145b3613-a3e8-11e4-8597-001517cff324 ONLINE 0 0 0 errors: No known data errors pool: tank state: ONLINE scan: scrub repaired 0 in 4h39m with 0 errors on Sun Feb 19 04:39:18 2017 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/1bcf0dfa-3c6a-11e3-a467-38eaa7a92833 ONLINE 0 0 0 gptid/1ca4547c-3c6a-11e3-a467-38eaa7a92833 ONLINE 0 0 0 gptid/1da21f6c-3c6a-11e3-a467-38eaa7a92833 ONLINE 0 0 0 gptid/1e8c3084-3c6a-11e3-a467-38eaa7a92833 ONLINE 0 0 0 errors: No known data errors 今回発生したCRITICALエラーには、「Device: /dev/ada1, 1 Currently unreadable (pending) sectors」とあったので、/dev/sda1でエラーが起きていると判断して、zpool statusを叩いてみたのですが、結果はエラー無し。 smartctl -a /dev/ada1 CRITICALエラー発生しているのに何で・・・？と思いながら/dev/sda1のオフラインテストを行った後、SMART値を確認してみました。 オフラインテスト実施（2TBのHDDで7時間ほどかかりました） smartctl -t long /dev/ada1 SMART値の確認 [root@kaga] ~# smartctl -a /dev/ada1 smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green (AF) Device Model: WDC WD20EARS-00MVWB0 Serial Number: WD-WCAZA6097262 LU WWN Device Id: 5 0014ee 25b069974 Firmware Version: 51.0AB51 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 2.6, 3.0 Gb/s Local Time is: Wed Apr 26 11:50:06 2017 JST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: (36780) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 355) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 253 165 021 Pre-fail Always - 950 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2398 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 033 033 000 Old_age Always - 49443 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 85 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 49 193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 1648955 194 Temperature_Celsius 0x0022 116 084 000 Old_age Always - 34 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 49429 19506320 # 2 Extended offline Interrupted (host reset) 60% 34995 - # 3 Extended offline Completed without error 00% 34992 - # 4 Extended offline Completed without error 00% 34986 - # 5 Extended offline Completed without error 00% 34979 - # 6 Extended offline Completed without error 00% 34972 - # 7 Extended offline Completed without error 00% 34965 - # 8 Extended offline Completed without error 00% 34958 - # 9 Extended offline Completed without error 00% 34951 - #10 Extended offline Completed without error 00% 34944 - #11 Extended offline Completed without error 00% 34937 - #12 Extended offline Completed without error 00% 34930 - #13 Extended offline Completed without error 00% 34923 - #14 Extended offline Completed without error 00% 34916 - #15 Extended offline Completed without error 00% 34909 - #16 Extended offline Completed without error 00% 34902 - #17 Extended offline Completed without error 00% 34895 - #18 Extended offline Completed without error 00% 34888 - #19 Extended offline Completed without error 00% 34881 - #20 Extended offline Completed without error 00% 34874 - #21 Extended offline Completed without error 00% 34867 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. 結果を確認してみると、確かにCurrent_Pending_Sectorが1つあるようですね。 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 Current Pending Sectorは不安定な状態にはなっているけど、読み込める可能性もあるのでremapせずにひとまず放置しているセクターになります。 直ちに影響が出るものでもありませんが、こちらが解決されない限りずっとアラートが出続けるので、サクッと対処します。 上記のSMART値の結果の内、 Sector Size: 512 bytes logical/physical と Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 49429 19506320 を確認します。 セクターサイズが512バイトであり、Current Pending Sectorが19506320セクターであることが分かります。 ここのセクターに対して書き込みを行います。 [root@kaga] ~# sysctl kern.geom.debugflags=16 kern.geom.debugflags: 0 -&gt; 16 [root@kaga] ~# dd if=/dev/ada1 of=/dev/ada1 bs=512 count=1 iseek=19506320 oseek=19506320 conv=noerror,sync dd: /dev/ada1: Input/output error 0+0 records in 0+0 records out 0 bytes transferred in 13.892026 secs (0 bytes/sec) dd: /dev/ada1: Input/output error 1+0 records in 1+0 records out 512 bytes transferred in 13.981968 secs (37 bytes/sec) この状態で再度、SMART値を確認します。 [root@kaga] ~# smartctl -a /dev/ada1 smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green (AF) Device Model: WDC WD20EARS-00MVWB0 Serial Number: WD-WCAZA6097262 LU WWN Device Id: 5 0014ee 25b069974 Firmware Version: 51.0AB51 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 2.6, 3.0 Gb/s Local Time is: Wed Apr 26 12:00:53 2017 JST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: (36780) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 355) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 253 165 021 Pre-fail Always - 950 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2398 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 033 033 000 Old_age Always - 49443 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 85 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 49 193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 1648958 194 Temperature_Celsius 0x0022 116 084 000 Old_age Always - 34 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 49429 19506320 # 2 Extended offline Interrupted (host reset) 60% 34995 - # 3 Extended offline Completed without error 00% 34992 - # 4 Extended offline Completed without error 00% 34986 - # 5 Extended offline Completed without error 00% 34979 - # 6 Extended offline Completed without error 00% 34972 - # 7 Extended offline Completed without error 00% 34965 - # 8 Extended offline Completed without error 00% 34958 - # 9 Extended offline Completed without error 00% 34951 - #10 Extended offline Completed without error 00% 34944 - #11 Extended offline Completed without error 00% 34937 - #12 Extended offline Completed without error 00% 34930 - #13 Extended offline Completed without error 00% 34923 - #14 Extended offline Completed without error 00% 34916 - #15 Extended offline Completed without error 00% 34909 - #16 Extended offline Completed without error 00% 34902 - #17 Extended offline Completed without error 00% 34895 - #18 Extended offline Completed without error 00% 34888 - #19 Extended offline Completed without error 00% 34881 - #20 Extended offline Completed without error 00% 34874 - #21 Extended offline Completed without error 00% 34867 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Current_Pending_Sectorの値が1から0になっていることが確認できます。 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 なお、末尾にRead failureのログが表示されていますが、前回実施したテストの結果ですので、上記のCurrent_Pending_Sectorの値が1になっていれば問題ありません。 まとめ CRITICALエラーのアラートを見てかなり焦りましたが、そこまで深刻な物ではなく安心しました。 SMARTの値をCUIで確認するなんて初めての経験でしたので、ちょうど良い障害対応でした。