FreeNAS でCurrently unreadable (pending) sectorsが発生、その対処方法は

FreeNAS サーバー
スポンサーリンク

FreeNAS でCurrently unreadable (pending) sectorsが発生、その対処方法は

昨日FreeNASを導入していたファイルサーバーがスタックしてしまい、電源を強制断したところWEB管理画面にCRITICALエラー発生のアラートが表示されました。

導入から5年、ついに来てしまったかと思いながら確認したところ、どうもエラーが発生したような形跡がないんです。

Currently unreadable (pending) sectorsの発見から対処までの流れをまとめてみます。

zpool statusの確認

[root@kaga] ~# zpool status
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h1m with 0 errors on Wed Apr 26 03:46:15 2017
config:

        NAME                                          STATE     READ WRITE CKSUM
        freenas-boot                                  ONLINE       0     0     0
          gptid/145b3613-a3e8-11e4-8597-001517cff324  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
  scan: scrub repaired 0 in 4h39m with 0 errors on Sun Feb 19 04:39:18 2017
config:

        NAME                                            STATE     READ WRITE CKSUM
        tank                                            ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/1bcf0dfa-3c6a-11e3-a467-38eaa7a92833  ONLINE       0     0     0
            gptid/1ca4547c-3c6a-11e3-a467-38eaa7a92833  ONLINE       0     0     0
            gptid/1da21f6c-3c6a-11e3-a467-38eaa7a92833  ONLINE       0     0     0
            gptid/1e8c3084-3c6a-11e3-a467-38eaa7a92833  ONLINE       0     0     0

errors: No known data errors

今回発生したCRITICALエラーには、「Device: /dev/ada1, 1 Currently unreadable (pending) sectors」とあったので、/dev/sda1でエラーが起きていると判断して、zpool statusを叩いてみたのですが、結果はエラー無し。

smartctl -a /dev/ada1

CRITICALエラー発生しているのに何で・・・?と思いながら/dev/sda1のオフラインテストを行った後、SMART値を確認してみました。

オフラインテスト実施(2TBのHDDで7時間ほどかかりました)

smartctl -t long /dev/ada1

SMART値の確認

[root@kaga] ~# smartctl -a /dev/ada1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (AF)
Device Model:     WDC WD20EARS-00MVWB0
Serial Number:    WD-WCAZA6097262
LU WWN Device Id: 5 0014ee 25b069974
Firmware Version: 51.0AB51
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Wed Apr 26 11:50:06 2017 JST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 121) The previous self-test completed having
                                        the read element of the test failed.
Total time to complete Offline
data collection:                (36780) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 355) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3035) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   253   165   021    Pre-fail  Always       -       950
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2398
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   033   033   000    Old_age   Always       -       49443
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       85
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       49
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       1648955
194 Temperature_Celsius     0x0022   116   084   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       1
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     49429         19506320
# 2  Extended offline    Interrupted (host reset)      60%     34995         -
# 3  Extended offline    Completed without error       00%     34992         -
# 4  Extended offline    Completed without error       00%     34986         -
# 5  Extended offline    Completed without error       00%     34979         -
# 6  Extended offline    Completed without error       00%     34972         -
# 7  Extended offline    Completed without error       00%     34965         -
# 8  Extended offline    Completed without error       00%     34958         -
# 9  Extended offline    Completed without error       00%     34951         -
#10  Extended offline    Completed without error       00%     34944         -
#11  Extended offline    Completed without error       00%     34937         -
#12  Extended offline    Completed without error       00%     34930         -
#13  Extended offline    Completed without error       00%     34923         -
#14  Extended offline    Completed without error       00%     34916         -
#15  Extended offline    Completed without error       00%     34909         -
#16  Extended offline    Completed without error       00%     34902         -
#17  Extended offline    Completed without error       00%     34895         -
#18  Extended offline    Completed without error       00%     34888         -
#19  Extended offline    Completed without error       00%     34881         -
#20  Extended offline    Completed without error       00%     34874         -
#21  Extended offline    Completed without error       00%     34867         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

結果を確認してみると、確かにCurrent_Pending_Sectorが1つあるようですね。

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       1

Current Pending Sectorは不安定な状態にはなっているけど、読み込める可能性もあるのでremapせずにひとまず放置しているセクターになります。

直ちに影響が出るものでもありませんが、こちらが解決されない限りずっとアラートが出続けるので、サクッと対処します。

上記のSMART値の結果の内、

Sector Size:      512 bytes logical/physical

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     49429         19506320

を確認します。 セクターサイズが512バイトであり、Current Pending Sectorが19506320セクターであることが分かります。

ここのセクターに対して書き込みを行います。

[root@kaga] ~# sysctl kern.geom.debugflags=16
kern.geom.debugflags: 0 -> 16
[root@kaga] ~# dd if=/dev/ada1 of=/dev/ada1 bs=512 count=1 iseek=19506320 oseek=19506320 conv=noerror,sync
dd: /dev/ada1: Input/output error
0+0 records in
0+0 records out
0 bytes transferred in 13.892026 secs (0 bytes/sec)
dd: /dev/ada1: Input/output error
1+0 records in
1+0 records out
512 bytes transferred in 13.981968 secs (37 bytes/sec)

この状態で再度、SMART値を確認します。

[root@kaga] ~# smartctl -a /dev/ada1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (AF)
Device Model:     WDC WD20EARS-00MVWB0
Serial Number:    WD-WCAZA6097262
LU WWN Device Id: 5 0014ee 25b069974
Firmware Version: 51.0AB51
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Wed Apr 26 12:00:53 2017 JST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 121) The previous self-test completed having
                                        the read element of the test failed.
Total time to complete Offline
data collection:                (36780) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 355) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3035) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   253   165   021    Pre-fail  Always       -       950
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2398
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   033   033   000    Old_age   Always       -       49443
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       85
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       49
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       1648958
194 Temperature_Celsius     0x0022   116   084   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     49429         19506320
# 2  Extended offline    Interrupted (host reset)      60%     34995         -
# 3  Extended offline    Completed without error       00%     34992         -
# 4  Extended offline    Completed without error       00%     34986         -
# 5  Extended offline    Completed without error       00%     34979         -
# 6  Extended offline    Completed without error       00%     34972         -
# 7  Extended offline    Completed without error       00%     34965         -
# 8  Extended offline    Completed without error       00%     34958         -
# 9  Extended offline    Completed without error       00%     34951         -
#10  Extended offline    Completed without error       00%     34944         -
#11  Extended offline    Completed without error       00%     34937         -
#12  Extended offline    Completed without error       00%     34930         -
#13  Extended offline    Completed without error       00%     34923         -
#14  Extended offline    Completed without error       00%     34916         -
#15  Extended offline    Completed without error       00%     34909         -
#16  Extended offline    Completed without error       00%     34902         -
#17  Extended offline    Completed without error       00%     34895         -
#18  Extended offline    Completed without error       00%     34888         -
#19  Extended offline    Completed without error       00%     34881         -
#20  Extended offline    Completed without error       00%     34874         -
#21  Extended offline    Completed without error       00%     34867         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Current_Pending_Sectorの値が1から0になっていることが確認できます。

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0

なお、末尾にRead failureのログが表示されていますが、前回実施したテストの結果ですので、上記のCurrent_Pending_Sectorの値が1になっていれば問題ありません。

まとめ

CRITICALエラーのアラートを見てかなり焦りましたが、そこまで深刻な物ではなく安心しました。

SMARTの値をCUIで確認するなんて初めての経験でしたので、ちょうど良い障害対応でした。

コメント