How to rebuild a degraded vdisk in a HP MSA 2012fc

December 21, 2010 30 Comments

Recently our HP MSA suffered a VDisk Failure, reporting the following messages:
Degraded.
Virtual-disk is not fault tolerant.

 

 Event 55: A disk drive reported a SMART event
 Event 314: There is a problem with a FRU.

 

There did not seem to be much information online as to how to resolve the issue. The Physical Hard Disk Drive had not failed, but the vdisk array was in a degraded state.

The first thing that needs to be done, is the meta-data on the leftover disk needs to be cleared before it can re-join the RAID as either a disk or a dedicated spare.

Clearing Disk Meta Data

Each disk has metadata that identifies whether the disk is a member of a vdisk, and identifies other members of that vdisk. If a disk’s metadata says the disk is a member of a vdisk but other members’ metadata say the disk isn’t a member, the disk becomes a leftover. The system overview and enclosure overview pages show the disk’s How Used value as LEFTOVR. A leftover disk’s Fault/UID LED is illuminated amber.

Before you can use the disk in a new vdisk or as a spare, you must clear the disk’s metadata.

To Clear meta data from leftover disks

  1. In the Configuration View panel, right-click the system and then select Tools > Clear Disk Metadata.
  2. In the main panel, select disks to clear metadata from.
  3. Click Clear Metadata. When processing is complete a success dialog appears.

  4. Click Ok

Re-Join Physical Disk to VDisk RAID

  1. In the HP MSA Storage Managment Utility, rRight click on the affected vdisk and select Configuration > Manage Dedicated Spares
  2. Your disk should appear in the list of available drives with a state of AVAIL. Tick the drive, then click the Modify Spares button.
  3. The disk will be re-joined to the array. Initially it may be listed as a spare, but the MSA will automatically re-join the disk as an active member of the RAID if this is how it was originally configured.
  4. The array will begin the Reconstruction process automatically. This can take a very long time depending on the size of your drives.

30 thoughts on “How to rebuild a degraded vdisk in a HP MSA 2012fc”

  1. Thanks for this. The MSA 2300 Series has to be the worst supported Stroage Solution in the world. There is so little documentation online for it, and what is, never matches up with what I actually see in my managment environment.

    It was nice to find your comments and help confirm that what I thought had I do to was the right thing.

    just to confirm, that this process resolved my issue, and cleared the aleats (after giving the MSA a while to rebuild the array).

  4. Very much appreciated, as the vendor support team wasn’t really helpful here. They asked me to telnet onto the disk array and perform a trust vdisk on the affected vdisks. I was wary, as the number of times they asked me to back up the data was making me pretty nervous.

    One other step that I performed after the above was to verify the vdisk to ensure no issues existed. You can right click on the vdisk and go to tools to do this.

    Thanks very much again for above.

  5. Hello,
    I am adding myself with appreciation of this as it really helped me yesterday with rebuilding the array.
    Big thanks.

  8. Hi,

    What should i do if those those assign global spare and dedicated spare frequently turn to available or leftover. After re-assign those hdd to global or dedicated spare.

    Pleased advice. Thank You

  10. Hi I have facing problem in MSA2312fc, My Vdisk are degraded & 2 Hard disks leftover. I want to know, If am clear the metadata, my backup files will be available? Its a RAID 5 with 300 GB HDD*5.

  11. Hello Steve,

    Unfortunately we have three disks with the value as LEFTOVR and the amber light on, obviously the system is degraded and it also warns me that the Virtual-disk is not fault tolerant.

    Therefore my question is, can I follow your instructions even with 3 “LEFTOVR” disks?

    Thank you very much in advance

  12. Hi Paul,
    I have never been in the situation with more than 1 LEFTOVR disks.
    My instinct would tell me that it is a bad idea to clear them all. However I am not sure of the correct answer.
    Are you in a position to be able to call HP support, or a HP Raid specialist for support?

    Good luck.

  13. Hi Raaja,
    I have never been in the situation with more than 1 LEFTOVR disks.
    My instinct would tell me that it is a bad idea to clear them all. However I am not sure of the correct answer.
    Are you in a position to be able to call HP support, or a HP Raid specialist for support?

    Good luck.

  14. Hi,
    I have had this situation before.
    First thing to try is to put the drive into a different bay in the MSA to rule out the bay is faulty.

    Unfortunately, my problem required me to replace the drive completely.
    This will be covered by warranty if you still have it.

    Good luck.

  15. Very useful information indeed.

    We were in the same situation as Paul with three LEFTOVR disks in the array. I did have one available drive so added that to the group before clearing the three drives. Two of the three are back in the array and the third is now the available spare.

    Works like a charm.

  16. Thank you, Steve!

    We had _four_ disks with LEFTOVR and the system was degraded etc. After clearing meta data from all leftover disks according your manual it successfully started Reconstruct job. 300.0GB disks were physically OK. It was only logical issue, I think.

  17. Hi all,
    I have the problem of vdisk failure on my msa. The VRAID level is Vraid10 without any spare. I replace the disk and configure it as “dedicated Spare” since 26 hours. But the revuilding is not start till now. How can I do this manualy?

  18. @Steve you’re the man very good experience in HP storagework.well done for posting such article it helps me in a dangerous state..much love…
    cheerrrrrrrrrrrrrrrr..

  20. Hi all,
    I have the problem of vdisk failure on my msa. The VRAID level is Vraid10 without any spare. I replace the disk and configure it as “dedicated Spare” since 26 hours. But the revuilding is not start till now. How can I do this manualy

  21. I have an HP Storage on which a 16 Disks were configured on a RAID5 VDisk. One of the failed and another in a Leftover state leaving the VDisk in a degraded state. The failed disk has been replaced and in the Available state while the other is still in Leftover state. The RAID5 VDisk has 14 Disks now and in a degraded state. Will the Re-join advise work without losing any data?

  24. Thanks for the info, array has just finished rebuilding.

    For those looking for more useful info that hp supply, these arrays are rebranded Dot Hill equipment. There website tends to have much better information available.

  25. Thanks Steve, I have 11 drives configured as RAID 6 and 1 global spare.

    I had 3 drives light up amber and go to LEFTOVER state.

    VDISK RECON is going at the moment.

    The MSA OS has to be the poorest design around.

    Thanks again for your help.

    Al

  27. thank you steve. i had an issue with a replaced disk on an old 2212fc the gui is completely different but the manual gave me the right steps to solve the problem

  29. Thank you. Very much appreciated because this helped me very much.
    Thanks for the explanation about the metadata as well.
    Now verry clear for me

Comments are closed.