Linux MD problem

[Shane Hathaway]

Mike Lovell wrote:
> I have a machine that has 4 disks in a raid 10 using md. The machine 
> went through an unclean shutdown yesterday and when the box came up, I 
> saw errors like the following in the kernel log and the array no longer 
> works.
> 
> [   28.575149] md: raid10 personality registered for level 10
> [   28.610827] md: md0 stopped.
> [   28.688678] md: bind<sdu1>
> [   28.688981] md: bind<sdv1>
> [   28.689269] md: bind<sdw1>
> [   28.689566] md: bind<sdx1>
> [   28.689748] md: kicking non-fresh sdw1 from array!
> [   28.689890] md: unbind<sdw1>
> [   28.690036] md: export_rdev(sdw1)
> [   28.690175] md: kicking non-fresh sdv1 from array!
> [   28.690316] md: unbind<sdv1>
> [   28.690452] md: export_rdev(sdv1)
> [   28.690589] md: kicking non-fresh sdu1 from array!
> [   28.690730] md: unbind<sdu1>
> [   28.690866] md: export_rdev(sdu1)
> [   28.704706] raid10: not enough operational mirrors for md0
> [   28.704706] md: pers->run() failed ...
> 
> Anyone have some suggestions on what can be done to get the array 
> working again? There are a lot of virtual machines that are on the array 
> that I really don't want to have to go through and rebuild. Thanks in 
> advance for any advice you guys have.

I would try assembling the array without sdx1:

   mdadm --assemble /dev/md0 /dev/sd[uvw]1 missing

If that works, I would check the filesystem for errors *without 
repairing anything yet*, since I don't want to change any bits at all yet:

   e2fsck -n -f /dev/md0

If that shows no errors, or a small number of easily repaired errors, 
then I would try to re-add sdx1:

   mdadm --manage --re-add /dev/md0 /dev/sdx1

If that second command fails, I would use "--add" instead of "--re-add", 
causing sdx1 to be rebuilt from scratch.

Then I would perform the real filesystem repair:

   e2fsck -p -f /dev/md0

If the first command I gave doesn't work, I would use "dd" to copy the 
partitions to backup drives before doing anything else, then I would do 
more aggressive things like "mdadm --assemble --force --update resync".

Shane