View Issue Details

IDProjectCategoryView StatusLast Update
0001266bareos-corevmware pluginpublic2022-08-24 14:57
Reportervrurg Assigned Tobruno-at-bareos  
PriorityurgentSeveritymajorReproducibilityalways
Status closedResolutionfixed 
PlatformLinuxOSCentOSOS Version7
Product Version18.2.9 
Summary0001266: A VMDK file restored using localvmdk=yes is not recognized as a virtual disk.
DescriptionAfter loosing a bunch of servers in an accident I needed to restore a VM into a file as restoring into a vCenter is not an option. Unfortunately, the resulting VMDK is not recognized by none of the applications and utilities I tried: VMWare Fusion (including vmware-vdiskmanager), ESXi (including vmkfstool in the shell), VirtualBox, qemu-img.

So far, after communicating to many people including VMWare suuport and community, the problem looks like the file I've got looks somewhat alike to VMDK monolithic format with the descriptor embedded into the header. Except that normal monolithic is supposed to start with 'KMDV' signature and have the embedded descriptor in a form of a normal text file. In fact, the VMDK I have at my hands starts with some kind of binary header, mostly zeroed out, and contains incomplete descriptor data in binary form where text configuration variable names are separated from their values with zeroes. In turn, the header is followed by the actual VMDK flat file starting somewhere at offset 918 (I guess, the actual offset may vary depending on the header content).

I tried different method of recovering the VMDK including cutting of the 918 bytes of the header to get the flat version and restoring the descriptor the way a VMWare support KB suggest (used vmkfstools). Unfortunately, the result suffers from wrong disk geometry causing boot failure and worse – FreeBSD zfs not being able to recover the ZFS partition on the disk.

So far, I barely can guess what is the source of the problem neither offer a way to reproduce it. We have tried restoring some test VMs the very same way and everything worked fine because the VMDKs created in these cases were all-fine monolithics with the expected KMDV signature and text descriptor embedded. The only difference of the problematic case from the test ones is that it had at least one snapshot at the time of backup. And, perhaps, bigger size though I don't think that 200GB should be of any problem here.
Additional InformationAttached is the hex dump of the restored VMDK file header.
TagsNo tags attached.

Activities

vrurg

vrurg

2020-08-08 22:29

reporter  

vmdk-header-hex (7,669 bytes)   
00000000  12 20 12 12 01 00 00 00  00 00 00 00 32 00 00 00  |. ..........2...|
00000010  00 00 00 00 00 00 00 00  fc 65 00 00 ff 00 00 00  |.........e......|
00000020  3f 00 00 00 fc 65 00 00  ff 00 00 00 3f 00 00 00  |?....e......?...|
00000030  00 00 00 19 00 00 00 00  03 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000070  00 00 00 00 00 00 00 00  00 00 00 00 12 20 12 12  |............. ..|
00000080  12 20 12 12 0a 00 00 00  06 00 00 00 12 20 12 12  |. ........... ..|
00000090  64 65 6c 65 74 61 62 6c  65 00 74 72 75 65 00 00  |deletable.true..|
000000a0  12 20 12 12 0e 00 00 00  22 00 00 00 12 20 12 12  |. ......".... ..|
000000b0  6c 6f 6e 67 43 6f 6e 74  65 6e 74 49 44 00 30 38  |longContentID.08|
000000c0  32 38 62 63 38 64 34 66  64 39 30 39 30 61 31 61  |28bc8d4fd9090a1a|
000000d0  33 61 31 34 34 38 36 63  61 33 39 36 33 63 00 00  |3a14486ca3963c..|
000000e0  12 20 12 12 0d 00 00 00  0c 00 00 00 12 20 12 12  |. ........... ..|
000000f0  74 6f 6f 6c 73 56 65 72  73 69 6f 6e 00 32 31 34  |toolsVersion.214|
00000100  37 34 38 33 36 34 37 00  6e 12 20 12 12 0a 00 00  |7483647.n. .....|
00000110  00 06 00 00 00 12 20 12  12 64 65 6c 65 74 61 62  |...... ..deletab|
00000120  6c 65 00 74 72 75 65 00  00 12 20 12 12 0e 00 00  |le.true... .....|
00000130  00 22 00 00 00 12 20 12  12 6c 6f 6e 67 43 6f 6e  |.".... ..longCon|
00000140  74 65 6e 74 49 44 00 30  38 32 38 62 63 38 64 34  |tentID.0828bc8d4|
00000150  66 64 39 30 39 30 61 31  61 33 61 31 34 34 38 36  |fd9090a1a3a14486|
00000160  63 61 33 39 36 33 63 00  00 12 20 12 12 0d 00 00  |ca3963c... .....|
00000170  00 0c 00 00 00 12 20 12  12 74 6f 6f 6c 73 56 65  |...... ..toolsVe|
00000180  72 73 69 6f 6e 00 32 31  34 37 34 38 33 36 34 37  |rsion.2147483647|
00000190  00 6e 12 20 12 12 0c 00  00 00 0a 00 00 00 12 20  |.n. ........... |
000001a0  12 12 61 64 61 70 74 65  72 54 79 70 65 00 6c 73  |..adapterType.ls|
000001b0  69 6c 6f 67 69 63 00 70  12 20 12 12 0a 00 00 00  |ilogic.p. ......|
000001c0  06 00 00 00 12 20 12 12  64 65 6c 65 74 61 62 6c  |..... ..deletabl|
000001d0  65 00 74 72 75 65 00 00  12 20 12 12 13 00 00 00  |e.true... ......|
000001e0  07 00 00 00 12 20 12 12  67 65 6f 6d 65 74 72 79  |..... ..geometry|
000001f0  2e 63 79 6c 69 6e 64 65  72 73 00 32 36 31 30 38  |.cylinders.26108|
00000200  00 00 12 20 12 12 0f 00  00 00 05 00 00 00 12 20  |... ........... |
00000210  12 12 67 65 6f 6d 65 74  72 79 2e 68 65 61 64 73  |..geometry.heads|
00000220  00 32 35 35 00 00 12 20  12 12 11 00 00 00 04 00  |.255... ........|
00000230  00 00 12 20 12 12 67 65  6f 6d 65 74 72 79 2e 73  |... ..geometry.s|
00000240  65 63 74 6f 72 73 00 36  33 00 01 12 20 12 12 0e  |ectors.63... ...|
00000250  00 00 00 22 00 00 00 12  20 12 12 6c 6f 6e 67 43  |...".... ..longC|
00000260  6f 6e 74 65 6e 74 49 44  00 30 38 32 38 62 63 38  |ontentID.0828bc8|
00000270  64 34 66 64 39 30 39 30  61 31 61 33 61 31 34 34  |d4fd9090a1a3a144|
00000280  38 36 63 61 33 39 36 33  63 00 00 12 20 12 12 10  |86ca3963c... ...|
00000290  00 00 00 03 00 00 00 12  20 12 12 74 68 69 6e 50  |........ ..thinP|
000002a0  72 6f 76 69 73 69 6f 6e  65 64 00 31 00 40 12 20  |rovisioned.1.@. |
000002b0  12 12 11 00 00 00 03 00  00 00 12 20 12 12 74 6f  |........... ..to|
000002c0  6f 6c 73 49 6e 73 74 61  6c 6c 54 79 70 65 00 30  |olsInstallType.0|
000002d0  00 40 12 20 12 12 0d 00  00 00 0c 00 00 00 12 20  |.@. ........... |
000002e0  12 12 74 6f 6f 6c 73 56  65 72 73 69 6f 6e 00 32  |..toolsVersion.2|
000002f0  31 34 37 34 38 33 36 34  37 00 6e 12 20 12 12 05  |147483647.n. ...|
00000300  00 00 00 31 00 00 00 12  20 12 12 75 75 69 64 00  |...1.... ..uuid.|
00000310  36 30 20 30 30 20 43 32  20 39 32 20 35 37 20 34  |60 00 C2 92 57 4|
00000320  65 20 63 39 20 66 30 2d  39 33 20 37 35 20 34 39  |e c9 f0-93 75 49|
00000330  20 38 36 20 63 30 20 38  64 20 31 34 20 31 61 00  | 86 c0 8d 14 1a.|
00000340  6f 12 20 12 12 11 00 00  00 04 00 00 00 12 20 12  |o. ........... .|
00000350  12 76 69 72 74 75 61 6c  48 57 56 65 72 73 69 6f  |.virtualHWVersio|
00000360  6e 00 31 31 00 01 12 20  12 12 00 00 00 00 00 00  |n.11... ........|
00000370  00 00 12 20 12 12 12 20  12 12 00 00 00 00 00 00  |... ... ........|
00000380  00 00 00 00 00 00 00 00  20 00 03 00 00 00 12 20  |........ ...... |
00000390  12 12 00 00 00 00 fc 31  c0 8e c0 8e d8 8e d0 bc  |.......1........|
000003a0  00 0e be 1a 7c bf 1a 06  b9 e6 01 f3 a4 e9 00 8a  |....|...........|
000003b0  80 fa 80 72 0b 8a 36 75  04 80 c6 80 38 f2 72 02  |...r..6u....8.r.|
000003c0  b2 80 e8 c8 00 b6 01 bb  00 0e be 88 07 e8 a3 00  |................|
000003d0  66 81 3e 00 0e 45 46 49  20 75 0d 66 81 3e 04 0e  |f.>..EFI u.f.>..|
000003e0  50 41 52 54 75 02 eb 19  80 fe 01 0f 85 b2 00 b6  |PARTu...........|
000003f0  00 be 10 12 bf 88 07 66  ff 0c b9 02 00 f3 a5 eb  |.......f........|
00000400  c6 be 48 0e bb 00 10 e8  69 00 89 de bf 90 07 b1  |..H.....i.......|
00000410  10 f3 a6 75 3f 89 df 8d  75 20 bb c0 07 8e c3 31  |...u?...u .....1|
00000420  db 56 e8 4e 00 5e 66 8b  45 28 66 3b 04 75 0f 66  |.V.N.^f.E(f;.u.f|
00000430  8b 45 2c 66 3b 44 04 75  05 8e c3 e9 58 75 66 ff  |.E,f;D.u....Xuf.|
00000440  04 66 83 54 04 00 8c c0  83 c0 20 3d 00 90 73 4c  |.f.T...... =..sL|
00000450  8e c0 eb cd 66 ff 0e 50  0e 74 50 a1 54 0e 01 c3  |....f..P.tP.T...|
00000460  81 fb 00 12 72 a4 66 ff  06 48 0e 66 83 16 4c 0e  |....r.f..H.f..L.|
00000470  00 eb 8e 66 ff 74 04 66  ff 34 06 53 6a 01 6a 10  |...f.t.f.4.Sj.j.|
00000480  89 e6 b8 00 42 cd 13 83  c4 10 72 1a c3 be 00 12  |....B.....r.....|
00000490  c7 04 1e 00 b8 00 48 cd  13 72 0b c3 be 28 07 eb  |......H..r...(..|
000004a0  16 be 3e 07 eb 11 be 56  07 eb 0c be 74 07 eb 07  |..>....V....t...|
000004b0  bb 07 00 b4 0e cd 10 ac  84 c0 75 f4 eb fe 42 6f  |..........u...Bo|
000004c0  6f 74 20 6c 6f 61 64 65  72 20 74 6f 6f 20 6c 61  |ot loader too la|
000004d0  72 67 65 00 49 6e 76 61  6c 69 64 20 70 61 72 74  |rge.Invalid part|
000004e0  69 74 69 6f 6e 20 74 61  62 6c 65 00 49 2f 4f 20  |ition table.I/O |
000004f0  65 72 72 6f 72 20 6c 6f  61 64 69 6e 67 20 62 6f  |error loading bo|
00000500  6f 74 20 6c 6f 61 64 65  72 00 4d 69 73 73 69 6e  |ot loader.Missin|
00000510  67 20 62 6f 6f 74 20 6c  6f 61 64 65 72 00 01 00  |g boot loader...|
00000520  00 00 00 00 00 00 9d 6b  bd 83 41 7f dc 11 be 0b  |.......k..A.....|
00000530  00 15 60 b8 4f 0f 90 90  90 90 90 90 90 90 90 90  |..`.O...........|
00000540  90 90 90 90 90 90 90 90  90 90 90 90 90 90 00 00  |................|
00000550  00 00 00 00 00 00 02 00  ee ff ff ff 01 00 00 00  |................|
00000560  ff ff ff 18 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000570  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000590  00 00 00 00 55 aa 45 46  49 20 50 41 52 54 00 00  |....U.EFI PART..|
000005a0  01 00 5c 00 00 00 cc 2a  49 2c 00 00 00 00 01 00  |..\....*I,......|
000005b0  00 00 00 00 00 00 ff ff  ff 18 00 00 00 00 28 00  |..............(.|
000005c0  00 00 00 00 00 00 d7 ff  ff 18 00 00 00 00 4b 4d  |..............KM|
000005d0  e4 a9 93 f7 e6 11 af 8c  00 50 56 84 a7 cc 02 00  |.........PV.....|
000005e0  00 00 00 00 00 00 80 00  00 00 80 00 00 00 f1 89  |................|
000005f0  5b 25 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |[%..............|
00000600  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000790  00 00 00 00 00 00 9d 6b  bd 83 41 7f dc 11 be 0b  |.......k..A.....|
000007a0  00 15 60 b8 4f 0f ab e2  e7 a9 93 f7 e6 11 af 8c  |..`.O...........|
000007b0  00 50 56 84 a7 cc 28 00  00 00 00 00 00 00 27 04  |.PV...(.......'.|
vmdk-header-hex (7,669 bytes)   
arogge

arogge

2020-08-21 12:22

manager   ~0004032

I'm afraid there is nothing we can do here.
When restoring with the VMWare Plugin the VMDK gets written by the VMWare SDK. Bareos does not have any knowledge of the mostly proprietary format.
If the VMware SDK wrote the VMDK, VMware should be able to read it, otherwise the VMware SDK should have noticed an error.

If you have any further insight on what might have gone wrong, feel free to share it.
sedlmeier

sedlmeier

2022-08-08 16:07

reporter   ~0004716

Restoring VMware disk images is a bit tricky and we had the same problem. I've tested it with Bareos 21 and the documentation doesn't seem to be very accurate on this.
What me surprises is that for a restore to a file, it also needs a valid connection to the vCenter and this might be a bug. This makes a restore in a worst case scenario very difficult because you need a working vCenter installation for a restore.

For us, the restore to a file worked when we gave the same plugin options as for the backup and added localvmdk=yes
The RAW file that Bareos backs up seems not compatible with VMware. You have to restore it through the plugin.

Here is an example that worked for us:

restore where=/backup/restores client=bareos-sd01 fileset=RESTORE-TEST.fileset replace=always pluginoptions="python:module_path=/usr/lib/bareos/plugins:module_name=bareos-fd-vmware:localvmdk=yes:config_file=/etc/bareos/vmware-plugin.ini:dc=DC1:folder=/:vmname=RESTORE-TEST" select current done all
stephand

stephand

2022-08-09 12:23

developer   ~0004718

There's a howto at http://www.voleg.info/bareos-restore-vmware.html
It also includes using vmkfstools on a ESXi server to convert the VMDK file to a format that can be used to add it to a VM.
At least for Linux VMs this procedure is known to work successfully.
We will probably add this information to the Bareos documentation in the future.
vrurg

vrurg

2022-08-09 16:29

reporter   ~0004719

It's a bit too late for me as we don't use bareos for vmware anymore. But with regard to the mentioned ways of restoring and fixing problems – none was useful back then. I had to use other means to recover at least some of the data I had on that VM. Having that experience in mind, I will never use bareos that way anymore.

What is fundamentally wrong anyway is that in order to restore a VM one needs a connection to the original ESXi server. The point of backups is to be able to recover from disasters, and a disaster may result in the original server to be gone for good. Anybody ever considered a scenario of the kind?
bruno-at-bareos

bruno-at-bareos

2022-08-11 10:27

manager   ~0004723

@sedlmeier thanks for the PR proposal code fix in
https://github.com/bareos/bareos/pull/1219
bruno-at-bareos

bruno-at-bareos

2022-08-24 14:57

manager   ~0004732

PR1219 merged

Issue History

Date Modified Username Field Change
2020-08-08 22:29 vrurg New Issue
2020-08-08 22:29 vrurg File Added: vmdk-header-hex
2020-08-21 12:22 arogge Status new => feedback
2020-08-21 12:22 arogge Note Added: 0004032
2022-08-08 16:07 sedlmeier Note Added: 0004716
2022-08-09 12:23 stephand Note Added: 0004718
2022-08-09 16:29 vrurg Note Added: 0004719
2022-08-09 16:29 vrurg Status feedback => new
2022-08-11 10:27 bruno-at-bareos Note Added: 0004723
2022-08-24 14:57 bruno-at-bareos Assigned To => bruno-at-bareos
2022-08-24 14:57 bruno-at-bareos Status new => closed
2022-08-24 14:57 bruno-at-bareos Resolution open => fixed
2022-08-24 14:57 bruno-at-bareos Note Added: 0004732