0001266: A VMDK file restored using localvmdk=yes is not recognized as a virtual disk. - Bareos Bug Tracker

ID	Project	Category	View Status	Date Submitted	Last Update

0001266	bareos-core	vmware plugin	public	2020-08-08 22:29	2022-08-24 14:57

Reporter	vrurg	Assigned To	bruno-at-bareos
Priority	urgent	Severity	major	Reproducibility	always
Status	closed	Resolution	fixed
Platform	Linux	OS	CentOS	OS Version	7
Product Version	18.2.9

Summary	0001266: A VMDK file restored using localvmdk=yes is not recognized as a virtual disk.
Description	After loosing a bunch of servers in an accident I needed to restore a VM into a file as restoring into a vCenter is not an option. Unfortunately, the resulting VMDK is not recognized by none of the applications and utilities I tried: VMWare Fusion (including vmware-vdiskmanager), ESXi (including vmkfstool in the shell), VirtualBox, qemu-img. So far, after communicating to many people including VMWare suuport and community, the problem looks like the file I've got looks somewhat alike to VMDK monolithic format with the descriptor embedded into the header. Except that normal monolithic is supposed to start with 'KMDV' signature and have the embedded descriptor in a form of a normal text file. In fact, the VMDK I have at my hands starts with some kind of binary header, mostly zeroed out, and contains incomplete descriptor data in binary form where text configuration variable names are separated from their values with zeroes. In turn, the header is followed by the actual VMDK flat file starting somewhere at offset 918 (I guess, the actual offset may vary depending on the header content). I tried different method of recovering the VMDK including cutting of the 918 bytes of the header to get the flat version and restoring the descriptor the way a VMWare support KB suggest (used vmkfstools). Unfortunately, the result suffers from wrong disk geometry causing boot failure and worse – FreeBSD zfs not being able to recover the ZFS partition on the disk. So far, I barely can guess what is the source of the problem neither offer a way to reproduce it. We have tried restoring some test VMs the very same way and everything worked fine because the VMDKs created in these cases were all-fine monolithics with the expected KMDV signature and text descriptor embedded. The only difference of the problematic case from the test ones is that it had at least one snapshot at the time of backup. And, perhaps, bigger size though I don't think that 200GB should be of any problem here.
Additional Information	Attached is the hex dump of the restored VMDK file header.
Tags	No tags attached.

vrurg 2020-08-08 22:29 reporter	vmdk-header-hex (7,669 bytes) 00000000 12 20 12 12 01 00 00 00 00 00 00 00 32 00 00 00 \|. ..........2...\| 00000010 00 00 00 00 00 00 00 00 fc 65 00 00 ff 00 00 00 \|.........e......\| 00000020 3f 00 00 00 fc 65 00 00 ff 00 00 00 3f 00 00 00 \|?....e......?...\| 00000030 00 00 00 19 00 00 00 00 03 00 00 00 00 00 00 00 \|................\| 00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 \|................\| * 00000070 00 00 00 00 00 00 00 00 00 00 00 00 12 20 12 12 \|............. ..\| 00000080 12 20 12 12 0a 00 00 00 06 00 00 00 12 20 12 12 \|. ........... ..\| 00000090 64 65 6c 65 74 61 62 6c 65 00 74 72 75 65 00 00 \|deletable.true..\| 000000a0 12 20 12 12 0e 00 00 00 22 00 00 00 12 20 12 12 \|. ......".... ..\| 000000b0 6c 6f 6e 67 43 6f 6e 74 65 6e 74 49 44 00 30 38 \|longContentID.08\| 000000c0 32 38 62 63 38 64 34 66 64 39 30 39 30 61 31 61 \|28bc8d4fd9090a1a\| 000000d0 33 61 31 34 34 38 36 63 61 33 39 36 33 63 00 00 \|3a14486ca3963c..\| 000000e0 12 20 12 12 0d 00 00 00 0c 00 00 00 12 20 12 12 \|. ........... ..\| 000000f0 74 6f 6f 6c 73 56 65 72 73 69 6f 6e 00 32 31 34 \|toolsVersion.214\| 00000100 37 34 38 33 36 34 37 00 6e 12 20 12 12 0a 00 00 \|7483647.n. .....\| 00000110 00 06 00 00 00 12 20 12 12 64 65 6c 65 74 61 62 \|...... ..deletab\| 00000120 6c 65 00 74 72 75 65 00 00 12 20 12 12 0e 00 00 \|le.true... .....\| 00000130 00 22 00 00 00 12 20 12 12 6c 6f 6e 67 43 6f 6e \|.".... ..longCon\| 00000140 74 65 6e 74 49 44 00 30 38 32 38 62 63 38 64 34 \|tentID.0828bc8d4\| 00000150 66 64 39 30 39 30 61 31 61 33 61 31 34 34 38 36 \|fd9090a1a3a14486\| 00000160 63 61 33 39 36 33 63 00 00 12 20 12 12 0d 00 00 \|ca3963c... .....\| 00000170 00 0c 00 00 00 12 20 12 12 74 6f 6f 6c 73 56 65 \|...... ..toolsVe\| 00000180 72 73 69 6f 6e 00 32 31 34 37 34 38 33 36 34 37 \|rsion.2147483647\| 00000190 00 6e 12 20 12 12 0c 00 00 00 0a 00 00 00 12 20 \|.n. ........... \| 000001a0 12 12 61 64 61 70 74 65 72 54 79 70 65 00 6c 73 \|..adapterType.ls\| 000001b0 69 6c 6f 67 69 63 00 70 12 20 12 12 0a 00 00 00 \|ilogic.p. ......\| 000001c0 06 00 00 00 12 20 12 12 64 65 6c 65 74 61 62 6c \|..... ..deletabl\| 000001d0 65 00 74 72 75 65 00 00 12 20 12 12 13 00 00 00 \|e.true... ......\| 000001e0 07 00 00 00 12 20 12 12 67 65 6f 6d 65 74 72 79 \|..... ..geometry\| 000001f0 2e 63 79 6c 69 6e 64 65 72 73 00 32 36 31 30 38 \|.cylinders.26108\| 00000200 00 00 12 20 12 12 0f 00 00 00 05 00 00 00 12 20 \|... ........... \| 00000210 12 12 67 65 6f 6d 65 74 72 79 2e 68 65 61 64 73 \|..geometry.heads\| 00000220 00 32 35 35 00 00 12 20 12 12 11 00 00 00 04 00 \|.255... ........\| 00000230 00 00 12 20 12 12 67 65 6f 6d 65 74 72 79 2e 73 \|... ..geometry.s\| 00000240 65 63 74 6f 72 73 00 36 33 00 01 12 20 12 12 0e \|ectors.63... ...\| 00000250 00 00 00 22 00 00 00 12 20 12 12 6c 6f 6e 67 43 \|...".... ..longC\| 00000260 6f 6e 74 65 6e 74 49 44 00 30 38 32 38 62 63 38 \|ontentID.0828bc8\| 00000270 64 34 66 64 39 30 39 30 61 31 61 33 61 31 34 34 \|d4fd9090a1a3a144\| 00000280 38 36 63 61 33 39 36 33 63 00 00 12 20 12 12 10 \|86ca3963c... ...\| 00000290 00 00 00 03 00 00 00 12 20 12 12 74 68 69 6e 50 \|........ ..thinP\| 000002a0 72 6f 76 69 73 69 6f 6e 65 64 00 31 00 40 12 20 \|rovisioned.1.@. \| 000002b0 12 12 11 00 00 00 03 00 00 00 12 20 12 12 74 6f \|........... ..to\| 000002c0 6f 6c 73 49 6e 73 74 61 6c 6c 54 79 70 65 00 30 \|olsInstallType.0\| 000002d0 00 40 12 20 12 12 0d 00 00 00 0c 00 00 00 12 20 \|.@. ........... \| 000002e0 12 12 74 6f 6f 6c 73 56 65 72 73 69 6f 6e 00 32 \|..toolsVersion.2\| 000002f0 31 34 37 34 38 33 36 34 37 00 6e 12 20 12 12 05 \|147483647.n. ...\| 00000300 00 00 00 31 00 00 00 12 20 12 12 75 75 69 64 00 \|...1.... ..uuid.\| 00000310 36 30 20 30 30 20 43 32 20 39 32 20 35 37 20 34 \|60 00 C2 92 57 4\| 00000320 65 20 63 39 20 66 30 2d 39 33 20 37 35 20 34 39 \|e c9 f0-93 75 49\| 00000330 20 38 36 20 63 30 20 38 64 20 31 34 20 31 61 00 \| 86 c0 8d 14 1a.\| 00000340 6f 12 20 12 12 11 00 00 00 04 00 00 00 12 20 12 \|o. ........... .\| 00000350 12 76 69 72 74 75 61 6c 48 57 56 65 72 73 69 6f \|.virtualHWVersio\| 00000360 6e 00 31 31 00 01 12 20 12 12 00 00 00 00 00 00 \|n.11... ........\| 00000370 00 00 12 20 12 12 12 20 12 12 00 00 00 00 00 00 \|... ... ........\| 00000380 00 00 00 00 00 00 00 00 20 00 03 00 00 00 12 20 \|........ ...... \| 00000390 12 12 00 00 00 00 fc 31 c0 8e c0 8e d8 8e d0 bc \|.......1........\| 000003a0 00 0e be 1a 7c bf 1a 06 b9 e6 01 f3 a4 e9 00 8a \|....\|...........\| 000003b0 80 fa 80 72 0b 8a 36 75 04 80 c6 80 38 f2 72 02 \|...r..6u....8.r.\| 000003c0 b2 80 e8 c8 00 b6 01 bb 00 0e be 88 07 e8 a3 00 \|................\| 000003d0 66 81 3e 00 0e 45 46 49 20 75 0d 66 81 3e 04 0e \|f.>..EFI u.f.>..\| 000003e0 50 41 52 54 75 02 eb 19 80 fe 01 0f 85 b2 00 b6 \|PARTu...........\| 000003f0 00 be 10 12 bf 88 07 66 ff 0c b9 02 00 f3 a5 eb \|.......f........\| 00000400 c6 be 48 0e bb 00 10 e8 69 00 89 de bf 90 07 b1 \|..H.....i.......\| 00000410 10 f3 a6 75 3f 89 df 8d 75 20 bb c0 07 8e c3 31 \|...u?...u .....1\| 00000420 db 56 e8 4e 00 5e 66 8b 45 28 66 3b 04 75 0f 66 \|.V.N.^f.E(f;.u.f\| 00000430 8b 45 2c 66 3b 44 04 75 05 8e c3 e9 58 75 66 ff \|.E,f;D.u....Xuf.\| 00000440 04 66 83 54 04 00 8c c0 83 c0 20 3d 00 90 73 4c \|.f.T...... =..sL\| 00000450 8e c0 eb cd 66 ff 0e 50 0e 74 50 a1 54 0e 01 c3 \|....f..P.tP.T...\| 00000460 81 fb 00 12 72 a4 66 ff 06 48 0e 66 83 16 4c 0e \|....r.f..H.f..L.\| 00000470 00 eb 8e 66 ff 74 04 66 ff 34 06 53 6a 01 6a 10 \|...f.t.f.4.Sj.j.\| 00000480 89 e6 b8 00 42 cd 13 83 c4 10 72 1a c3 be 00 12 \|....B.....r.....\| 00000490 c7 04 1e 00 b8 00 48 cd 13 72 0b c3 be 28 07 eb \|......H..r...(..\| 000004a0 16 be 3e 07 eb 11 be 56 07 eb 0c be 74 07 eb 07 \|..>....V....t...\| 000004b0 bb 07 00 b4 0e cd 10 ac 84 c0 75 f4 eb fe 42 6f \|..........u...Bo\| 000004c0 6f 74 20 6c 6f 61 64 65 72 20 74 6f 6f 20 6c 61 \|ot loader too la\| 000004d0 72 67 65 00 49 6e 76 61 6c 69 64 20 70 61 72 74 \|rge.Invalid part\| 000004e0 69 74 69 6f 6e 20 74 61 62 6c 65 00 49 2f 4f 20 \|ition table.I/O \| 000004f0 65 72 72 6f 72 20 6c 6f 61 64 69 6e 67 20 62 6f \|error loading bo\| 00000500 6f 74 20 6c 6f 61 64 65 72 00 4d 69 73 73 69 6e \|ot loader.Missin\| 00000510 67 20 62 6f 6f 74 20 6c 6f 61 64 65 72 00 01 00 \|g boot loader...\| 00000520 00 00 00 00 00 00 9d 6b bd 83 41 7f dc 11 be 0b \|.......k..A.....\| 00000530 00 15 60 b8 4f 0f 90 90 90 90 90 90 90 90 90 90 \|..`.O...........\| 00000540 90 90 90 90 90 90 90 90 90 90 90 90 90 90 00 00 \|................\| 00000550 00 00 00 00 00 00 02 00 ee ff ff ff 01 00 00 00 \|................\| 00000560 ff ff ff 18 00 00 00 00 00 00 00 00 00 00 00 00 \|................\| 00000570 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 \|................\| * 00000590 00 00 00 00 55 aa 45 46 49 20 50 41 52 54 00 00 \|....U.EFI PART..\| 000005a0 01 00 5c 00 00 00 cc 2a 49 2c 00 00 00 00 01 00 \|..\....I,......\| 000005b0 00 00 00 00 00 00 ff ff ff 18 00 00 00 00 28 00 \|..............(.\| 000005c0 00 00 00 00 00 00 d7 ff ff 18 00 00 00 00 4b 4d \|..............KM\| 000005d0 e4 a9 93 f7 e6 11 af 8c 00 50 56 84 a7 cc 02 00 \|.........PV.....\| 000005e0 00 00 00 00 00 00 80 00 00 00 80 00 00 00 f1 89 \|................\| 000005f0 5b 25 00 00 00 00 00 00 00 00 00 00 00 00 00 00 \|[%..............\| 00000600 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 \|................\| 00000790 00 00 00 00 00 00 9d 6b bd 83 41 7f dc 11 be 0b \|.......k..A.....\| 000007a0 00 15 60 b8 4f 0f ab e2 e7 a9 93 f7 e6 11 af 8c \|..`.O...........\| 000007b0 00 50 56 84 a7 cc 28 00 00 00 00 00 00 00 27 04 \|.PV...(.......'.\| vmdk-header-hex (7,669 bytes)

arogge 2020-08-21 12:22 manager ~0004032	I'm afraid there is nothing we can do here. When restoring with the VMWare Plugin the VMDK gets written by the VMWare SDK. Bareos does not have any knowledge of the mostly proprietary format. If the VMware SDK wrote the VMDK, VMware should be able to read it, otherwise the VMware SDK should have noticed an error. If you have any further insight on what might have gone wrong, feel free to share it.

sedlmeier 2022-08-08 16:07 reporter ~0004716	Restoring VMware disk images is a bit tricky and we had the same problem. I've tested it with Bareos 21 and the documentation doesn't seem to be very accurate on this. What me surprises is that for a restore to a file, it also needs a valid connection to the vCenter and this might be a bug. This makes a restore in a worst case scenario very difficult because you need a working vCenter installation for a restore. For us, the restore to a file worked when we gave the same plugin options as for the backup and added localvmdk=yes The RAW file that Bareos backs up seems not compatible with VMware. You have to restore it through the plugin. Here is an example that worked for us: restore where=/backup/restores client=bareos-sd01 fileset=RESTORE-TEST.fileset replace=always pluginoptions="python:module_path=/usr/lib/bareos/plugins:module_name=bareos-fd-vmware:localvmdk=yes:config_file=/etc/bareos/vmware-plugin.ini:dc=DC1:folder=/:vmname=RESTORE-TEST" select current done all

stephand 2022-08-09 12:23 developer ~0004718	There's a howto at http://www.voleg.info/bareos-restore-vmware.html It also includes using vmkfstools on a ESXi server to convert the VMDK file to a format that can be used to add it to a VM. At least for Linux VMs this procedure is known to work successfully. We will probably add this information to the Bareos documentation in the future.

vrurg 2022-08-09 16:29 reporter ~0004719	It's a bit too late for me as we don't use bareos for vmware anymore. But with regard to the mentioned ways of restoring and fixing problems – none was useful back then. I had to use other means to recover at least some of the data I had on that VM. Having that experience in mind, I will never use bareos that way anymore. What is fundamentally wrong anyway is that in order to restore a VM one needs a connection to the original ESXi server. The point of backups is to be able to recover from disasters, and a disaster may result in the original server to be gone for good. Anybody ever considered a scenario of the kind?

bruno-at-bareos 2022-08-11 10:27 manager ~0004723	@sedlmeier thanks for the PR proposal code fix in https://github.com/bareos/bareos/pull/1219

bruno-at-bareos 2022-08-24 14:57 manager ~0004732	PR1219 merged

Date Modified	Username	Field	Change
2020-08-08 22:29	vrurg	New Issue
2020-08-08 22:29	vrurg	File Added: vmdk-header-hex
2020-08-21 12:22	arogge	Status	new => feedback
2020-08-21 12:22	arogge	Note Added: 0004032
2022-08-08 16:07	sedlmeier	Note Added: 0004716
2022-08-09 12:23	stephand	Note Added: 0004718
2022-08-09 16:29	vrurg	Note Added: 0004719
2022-08-09 16:29	vrurg	Status	feedback => new
2022-08-11 10:27	bruno-at-bareos	Note Added: 0004723
2022-08-24 14:57	bruno-at-bareos	Assigned To	=> bruno-at-bareos
2022-08-24 14:57	bruno-at-bareos	Status	new => closed
2022-08-24 14:57	bruno-at-bareos	Resolution	open => fixed
2022-08-24 14:57	bruno-at-bareos	Note Added: 0004732