View Issue Details

IDProjectCategoryView StatusLast Update
0000670bareos-core[All Projects] vmware pluginpublic2019-05-12 09:02
ReporterzachaAssigned Tostephand 
PriorityhighSeveritycrashReproducibilityalways
Status assignedResolutionopen 
PlatformLinuxOSDebianOS Version8
Product Version15.4.0 
Target VersionFixed in Version 
Summary0000670: backup of large vmdks corrupt
Descriptionbareos-vmware-plugin 16.2.2
bareos-vmware-vix-disklib5 5.5.4-2454786
bareos-vadp-dumper 16.2.2
esxi 5.5

backup of large vmdk files (>2TB nominal capacity) is corrupt.
Steps To Reproducecreate a virtual disk of size > 2TB, can be thin provisioned.
write any data on the disk

create a second disk e.g. of size 1TB, thin provisioned too.
copy the exact same data to the smaller disk. Does not need to be much data, just a gig or so.

take a full backup of the virtual machine.

try to restore the machine

1. the size of the backup will be smaller than the actual disk
(we realized backup up a 4TB disk which is currently filled with 1,8TB data resulted in a ~240GB backup size which if of course not possible - no compression used in the backup job, we are compressing transparently in the fc target)
2. when trying to restore it will result in something like

25-Jun 16:13 valerian-fd JobId 200273: Fatal error: python-fd: check_dumper(): bareos_vadp_dumper returncode: 1 error output:
Warning: VixDiskLib: Invalid configuration file parameter. Failed to read configuration file.
Failed to create Logical Disk for /tmp/bareos-restores/[ha-10k] augustus-old/augustus-old_2.vmdk, One of the parameters supplied is invalid [16000]

25-Jun 16:13 valerian-sd JobId 200273: Error: bsock_tcp.c:405 Write error sending 65536 bytes to client:127.0.0.1:9103: ERR=Connection reset by peer 25-Jun 16:13 valerian-fd JobId 200273: Error: restore.c:1239 Write error on /tmp/bareos-restores/VMS/onesty-tech-cb/support/augustus-old/[ha-10k] augustus-old/augustus-old_2.vmdk: Broken pipe 25-Jun 16:13 valerian-sd JobId 200273: Fatal error: read.c:154 Error sending to File daemon. ERR=Connection reset by peer 25-Jun 16:13 valerian-fd JobId 200273: Fatal error: python-fd: check_dumper(): bareos_vadp_dumper returncode: 1 error output:
Warning: VixDiskLib: Invalid configuration file parameter. Failed to read configuration file.
Failed to create Logical Disk for /tmp/bareos-restores/[ha-10k] augustus-old/augustus-old_2.vmdk, One of the parameters supplied is invalid [16000]


I assume that this might be related in change of metadata possibly with disks > 2TB?

if you like I can provide two example disks filled with exactly the same data, one will backup correctly, one not.

what is really ugly is that the backup completes without any error (backup ok). we just noticed this occasionaly as we wondered the backup was much smaller as expected.
TagsNo tags attached.
bareos-master: impact
bareos-master: action
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action

Activities

Carl Lau

Carl Lau

2016-07-30 02:13

reporter   ~0002327

Last edited: 2016-07-30 02:14

View 2 revisions

maybe the problem is relative to this issue that have resolved by VDDK 6.0.2 release.
Pleae refer to this link: https://www.vmware.com/support/developer/vddk/vddk-602-releasenotes.html

"VDDK cannot HotAdd a > 2TB disk on VVol datastores.
When trying to open a > 2TB virtual disk for writing on VVol datastores, the following error message appeared: “Failed to hot-add SCSI targets: Vmomi::MethodFault::Exception: vim.fault.GenericVmConfigFault.” The fix is actually not in VDDK but in ESXi hosts, which should be upgraded to vSphere 6.0 U2."

zacha

zacha

2016-11-28 09:04

reporter   ~0002453

I cannot see any connection between the both issues at least it is not obvious to me. What did you lead to the assumption those bose might be related? Just that both stand in connection with > 2TB vmdk files? We don't even use virtual volumes on vsphere. Just plain old data stores.

The > 2TB virtual disk works correctly just that the backed up files are corrupt.

Anyone able to back up vmdk files > 2TB CONFIGURED size? Again, the disk does not have to have 2TB to test this, just have > 2TB configured. So can be easily tested with ANY datastore.
stephand

stephand

2019-01-16 18:12

developer   ~0003201

This bug entry is very old, and nobody else added any comment.

To be sure that this is a bug in the Bareos code, any information would be appreciated to know if this is problem still exists with newer vSphere versions (6.0/6.5) and the current Bareos version 17.2 where the VMWare Plugin is using VDDK 6.5.
zacha

zacha

2019-01-16 22:03

reporter   ~0003202

I never resolved this. It was easy to reproduce as shown above. But I don't have no possibility anymore to reproduce because I am not working in this environment anymore. So I am not able to contribute.
cmlara

cmlara

2019-05-12 09:02

reporter   ~0003364

Tested on
Vpshere 6.7
ESXi 6.7u2 -- Local storage only (single host)

Created a new vm with a single disk 3TB in size.
Install OS
Reset CBT (needed because some of my testing messed with the CBT tracking making it think the entire disk needed to be backed up, this shouldn't normally occur to others and activating CBT per manual should be fine)
Backup as per instructions in manual (backup is about 1gb so maybe this isn't enough to trigger but I think were good based on reading the docs)
(note: I ended up accidentally performing the backup while the machine was powered off but since the backup works on a snapshot this shouldn't matter)

Restored as per instructions in manual
Boot VM and all looks good

I'll be putting this into my environment soon enough to backup some larger VM's though I'm not sure yet if I am going to put it on the 2TB vm's or not (I'm starting to become inclined to do so as this may make life much easier) this will however take a bit longer as I'm still moving datasets around and the 2TB+ machines haven been provisioned yet.

Per: https://kb.vmware.com/s/article/2058287 HotAdd/Extend was not permitted past 2TB for ESXi below 6.5.
In addition per: https://code.vmware.com/doc/vddkDataStruct.5.5.html (linked from the bareos manual) the HotAdd limit for VMFS3 was also determined by block size (8MB block was 2TB, 4mb was 1TB, etc) this went away in VMFS5

Developers probably already know this but for a bit of history to make it easier to find:
 vSphere Storage APIs – Data Protection (formerly known as VADP) as noted on the vddkDataStruct page will choose 1 of 3 methods for how to download the virtual disk image.

When the bareos-fd is running inside a GuestVM that has direct access to the same storage the preferred method is via HotAdd to the VM (the VMWARE API actually does a guest config change to the system holding bareos-fd to map the existing VMDK to it as an additional attached disk) which for files larger than 2TB will require ESXI 6.5 or above.

If anyone can not upgrade to 6.5 and still hits this issue I believe they should be able to use the bareos-vmware plugin transport parameter and switch to to one of the other transports to avoid this.

If I put this on the larger datasets and see it occur I will post so, but at the moment I'm feeling pretty confident based on VMWare's documentation that I shouldn't hit this as I'm running 6.7

Issue History

Date Modified Username Field Change
2016-06-25 16:30 zacha New Issue
2016-07-30 02:13 Carl Lau Note Added: 0002327
2016-07-30 02:14 Carl Lau Note Edited: 0002327 View Revisions
2016-11-28 09:04 zacha Note Added: 0002453
2019-01-16 18:12 stephand Assigned To => stephand
2019-01-16 18:12 stephand Status new => feedback
2019-01-16 18:12 stephand Note Added: 0003201
2019-01-16 22:03 zacha Note Added: 0003202
2019-01-16 22:03 zacha Status feedback => assigned
2019-05-12 09:02 cmlara Note Added: 0003364