View Issue Details

IDProjectCategoryView StatusLast Update
0001380bareos-core[All Projects] file daemonpublic2021-08-13 21:13
Reportermatus22Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
PlatformLinuxOSCentOSOS Version8
Product Version20.0.2 
Fixed in Version 
Summary0001380: Backup failures on concurent backups using ovirt-plugin
DescriptionWe are running multiple backups from ovirt to utilize bandwidth and speedup process of backup.

When running concurrent backups started on same time, only single backup will succeed.
Second and other backups will fail right after successful image download.

Looking at the logs, it looks like bareos-fd is unable to send event to ovirt because of duplicate event_id.

I think, event_id variable generation mechanism is insufficient.
Running multiple backups at the same time. time.time() will generate same number which will ovirt-engine deny
https://github.com/bareos/bareos/blob/de698088e4de495eb8bef0e1cb267e9526be31db/core/src/plugins/filed/python/ovirt/BareosFdPluginOvirt.py#L830

I was able to partially mitigate this issue subtracting small random number from event_id variable.

PS: i am sorry for DUP of this issue ( https://bugs.bareos.org/view.php?id=1302 ), but it was in wrong category and missing many details
Steps To Reproduce- configure host for backups of multiple VMs utilizing ovirt-plugin
- schedule backup of multiple VMs to same Schedule
- concurent backups needs to be started in the same second.
Additional InformationBareos using python3, ovirtsdk-4.

Bareos Exception:
Traceback (most recent call last):
  File "/usr/lib64/bareos/plugins/BareosFdWrapper.py", line 38, in handle_plugin_event
    return bareos_fd_plugin_object.handle_plugin_event(event)
  File "/usr/lib64/bareos/plugins/BareosFdPluginOvirt.py", line 561, in handle_plugin_event
    return self.start_backup_job()
  File "/usr/lib64/bareos/plugins/BareosFdPluginOvirt.py", line 160, in start_backup_job
    return self.ovirt.prepare_vm_backup()
  File "/usr/lib64/bareos/plugins/BareosFdPluginOvirt.py", line 893, in prepare_vm_backup
    self.create_vm_snapshot()
  File "/usr/lib64/bareos/plugins/BareosFdPluginOvirt.py", line 943, in create_vm_snapshot
    "starting." % (self.vm.name, snap_description)
  File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/services.py", line 9014, in add
    return self._internal_add(event, headers, query, wait)
  File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 232, in _internal_add
    return future.wait() if wait else future
  File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 55, in wait
    return self._code(response)
  File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 229, in callback
    self._check_fault(response)
  File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 132, in _check_fault
    self._raise_error(response, body)
  File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 118, in _raise_error
    raise error
ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Internal Engine Error]". HTTP response code is 400.


Ovirt-engine error:
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "audit_log_origin_custom_event_id_idx"
  Detail: Key (origin, custom_event_id)=(Bareos oVirt plugin, 1628599716) already exists.
TagsNo tags attached.
bareos-master: impact
bareos-master: action
bareos-19.2: impact
bareos-19.2: action
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2021-08-13 21:13 matus22 New Issue