0001380: Backup failures on concurent backups using ovirt-plugin - Bareos Bug Tracker

ID	Project	Category	View Status	Date Submitted	Last Update

0001380	bareos-core	file daemon	public	2021-08-13 21:13	2023-08-31 09:48

Reporter	matus22	Assigned To	bruno-at-bareos
Priority	normal	Severity	major	Reproducibility	always
Status	closed	Resolution	not fixable
Platform	Linux	OS	CentOS	OS Version	8
Product Version	20.0.2

Summary	0001380: Backup failures on concurent backups using ovirt-plugin
Description	We are running multiple backups from ovirt to utilize bandwidth and speedup process of backup. When running concurrent backups started on same time, only single backup will succeed. Second and other backups will fail right after successful image download. Looking at the logs, it looks like bareos-fd is unable to send event to ovirt because of duplicate event_id. I think, event_id variable generation mechanism is insufficient. Running multiple backups at the same time. time.time() will generate same number which will ovirt-engine deny https://github.com/bareos/bareos/blob/de698088e4de495eb8bef0e1cb267e9526be31db/core/src/plugins/filed/python/ovirt/BareosFdPluginOvirt.py#L830 I was able to partially mitigate this issue subtracting small random number from event_id variable. PS: i am sorry for DUP of this issue ( https://bugs.bareos.org/view.php?id=1302 ), but it was in wrong category and missing many details
Steps To Reproduce	- configure host for backups of multiple VMs utilizing ovirt-plugin - schedule backup of multiple VMs to same Schedule - concurent backups needs to be started in the same second.
Additional Information	Bareos using python3, ovirtsdk-4. Bareos Exception: Traceback (most recent call last): File "/usr/lib64/bareos/plugins/BareosFdWrapper.py", line 38, in handle_plugin_event return bareos_fd_plugin_object.handle_plugin_event(event) File "/usr/lib64/bareos/plugins/BareosFdPluginOvirt.py", line 561, in handle_plugin_event return self.start_backup_job() File "/usr/lib64/bareos/plugins/BareosFdPluginOvirt.py", line 160, in start_backup_job return self.ovirt.prepare_vm_backup() File "/usr/lib64/bareos/plugins/BareosFdPluginOvirt.py", line 893, in prepare_vm_backup self.create_vm_snapshot() File "/usr/lib64/bareos/plugins/BareosFdPluginOvirt.py", line 943, in create_vm_snapshot "starting." % (self.vm.name, snap_description) File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/services.py", line 9014, in add return self._internal_add(event, headers, query, wait) File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 232, in _internal_add return future.wait() if wait else future File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 229, in callback self._check_fault(response) File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/usr/local/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Internal Engine Error]". HTTP response code is 400. Ovirt-engine error: Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "audit_log_origin_custom_event_id_idx" Detail: Key (origin, custom_event_id)=(Bareos oVirt plugin, 1628599716) already exists.
Tags	No tags attached.

bruno-at-bareos 2023-07-19 17:04 manager ~0005231	Hello is this still the case with the Bareos 22.1.0 version ?

bruno-at-bareos 2023-08-31 09:48 manager ~0005356	After a round table about the pro/con with the ovirt plugin, I'll give you our conclusions: RedHat already announced last year the EOL of their product RedHat Virtualization (mid 2024). They also drop a lot of contribution in ovirt. This puts the project in a delicate situation (also moving from CentOS to something else). The crash you are experiencing is probably related to Python module single phase initialization. We finally tracked this down to pycurl being the problem, not necessarily the oVirt XML module. It's sufficient to simply add import pycurl to any Python plugin without even using it to reproduce the segfault. Multi-phase initialization is defined in PEP 489, see also this article https://vstinner.github.io/isolate-subinterpreters.html

Date Modified	Username	Field	Change
2021-08-13 21:13	matus22	New Issue
2023-07-19 17:04	bruno-at-bareos	Assigned To	=> bruno-at-bareos
2023-07-19 17:04	bruno-at-bareos	Status	new => feedback
2023-07-19 17:04	bruno-at-bareos	Note Added: 0005231
2023-08-31 09:48	bruno-at-bareos	Status	feedback => closed
2023-08-31 09:48	bruno-at-bareos	Resolution	open => not fixable
2023-08-31 09:48	bruno-at-bareos	Note Added: 0005356