0000950: MariaDB backup with percona plugin - Bareos Bug Tracker

ID	Project	Category	View Status	Date Submitted	Last Update

0000950	bareos-core	file daemon	public	2018-05-22 10:42	2023-03-23 16:41

Reporter	beckzg	Assigned To	bruno-at-bareos
Priority	high	Severity	major	Reproducibility	always
Status	closed	Resolution	won't fix
Platform	Linux	OS	Debian	OS Version	9
Product Version	17.2.4

Summary	0000950: MariaDB backup with percona plugin
Description	Hi, I've a strange problem and get find out yet whats wrong, I use the percona plugin to backup up MariaDB 10.2.4 database. The fileset configuration looks like this: FileSet { Name = "MariaDB" Description = "Backup MariaDB" Include { Options { Signature = MD5 # calculate md5 checksum per file } Plugin = "python:module_path=/usr/lib/bareos/plugins:module_name=bareos-fd-percona:dumpbinary=mariabackup:dumpoptions=--backup --compress --stream=xbstream --parallel=4 --compress-threads=4:mycnf=/etc/mysql/backup.cnf" } } ---------------------------------------------------------------------------------------------------- The first run (Full backup) went well: bareos-dir JobId 25: Bareos bareos-dir 17.2.4 (21Sep17): Build OS: x86_64-pc-linux-gnu debian Debian GNU/Linux 9.3 (stretch) JobId: 25 Job: sql01.local-MariaDB.2018-05-18_12.41.14_32 Backup Level: Full (upgraded from Incremental) Client: "sql01.local" 17.2.4 (21Sep17) x86_64-pc-linux-gnu,debian,Debian GNU/Linux 9.3 (stretch),Debian_9.0,x86_64 FileSet: "MariaDB" 2018-05-18 11:02:32 Pool: "Full" (From Job FullPool override) Catalog: "MyCatalog" (From Client resource) Storage: "File" (From Job resource) Scheduled time: 18-May-2018 12:41:13 Start time: 18-May-2018 12:41:16 End time: 18-May-2018 12:54:47 Elapsed time: 13 mins 31 secs Priority: 10 FD Files Written: 2 SD Files Written: 2 FD Bytes Written: 50,554,175,708 (50.55 GB) SD Bytes Written: 50,554,176,649 (50.55 GB) Rate: 62335.6 KB/s Software Compression: None VSS: no Encryption: no Accurate: no Volume name(s): Full-0001\|Full-0003\|Full-0004\|Full-0005\|Full-0006\|Full-0007\|Full-0008\|Full-0009\|Full-0010\|Full-0011\|Full-0012\|Full-0013\|Full-0014\|Full-0015\|Full-0016\|Full-0017\|Full-0018\|Full-0019\|Full-0020\|Full-0021\|Full-0022\|Full-0023\|Full-0024\|Full-0025\|Full-0026\|Full-0027\|Full-0028\|Full-0029\|Full-0030\|Full-0031\|Full-0032\|Full-0033\|Full-0034\|Full-0035\|Full-0036\|Full-0037\|Full-0038\|Full-0039\|Full-0040\|Full-0041\|Full-0042\|Full-0043\|Full-0044\|Full-0045\|Full-0046\|Full-0047\|Full-0048\|Full-0049 Volume Session Id: 1 Volume Session Time: 1526639588 Last Volume Bytes: 125,991,399 (125.9 MB) Non-fatal FD errors: 0 SD Errors: 0 FD termination status: OK SD termination status: OK Termination: Backup OK ---------------------------------------------------------------------------------------------------- But when I try to run again (Incremental) I get the following error: 8-May 13:11 bareos-dir JobId 30: Start Backup JobId 30, Job=sql01.local-MariaDB.2018-05-18_13.11.14_59 18-May 13:11 bareos-dir JobId 30: Using Device "FileStorage" to write. 18-May 13:11 sql01.local JobId 30: Fatal error: python-fd: Traceback (most recent call last): File "/usr/lib/bareos/plugins/BareosFdWrapper.py", line 58, in restore_object_data return bareos_fd_plugin_object.restore_object_data(context, ROP) File "/usr/lib/bareos/plugins/BareosFdPercona.py", line 423, in restore_object_data self.rop_data[ROP.jobid] = json.loads(str(self.row_rop_raw)) File "/usr/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode obj, end = self.scan_once(s, idx) File "/usr/lib/python2.7/json/decoder.py", line 36, in errmsg lineno, colno = linecol(doc, pos) TypeError: 'NoneType' object is not callable 18-May 13:11 sql01.local JobId 30: Fatal error: Failed to authenticate Storage daemon. 18-May 13:11 bareos-dir JobId 30: Fatal error: Bad response to Storage command: wanted 2000 OK storage , got 2902 Bad storage 18-May 13:11 bareos-dir JobId 30: Error: Bareos bareos-dir 17.2.4 (21Sep17): Build OS: x86_64-pc-linux-gnu debian Debian GNU/Linux 9.3 (stretch) JobId: 30 Job: sql01.local-MariaDB.2018-05-18_13.11.14_59 Backup Level: Incremental, since=2018-05-18 12:41:16 Client: "sql01.local" 17.2.4 (21Sep17) x86_64-pc-linux-gnu,debian,Debian GNU/Linux 9.3 (stretch),Debian_9.0,x86_64 FileSet: "MariaDB" 2018-05-18 11:02:32 Pool: "Incremental" (From Job IncPool override) Catalog: "MyCatalog" (From Client resource) Storage: "File" (From Job resource) Scheduled time: 18-May-2018 13:11:13 Start time: 18-May-2018 13:11:16 End time: 18-May-2018 13:11:16 Elapsed time: 0 secs Priority: 10 FD Files Written: 0 SD Files Written: 0 FD Bytes Written: 0 (0 B) SD Bytes Written: 0 (0 B) Rate: 0.0 KB/s Software Compression: None VSS: no Encryption: no Accurate: no Volume name(s): Volume Session Id: 5 Volume Session Time: 1526640985 Last Volume Bytes: 0 (0 B) Non-fatal FD errors: 2 SD Errors: 0 FD termination status: Fatal Error SD termination status: Waiting on FD Termination: * Backup Error * Any ideas what should be my problem? Kind Regards, Zoltan
Steps To Reproduce	First run a Full backup and after an Incremental backup.
Tags	No tags attached.

beckzg 2018-05-22 13:44 reporter	sql01.local.trace (15,581 bytes)

beckzg 2018-05-22 16:26 reporter ~0003011	The same error occurs with percona-xtrabackup and percona-xtrabackup-24 packages: root@sql01:/etc/mysql# xtrabackup --version xtrabackup version 2.3.10 based on MySQL server 5.6.24 Linux (x86_64) (revision id: bd0d4403f36) root@sql01:/etc/mysql# xtrabackup --version xtrabackup version 2.4.11 based on MySQL server 5.7.19 Linux (x86_64) (revision id: b4e0db5) 22-May 16:17 bareos-dir JobId 57: Start Backup JobId 57, Job=sql01.local-MariaDB.2018-05-22_16.17.07_19 22-May 16:17 bareos-dir JobId 57: Using Device "FileStorage" to write. 22-May 16:17 sql01.local JobId 57: Fatal error: python-fd: Traceback (most recent call last): File "/usr/lib/bareos/plugins/BareosFdWrapper.py", line 58, in restore_object_data return bareos_fd_plugin_object.restore_object_data(context, ROP) File "/usr/lib/bareos/plugins/BareosFdPercona.py", line 423, in restore_object_data self.rop_data[ROP.jobid] = json.loads(str(self.row_rop_raw)) File "/usr/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode obj, end = self.scan_once(s, idx) File "/usr/lib/python2.7/json/decoder.py", line 36, in errmsg lineno, colno = linecol(doc, pos) TypeError: 'NoneType' object is not callable 22-May 16:17 sql01.local JobId 57: Fatal error: Failed to authenticate Storage daemon. 22-May 16:17 bareos-dir JobId 57: Fatal error: Bad response to Storage command: wanted 2000 OK storage , got 2902 Bad storage

beckzg 2018-05-22 16:29 reporter ~0003012	root@sql01:/etc/mysql# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.4 (stretch) Release: 9.4 Codename: stretch root@sql01:/etc/mysql# mysql --version mysql Ver 15.1 Distrib 10.2.15-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

beckzg 2018-05-22 16:32 reporter ~0003013	root@sql01:/etc/mysql# mariabackup --version mariabackup based on MariaDB server 10.2.15-MariaDB debian-linux-gnu (x86_64)

beckzg 2018-06-26 10:07 reporter ~0003052	I installed a completely new system with CentOS (CentOS Linux release 7.5.1804 (Core)) and Percona Server (Percona-Server-server-57-5.7.22-22.1.el7.x86_64). The symptom is absolutely the same, the full backup runs but the incremental backup fails: 26-Jun 10:02 bareos-dir JobId 286: Using Device "FileStorage" to write. 26-Jun 10:02 sql03.local JobId 286: Fatal error: python-fd: Traceback (most recent call last): File "/usr/lib64/bareos/plugins/BareosFdWrapper.py", line 58, in restore_object_data return bareos_fd_plugin_object.restore_object_data(context, ROP) File "/usr/lib64/bareos/plugins/BareosFdPercona.py", line 423, in restore_object_data self.rop_data[ROP.jobid] = json.loads(str(self.row_rop_raw)) File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode obj, end = self.scan_once(s, idx) File "/usr/lib64/python2.7/json/decoder.py", line 38, in errmsg lineno, colno = linecol(doc, pos) TypeError: 'NoneType' object is not callable 26-Jun 10:02 sql03.local JobId 286: Fatal error: Failed to authenticate Storage daemon. 26-Jun 10:02 bareos-dir JobId 286: Fatal error: Bad response to Storage command: wanted 2000 OK storage , got 2902 Bad storage

beckzg 2018-07-05 23:32 reporter ~0003063	I found the problem, this error only occurs if sqlite3 is used for catalog, now I changed to mysql and the jobs are running. So it looks like it is a bug somewhere in the code related to sqlite3.

bruno-at-bareos 2023-03-23 16:41 manager ~0004948	sqlite3 database catalog is no more possible to used.

Date Modified	Username	Field	Change
2018-05-22 10:42	beckzg	New Issue
2018-05-22 13:44	beckzg	File Added: sql01.local.trace
2018-05-22 16:26	beckzg	Note Added: 0003011
2018-05-22 16:29	beckzg	Note Added: 0003012
2018-05-22 16:32	beckzg	Note Added: 0003013
2018-06-26 10:07	beckzg	Note Added: 0003052
2018-07-05 23:32	beckzg	Note Added: 0003063
2023-03-23 16:41	bruno-at-bareos	Assigned To	=> bruno-at-bareos
2023-03-23 16:41	bruno-at-bareos	Status	new => closed
2023-03-23 16:41	bruno-at-bareos	Resolution	open => won't fix
2023-03-23 16:41	bruno-at-bareos	Note Added: 0004948