View Issue Details

IDProjectCategoryView StatusLast Update
0000473bareos-core[All Projects] file daemonpublic2015-10-02 20:03
ReporterSergeyMorozovAssigned To 
PrioritynormalSeveritycrashReproducibilitysometimes
Status closedResolutionfixed 
PlatformLinuxOSGentooOS Version
Product Version14.2.5 
Target Version14.2.6Fixed in Version 
Summary0000473: Incremental backup failure
Descriptionbareos-fd crashing on linux.
Steps To ReproduceI don't know how to reproduce it, but sometimes bareos-fd crashing during an incremental backup. Version 14.2.4 is also have this problem.
Additional InformationJob and fileset configuration:

Job {
  Name = "ownCloud"
  JobDefs = "DefaultJob"
  Level = Incremental
  Pool = Incremental
  FileSet = "ownCloud"
  Schedule = "WeeklyNightCycle"
  Accurate = yes
}

FileSet {
  Name = "ownCloud"
  Include {
    Plugin = "bpipe:file=/MYSQL/ownCloud.sql:reader=mysqldump --user=owncloud --password=8hfSQJ2jetzLRS5Q owncloud:writer=my
sql --user=owncloud --password=8hfSQJ2jetzLRS5Q owncloud"
    Options {
      Signature = MD5 # calculate md5 checksum per file
    }
    File = /var/www/localhost/htdocs/owncloud
    File = /var/local/owncloud
  }
}

Traceback:

Created /var/lib/bareos/bareos-fd.core.10415 for doing postmortem debugging
[New LWP 10419]
[New LWP 10420]
[New LWP 17836]
[New LWP 17849]
[New LWP 10415]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/bareos-fd'.
#0 0x0000003152e0f71d in nanosleep () from /lib64/libpthread.so.0
$1 = '\000' <repeats 127 times>
$2 = 0x62e058 "bareos-fd"
$3 = 0x0
$4 = 0x0
$5 = 0x7f20a2d77eb4 "14.2.5 (27 May 2015)"
$6 = 0x7f20a2d77e98 "x86_64-pc-linux-gnu"
$7 = 0x7f20a2d77e91 "gentoo"
$8 = 0x7f20a2d77afb ""
$9 = "server", '\000' <repeats 249 times>
$10 = 0x7f20a2d77eac "gentoo "
Environment variable "TestName" not defined.
#0 0x0000003152e0f71d in nanosleep () from /lib64/libpthread.so.0
0000001 0x00007f20a2d4bbb4 in bmicrosleep (sec=sec@entry=30, usec=usec@entry=0) at bsys.c:100
0000002 0x00007f20a2d5b971 in check_deadlock () at lockmgr.c:567
0000003 0x0000003152e0824d in start_thread () from /lib64/libpthread.so.0
0000004 0x00000031526e955d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f20a2d1a740 (LWP 10415)):
#0 0x00000031526e068d in poll () from /lib64/libc.so.6
0000001 0x00007f20a2d40f3b in bnet_thread_server_tcp (addr_list=<optimized out>, max_clients=<optimized out>, sockfds=<optimized out>, client_wq=client_wq@entry=0x62c800 <dir_workq>, nokeepalive=<optimized out>, handle_client_request=<optimized out>) at bnet_server_tcp.c:298
0000002 0x0000000000408c9a in main (argc=<optimized out>, argv=<optimized out>) at filed.c:276

Thread 4 (Thread 0x7f2093fff700 (LWP 17849)):
#0 0x00000031526e068d in poll () from /lib64/libc.so.6
0000001 0x00007f20a2d63dd8 in wait_for_readable_fd (fd=<optimized out>, msec=5000, ignore_interupts=ignore_interupts@entry=false) at poll.c:62
0000002 0x00007f20a2d499e1 in BSOCK_TCP::wait_data_intr (this=0x7f208c001078, sec=<optimized out>, usec=<optimized out>) at bsock_tcp.c:817
0000003 0x000000000041a784 in sd_heartbeat_thread (arg=arg@entry=0x7f2098023a68) at heartbeat.c:73
0000004 0x00007f20a2d5ba00 in lmgr_thread_launcher (x=0x7f2098019008) at lockmgr.c:932
0000005 0x0000003152e0824d in start_thread () from /lib64/libpthread.so.0
0000006 0x00000031526e955d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f20a0b92700 (LWP 17836)):
#0 0x0000003152e0faa2 in waitpid () from /lib64/libpthread.so.0
0000001 0x00007f20a2d68414 in signal_handler (sig=11) at signal.c:240
0000002 <signal handler called>
0000003 0x000000315528040b in PyDict_GetItem () from /usr/lib64/libpython2.7.so.1.0
0000004 0x0000003155281e64 in PyDict_GetItemString () from /usr/lib64/libpython2.7.so.1.0
0000005 0x00007f20a191630a in PyCheckFile (ctx=0x7f209802f158, fname=0x7f20933622a8 "/var/local/owncloud/data/sergey/files_trashbin/versions/MyDB.kdbx.v1393235275.d1430165584") at python-fd.c:2054
0000006 0x00007f20a19135b5 in checkFile (ctx=0x7f209802f158, fname=0x7f20933622a8 "/var/local/owncloud/data/sergey/files_trashbin/versions/MyDB.kdbx.v1393235275.d1430165584") at python-fd.c:726
0000007 0x0000000000415a9f in plugin_check_file (jcr=0x7f2098023a68, fname=0x7f20933622a8 "/var/local/owncloud/data/sergey/files_trashbin/versions/MyDB.kdbx.v1393235275.d1430165584") at fd_plugins.c:362
0000008 0x000000000040a124 in B_ACCURATE_HTABLE::send_deleted_list (this=0x7f2098048ae8, jcr=0x7f2098023a68) at accurate_htable.c:179
0000009 0x000000000040948b in accurate_finish (jcr=jcr@entry=0x7f2098023a68) at accurate.c:88
0000010 0x000000000040b520 in blast_data_to_storage_daemon (jcr=jcr@entry=0x7f2098023a68, addr=addr@entry=0x0) at backup.c:153
0000011 0x00000000004123f5 in backup_cmd (jcr=0x7f2098023a68) at dir_cmd.c:1730
0000012 0x000000000041397e in handle_director_connection (dir=0x6db948) at dir_cmd.c:482
0000013 handle_connection_request (arg=0x6db948) at dir_cmd.c:590
0000014 0x00007f20a2d71e95 in workq_server (arg=arg@entry=0x62c800 <dir_workq>) at workq.c:335
0000015 0x00007f20a2d5ba00 in lmgr_thread_launcher (x=0x6c6228) at lockmgr.c:932
0000016 0x0000003152e0824d in start_thread () from /lib64/libpthread.so.0
0000017 0x00000031526e955d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f20a1393700 (LWP 10420)):
#0 0x0000003152e0c978 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
0000001 0x00007f20a2d5be8c in bthread_cond_timedwait_p (cond=cond@entry=0x7f20a2f8b540 <_ZL5timer>, m=m@entry=0x7f20a2f8b580 <_ZL11timer_mutex>, abstime=abstime@entry=0x7f20a1392e90, file=file@entry=0x7f20a2d7b03a "watchdog.c", line=line@entry=313) at lockmgr.c:817
0000002 0x00007f20a2d71539 in watchdog_thread (arg=arg@entry=0x0) at watchdog.c:313
0000003 0x00007f20a2d5ba00 in lmgr_thread_launcher (x=0x6cd9a8) at lockmgr.c:932
0000004 0x0000003152e0824d in start_thread () from /lib64/libpthread.so.0
0000005 0x00000031526e955d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f20a2320700 (LWP 10419)):
#0 0x0000003152e0f71d in nanosleep () from /lib64/libpthread.so.0
0000001 0x00007f20a2d4bbb4 in bmicrosleep (sec=sec@entry=30, usec=usec@entry=0) at bsys.c:100
0000002 0x00007f20a2d5b971 in check_deadlock () at lockmgr.c:567
0000003 0x0000003152e0824d in start_thread () from /lib64/libpthread.so.0
0000004 0x00000031526e955d in clone () from /lib64/libc.so.6
#0 0x0000003152e0f71d in nanosleep () from /lib64/libpthread.so.0
No symbol table info available.
0000001 0x00007f20a2d4bbb4 in bmicrosleep (sec=sec@entry=30, usec=usec@entry=0) at bsys.c:100
100 status = nanosleep(&timeout, NULL);
timeout = {tv_sec = 30, tv_nsec = 0}
tv = {tv_sec = 0, tv_usec = 0}
tz = {tz_minuteswest = -1573779712, tz_dsttime = 32544}
status = <optimized out>
0000002 0x00007f20a2d5b971 in check_deadlock () at lockmgr.c:567
567 while (!bmicrosleep(30, 0)) {
__clframe = <optimized out>
old = 0
0000003 0x0000003152e0824d in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
0000004 0x00000031526e955d in clone () from /lib64/libc.so.6
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
TagsNo tags attached.
bareos-master: impactyes
bareos-master: actionwill care
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impactyes
bareos-14.2: actionwill care
bareos-13.2: impactno
bareos-13.2: action
bareos-12.4: impactno
bareos-12.4: action

Relationships

child of 0000474 closed Release bareos-14.2.6 

Activities

mvwieringen

mvwieringen

2015-05-30 12:13

developer   ~0001764

First of all your config makes little sense you cannot use the bpipe plugin
as "option" plugin.

It does indeed trigger a problem in the python-fd plugin which is loaded
but inactive but tries to resolve an entry point into a python script via
a python dictionary that is not there so leads to a null pointer deference.

A fix will be in master soon but I would concentrate first on creating a
proper fileset, either remove the whole bpipe from the option block or
create a proper bpipe fileset e.g. split the current into two.
mvwieringen

mvwieringen

2015-06-02 22:36

developer   ~0001766

Fix committed to bareos master branch with changesetid 5370.
mvwieringen

mvwieringen

2015-06-17 20:26

developer   ~0001771

Fix committed to bareos bareos-14.2 branch with changesetid 5380.

Related Changesets

bareos: master ecd961f0

2015-05-30 12:01:56

mvwieringen

Ported: N/A

Details Diff
Don't call PyCheckFile()

When we have a inactive python-fd instance in a Job e.g. no option or
file plugin definition in the fileset we are still called as part of the
Checkfile method which calls each plugin loaded. This gives problems as
we try to lookup the entry point into a non-existing python script and
as such we have no dictonary at all which means the lookup will
dereference a null pointer.

Fixes 0000473: Incremental backup failure
Affected Issues
0000473
mod - src/plugins/filed/python-fd.c Diff File

bareos: bareos-14.2 5b4afdd5

2015-05-30 12:01:56

mvwieringen

Ported: N/A

Details Diff
Don't call PyCheckFile()

When we have a inactive python-fd instance in a Job e.g. no option or
file plugin definition in the fileset we are still called as part of the
Checkfile method which calls each plugin loaded. This gives problems as
we try to lookup the entry point into a non-existing python script and
as such we have no dictonary at all which means the lookup will
dereference a null pointer.

Fixes 0000473: Incremental backup failure
Affected Issues
0000473
mod - src/plugins/filed/python-fd.c Diff File

Issue History

Date Modified Username Field Change
2015-05-30 02:29 SergeyMorozov New Issue
2015-05-30 12:13 mvwieringen Note Added: 0001764
2015-05-30 12:13 mvwieringen Status new => feedback
2015-06-01 15:27 mvwieringen bareos-master: impact => yes
2015-06-01 15:27 mvwieringen bareos-master: action => will care
2015-06-01 15:27 mvwieringen bareos-14.2: impact => yes
2015-06-01 15:27 mvwieringen bareos-14.2: action => will care
2015-06-01 15:27 mvwieringen bareos-13.2: impact => no
2015-06-01 15:27 mvwieringen bareos-12.4: impact => no
2015-06-02 22:36 mvwieringen Changeset attached => bareos master ecd961f0
2015-06-02 22:36 mvwieringen Note Added: 0001766
2015-06-02 22:36 mvwieringen Status feedback => resolved
2015-06-02 22:36 mvwieringen Resolution open => fixed
2015-06-17 12:14 joergs Relationship added child of 0000474
2015-06-17 20:26 mvwieringen Changeset attached => bareos bareos-14.2 5b4afdd5
2015-06-17 20:26 mvwieringen Note Added: 0001771
2015-10-02 20:03 mvwieringen adm Status resolved => closed
2015-10-02 20:03 mvwieringen adm Target Version => 14.2.6