View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001006bareos-corestorage daemonpublic2018-09-12 12:452019-12-18 15:24
ReporterMartin Svec Assigned Tojoergs  
PrioritynormalSeveritycrashReproducibilityalways
Status closedResolutionduplicate 
PlatformLinuxOSDebianOS Version9
Product Version17.2.4 
Summary0001006: Storage daemon segfaults in update_job_statistics when starting scheduled jobs
DescriptionIf job statistics are turned on and multiple backup jobs are scheduled and started at the same time, bareos-sd 17.2.4 almost always segfaults. When started in gdb, I got the following bareos-sd backtrace:

(gdb) bt
#0 0x000055555556c991 in update_job_statistics (jcr=0x7fffbc001078, now=1536531334) at sd_stats.c:296
0000001 0x000055555556cc93 in statistics_thread_runner (arg=arg@entry=0x0) at sd_stats.c:386
0000002 0x00007ffff74add9f in lmgr_thread_launcher (x=0x5555557b7328) at lockmgr.c:928
0000003 0x00007ffff6309494 in start_thread (arg=0x7fffeeffd700) at pthread_create.c:333
0000004 0x00007ffff519aacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

As a workaround, it suffices to set "Collect Job Statistics = no".
Steps To ReproduceTurn on job statistics, schedule multiple backup-to-disk jobs using one common Schedule and let the director to start all the jobs according to the Schedule. Note that the concurrency is important. If the jobs are started manually one by one, the bug doesn't occur.
Additional InformationOur setup: Bareos 17.2.4 installed from bareos.org Debian packages (http://download.bareos.org/bareos/release/latest/Debian_9.0/). Director, storage daemon and webui are all on one physical server, catalog is in MySQL database. Backups are stored on a local disk device. Storage daemon has twenty identical disk Device resources, pointing to the same location:

Device {
  Name = Disk-Bareos-01-000
  Media Type = Disk
  Device Type = File
  Archive Device = /backup
  LabelMedia = yes
  Random Access = yes
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen = no
  Maximum Concurrent Jobs = 1
}

...

Device {
  Name = Disk-Bareos-01-019
  Media Type = Disk
  Device Type = File
  Archive Device = /backup
  LabelMedia = yes
  Random Access = yes
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen = no
  Maximum Concurrent Jobs = 1
}


We've 88 jobs and four different nightly Schedules, Maximum Concurrent Jobs is set to 20 both in director and storage daemon. That is, director typically starts up to 20 jobs at the same time.
TagsNo tags attached.

Relationships

duplicate of 0000983 closedfranku SD crashes sometimes with statistics collection on 
has duplicate 0001024 closedarogge crash SD if 2 jobs are running using one tape 

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2018-09-12 12:45 Martin Svec New Issue
2018-09-14 15:13 joergs Relationship added duplicate of 0000983
2018-09-14 15:14 joergs Status new => resolved
2018-09-14 15:14 joergs Resolution open => duplicate
2018-09-14 15:14 joergs Assigned To => joergs
2019-07-15 10:33 arogge Relationship added has duplicate 0001024
2019-12-18 15:24 arogge Status resolved => closed