View Issue Details

IDProjectCategoryView StatusLast Update
0000935bareos-corefile daemonpublic2023-07-04 15:14
ReporterMarceloRuiz Assigned Tobruno-at-bareos  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
PlatformLinuxOSUbuntuOS Version17.10
Product Version16.2.4 
Summary0000935: Unable to stop filedaemon
DescriptionThe file daemon cannot be stopped (or restarted) on multiple computers with the same configuration. It also delays the shutdown process for about 1 min and 30 seconds until it is terminated by the OS.
Steps To ReproduceJust install packages from http://download.bareos.org/bareos/release/16.2/xUbuntu_16.04/amd64/ into Ubuntu (dependencies are met, so no hack is necessary for the installation to succeed and Bareos to properly work).
After the system is configured (in my case with TLS and both Client to Director and Director to Client communications) Bareos will start with no problem, but it won't stop normally when 'sudo service bareos-fd stop' is issued after it connects to the Director. If no connection is active (Director is down), it will stop with no problem.

Additional InformationThere is a report for the Bacula file daemon not stopping: "https://serverfault.com/questions/604645/bacula-daemon-fails-to-stop-or-restart"
that states that somehow the port for the file daemon is hard coded. I don't know if it is relevant to this bug.


I also noticed huge files under /var/lib/bareos on my system:

----------------------------------------
marcelo  /  var  lib  bareos  ls -l
----------------------------------------

total 1553776
-rw-r----- 1 root bareos 740 Mar 23 16:13 bareos.15815.traceback
-rw-r----- 1 root bareos 724 Mar 23 16:15 bareos.15925.traceback
-rw-r----- 1 root bareos 744 Mar 23 16:09 bareos.993.traceback
-rw-r----- 1 root bareos 4 Mar 27 10:49 bareos-fd.9102.pid
-rw-r----- 1 root bareos 2196 Mar 27 11:00 bareos-fd.9102.state
-rw-r----- 1 root bareos 462759528 Mar 23 16:13 bareos-fd.core.15815
-rw-r----- 1 root bareos 438955680 Mar 23 16:15 bareos-fd.core.15925
-rw-r----- 1 root bareos 689301752 Mar 23 16:09 bareos-fd.core.993
-rw-r----- 1 root bareos 1112 Mar 23 16:13 marcelo-qosmio-fd.15815.bactrace
-rw-r----- 1 root bareos 1071 Mar 23 16:15 marcelo-qosmio-fd.15925.bactrace
-rw-r----- 1 root bareos 68 Mar 23 16:09 marcelo-qosmio-fd.993.bactrace
drwxrwxr-x 2 bareos bareos 4096 Oct 17 2016 storage


----------------------------------------
service bareos-fd status
----------------------------------------

● bareos-filedaemon.service - Bareos File Daemon service
   Loaded: loaded (/lib/systemd/system/bareos-filedaemon.service; enabled; vendor preset: enabled)
   Active: failed (Result: timeout) since Tue 2018-03-27 11:44:24 EDT; 2min 30s ago
     Docs: man:bareos-fd(8)
  Process: 7114 ExecStart=/usr/sbin/bareos-fd (code=exited, status=0/SUCCESS)
 Main PID: 7115 (code=killed, signal=KILL)

Mar 27 11:40:00 marcelo-qosmio systemd[1]: bareos-filedaemon.service: PID file /var/lib/bareos/bareos-fd.9102.pid not readable (yet?) after start: No such
Mar 27 11:40:00 marcelo-qosmio systemd[1]: Started Bareos File Daemon service.
Mar 27 11:42:54 marcelo-qosmio bareos-fd[7115]: Shutting down BAREOS service: marcelo-qosmio-fd ...
Mar 27 11:42:54 marcelo-qosmio systemd[1]: Stopping Bareos File Daemon service...
Mar 27 11:44:24 marcelo-qosmio systemd[1]: bareos-filedaemon.service: State 'stop-sigterm' timed out. Killing.
Mar 27 11:44:24 marcelo-qosmio systemd[1]: bareos-filedaemon.service: Killing process 7115 (bareos-fd) with signal SIGKILL.
Mar 27 11:44:24 marcelo-qosmio systemd[1]: bareos-filedaemon.service: Main process exited, code=killed, status=9/KILL
Mar 27 11:44:24 marcelo-qosmio systemd[1]: Stopped Bareos File Daemon service.
Mar 27 11:44:24 marcelo-qosmio systemd[1]: bareos-filedaemon.service: Unit entered failed state.
Mar 27 11:44:24 marcelo-qosmio systemd[1]: bareos-filedaemon.service: Failed with result 'timeout'.

----------------------------------------
dpkg --status bareos-filedaemon
----------------------------------------

Package: bareos-filedaemon
Status: install ok installed
Priority: optional
Section: admin
Installed-Size: 346
Maintainer: Joerg Steffens <joerg.steffens@bareos.com>
Architecture: amd64
Source: bareos
Version: 16.2.4-12.1
Replaces: bacula-fd
Depends: bareos-common (= 16.2.4-12.1), lsb-base (>= 3.2-13), lsof, libc6 (>= 2.14), libfastlz, libjansson4 (>= 2.0.1), zlib1g (>= 1:1.1.4), init-system-helpers (>= 1.18~)
Pre-Depends: debconf (>= 1.4.30) | debconf-2.0, adduser
Conflicts: bacula-fd
Conffiles:
 /etc/init.d/bareos-fd bcc61ad57fde8a771a5002365130c3ec
Description: Backup Archiving Recovery Open Sourced - file daemon
 Bareos is a set of programs to manage backup, recovery and verification of
 data across a network of computers of different kinds.
 .
 The file daemon has to be installed on the machine to be backed up. It is
 responsible for providing the file attributes and data when requested by
 the Director, and also for the file system-dependent part of restoration.
 .
 This package contains the Bareos File daemon.
Homepage: http://www.bareos.org/

----------------------------------------
dpkg -l "*bareos*"
----------------------------------------

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-================-=============-=============-======================================
ii bareos-bconsole 16.2.4-12.1 amd64 Backup Archiving Recovery Open Sourced
ii bareos-client 16.2.4-12.1 amd64 Backup Archiving Recovery Open Sourced
ii bareos-common 16.2.4-12.1 amd64 Backup Archiving Recovery Open Sourced
un bareos-database- <none> <none> (no description available)
un bareos-database- <none> <none> (no description available)
un bareos-database- <none> <none> (no description available)
un bareos-database- <none> <none> (no description available)
un bareos-database- <none> <none> (no description available)
ii bareos-dbg 16.2.4-12.1 amd64 Backup Archiving Recovery Open Sourced
un bareos-director <none> <none> (no description available)
ii bareos-filedaemo 16.2.4-12.1 amd64 Backup Archiving Recovery Open Sourced
un bareos-storage <none> <none> (no description available)
un bareos-storage-f <none> <none> (no description available)
un bareos-storage-t <none> <none> (no description available)
un bareos-tools <none> <none> (no description available)
un bareos-traymonit <none> <none> (no description available)

----------------------------------------
ls -la /etc/systemd/system/bareos*
----------------------------------------

lrwxrwxrwx 1 root root 45 Mar 16 22:47 /etc/systemd/system/bareos-fd.service -> /lib/systemd/system/bareos-filedaemon.service


----------------------------------------
ls -la /lib/systemd/system/bareos*
----------------------------------------

-rw-r--r-- 1 root root 985 Oct 17 2016 /lib/systemd/system/bareos-filedaemon.service


----------------------------------------
cat bareos.15925.traceback
----------------------------------------

Created /var/lib/bareos/bareos-fd.core.15925 for doing postmortem debugging
[New LWP 15925]
[New LWP 15926]
[New LWP 15927]
[New LWP 15928]
[New LWP 15934]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/bareos-fd'.
#0 0x00007f3ff057d951 in __GI___poll (fds=0x7ffdb27f9e10, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
29 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.
[Current thread is 1 (Thread 0x7f3ff1b47740 (LWP 15925))]
$1 = 1668440429
$2 = 21503672
$3 = 21503736
/usr/lib/bareos/scripts/btraceback.gdb:4: Error in sourced command file:
No symbol "catalog_db" in current context.


----------------------------------------
cat marcelo-qosmio-fd.15925.bactrace
----------------------------------------
 
Attempt to dump locks
threadid=0x7f3fdfdee700 max=1 current=-1
threadid=0x7f3fed128700 max=2 current=-1
threadid=0x7f3fed929700 max=0 current=-1
threadid=0x7f3fee12a700 max=0 current=-1
threadid=0x7f3ff1b47740 max=0 current=-1
Attempt to dump current JCRs. njcrs=2
threadid=0x7f3fdfdee700 JobId=43 JobStatus=C jcr=0x7f3fe40012c8 name=marcelo-qosmio-backup-install.2018-03-23_16.14.43_06
threadid=0x7f3fdfdee700 killable=1 JobId=43 JobStatus=C jcr=0x7f3fe40012c8 name=marcelo-qosmio-backup-install.2018-03-23_16.14.43_06
    use_count=1
    JobType=I JobLevel=I
    sched_time=23-Mar-2018 16:13 start_time=23-Mar-2018 16:13
    end_time=31-Dec-1969 19:00 wait_time=31-Dec-1969 19:00
    db=(nil) db_batch=(nil) batch_started=0
threadid=0x7f3fed128700 JobId=0 JobStatus=C jcr=0x7f3fe4029248 name=*Director*
threadid=0x7f3fed128700 killable=1 JobId=0 JobStatus=C jcr=0x7f3fe4029248 name=*Director*
    use_count=1
    JobType=I JobLevel=
    sched_time=23-Mar-2018 16:14 start_time=23-Mar-2018 16:14
    end_time=31-Dec-1969 19:00 wait_time=31-Dec-1969 19:00
    db=(nil) db_batch=(nil) batch_started=0

Unfortunately, I cannot upgrade to Bareos 17.2 because the Director and Storage daemon are running on FreeBSD (FreeNAS) and Bareos port seems not to be updated in the near future or at all (maintainer said it was going to update ASAP on January and never answered e-mails after that).

I will be happy to provide more information if needed.


TagsNo tags attached.

Relationships

related to 0000385 closedfranku bareos daemon stop restart hang if bareos-tray-monitor is connected 

Activities

joergs

joergs

2018-04-05 12:44

developer   ~0002959

Yes, this is a known problem. As soon as the fieldaemon has an open network connection (either client initiated connection or e.g. the traymonitor) it will not exit smoothly.
bruno-at-bareos

bruno-at-bareos

2023-07-04 15:14

manager   ~0005126

fixed in 19.2.1 see 385

Issue History

Date Modified Username Field Change
2018-03-27 18:10 MarceloRuiz New Issue
2018-04-05 12:44 joergs Note Added: 0002959
2018-04-05 12:44 joergs Status new => confirmed
2018-04-05 18:26 joergs Relationship added related to 0000385
2023-07-04 15:14 bruno-at-bareos Assigned To => bruno-at-bareos
2023-07-04 15:14 bruno-at-bareos Status confirmed => closed
2023-07-04 15:14 bruno-at-bareos Resolution open => fixed
2023-07-04 15:14 bruno-at-bareos Note Added: 0005126