View Issue Details

IDProjectCategoryView StatusLast Update
0001364bareos-core[All Projects] file daemonpublic2021-06-16 16:47
ReporterphsmAssigned To 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
Product Version 
Fixed in Version 
Summary0001364: Backup job stucks when RunBeforeJob script takes long time
DescriptionOS: Ubuntu 20.04
BareOS version: 20.0.1
Media type used: File

Hello,
I created a job to backup a database dump (see jobdef and job configs in the attachment) . The database is quite big so it takes good half an hour to proceed RunBeforeJob command.
Once the command successfully finishes, there is no activity on this job anymore, and the only way to deal with it is to cancel it.

The only clue I've seen so far is the storage daemon status saying "Device is BLOCKED waiting to create a volume".

If I switch off the RunBeforeJob script, the job starts backing up file immediately without any issue.

I've attached the files:
jobdef.txt - Job defaults config
job.txt - the actual job config
storagedaemon_device.txt - the config of storage-daemon Device
storagestatus.txt - the status message of the storage-daemon while the job already stuck
job_log.txt - the log of the job at the moment the job is stuck.

TagsNo tags attached.
bareos-master: impact
bareos-master: action
bareos-19.2: impact
bareos-19.2: action
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action

Activities

phsm

phsm

2021-06-16 16:47

reporter  

jobdef.txt (695 bytes)
JobDefs {
  Name = PostgresAllDatabases
  Type = Backup
  Level = Full
  FileSet = Postgres
  Schedule = FullFourTimesADay
  Messages = Standard
  Max Run Time = 3 hours
  Max Start Delay = 2 hours
  Max Wait Time = 2 hours
  Pool = Full
  Priority = 10
  Write Bootstrap = "/var/lib/bareos/%c.bsr"

  Run Script {
    Console = ".bvfs_update jobid=%i"
    RunsWhen = After
    RunsOnClient = No
  }

  Run Script {
    FailJobOnError = Yes
    RunsOnClient = Yes
    RunsWhen = Before
    Command = "/usr/local/bin/bareos-pgdump.sh /var/backups/postgres"
  }

  Run Script {
    RunsOnSuccess = Yes
    RunsOnClient = Yes
    RunsWhen = After
    Command = "rm -rf /var/backups/postgres"
  }
}
jobdef.txt (695 bytes)
job.txt (128 bytes)
Job {
  Name = Backup_Client_Database
  JobDefs = PostgresAllDatabases
  Client = client.example.com
  Storage = CephExternal
}
job.txt (128 bytes)
job_log.txt (9,139 bytes)
16-Jun 13:33 bareos.example.com JobId 5261: Start Backup JobId 5261, Job=Backup_Client_Database.2021-06-16_13.33.41_01
16-Jun 13:33 bareos.example.com JobId 5261: Connected Storage daemon at 10.0.0.1:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
16-Jun 13:33 bareos.example.com JobId 5261: Using Device "FileStorage" to write.
16-Jun 13:33 bareos.example.com JobId 5261: Connected Client: client.example.com at client.example.com:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
16-Jun 13:33 bareos.example.com JobId 5261:  Handshake: Immediate TLS 
16-Jun 13:33 client.example.com JobId 5261: Connected Storage daemon at 10.0.0.1:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
16-Jun 13:33 client.example.com JobId 5261: shell command: run ClientBeforeJob "/usr/local/bin/bareos-pgdump.sh /var/backups/postgres"
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: warning: there are circular foreign-key constraints on this table:
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump:   hypertable
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: Consider using a full dump instead of a --data-only dump to avoid this problem.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: warning: there are circular foreign-key constraints on this table:
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump:   chunk
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: Consider using a full dump instead of a --data-only dump to avoid this problem.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
16-Jun 14:09 client.example.com JobId 5261: Extended attribute support is enabled
16-Jun 14:09 client.example.com JobId 5261: ACL support is enabled
< At this point all activity on the job stops >
job_log.txt (9,139 bytes)
storagestatus.txt (1,140 bytes)
Connecting to Storage daemon CephExternal at 10.0.0.1:9103

CephExternal Version: 20.0.1 (02 March 2021) Ubuntu 20.04.1 LTS 
Daemon started 16-Jun-21 13:25. Jobs: run=2, running=0, bareos.org build binary
 Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 bwlimit=0kB/s

Running Jobs:
Writing: Full Backup job Backup_Client_Database JobId=5261 Volume=""
    pool="Full" device="FileStorage" (/mnt/ceph/bareos-external)
    spooling=0 despooling=0 despool_wait=0
    Files=0 Bytes=0 AveBytes/sec=0 LastBytes/sec=0
    FDReadSeqNo=6 in_msg=6 out_msg=4 fd=5
====

Jobs waiting to reserve a drive:
====

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name 
===================================================================
  5259  Full         45    32.01 K  OK       16-Jun-21 13:33 Backup_Client_Files
  5260  Incr          0         0   OK       16-Jun-21 13:33 Backup_Client_Files
====

Device status:

Device "FileStorage" (/mnt/ceph/bareos-external) is not open.
    Device is BLOCKED waiting to create a volume for:
       Pool:        Full
       Media type:  File
==
====

Used Volume status:
====

====
storagestatus.txt (1,140 bytes)
storagedaemon_device.txt (318 bytes)
Device {
  Device Type = File
  Name = FileStorage
  Media Type = File
  Archive Device = /mnt/ceph/bareos-external
  Label Media = yes
  Random Access = yes
  Automatic Mount = yes
  Removable Media = no
  Always Open = no
  Description = "File device. A connecting Director must have the same Name and MediaType."
}

Issue History

Date Modified Username Field Change
2021-06-16 16:47 phsm New Issue
2021-06-16 16:47 phsm File Added: jobdef.txt
2021-06-16 16:47 phsm File Added: job.txt
2021-06-16 16:47 phsm File Added: job_log.txt
2021-06-16 16:47 phsm File Added: storagestatus.txt
2021-06-16 16:47 phsm File Added: storagedaemon_device.txt