View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001364 | bareos-core | file daemon | public | 2021-06-16 16:47 | 2023-04-14 11:46 |
| Reporter | phsm | Assigned To | bruno-at-bareos | ||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | closed | Resolution | unable to reproduce | ||
| Summary | 0001364: Backup job stucks when RunBeforeJob script takes long time | ||||
| Description | OS: Ubuntu 20.04 BareOS version: 20.0.1 Media type used: File Hello, I created a job to backup a database dump (see jobdef and job configs in the attachment) . The database is quite big so it takes good half an hour to proceed RunBeforeJob command. Once the command successfully finishes, there is no activity on this job anymore, and the only way to deal with it is to cancel it. The only clue I've seen so far is the storage daemon status saying "Device is BLOCKED waiting to create a volume". If I switch off the RunBeforeJob script, the job starts backing up file immediately without any issue. I've attached the files: jobdef.txt - Job defaults config job.txt - the actual job config storagedaemon_device.txt - the config of storage-daemon Device storagestatus.txt - the status message of the storage-daemon while the job already stuck job_log.txt - the log of the job at the moment the job is stuck. | ||||
| Tags | No tags attached. | ||||
 |
jobdef.txt (695 bytes)
JobDefs {
Name = PostgresAllDatabases
Type = Backup
Level = Full
FileSet = Postgres
Schedule = FullFourTimesADay
Messages = Standard
Max Run Time = 3 hours
Max Start Delay = 2 hours
Max Wait Time = 2 hours
Pool = Full
Priority = 10
Write Bootstrap = "/var/lib/bareos/%c.bsr"
Run Script {
Console = ".bvfs_update jobid=%i"
RunsWhen = After
RunsOnClient = No
}
Run Script {
FailJobOnError = Yes
RunsOnClient = Yes
RunsWhen = Before
Command = "/usr/local/bin/bareos-pgdump.sh /var/backups/postgres"
}
Run Script {
RunsOnSuccess = Yes
RunsOnClient = Yes
RunsWhen = After
Command = "rm -rf /var/backups/postgres"
}
}
job.txt (128 bytes)
Job {
Name = Backup_Client_Database
JobDefs = PostgresAllDatabases
Client = client.example.com
Storage = CephExternal
}
job_log.txt (9,139 bytes)
16-Jun 13:33 bareos.example.com JobId 5261: Start Backup JobId 5261, Job=Backup_Client_Database.2021-06-16_13.33.41_01 16-Jun 13:33 bareos.example.com JobId 5261: Connected Storage daemon at 10.0.0.1:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 16-Jun 13:33 bareos.example.com JobId 5261: Using Device "FileStorage" to write. 16-Jun 13:33 bareos.example.com JobId 5261: Connected Client: client.example.com at client.example.com:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 16-Jun 13:33 bareos.example.com JobId 5261: Handshake: Immediate TLS 16-Jun 13:33 client.example.com JobId 5261: Connected Storage daemon at 10.0.0.1:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 16-Jun 13:33 client.example.com JobId 5261: shell command: run ClientBeforeJob "/usr/local/bin/bareos-pgdump.sh /var/backups/postgres" 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: warning: there are circular foreign-key constraints on this table: 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: hypertable 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: Consider using a full dump instead of a --data-only dump to avoid this problem. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: warning: there are circular foreign-key constraints on this table: 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: chunk 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: Consider using a full dump instead of a --data-only dump to avoid this problem. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 13:33 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: pg_dump: NOTICE: hypertable data are in the chunks, no data will be copied 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: DETAIL: Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data. 16-Jun 14:09 client.example.com JobId 5261: ClientBeforeJob: HINT: Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually. 16-Jun 14:09 client.example.com JobId 5261: Extended attribute support is enabled 16-Jun 14:09 client.example.com JobId 5261: ACL support is enabled < At this point all activity on the job stops > storagestatus.txt (1,140 bytes)
Connecting to Storage daemon CephExternal at 10.0.0.1:9103
CephExternal Version: 20.0.1 (02 March 2021) Ubuntu 20.04.1 LTS
Daemon started 16-Jun-21 13:25. Jobs: run=2, running=0, bareos.org build binary
Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 bwlimit=0kB/s
Running Jobs:
Writing: Full Backup job Backup_Client_Database JobId=5261 Volume=""
pool="Full" device="FileStorage" (/mnt/ceph/bareos-external)
spooling=0 despooling=0 despool_wait=0
Files=0 Bytes=0 AveBytes/sec=0 LastBytes/sec=0
FDReadSeqNo=6 in_msg=6 out_msg=4 fd=5
====
Jobs waiting to reserve a drive:
====
Terminated Jobs:
JobId Level Files Bytes Status Finished Name
===================================================================
5259 Full 45 32.01 K OK 16-Jun-21 13:33 Backup_Client_Files
5260 Incr 0 0 OK 16-Jun-21 13:33 Backup_Client_Files
====
Device status:
Device "FileStorage" (/mnt/ceph/bareos-external) is not open.
Device is BLOCKED waiting to create a volume for:
Pool: Full
Media type: File
==
====
Used Volume status:
====
====
storagedaemon_device.txt (318 bytes)
Device {
Device Type = File
Name = FileStorage
Media Type = File
Archive Device = /mnt/ceph/bareos-external
Label Media = yes
Random Access = yes
Automatic Mount = yes
Removable Media = no
Always Open = no
Description = "File device. A connecting Director must have the same Name and MediaType."
}
|
|
|
Did you already retry with recent Bareos code like 21 or 22 for example and also without playing with the following parameters Max Run Time = 3 hours Max Start Delay = 2 hours Max Wait Time = 2 hours |
|
|
to get update informations |
|
|
closing no anwser, and we know runscript running several hours working as expected |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2021-06-16 16:47 | phsm | New Issue | |
| 2021-06-16 16:47 | phsm | File Added: jobdef.txt | |
| 2021-06-16 16:47 | phsm | File Added: job.txt | |
| 2021-06-16 16:47 | phsm | File Added: job_log.txt | |
| 2021-06-16 16:47 | phsm | File Added: storagestatus.txt | |
| 2021-06-16 16:47 | phsm | File Added: storagedaemon_device.txt | |
| 2023-03-21 17:31 | bruno-at-bareos | Note Added: 0004919 | |
| 2023-03-29 15:07 | bruno-at-bareos | Assigned To | => bruno-at-bareos |
| 2023-03-29 15:07 | bruno-at-bareos | Status | new => feedback |
| 2023-03-29 15:07 | bruno-at-bareos | Note Added: 0004959 | |
| 2023-04-14 11:46 | bruno-at-bareos | Status | feedback => closed |
| 2023-04-14 11:46 | bruno-at-bareos | Resolution | open => unable to reproduce |
| 2023-04-14 11:46 | bruno-at-bareos | Note Added: 0004967 |
