View Issue Details

IDProjectCategoryView StatusLast Update
0000865bareos-core[All Projects] directorpublic2017-11-16 12:46
Reporterchaos_prevailsAssigned To 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
PlatformLinux UbuntuOSServer x64OS Version16.04 latest
Product Version16.2.4 
Fixed in Version 
Summary0000865: mailcommand not working with big job and too many files (timeout?) sendmail script works with short jobs
DescriptionA full backup job contains a couple of thousand files. I want to have a list of all files (to know if all directories are fine, verify some important files in some directories, ... ) sent to me by email after job completion.

This works if the number of files is relatively low, but stops working with a couple of thousand files.

There is nothing in /var/log/syslog, or /var/log/bareos.log (beside the successful job).

/var/log/bareos-audit.log shows the bconsole queries of the sendmail script (same queries as successful sent emails):
...
13-Oct 08:37 bareos-dir: Console [default] from [127.0.0.1] cmdline list joblog jobid=73 << job for which I don't receive email
13-Oct 08:37 bareos-dir: Console [default] from [127.0.0.1] cmdline list files jobid=73 << job for which I don't receive email
13-Oct 08:38 bareos-dir: Console [default] from [127.0.0.1] cmdline list joblog jobid=74 << job for which I receive email
13-Oct 08:38 bareos-dir: Console [default] from [127.0.0.1] cmdline list files jobid=74 << job for which I receive email
...

/var/log/mail does NOT show that the email is sent for the job in question (while it shows sent emails of other jobs, which I also receive in my mailbox)

Strangely enough, if I start a different small job (because testing receiving emails with a job taking 2 days is not so efficient), with the jobid of the big job manually added to the sendmail script, it works.

So it seems not to be the sendmail script, but rather a problem with a job which takes very long + a long list of files to be added to an email.

BTW: it works neither with sendemail nor with ssmtp (my mailserver needs smtp authentification)

Please find my configuration in the additional information
Steps To Reproduce1) have the list of files added to your email (e.g. as attachment) after job completion
2) start a job with a lot of files to backup
3) the email is not sent (no entry in /var/log/mail.log)
Additional Information/etc/bareos/bareos-dir.d/messages/Standard.conf

Messages {
  Name = Standard
  Description = "Reasonable message delivery -- send most everything to email address and to the console."
  operatorcommand = "/usr/local/bin/bareos-messages.sh \"[Operator] %d: %j Messages (jobid %i)\" %i \"%r\""
  mailcommand = "/usr/local/bin/bareos-messages.sh \"[SERVICE %e] %d: %t %n %l on %c (jobid %i)\" %i \"%r\""
  operator = it@XXXXX = mount # (0000003)
  mail = it@XXXXXX = all, !skipped, !saved, !audit # (0000002)
  console = all, !skipped, !saved, !audit
  append = "/var/log/bareos/bareos.log" = all, !skipped, !saved, !audit
  catalog = all, !skipped, !saved, !audit
}


/usr/local/bin/bareos-messages.sh

#!/bin/bash

SUBJECT=$1
#JOBID=$2
JOBID=73
RECIPIENTS=$3
ATTACHMENT_PATH=/tmp/
ATTACHMENT_FILE=bareosjobfilelist.gz

#if Operator or Daemon message
if [[ $SUBJECT == *"[Operator]"* || $SUBJECT == *"[Daemon]"* ]]; then

BODY=$(cat <<EOF
*Messages*
`echo messages | bconsole`
EOF
)

#send operator or daemon email
echo "$BODY" | mail -s "$SUBJECT" "$RECIPIENTS"

else #Service start

BODY=$(cat <<EOF
*Joblog*
`echo list joblog jobid=$JOBID | bconsole`
EOF
)
FILES=`echo list files jobid=$JOBID | bconsole`
FILESNO=`echo "$FILES" | wc -l`

if (( "$FILESNO" > 2000 )); then #start create attachment

echo "$FILES" | gzip > $ATTACHMENT_PATH$ATTACHMENT_FILE

#send email with attachment
(echo "$BODY"; uuencode $ATTACHMENT_PATH$ATTACHMENT_FILE $ATTACHMENT_FILE) | mail -s "$SUBJECT" "$RECIPIENTS"
rm `echo $ATTACHMENT_PATH$ATTACHMENT_FILE`

else # start add files to body

BODY+=$(cat <<EOF


*Files backed up*
`echo "$FILES"`
EOF
)

#send email without attachment
echo "$BODY" | mail -s "$SUBJECT" "$RECIPIENTS"

fi #end create attachment
fi #end Service

TagsNo tags attached.
bareos-master: impact
bareos-master: action
bareos-19.2: impact
bareos-19.2: action
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action

Activities

Shodan

Shodan

2017-11-16 12:46

reporter   ~0002816

Similar problem for me, I'm using zabbix.sh script for zabbix monitoring

Messages {
  Name = Standard
  Mail Command = "/etc/bareos/zabbix/zabbix.sh %i %t %e %c %l"
  Mail = admin@local.com = all, !skipped
}

The code fragment below, verify job status code

sql="/usr/bin/mysql --defaults-file=$bareosDbPass -h$bareosDbAddr -P$bareosDbPort -u$bareosDbUser -D$bareosDbName -NB -e"
bareosJobStatus=$($sql "select JobStatus from Job where JobId=$bareosJobId;" 2>/dev/null)
if [ -z $bareosJobStatus ] ; then exit 13 ; fi
case $bareosJobStatus in
        "T") status=0 ;; (success)
        "W") status=1 ;; (warning)
        *) status=2 ;; (any other status)
esac

Bareos director and script are using the same remote mysql database

Catalog {
  Name = MyCatalog
  Reconnect = yes
  dbdriver = mysql
  dbname = bareos
  dbaddress = 10.1.1.1
  dbport = 3306
  dbuser = bareos
  dbpassword = "bareos"
}

All works perfectly for version 16, but after upgrade bareos to version 17RC, I've got a lot of zabbix alerts about failed backup jobs. The failed jobs has random order (aka race conditional)
After investigation, I've found that jobs have status "T" (success) and zabbix alerts are false positive, also zabbix.sh script got status "R" (running) for the same backup jobs.

My test
1) Run some lightweight backup job, job is is 1111 for example, bareos director and mysql database have no load
2) Job finished successfully with status OK, message appears in bconsole.
3) job with ID 1111 still has status "R" (running) in mysql database.
4) zabbix.sh script got status "R" for job id 1111 and exit with code 2 (failed)
5) After some time (1-2 seconds), job id 1111 changed status to "T" (success) in mysql database.

Issue History

Date Modified Username Field Change
2017-10-17 13:36 chaos_prevails New Issue
2017-11-16 12:46 Shodan Note Added: 0002816