View Issue Details

IDProjectCategoryView StatusLast Update
0000334bareos-coredirectorpublic2015-03-25 19:19
Reporterogagnon Assigned To 
PrioritynormalSeverityfeatureReproducibilityN/A
Status closedResolutionfixed 
Fixed in Version14.2.0 
Summary0000334: Give access to job size to the plugins + enhancements/corrections
DescriptionHello,
I'm building a python plugin to output jobs summary in JSON to send it to an Elasticsearch cluster.

The JobBytes, ReadBytes and LastRate values were not available to the plugins of the Director.

Also, since the plugin needs to know about Canceled jobs, I moved the is_job_canceled() function after it passed through the plugins.

While testing my changes, I came unto a signed/unsigned problem with values >2^31 (0000002:0000002,100,000,000). Python does not sign its integers so when passing an unsigned value to Python, it needs to be made explicitly. I added a function to correctly send the added variables (JobBytes, ReadBytes and LastRate) to Python as unsigned but I also moved some variables sent as signed integer to Python while they may be over 2^31 to that function (*JobFiles and *Errors) since they are defined as unsigned in the C code.
Additional InformationAdding a feature to output messages in JSON or directly to Elasticsearch would be nice.

I can code it to make it native in Bareos instead of using a plugin to do it.
TagsNo tags attached.

Activities

ogagnon

ogagnon

2014-08-29 02:40

reporter  

add_var_avail_to_plugins.patch (3,747 bytes)   
diff --git a/src/dird/dir_plugins.c b/src/dird/dir_plugins.c
index e7a2559..3229d4f 100644
--- a/src/dird/dir_plugins.c
+++ b/src/dird/dir_plugins.c  
@@ -135,9 +135,10 @@ int generate_plugin_event(JCR *jcr, bDirEventType eventType, void *value, bool r
       return bRC_OK;                  /* Return if no plugins loaded */
    }
 
-   if (jcr->is_job_canceled()) {
-      return bRC_Cancel;
-   }
+   // Canceled jobs should pass through the plugin too
+   //if (jcr->is_job_canceled()) {
+   //   return bRC_Cancel;
+   //} 
 
    bpContext *plugin_ctx_list = (bpContext *)jcr->plugin_ctx_list;
    event.eventType = eventType; 
@@ -164,6 +165,10 @@ int generate_plugin_event(JCR *jcr, bDirEventType eventType, void *value, bool r
          }
       }
    }
+   
+   if (jcr->is_job_canceled()) {
+      return bRC_Cancel;
+   }
 
    return rc;
 }
@@ -506,6 +511,18 @@ static bRC bareosGetValue(bpContext *ctx, brDirVariable var, void *value)
          *((int *)value) = jcr->SDJobStatus;
          Dmsg1(dbglvl, "BAREOS: return bDirVarSDJobStatus=%c\n", jcr->SDJobStatus);
          break;
+      case bDirVarLastRate:
+         *((int *)value) = jcr->LastRate;
+         Dmsg1(dbglvl, "BAREOS: return bDirVarLastRate=%d\n", jcr->LastRate);
+         break;
+      case bDirVarJobBytes:
+         *((int *)value) = jcr->JobBytes;
+         Dmsg1(dbglvl, "BAREOS: return bDirVarJobBytes=%d\n", jcr->JobBytes);
+         break;
+      case bDirVarReadBytes:
+         *((int *)value) = jcr->ReadBytes;
+         Dmsg1(dbglvl, "BAREOS: return bDirVarReadBytes=%d\n", jcr->ReadBytes);
+         break;
       default:
          break;
       }
diff --git a/src/dird/dir_plugins.h b/src/dird/dir_plugins.h
index 2069793..cf3d02b 100644
--- a/src/dird/dir_plugins.h
+++ b/src/dird/dir_plugins.h
@@ -80,7 +80,10 @@ typedef enum {
   bDirVarSDErrors = 21, 
   bDirVarFDJobStatus = 22,
   bDirVarSDJobStatus = 23,
-  bDirVarPluginDir = 24 
+  bDirVarPluginDir = 24,
+  bDirVarLastRate = 25,
+  bDirVarJobBytes = 26,
+  bDirVarReadBytes = 27
 } brDirVariable;
   
 /*
diff --git a/src/plugins/dird/bareos_dir_consts.py b/src/plugins/dird/bareos_dir_consts.py
index 941adfe..d2a82f5 100644
--- a/src/plugins/dird/bareos_dir_consts.py
+++ b/src/plugins/dird/bareos_dir_consts.py
@@ -40,7 +40,10 @@ brDirVariable = dict(
     bDirVarSDJobFiles = 20, 
     bDirVarSDErrors = 21,   
     bDirVarFDJobStatus = 22,
-    bDirVarSDJobStatus = 23    
+    bDirVarSDJobStatus = 23,   
+    bDirVarLastRate = 25,      
+    bDirVarJobBytes = 26,            
+    bDirVarReadBytes = 27
 )
 
 bwDirVariable = dict(
diff --git a/src/plugins/dird/python-dir.c b/src/plugins/dird/python-dir.c
index e713205..f79f5af 100644
--- a/src/plugins/dird/python-dir.c
+++ b/src/plugins/dird/python-dir.c
@@ -545,10 +545,6 @@ static PyObject *PyBareosGetValue(PyObject *self, PyObject *args)
    case bDirVarNumVols:  
    case bDirVarJobStatus:
    case bDirVarPriority: 
-   case bDirVarJobErrors: 
-   case bDirVarJobFiles:  
-   case bDirVarSDJobFiles: 
-   case bDirVarSDErrors:     
    case bDirVarFDJobStatus:  
    case bDirVarSDJobStatus: {
       int value;
@@ -559,6 +555,21 @@ static PyObject *PyBareosGetValue(PyObject *self, PyObject *args)
       }
       break;
    }
+   case bDirVarJobErrors:
+   case bDirVarSDErrors:  
+   case bDirVarJobFiles:  
+   case bDirVarSDJobFiles:
+   case bDirVarLastRate:   
+   case bDirVarJobBytes:   
+   case bDirVarReadBytes: {
+      uint64_t value;
+
+      ctx = PyGetbpContext(pyCtx);
+      if (bfuncs->getBareosValue(ctx, (brDirVariable)var, &value) == bRC_OK) {
+         pRetVal = PyLong_FromUnsignedLong(value);
+      }
+      break;
+   }
    case bDirVarJobName:
    case bDirVarJob:  
    case bDirVarClient:

add_var_avail_to_plugins.patch (3,747 bytes)   
ogagnon

ogagnon

2014-08-29 09:58

reporter   ~0000966

Fix committed to bareos master branch with changesetid 2040.
ogagnon

ogagnon

2014-09-09 15:18

reporter   ~0000972

Fix committed to bareos bareos-14.2 branch with changesetid 2067.
mvwieringen

mvwieringen

2015-03-25 16:51

developer   ~0001463

Fix committed to bareos2015 bareos-14.2 branch with changesetid 4663.
joergs

joergs

2015-03-25 19:19

developer   ~0001613

Due to the reimport of the Github repository to bugs.bareos.org, the status of some tickets have been changed. These tickets will be closed again.
Sorry for the noise.

Related Changesets

bareos: master b96a3694

2014-08-29 08:54

ogagnon

Ported: N/A

Details Diff
Give access to job size to the plugins + enhancements/corrections

The JobBytes, ReadBytes and LastRate values were not available to the
plugins of the Director.

Also, since the plugin needs to know about Canceled jobs, I moved the
is_job_canceled() function after it passed through the plugins.

While testing my changes, I came unto a signed/unsigned problem with
values >2^31 (0000002:0000002,100,000,000). Python does not sign its
integers so when passing an unsigned value to Python, it needs to be
made explicitly. I added a function to correctly send the added
variables (JobBytes, ReadBytes and LastRate) to Python as unsigned but I
also moved some variables sent as signed integer to Python while they
may be over 2^31 to that function (*JobFiles and *Errors) since they are
defined as unsigned in the C code.

Fixes 0000334: Give access to job size to the plugins +
enhancements/corrections

Signed-off-by: Marco van Wieringen <marco.van.wieringen@bareos.com>
Affected Issues
0000334
mod - AUTHORS Diff File
mod - src/dird/dir_plugins.c Diff File
mod - src/dird/dir_plugins.h Diff File
mod - src/plugins/dird/bareos_dir_consts.py Diff File
mod - src/plugins/dird/python-dir.c Diff File

bareos: bareos-14.2 ab271606

2014-08-29 08:54

ogagnon

Ported: N/A

Details Diff
Give access to job size to the plugins + enhancements/corrections

The JobBytes, ReadBytes and LastRate values were not available to the
plugins of the Director.

Also, since the plugin needs to know about Canceled jobs, I moved the
is_job_canceled() function after it passed through the plugins.

While testing my changes, I came unto a signed/unsigned problem with
values >2^31 (0000002:0000002,100,000,000). Python does not sign its
integers so when passing an unsigned value to Python, it needs to be
made explicitly. I added a function to correctly send the added
variables (JobBytes, ReadBytes and LastRate) to Python as unsigned but I
also moved some variables sent as signed integer to Python while they
may be over 2^31 to that function (*JobFiles and *Errors) since they are
defined as unsigned in the C code.

Fixes 0000334: Give access to job size to the plugins +
enhancements/corrections

Signed-off-by: Marco van Wieringen <marco.van.wieringen@bareos.com>
Affected Issues
0000334
mod - AUTHORS Diff File
mod - src/dird/dir_plugins.c Diff File
mod - src/dird/dir_plugins.h Diff File
mod - src/plugins/dird/bareos_dir_consts.py Diff File
mod - src/plugins/dird/python-dir.c Diff File

bareos2015: bareos-14.2 5291834e

2014-08-29 10:54

ogagnon


Committer: mvwieringen

Ported: N/A

Details Diff
Give access to job size to the plugins + enhancements/corrections

The JobBytes, ReadBytes and LastRate values were not available to the
plugins of the Director.

Also, since the plugin needs to know about Canceled jobs, I moved the
is_job_canceled() function after it passed through the plugins.

While testing my changes, I came unto a signed/unsigned problem with
values >2^31 (0000002:0000002,100,000,000). Python does not sign its
integers so when passing an unsigned value to Python, it needs to be
made explicitly. I added a function to correctly send the added
variables (JobBytes, ReadBytes and LastRate) to Python as unsigned but I
also moved some variables sent as signed integer to Python while they
may be over 2^31 to that function (*JobFiles and *Errors) since they are
defined as unsigned in the C code.

Fixes 0000334: Give access to job size to the plugins +
enhancements/corrections

Signed-off-by: Marco van Wieringen <marco.van.wieringen@bareos.com>
Affected Issues
0000334
mod - AUTHORS Diff File
mod - src/dird/dir_plugins.c Diff File
mod - src/dird/dir_plugins.h Diff File
mod - src/plugins/dird/bareos_dir_consts.py Diff File
mod - src/plugins/dird/python-dir.c Diff File

Issue History

Date Modified Username Field Change
2014-08-29 02:40 ogagnon New Issue
2014-08-29 02:40 ogagnon File Added: add_var_avail_to_plugins.patch
2014-08-29 09:58 ogagnon Changeset attached => bareos master b96a3694
2014-08-29 09:58 ogagnon Note Added: 0000966
2014-08-29 09:58 ogagnon Assigned To => ogagnon
2014-08-29 09:58 ogagnon Status new => resolved
2014-08-29 09:58 ogagnon Resolution open => fixed
2014-09-09 15:18 ogagnon Changeset attached => bareos bareos-14.2 ab271606
2014-09-09 15:18 ogagnon Note Added: 0000972
2014-09-24 13:18 joergs Assigned To ogagnon =>
2014-09-24 13:18 joergs Status resolved => closed
2014-09-24 13:18 joergs Fixed in Version => 14.2.0
2015-03-25 16:51 mvwieringen Changeset attached => bareos2015 bareos-14.2 5291834e
2015-03-25 16:51 mvwieringen Note Added: 0001463
2015-03-25 16:51 mvwieringen Status closed => resolved
2015-03-25 19:19 joergs Note Added: 0001613
2015-03-25 19:19 joergs Status resolved => closed