View Issue Details

IDProjectCategoryView StatusLast Update
0000719bareos-coredirectorpublic2017-12-18 12:12
Reporterhop Assigned To 
PriorityhighSeveritymajorReproducibilityalways
Status closedResolutionfixed 
PlatformLinuxOSUbuntuOS Version14.04
Product Version16.2.4 
Fixed in Version17.2.4 
Summary0000719: ERROR in output_formatter.c:822 Failed to send result as json.
DescriptionWebui fails.


01-Nov 19:05 qdehd4-dir: ERROR in output_formatter.c:822 Failed to send result as json. Maybe result message to long?
Message length = 3318667

I did already read those other simmilar issues, and did learn, it was a "design decision". BUT I WANT TO DOUBT THAT DECISION HERE.

HOW can it be, that the GUI defines how many volumes I have to have?
(Or, as others wrote in the other reports, the number of backuped files?)

So either there has to be a parameter to configure the message-size, or it has to be set to a higher value.

Steps To Reproduceuse "too" much volumes
Additional InformationAnd maybe, as a goodie, there could be a way to choose what the gui shows (and therefor loads/sends) when opening the "volumes" page.
TagsNo tags attached.

Relationships

related to 0000794 closedstephand Release bareos-16.2.6 
child of 0000861 closedjoergs Release bareos-17.2.4 

Activities

joergs

joergs

2016-11-02 10:21

developer   ~0002424

The maximum message length for TCP connections is 1000000 bytes, see https://github.com/bareos/bareos/blob/master/src/lib/bsock_tcp.h#L40

There have been a change that with 16.2 arbitrary message length are possible, however, other issues did arise from this, so it have been removed again.

This is not a limitation of the webui, but of the Director->Console->JSON-Backend.

How many volumes do you use? Have you updated from earlier versions? If yes, this the problem did not occur there?
hop

hop

2016-11-02 11:11

reporter   ~0002425

now we see none at all, which is worse than with the version 15.2, where the error Message popped up, but the gui still showed some (all? I didn't count them :-) )

But if it is a problem in the backend, why do I never have those issues with bconsole ??
joergs

joergs

2016-11-02 11:14

developer   ~0002426

You will have the same error, when you set the bconsole in JSON mode (command: ".api json") und sent the same command (llist volumes?).
hop

hop

2016-11-03 11:04

reporter   ~0002428

Last edited: 2016-11-03 13:39

Which shows, either the (json) platform or the way it is used is not suitable for this backup tool. Backups and especially their reports of the backuped files etc. can be VERY, VERY long. A backup-tool has to be able to cope with that.
(BTW: Bareos fails in sending the Backup report, too. As this mail intends to have the number of volumes purged/used inside)
It is "not the best thing", that a backup tool gets irritated by the number of files, volumes or whatever makes status or index reports "too long" for the tool.
How can i be sure, i restored everything... etc etc

(Note edited by author to be less offensive)

magedshaker

magedshaker

2016-11-21 17:09

reporter   ~0002450

I have the same issue , so i still have the same error when make go to restore tab and browser incremental backup , so all full backup are listed properly you see below
21-Nov 16:29 bdr01-dir: ERROR in output_formatter.c:822 Failed to send result as json. Maybe result message to long?
Message length = 582
21-Nov 16:33 bdr01-dir: ERROR in output_formatter.c:822 Failed to send result as json. Maybe result message to long?
Message length = 582
21-Nov 17:00 bdr01-dir: ERROR in output_formatter.c:822 Failed to send result as json. Maybe result message to long?
Message length = 582
21-Nov 17:05 bdr01-dir: ERROR in output_formatter.c:822 Failed to send result as json. Maybe result message to long?
Message length = 582
21-Nov 17:45 bdr01-dir: ERROR in output_formatter.c:822 Failed to send result as json. Maybe result message to long?
Message length = 317
magedshaker

magedshaker

2016-11-21 17:11

reporter   ~0002451

Also that happend when update to version 16.2
# @/etc/bareos/bareos-dir.d/webui-profiles.conf
#

#
# bareos-webui default profile resource
#
Profile {
  Name = webui
  CommandACL = !.bvfs_clear_cache, !.exit, !.sql, !configure, !create, !delete, !purge, !sqlquery, !umount, !unmount, *all*
joergs

joergs

2016-11-22 11:06

developer   ~0002452

@magedshaher: also the same code issues the error message, your problem is not related to the original problem of this ticket.
The original problem is about too long messages (Message length = 3318667). You logs to only show short messages, that can not be sent for unknown reasons.

Maybe you can write a message to the bareos-users mailing list and explain,
  * when the problem started
  * what exactly happend
Ryushin

Ryushin

2017-01-26 15:49

reporter   ~0002532

Last edited: 2017-01-26 15:51

I'm also seeing the same error on two different installations.
Installation one has 1680 volumes and installation two has 1336 volumes.

Backing up to multiple 8TB drives with 50GB volumes using vchanger. I'm seeing this error when trying to view the pools or volumes under the storages tab.

Running version: 16.4.3.1482412748 on both installations.

BTW, as time goes on, we will be adding many more 8TB drives. So we could easily see 10,000 volumes as time goes on.

penn

penn

2017-02-09 07:59

reporter   ~0002553

I have the same Error when exec command "llist jobs".
My bareos has 100 vms, and backup job will trigger 1hour every day
Signum

Signum

2017-03-06 16:53

reporter   ~0002602

Same here. "llist jobs days=3" is often sufficient to trigger the error. It also happens when I want to see the contents of our tape library. I just tried the experimental bareos-webui-16.4.3.1486047414.2712657 version and it still has the problem.

In my opinion this is priority "block" or at least "high" because it make the UI barely usable on non-trivial sites.
frank

frank

2017-03-17 11:11

developer   ~0002612

I'm going to change the category of the report to "director", as it is the source of the limitation and we need to fix it there first.
pazion

pazion

2017-03-30 23:15

reporter   ~0002619

Last edited: 2017-03-30 23:17

Not sure if I hit the same error ( ERROR in output_formatter.c:822... Message length = 5312), but maybe found a way to fix.

1. I found out when I did not merge filesets it worked. This made me believe it might have to do something with the processing size/time.
2.a. I changed the max_input_values in php.ini ( is 1000 default ) to a high number.
2.b. Changed max_execution_time in php.ini
2.c. Ran .bvfs_update command in bconsole.
3. I am not sure what action ins step 2 triggered it, but error is gone

Maybe it helps someone.

pazion

pazion

2017-04-13 11:13

reporter   ~0002626

Turns out issue was not solved with my above comments.

13-Apr 10:56 bareos-dir: ERROR in output_formatter.c:822 Failed to send result as json. Maybe result message to long?
Message length = 155
asalvo

asalvo

2017-04-26 01:16

reporter   ~0002631

Probably related... this occurs, I think, when there are a lot of volumes in the Storages->Volumes tab.

An error occurred
An error occurred during execution; please try again later.
Additional information:
Zend\Json\Exception\RuntimeException
File:
/usr/share/bareos-webui/vendor/zendframework/zend-json/src/Json.php:66
Message:
Decoding failed: Unexpected control character found
Stack trace:
#0 /usr/share/bareos-webui/module/Media/src/Media/Model/MediaModel.php(36): Zend\Json\Json::decode('{ "jsonrpc": "2...', 1)
0000001 /usr/share/bareos-webui/module/Media/src/Media/Controller/MediaController.php(63): Media\Model\MediaModel->getVolumes(Object(Bareos\BSock\BareosBSock))
0000002 /usr/share/bareos-webui/vendor/zendframework/zend-mvc/src/Controller/AbstractActionController.php(82): Media\Controller\MediaController->indexAction()
0000003 [internal function]: Zend\Mvc\Controller\AbstractActionController->onDispatch(Object(Zend\Mvc\MvcEvent))
0000004 /usr/share/bareos-webui/vendor/zendframework/zend-eventmanager/src/EventManager.php(444): call_user_func(Array, Object(Zend\Mvc\MvcEvent))
0000005 /usr/share/bareos-webui/vendor/zendframework/zend-eventmanager/src/EventManager.php(205): Zend\EventManager\EventManager->triggerListeners('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
0000006 /usr/share/bareos-webui/vendor/zendframework/zend-mvc/src/Controller/AbstractController.php(118): Zend\EventManager\EventManager->trigger('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
0000007 /usr/share/bareos-webui/vendor/zendframework/zend-mvc/src/DispatchListener.php(93): Zend\Mvc\Controller\AbstractController->dispatch(Object(Zend\Http\PhpEnvironment\Request), Object(Zend\Http\PhpEnvironment\Response))
0000008 [internal function]: Zend\Mvc\DispatchListener->onDispatch(Object(Zend\Mvc\MvcEvent))
0000009 /usr/share/bareos-webui/vendor/zendframework/zend-eventmanager/src/EventManager.php(444): call_user_func(Array, Object(Zend\Mvc\MvcEvent))
0000010 /usr/share/bareos-webui/vendor/zendframework/zend-eventmanager/src/EventManager.php(205): Zend\EventManager\EventManager->triggerListeners('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
0000011 /usr/share/bareos-webui/vendor/zendframework/zend-mvc/src/Application.php(314): Zend\EventManager\EventManager->trigger('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
0000012 /usr/share/bareos-webui/public/index.php(24): Zend\Mvc\Application->run()
0000013 {main}


I also get an email with this message:
25-Apr 22:54 bareos-dir: ERROR in output_formatter.c:822 Failed to send result as json. Maybe result message to long?
Message length = 4994996
romans

romans

2017-05-19 08:47

reporter   ~0002642

We use one volume for every backup/job and according to it we have the same error.
Is there any ETA to get this issue fixed or any recommendations/workaround how to get it working now ?
joergs

joergs

2017-09-02 00:27

developer   ~0002727

Fix committed to bareos master branch with changesetid 7176.

Related Changesets

bareos: master 42d7e719

2017-06-30 20:18

joergs

Ported: N/A

Details Diff
bsock: allow sending long messages

While API modes 0 and 1 normally send data as small chunks (often line
based), API mode 2 (JSON) collects the result of a bcommand, forms a
JSON object and sends it. This can result in large messages.

The maximum valid size of the result of a bcommand used to be 4 MB.
The size of "llist jobs limit=1" is already close to 1 KB.
So before this patch, "llist jobs" silently fails if the Bareos
Director returns more than 5000 jobs.

With this patch, the maximum length of a bcommand result is int32 (2GB).

This is a modified version of commit 84ba7730bb4799f0793b2c0ee8572c5c75970ffe,
which had to be reverted because of problems.

Fixes 0000719: ERROR in output_formatter.c:822 Failed to send result as json.
Affected Issues
0000719
mod - src/lib/bsock_tcp.c Diff File

Issue History

Date Modified Username Field Change
2016-11-01 19:49 hop New Issue
2016-11-02 10:21 joergs Note Added: 0002424
2016-11-02 10:21 joergs Priority high => normal
2016-11-02 10:21 joergs Severity block => minor
2016-11-02 10:21 joergs Status new => feedback
2016-11-02 11:11 hop Note Added: 0002425
2016-11-02 11:11 hop Status feedback => new
2016-11-02 11:14 joergs Note Added: 0002426
2016-11-03 11:04 hop Note Added: 0002428
2016-11-03 13:39 hop Note Edited: 0002428
2016-11-21 17:09 magedshaker Note Added: 0002450
2016-11-21 17:11 magedshaker Note Added: 0002451
2016-11-22 11:06 joergs Note Added: 0002452
2017-01-26 15:49 Ryushin Note Added: 0002532
2017-01-26 15:51 Ryushin Note Edited: 0002532
2017-02-09 07:59 penn Note Added: 0002553
2017-03-02 13:22 frank Assigned To => frank
2017-03-02 13:22 frank Status new => assigned
2017-03-06 16:53 Signum Note Added: 0002602
2017-03-06 17:21 frank Priority normal => high
2017-03-06 17:21 frank Severity minor => major
2017-03-17 11:11 frank Note Added: 0002612
2017-03-17 11:11 frank Assigned To frank => joergs
2017-03-17 11:11 frank Category webui => director
2017-03-30 23:15 pazion Note Added: 0002619
2017-03-30 23:17 pazion Note Edited: 0002619
2017-04-13 11:13 pazion Note Added: 0002626
2017-04-26 01:16 asalvo Note Added: 0002631
2017-05-19 08:47 romans Note Added: 0002642
2017-06-16 15:08 stephand Relationship added related to 0000794
2017-09-02 00:27 joergs Changeset attached => bareos master 42d7e719
2017-09-02 00:27 joergs Note Added: 0002727
2017-09-02 00:27 joergs Status assigned => resolved
2017-09-02 00:27 joergs Resolution open => fixed
2017-12-18 12:06 joergs Relationship added child of 0000861
2017-12-18 12:12 joergs Status resolved => closed
2017-12-18 12:12 joergs Assigned To joergs =>
2017-12-18 12:12 joergs Fixed in Version => 17.2.4