View Issue Details

Related ChangesetsJump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001088bareos-coredirectorpublic2019-06-05 16:142019-12-18 15:24
Reporterstephand Assigned Tofrank  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Summary0001088: Integer out of range when using large amounts of files with Base Jobs
DescriptionWhen using Base Jobs (for details see https://docs.bareos.org/TasksAndConcepts/FileDeduplicationUsingBaseJobs.html) with large
amounts of files, after a longer of operation without problems, the jobs which are using a Base Jobs start to fail.
The job error message contains "Fatal error:" but does not contain the underlying database error message, but in the PostgreSQL
database logs, an "integer out of range" error is logged for the following kind of SQL statement:
INSERT INTO BaseFiles (BaseJobId, JobId, FileId, FileIndex) SELECT B.JobId AS BaseJobId, 89793 AS JobId, B.FileId, B.FileIndex FROM basefile89793 AS A, new_basefile86703 AS B WHERE A.Path = B.Path AND A.Name = B.Name ORDER BY B.FileId
Steps To ReproduceTo reproduce this issue, a lot of jobs based on Base Jobs must be run with large amounts of files.
Additional InformationCREATE TABLE basefiles
(
    BaseId SERIAL NOT NULL,
    JobId INTEGER NOT NULL,
    FileId BIGINT NOT NULL,
    FileIndex INTEGER,
    BaseJobId INTEGER,
    PRIMARY KEY (BaseId)
);

and for MySQL/MariaDB it is:

CREATE TABLE BaseFiles (
   BaseId INTEGER UNSIGNED AUTO_INCREMENT,
   BaseJobId INTEGER UNSIGNED NOT NULL REFERENCES Job,
   JobId INTEGER UNSIGNED NOT NULL REFERENCES Job,
   FileId BIGINT UNSIGNED NOT NULL REFERENCES File,
   FileIndex INTEGER UNSIGNED,
   PRIMARY KEY(BaseId)
);

In both cases, SERIAL and INTEGER is a 4 byte integer.

To fix this issue, the data type must be changed to 8 byte integer (BIGSERIAL/BIGINT).

For PostgreSQL this can be done manually by:
ALTER TABLE basefiles ALTER COLUMN baseid SET DATA TYPE bigint;

However, this should be fixed properly in the corresponding DDL files.
TagsNo tags attached.

Activities

frank

frank

2019-12-12 09:22

developer   ~0003658

Fix committed to bareos master branch with changesetid 12371.

Related Changesets

bareos: master a8b0f67e

2019-12-04 18:19

frank

Ported: N/A

Details Diff		 cats: Update DB schema from 2171 to 2192

- Add an INDEX on Job.JobTDate to improve performance (MySQL only)
- Fix integer out of range error by changing BaseFiles.BaseId to
BIGINT/BIGSERIAL
- Add new DDL update scripts for each supported DB backend
- Set BDB_VERSION in cats.h
- Remove unnecessary INDEX pathvisibility_jobid
- Collation handling PostgreSQL >= 10
- Update basefiles sequence PostgreSQL >= 10

Fixes 0001088: Integer out of range when using large amounts of files with Base Jobs
Fixes 0001061: Tremendous MySQL load		 Affected Issues
0001061, 0001088
mod - core/src/cats/cats.h Diff File
mod - core/src/cats/create_bareos_database.in Diff File
mod - core/src/cats/ddl/creates/mysql.sql Diff File
mod - core/src/cats/ddl/creates/postgresql.sql Diff File
mod - core/src/cats/ddl/creates/sqlite3.sql Diff File
add - core/src/cats/ddl/updates/mysql.2171_2192.sql Diff File
add - core/src/cats/ddl/updates/postgresql.2171_2192.sql Diff File
add - core/src/cats/ddl/updates/sqlite3.2171_2192.sql Diff File
mod - core/src/cats/ddl/versions.map.in Diff File
mod - core/src/cats/update_bareos_tables.in Diff File

Issue History

Date Modified Username Field Change
2019-06-05 16:14 stephand New Issue
2019-07-04 16:34 arogge Status new => acknowledged
2019-12-12 09:22 frank Changeset attached => bareos master a8b0f67e
2019-12-12 09:22 frank Note Added: 0003658
2019-12-12 09:22 frank Assigned To => frank
2019-12-12 09:22 frank Status acknowledged => resolved
2019-12-12 09:22 frank Resolution open => fixed
2019-12-18 15:24 arogge Status resolved => closed