Borg does not have a public API on the Python level. That does not keep you from writing import borg
,
but does mean that there are no release-to-release guarantees on what you might find in that package, not
even for point releases (1.1.x), and there is no documentation beyond the code and the internals documents.
Borg does on the other hand provide an API on a command-line level. In other words, a frontend should (for example) create a backup archive just invoke borg create, give commandline parameters/options as needed and parse JSON output from borg.
Important: JSON output is expected to be UTF-8, but currently borg depends on the locale being configured for that (must be a UTF-8 locale and not “C” or “ascii”), so that Python will choose to encode to UTF-8. The same applies to any inputs read by borg, they are expected to be UTF-8 encoded also.
We consider this a bug (see #2273) and might fix it later, so borg will use UTF-8 independent of the locale.
On POSIX systems, you can usually set environment vars to choose a UTF-8 locale:
export LANG=en_US.UTF-8
export LC_CTYPE=en_US.UTF-8
Especially for graphical frontends it is important to be able to convey and reformat progress information
in meaningful ways. The --log-json
option turns the stderr stream of Borg into a stream of JSON lines,
where each line is a JSON object. The type key of the object determines its other contents.
Warning
JSON logging requires successful argument parsing. Even with --log-json
specified, a
parsing error will be printed in plain text, because logging set-up happens after all arguments are
parsed.
Since JSON can only encode text, any string representing a file system path may miss non-text parts.
The following types are in use. Progress information is governed by the usual rules for progress information,
it is not produced unless --progress
is specified.
Output during operations creating archives (borg create and borg recreate). The following keys exist, each represents the current progress.
Original size of data processed so far (before compression and deduplication, may be empty/absent)
Compressed size (may be empty/absent)
Deduplicated size (may be empty/absent)
Number of (regular) files processed so far (may be empty/absent)
Current path (may be empty/absent)
Unix timestamp (float)
boolean indicating whether the operation has finished, only the last object for an operation can have this property set to true.
A message-based progress information with no concrete progress information, just a message saying what is currently being worked on.
unique, opaque integer ID of the operation
Message ID of the operation (may be null)
boolean indicating whether the operation has finished, only the last object for an operation can have this property set to true.
current progress message (may be empty/absent)
Unix timestamp (float)
Absolute progress information with defined end/total and current value.
unique, opaque integer ID of the operation
Message ID of the operation (may be null)
boolean indicating whether the operation has finished, only the last object for an operation can have this property set to true.
A formatted progress message, this will include the percentage and perhaps other information (absent for finished == true)
Current value (always less-or-equal to total, absent for finished == true)
Array that describes the current item, may be null, contents depend on msgid (absent for finished == true)
Total value (absent for finished == true)
Unix timestamp (float)
This is only output by borg create and borg recreate if --list
is specified. The usual
rules for the file listing applies, including the --filter
option.
Single-character status as for regular list output
Path of the file system object
Any regular log output invokes this type. Regular log options and filtering applies to these as well.
Unix timestamp (float)
Upper-case log level name (also called severity). Defined levels are: DEBUG, INFO, WARNING, ERROR, CRITICAL
Name of the emitting entity
Formatted log message
Message ID, may be null or absent
See Prompts for the types used by prompts.
Examples (reformatted, each object would be on exactly one line)
borg extract progress:
{"message": "100.0% Extracting: src/borgbackup.egg-info/entry_points.txt",
"current": 13000228, "total": 13004993, "info": ["src/borgbackup.egg-info/entry_points.txt"],
"operation": 1, "msgid": "extract", "type": "progress_percent", "finished": false}
{"message": "100.0% Extracting: src/borgbackup.egg-info/SOURCES.txt",
"current": 13004993, "total": 13004993, "info": ["src/borgbackup.egg-info/SOURCES.txt"],
"operation": 1, "msgid": "extract", "type": "progress_percent", "finished": false}
{"operation": 1, "msgid": "extract", "type": "progress_percent", "finished": true}
borg create file listing with progress:
{"original_size": 0, "compressed_size": 0, "deduplicated_size": 0, "nfiles": 0, "type": "archive_progress", "path": "src"}
{"type": "file_status", "status": "U", "path": "src/borgbackup.egg-info/entry_points.txt"}
{"type": "file_status", "status": "U", "path": "src/borgbackup.egg-info/SOURCES.txt"}
{"type": "file_status", "status": "d", "path": "src/borgbackup.egg-info"}
{"type": "file_status", "status": "d", "path": "src"}
{"original_size": 13176040, "compressed_size": 11386863, "deduplicated_size": 503, "nfiles": 277, "type": "archive_progress", "path": ""}
Internal transaction progress:
{"message": "Saving files cache", "operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": false}
{"message": "Saving cache config", "operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": false}
{"message": "Saving chunks cache", "operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": false}
{"operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": true}
A debug log message:
{"message": "35 self tests completed in 0.08 seconds",
"type": "log_message", "created": 1488278449.5575905, "levelname": "DEBUG", "name": "borg.archiver"}
Prompts assume a JSON form as well when the --log-json
option is specified. Responses
are still read verbatim from stdin, while prompts are JSON messages printed to stderr,
just like log messages.
Prompts use the question_prompt and question_prompt_retry types for the prompt itself, and question_invalid_answer, question_accepted_default, question_accepted_true, question_accepted_false and question_env_answer types for information about prompt processing.
The message property contains the same string displayed regularly in the same situation, while the msgid property may contain a msgid, typically the name of the environment variable that can be used to override the prompt. It is the same for all JSON messages pertaining to the same prompt.
Examples (reformatted, each object would be on exactly one line)
Providing an invalid answer:
{"type": "question_prompt", "msgid": "BORG_CHECK_I_KNOW_WHAT_I_AM_DOING",
"message": "... Type 'YES' if you understand this and want to continue: "}
incorrect answer # input on stdin
{"type": "question_invalid_answer", "msgid": "BORG_CHECK_I_KNOW_WHAT_I_AM_DOING", "is_prompt": false,
"message": "Invalid answer, aborting."}
Providing a false (negative) answer:
{"type": "question_prompt", "msgid": "BORG_CHECK_I_KNOW_WHAT_I_AM_DOING",
"message": "... Type 'YES' if you understand this and want to continue: "}
NO # input on stdin
{"type": "question_accepted_false", "msgid": "BORG_CHECK_I_KNOW_WHAT_I_AM_DOING",
"message": "Aborting.", "is_prompt": false}
Providing a true (affirmative) answer:
{"type": "question_prompt", "msgid": "BORG_CHECK_I_KNOW_WHAT_I_AM_DOING",
"message": "... Type 'YES' if you understand this and want to continue: "}
YES # input on stdin
# no further output, just like the prompt without --log-json
Passphrase prompts should be handled differently. Use the environment variables BORG_PASSPHRASE and BORG_NEW_PASSPHRASE (see Environment Variables for reference) to pass passphrases to Borg, don’t use the interactive passphrase prompts.
When setting a new passphrase (borg init, borg key change-passphrase) normally Borg prompts whether it should display the passphrase. This can be suppressed by setting the environment variable BORG_DISPLAY_PASSPHRASE to no.
When “confronted” with an unknown repository, where the application does not know whether the repository is encrypted, the following algorithm can be followed to detect encryption:
Set BORG_PASSPHRASE to gibberish (for example a freshly generated UUID4, which cannot possibly be the passphrase)
Invoke borg list repository ...
If this fails, due the repository being encrypted and the passphrase obviously being wrong, you’ll get an error with the PassphraseWrong msgid.
The repository is encrypted, for further access the application will need the passphrase.
If this does not fail, then the repository is not encrypted.
stdout is different and more command-dependent than logging. Commands like borg info, borg create
and borg list implement a --json
option which turns their regular output into a single JSON object.
Some commands, like borg list and borg diff, can produce a lot of JSON. Since many JSON implementations
don’t support a streaming mode of operation, which is pretty much required to deal with this amount of JSON, these
commands implement a --json-lines
option which generates output in the JSON lines format,
which is simply a number of JSON objects separated by new lines.
Dates are formatted according to ISO 8601 in local time. No explicit time zone is specified at this time
(subject to change). The equivalent strftime format string is ‘%Y-%m-%dT%H:%M:%S.%f’,
e.g. 2017-08-07T12:27:20.123456
.
The root object of ‘--json’ output will contain at least a repository key with an object containing:
The ID of the repository, normally 64 hex characters
Canonicalized repository path, thus this may be different from what is specified on the command line
Date when the repository was last modified by the Borg client
The encryption key, if present, contains:
Textual encryption mode name (same as borg init --encryption
names)
Path to the local key file used for access. Depending on mode this key may be absent.
The cache key, if present, contains:
Path to the local repository cache
Object containing cache stats:
Number of chunks
Number of unique chunks
Total uncompressed size of all chunks multiplied with their reference counts
Total compressed and encrypted size of all chunks multiplied with their reference counts
Uncompressed size of all chunks
Compressed and encrypted size of all chunks
Example borg info output:
{
"cache": {
"path": "/home/user/.cache/borg/0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
"stats": {
"total_chunks": 511533,
"total_csize": 17948017540,
"total_size": 22635749792,
"total_unique_chunks": 54892,
"unique_csize": 1920405405,
"unique_size": 2449675468
}
},
"encryption": {
"mode": "repokey"
},
"repository": {
"id": "0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
"last_modified": "2017-08-07T12:27:20.789123",
"location": "/home/user/testrepo"
},
"security_dir": "/home/user/.config/borg/security/0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
"archives": []
}
borg info uses an extended format for archives, which is more expensive to retrieve, while borg list uses a simpler format that is faster to retrieve. Either return archives in an array under the archives key, while borg create returns a single archive object under the archive key.
Both formats contain a name key with the archive name, the id key with the hexadecimal archive ID, and the start key with the start timestamp.
borg info and borg create further have:
End timestamp
Duration in seconds between start and end in seconds (float)
Archive statistics (freshly calculated, this is what makes “info” more expensive)
Size of files and metadata before compression
Size after compression
Deduplicated size (against the current repository, not when the archive was created)
Number of regular files in the archive
Object describing the utilization of Borg limits
Float between 0 and 1 describing how large this archive is relative to the maximum size allowed by Borg
Array of strings of the command line that created the archive
The note about paths from above applies here as well.
The chunker parameters the archive has been created with.
borg info further has:
Hostname of the creating host
Name of the creating user
Archive comment, if any
Some keys/values are more expensive to compute than others (e.g. because it requires opening the archive, not just the manifest). To optimize for speed, borg list repo does not determine these values except when they are requested. The --format option is used for that (for normal mode as well as for --json mode), so, to have the comment included in the json output, you will need:
borg list repo --format "{name}{comment}" --json`
Example of a simple archive listing (borg list --last 1 --json
):
{
"archives": [
{
"id": "80cd07219ad725b3c5f665c1dcf119435c4dee1647a560ecac30f8d40221a46a",
"name": "host-system-backup-2017-02-27",
"start": "2017-08-07T12:27:20.789123"
}
],
"encryption": {
"mode": "repokey"
},
"repository": {
"id": "0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
"last_modified": "2017-08-07T12:27:20.789123",
"location": "/home/user/repository"
}
}
The same archive with more information (borg info --last 1 --json
):
{
"archives": [
{
"chunker_params": [
"buzhash",
13,
23,
16,
4095
],
"command_line": [
"/home/user/.local/bin/borg",
"create",
"/home/user/repository",
"..."
],
"comment": "",
"duration": 5.641542,
"end": "2017-02-27T12:27:20.789123",
"hostname": "host",
"id": "80cd07219ad725b3c5f665c1dcf119435c4dee1647a560ecac30f8d40221a46a",
"limits": {
"max_archive_size": 0.0001330855110409714
},
"name": "host-system-backup-2017-02-27",
"start": "2017-02-27T12:27:20.789123",
"stats": {
"compressed_size": 1880961894,
"deduplicated_size": 2791,
"nfiles": 53669,
"original_size": 2400471280
},
"username": "user"
}
],
"cache": {
"path": "/home/user/.cache/borg/0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
"stats": {
"total_chunks": 511533,
"total_csize": 17948017540,
"total_size": 22635749792,
"total_unique_chunks": 54892,
"unique_csize": 1920405405,
"unique_size": 2449675468
}
},
"encryption": {
"mode": "repokey"
},
"repository": {
"id": "0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
"last_modified": "2017-08-07T12:27:20.789123",
"location": "/home/user/repository"
}
}
Each archive item (file, directory, …) is described by one object in the borg list output. Refer to the borg list documentation for the available keys and their meaning.
Example (excerpt) of borg list --json-lines
:
{"type": "d", "mode": "drwxr-xr-x", "user": "user", "group": "user", "uid": 1000, "gid": 1000, "path": "linux", "healthy": true, "source": "", "linktarget": "", "flags": null, "mtime": "2017-02-27T12:27:20.023407", "size": 0}
{"type": "d", "mode": "drwxr-xr-x", "user": "user", "group": "user", "uid": 1000, "gid": 1000, "path": "linux/baz", "healthy": true, "source": "", "linktarget": "", "flags": null, "mtime": "2017-02-27T12:27:20.585407", "size": 0}
Each archive difference item (file contents, user/group/mode) output by borg diff is represented by an ItemDiff object. The propertiese of an ItemDiff object are:
The filename/path of the Item (file, directory, symlink).
A list of Change objects describing the changes made to the item in the two archives. For example, there will be two changes if the contents of a file are changed, and its ownership are changed.
The Change object can contain a number of properties depending on the type of change that occurred. If a ‘property’ is not required for the type of change, it is not output. The possible properties of a Change object are:
The type property is always present. It identifies the type of change and will be one of these values:
modified - file contents changed.
added - the file was added.
removed - the file was removed.
added directory - the directory was added.
removed directory - the directory was removed.
added link - the symlink was added.
removed link - the symlink was removed.
changed link - the symlink target was changed.
mode - the file/directory/link mode was changed. Note - this could indicate a change from a file/directory/link type to a different type (file/directory/link), such as -- a file is deleted and replaced with a directory of the same name.
owner - user and/or group ownership changed.
If type == ‘added’ or ‘removed’, then size provides the size of the added or removed file.
If type == ‘modified’ and chunk ids can be compared, then added and removed indicate the amount of data ‘added’ and ‘removed’. If chunk ids can not be compared, then added and removed properties are not provided and the only information available is that the file contents were modified.
See added property.
If type == ‘mode’, then old_mode and new_mode provide the mode and permissions changes.
See old_mode property.
If type == ‘owner’, then old_user, new_user, old_group and new_group provide the user and group ownership changes.
See old_user property.
See old_user property.
See old_user property.
Example (excerpt) of borg diff --json-lines
:
{"path": "file1", "changes": [{"path": "file1", "changes": [{"type": "modified", "added": 17, "removed": 5}, {"type": "mode", "old_mode": "-rw-r--r--", "new_mode": "-rwxr-xr-x"}]}]}
{"path": "file2", "changes": [{"type": "modified", "added": 135, "removed": 252}]}
{"path": "file4", "changes": [{"type": "added", "size": 0}]}
{"path": "file3", "changes": [{"type": "removed", "size": 0}]}
Message IDs are strings that essentially give a log message or operation a name, without actually using the full text, since texts change more frequently. Message IDs are unambiguous and reduce the need to parse log messages.
Assigned message IDs and related error RCs (exit codes) are:
Error: {}
Error: {}
Requested buffer size {} is above the limit of {}.
Could not pop_front first {} elements, collection only has {} elements..
Runtime Error: {}
Cancelled by user.
Command Error: {}
Formatting Error: “{}”.format({}): {}({})
Invalid placeholder “{}” in string: {}
A repository already exists at {}.
Attic repository detected. Please run “borg upgrade {}”.
Inconsistency detected. Please run “borg check {}”.
Repository {} does not exist.
Insufficient free space to complete transaction (required: {}, available: {}).
{} is not a valid repository. Check repo config.
{} does not have a valid configuration. Check repo config [{}].
Object with key {} not found in repository {}.
The parent path of the repo directory [{}] does not exist.
There is already something at {}.
The storage quota ({}) has been exceeded ({}). Try deleting some archives.
Permission denied to {}.
Unsupported repository feature(s) {}. A newer version of borg is required to access this repository.
Repository has no manifest.
Unsupported manifest envelope. A newer version is required to access this repository.
Archive {} already exists
Archive {} does not exist
Failed to encode filename “{}” into file system encoding “{}”. Consider configuring the LANG environment variable.
Invalid key data for repository {} found in {}.
Mismatch between repository {} and key file {}.
No key file for repository {} found in {}.
This file is not a borg key backup, aborting.
No key entry found in the config of repository {}.
This key backup seems to be for a different backup repository, aborting.
Key management not available for unencrypted repositories.
Key type {0} is unknown.
Unsupported payload type {}. A newer version is required to access this repository.
can not acquire a passphrase: {}
passcommand supplied in BORG_PASSCOMMAND failed: {}
passphrase supplied in BORG_PASSPHRASE, by BORG_PASSCOMMAND or via BORG_PASSPHRASE_FD is incorrect.
exceeded the maximum password retries
Cache initialization aborted
Repository encryption method changed since last access, refusing to continue
Repository access aborted
Cache is newer than repository - do you have multiple, independently updated repos with same ID?
Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)
Failed to acquire the lock {}.
Failed to acquire the lock {}.
Failed to create/acquire the lock {} ({}).
Failed to create/acquire the lock {} (timeout).
Failed to release the lock {} (was not locked).
Failed to release the lock {} (was/is locked, but not by me).
Connection closed by remote host
Connection closed by remote host. {}
RPC method {} is not valid
Repository path not allowed: {}
Borg server is too old for {}. Required version {}
Borg {}: Got unexpected RPC data format from client.
Got unexpected RPC data format from server: {}
Connection to remote host is broken. {}
Data integrity error: {}
File failed integrity check: {}
Decompression error: {}
Data integrity error: {}
Archive ‘{}’ is unauthenticated, but it is required for this repository.
Data integrity error: {}
Manifest is unauthenticated, but it is required for this repository.
Could not verify manifest: Unsupported suite {!r}; a newer version is needed.
Warning: {}
{}: {}
{}: file changed while we backed it up
Include pattern ‘{}’ never matched.
{}: backup error
{}: file type or inode changed while we backed it up (race condition, skipped file)
{}: {}
{}: {}
{}: {}
{}: {}
cache.begin_transaction
cache.download_chunks, appears with borg create --no-cache-sync
cache.commit
cache.sync
info is one string element, the name of the archive currently synced.
repository.compact_segments
repository.replay_segments
repository.check
check.verify_data
check.rebuild_manifest
check.rebuild_refcounts
extract
info is one string element, the name of the path currently extracted.
extract.permissions
archive.delete
archive.calc_stats
prune
upgrade.convert_segments
For “Warning: Attempting to access a previously unknown unencrypted repository”
For “Warning: The repository at location … was previously located at …”
For “This is a potentially dangerous function…” (check --repair)
For “You requested to completely DELETE the repository including all archives it contains:”