The path/filenames used as input for the pattern matching start from the currently active recursion root. You usually give the recursion root(s) when invoking borg and these can be either relative or absolute paths.
So, when you give relative/ as root, the paths going into the matcher will look like relative/…/file.ext. When you give /absolute/ as root, they will look like /absolute/…/file.ext. This is meant when we talk about “full path” below.
File patterns support these styles: fnmatch, shell, regular expressions,
path prefixes and path full-matches. By default, fnmatch is used for
--exclude
patterns and shell-style is used for the experimental --pattern
option.
If followed by a colon (‘:’) the first two characters of a pattern are used as a style selector. Explicit style selection is necessary when a non-default style is desired or when the desired pattern starts with two alphanumeric characters followed by a colon (i.e. aa:something/*).
--exclude
and --exclude-from
.
These patterns use a variant of shell pattern syntax, with ‘*’ matching
any number of characters, ‘?’ matching any single character, ‘[…]’
matching any single character specified, including ranges, and ‘[!…]’
matching any character not specified. For the purpose of these patterns,
the path separator (backslash for Windows and ‘/’ on other systems) is not
treated specially. Wrap meta-characters in brackets for a literal
match (i.e. [?] to match the literal character ?). For a path
to match a pattern, the full path must match, or it must match
from the start of the full path to just before a path separator. Except
for the root path, paths will never end in the path separator when
matching is attempted. Thus, if a given pattern ends in a path
separator, a ‘*’ is appended before matching is attempted.--pattern
and --patterns-from
.
Like fnmatch patterns these are similar to shell patterns. The difference
is that the pattern may include **/ for matching zero or more directory
levels, * for matching zero or more arbitrary characters with the
exception of any path separator.This pattern style is (only) useful to match full paths. This is kind of a pseudo pattern as it can not have any variable or unspecified parts - the full path must be given. pf:root/file.ext matches root/file.txt only.
Implementation note: this is implemented via very time-efficient O(1) hashtable lookups (this means you can have huge amounts of such patterns without impacting performance much). Due to that, this kind of pattern does not respect any context or order. If you use such a pattern to include a file, it will always be included (if the directory recursion encounters it). Other include/exclude patterns that would normally match will be ignored. Same logic applies for exclude.
Note
re:, sh: and fm: patterns are all implemented on top of the Python SRE engine. It is very easy to formulate patterns for each of these types which requires an inordinate amount of time to match paths. If untrusted users are able to supply patterns, ensure they cannot supply re: patterns. Further, ensure that sh: and fm: patterns only contain a handful of wildcards at most.
Exclusions can be passed via the command line option --exclude
. When used
from within a shell the patterns should be quoted to protect them from
expansion.
The --exclude-from
option permits loading exclusion patterns from a text
file with one pattern per line. Lines empty or starting with the number sign
(‘#’) after removing whitespace on both ends are ignored. The optional style
selector prefix is also supported for patterns loaded from a file. Due to
whitespace removal paths with whitespace at the beginning or end can only be
excluded using regular expressions.
Examples:
# Exclude '/home/user/file.o' but not '/home/user/file.odt':
$ borg create -e '*.o' backup /
# Exclude '/home/user/junk' and '/home/user/subdir/junk' but
# not '/home/user/importantjunk' or '/etc/junk':
$ borg create -e '/home/*/junk' backup /
# Exclude the contents of '/home/user/cache' but not the directory itself:
$ borg create -e /home/user/cache/ backup /
# The file '/home/user/cache/important' is *not* backed up:
$ borg create -e /home/user/cache/ backup / /home/user/cache/important
# The contents of directories in '/home' are not backed up when their name
# ends in '.tmp'
$ borg create --exclude 're:^/home/[^/]+\.tmp/' backup /
# Load exclusions from file
$ cat >exclude.txt <<EOF
# Comment line
/home/*/junk
*.tmp
fm:aa:something/*
re:^/home/[^/]\.tmp/
sh:/home/*/.thumbnails
EOF
$ borg create --exclude-from exclude.txt backup /
A more general and easier to use way to define filename matching patterns exists
with the experimental --pattern
and --patterns-from
options. Using these, you
may specify the backup roots (starting points) and patterns for inclusion/exclusion.
A root path starts with the prefix R, followed by a path (a plain path, not a
file pattern). An include rule starts with the prefix +, an exclude rule starts
with the prefix -, an exclude-norecurse rule starts with !, all followed by a pattern.
Inclusion patterns are useful to include paths that are contained in an excluded
path. The first matching pattern is used so if an include pattern matches before
an exclude pattern, the file is backed up. If an exclude-norecurse pattern matches
a directory, it won’t recurse into it and won’t discover any potential matches for
include rules below that directory.
Note that the default pattern style for --pattern
and --patterns-from
is
shell style (sh:), so those patterns behave similar to rsync include/exclude
patterns. The pattern style can be set via the P prefix.
Patterns (--pattern
) and excludes (--exclude
) from the command line are
considered first (in the order of appearance). Then patterns from --patterns-from
are added. Exclusion patterns from --exclude-from
files are appended last.
Examples:
# backup pics, but not the ones from 2018, except the good ones:
# note: using = is essential to avoid cmdline argument parsing issues.
borg create --pattern=+pics/2018/good --pattern=-pics/2018 repo::arch pics
# use a file with patterns:
borg create --patterns-from patterns.lst repo::arch
The patterns.lst file could look like that:
# "sh:" pattern style is the default, so the following line is not needed:
P sh
R /
# can be rebuild
- /home/*/.cache
# they're downloads for a reason
- /home/*/Downloads
# susan is a nice person
# include susans home
+ /home/susan
# don't backup the other home directories
- /home/*
Repository (or Archive) URLs, --prefix
and --remote-path
values support these
placeholders:
If literal curly braces need to be used, double them for escaping:
borg create /path/to/repo::{{literal_text}}
Examples:
borg create /path/to/repo::{hostname}-{user}-{utcnow} ...
borg create /path/to/repo::{hostname}-{now:%Y-%m-%d_%H:%M:%S} ...
borg prune --prefix '{hostname}-' ...
Note
systemd uses a difficult, non-standard syntax for command lines in unit files (refer to the systemd.unit(5) manual page).
When invoking borg from unit files, pay particular attention to escaping,
especially when using the now/utcnow placeholders, since systemd performs its own
%-based variable replacement even in quoted text. To avoid interference from systemd,
double all percent signs ({hostname}-{now:%Y-%m-%d_%H:%M:%S}
becomes {hostname}-{now:%%Y-%%m-%%d_%%H:%%M:%%S}
).
It is no problem to mix different compression methods in one repo, deduplication is done on the source data chunks (not on the compressed or encrypted data).
If some specific chunk was once compressed and stored into the repo, creating another backup that also uses this chunk will not change the stored chunk. So if you use different compression specs for the backups, whichever stores a chunk first determines its compression. See also borg recreate.
Compression is lz4 by default. If you want something else, you have to specify what you want.
Valid compression specifiers are:
Examples:
borg create --compression lz4 REPO::ARCHIVE data
borg create --compression zstd REPO::ARCHIVE data
borg create --compression zstd,10 REPO::ARCHIVE data
borg create --compression zlib REPO::ARCHIVE data
borg create --compression zlib,1 REPO::ARCHIVE data
borg create --compression auto,lzma,6 REPO::ARCHIVE data
borg create --compression auto,lzma ...