Skip to content

Instantly share code, notes, and snippets.

@csarn

csarn/README.md

Last active May 19, 2024 20:21
Show Gist options
  • Save csarn/3084913ba3e78da6c4d511df131c4d5f to your computer and use it in GitHub Desktop.
Save csarn/3084913ba3e78da6c4d511df131c4d5f to your computer and use it in GitHub Desktop.
Download ZIP
ZFS pull backup with minimal permissions
Raw
README.md

Usage

on system that should be backed up

  • put "restrict_commands.sh" in /usr/local/bin and make it executable
  • install ts, lzop and optionally mbuffer
useradd zfsbackup --create-home --system
mkdir /home/zfsbackup/.ssh
zfs allow -u zfsbackup send,hold tank/dataset
echo 'restrict,command="restrict_commands.sh" ssh-ed25519 ...' > /home/zfsbackup/.ssh/authorized_keys
chown zfsbackup:zfsbackup /home/zfsbackup/.ssh -R

on backup server

run cronjob with:

syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw zfsbackup@target:tank/dataset tank/dataset
Raw
restrict_commands.sh
#!/bin/bash
export PATH=/usr/sbin:$PATH
_RE_DATASET=$'[\"\']+[a-z0-9/_]+[\"\']+'
_RE_SNAPSHOT=$'[\"\']+[a-z0-9/_]+[\"\']*@[\"\']*[a-z0-9/_:-]+[\"\']+'
_RE_SIZE=$'[0-9]+[kMG]?'
_WHITELIST=(
-e "exit"
-e "echo -n"
-e "zpool get -o value -H feature@extensible_dataset $_RE_DATASET"
-e "zfs get -H syncoid:sync $_RE_DATASET"
-e "zfs get -Hpd 1 -t snapshot guid,creation $_RE_DATASET"
-e " *zfs send -R -w -nv?P $_RE_SNAPSHOT"
-e " *zfs send -R -w +$_RE_SNAPSHOT \| +lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
-e " *zfs send -R -w -nv?P +-I $_RE_SNAPSHOT $_RE_SNAPSHOT"
-e " *zfs send -R -w +-I $_RE_SNAPSHOT $_RE_SNAPSHOT \| lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
-e "command -v (lzop|mbuffer)"
)
## LOG non-whitelisted commands, execute whitelisted
echo "$SSH_ORIGINAL_COMMAND" |\
tee >(egrep -x -v "${_WHITELIST[@]}" \
| ts "non-whitelisted command issued: (client $SSH_CLIENT)" \
| logger -p local0.crit \
) |\
egrep -x "${_WHITELIST[@]}" | bash
@rajil
Copy link

rajil commented Dec 27, 2021 •
edited

Thanks for the instructions. I am able to do a transfer without the restrict_commands.sh script. However, once i enable it i get an error,

# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshport=2222 --sshkey=/root/.ssh/mykey zfsbackup@masterserver.com:tank/stuff tank/stuff
Use of uninitialized value $sendsize in substitution (s///) at /usr/bin/syncoid line 1784.
Use of uninitialized value $sendsize in scalar chomp at /usr/bin/syncoid line 1788.
Use of uninitialized value $sendsize in pattern match (m//) at /usr/bin/syncoid line 1791.
Resuming interrupted zfs send/receive from tank/stuff to tank/stuff (~ UNKNOWN remaining):
lzop: <stdin>: not a lzop file
cannot receive: failed to read from stream
CRITICAL ERROR: ssh   -p 2222 -i /root/.ssh/mykey -S /tmp/syncoid-zfsbackup@masterserver.com-1640603651 zfsbackup@masterserver.com ' zfs send -w  -t **redacted** | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 0 |  zfs receive  -s -F 'tank/stuff' 2>&1 failed: 256 at /usr/bin/syncoid line 580.

Is the --sshport=2222 causing the shell script to fail?

@Mikesco3
Copy link

Thanks for the instructions. I am able to do a transfer without the restrict_commands.sh script. However, once i enable it i get an error,

# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshport=2222 --sshkey=/root/.ssh/mykey zfsbackup@masterserver.com:tank/stuff tank/stuff
Use of uninitialized value $sendsize in substitution (s///) at /usr/bin/syncoid line 1784.
Use of uninitialized value $sendsize in scalar chomp at /usr/bin/syncoid line 1788.
Use of uninitialized value $sendsize in pattern match (m//) at /usr/bin/syncoid line 1791.
Resuming interrupted zfs send/receive from tank/stuff to tank/stuff (~ UNKNOWN remaining):
lzop: <stdin>: not a lzop file
cannot receive: failed to read from stream
CRITICAL ERROR: ssh   -p 2222 -i /root/.ssh/mykey -S /tmp/syncoid-zfsbackup@masterserver.com-1640603651 zfsbackup@masterserver.com ' zfs send -w  -t **redacted** | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 0 |  zfs receive  -s -F 'tank/stuff' 2>&1 failed: 256 at /usr/bin/syncoid line 580.

Is the --sshport=2222 causing the shell script to fail?

if you created a user called zfsbackup shouldn't you be referencing ssh keys in the /home/zfsbackup/.ssh/mykey location?
because I can't help notice that you're referencing the root's ssh key, which the zfsbackup user isn't supposed to have access to.

@Mikesco3
Copy link

Mikesco3 commented Mar 22, 2022 •
edited

Sorry I'm kind of new to github, but I wanted to make a few suggestions

  1. On the readme.md file
    please add mkdir /home/zfsbackup/.ssh
    else the echo 'restrict,command="restrict_commands.sh" ssh-ed25519 ...' > /home/zfsbackup/.ssh/authorized_keys fails.

  2. On the restrict_commands.sh:
    There is no mention where to place this file, I ended up creating a .local/bin folder in the zfsbackup user's home path
    mkdir -p /home/zfsbackup/.local/bin
    and adding that path in line 2 of the restrict_commands.sh so that it reads
    export PATH=$PATH:$HOME/.local/bin:/usr/sbin
    instead of export PATH=/usr/sbin:$PATH

  3. Finally, if the restrict_commands.sh file fails with unknown command on line 26 error, it is because it is likely missing the ts command
    so in order to fix that I installed moreutils
    apt-get install moreutils

NOTE: just to be on the safe side it may be useful to ensure the user zfsbackup has permission to the folders we created on his home folder,
so to be safe run chown zfsbackup:zfsbackup /home/zfsbackup --recursive

@csarn
Copy link
Author

csarn commented Mar 23, 2022

Thanks for the instructions. I am able to do a transfer without the restrict_commands.sh script. However, once i enable it i get an error,

# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshport=2222 --sshkey=/root/.ssh/mykey zfsbackup@masterserver.com:tank/stuff tank/stuff
Use of uninitialized value $sendsize in substitution (s///) at /usr/bin/syncoid line 1784.
Use of uninitialized value $sendsize in scalar chomp at /usr/bin/syncoid line 1788.
Use of uninitialized value $sendsize in pattern match (m//) at /usr/bin/syncoid line 1791.
Resuming interrupted zfs send/receive from tank/stuff to tank/stuff (~ UNKNOWN remaining):
lzop: <stdin>: not a lzop file
cannot receive: failed to read from stream
CRITICAL ERROR: ssh   -p 2222 -i /root/.ssh/mykey -S /tmp/syncoid-zfsbackup@masterserver.com-1640603651 zfsbackup@masterserver.com ' zfs send -w  -t **redacted** | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 0 |  zfs receive  -s -F 'tank/stuff' 2>&1 failed: 256 at /usr/bin/syncoid line 580.

Is the --sshport=2222 causing the shell script to fail?

if you created a user called zfsbackup shouldn't you be referencing ssh keys in the /home/zfsbackup/.ssh/mykey location? because I can't help notice that you're referencing the root's ssh key, which the zfsbackup user isn't supposed to have access to.

Well no, the ssh key usage is correct. The backup job running on the backup server has to be run as root (it is not possible on linux to allow a non-root user to "zfs receive"). So root@backup has the private key, and logs into the server-to-be-backupped using the zfsbackup user account.

Thanks for your suggestions, I updated the readme accordingly. And I'll try to find out if running ssh on a different port will cause trouble.

@alembiq
Copy link

alembiq commented Apr 1, 2024

any luck? i'm stucked on the lzop error also :(

syncoid --sshkey=/var/lib/syncoid/kubera-2022-05-31 backup@10.10.10.10:zpool/nixos/etc tank/test --no-privilege-elevation --debug --sendoptions="w c " --sshoption=StrictHostKeyChecking=off
DEBUG: SSHCMD: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31
loginShellInit
DEBUG: checking availability of lzop on source...
DEBUG: checking availability of lzop on target...
DEBUG: checking availability of lzop on local machine...
DEBUG: checking availability of mbuffer on source...
DEBUG: checking availability of mbuffer on target...
DEBUG: checking availability of pv on local machine...
DEBUG: checking availability of zfs resume feature on source...
DEBUG: checking availability of zfs resume feature on target...
DEBUG: syncing source zpool/nixos/etc to target tank/test.
DEBUG: getting current value of syncoid:sync on zpool/nixos/etc...
ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs get -H syncoid:sync ''"'"'zpool/nixos/etc'"'"''
DEBUG: checking to see if tank/test on  is already in zfs receive using  ps -Ao args= ...
DEBUG: checking to see if target filesystem exists using "  zfs get -H name 'tank/test' 2>&1 |"...
DEBUG: getting list of snapshots on zpool/nixos/etc using ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs get -Hpd 1 -t snapshot guid,creation ''"'"'zpool/nixos/etc'"'"'' |...
DEBUG: creating sync snapshot using "ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs snapshot ''"'"'zpool/nixos/etc'"'"''@syncoid_kubera_2024-04-01:21:54:02-GMT02:00
"...
loginShellInit
DEBUG: target tank/test does not exist.  Finding oldest available snapshot on source zpool/nixos/etc ...
DEBUG: getting estimated transfer size from source -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 using "ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs send -w -c  -nvP ''"'"'zpool/nixos/etc@autosnap_2024-03-28_22:13:01_monthly'"'"'' 2>&1 |"...
DEBUG: sendsize = 782528
INFO: Sending oldest full snapshot zpool/nixos/etc@autosnap_2024-03-28_22:13:01_monthly (~ 764 KB) to new target filesystem:
DEBUG: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 ' zfs send -w -c  '"'"'zpool/nixos/etc'"'"'@'"'"'autosnap_2024-03-28_22:13:01_monthly'"'"' | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 782528 |  zfs receive  -s -F 'tank/test'
DEBUG: checking to see if tank/test on  is already in zfs receive using  ps -Ao args= ...
lzop: <stdin>: not a lzop file
mbuffer: error: outputThread: error writing to <stdout> at offset 0x10000: Broken pipe
0.00 B 0:00:00 [0.00 B/s] [>                                                                                                                                                                                                                 ]  0%
cannot receive: failed to read from stream
mbuffer: warning: error during output to <stdout>: Broken pipe
CRITICAL ERROR: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 ' zfs send -w -c  '"'"'zpool/nixos/etc'"'"'@'"'"'autosnap_2024-03-28_22:13:01_monthly'"'"' | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 782528 |  zfs receive  -s -F 'tank/test' failed: 256 at /usr/sbin/syncoid line 492.

@lediur
Copy link

lediur commented Apr 27, 2024 •
edited

I also encountered the above mysterious lzop: <stdin>: not a lzop file error. I initially tried to work around it by specifying --compress=none, but that resulted in a different cannot receive: failed to read from stream. Turns out this is a bit of a red herring.

/var/log/syslog on the remote host (source of the dataset) revealed that it was catching multiple "non-whitelisted" commands. It turns out that the two _RE_DATASET and _RE_SNAPSHOT regex in the restrict_commands.sh above don't account for capital letters in either the dataset or snapshot, and syncoid was trying to send snapshots that started with TEMP_.

Changing those two lines to:

_RE_DATASET=$'[\"\']+[A-Za-z0-9/_]+[\"\']+'
_RE_SNAPSHOT=$'[\"\']+[A-Za-z0-9/_]+[\"\']*@[\"\']*[A-Za-z0-9/_:-]+[\"\']+'

resolved my issue. I'm now happily pulling snapshots from my primary server to my backup server. @Mikesco3 harder to tell if this resolves your issue specifically, but @alembiq I do see a capital GMT in your log output so you might give it a shot.

@alembiq
Copy link

alembiq commented May 16, 2024

me bad, i never used the script above, I got this error from using syncoid from default package, but I'm getting the error above

should have read the thread more in detail before posting my question :( sorry

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment