Skip to content

Instantly share code, notes, and snippets.

@csarn

csarn/README.md

Last active April 27, 2024 08:20
Show Gist options
  • Star 8 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save csarn/3084913ba3e78da6c4d511df131c4d5f to your computer and use it in GitHub Desktop.
Save csarn/3084913ba3e78da6c4d511df131c4d5f to your computer and use it in GitHub Desktop.
Download ZIP
ZFS pull backup with minimal permissions
Raw
README.md

Usage

on system that should be backed up

  • put "restrict_commands.sh" in /usr/local/bin and make it executable
  • install ts, lzop and optionally mbuffer
useradd zfsbackup --create-home --system
mkdir /home/zfsbackup/.ssh
zfs allow -u zfsbackup send,hold tank/dataset
echo 'restrict,command="restrict_commands.sh" ssh-ed25519 ...' > /home/zfsbackup/.ssh/authorized_keys
chown zfsbackup:zfsbackup /home/zfsbackup/.ssh -R

on backup server

run cronjob with:

syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw zfsbackup@target:tank/dataset tank/dataset
Raw
restrict_commands.sh
#!/bin/bash
export PATH=/usr/sbin:$PATH
_RE_DATASET=$'[\"\']+[a-z0-9/_]+[\"\']+'
_RE_SNAPSHOT=$'[\"\']+[a-z0-9/_]+[\"\']*@[\"\']*[a-z0-9/_:-]+[\"\']+'
_RE_SIZE=$'[0-9]+[kMG]?'
_WHITELIST=(
-e "exit"
-e "echo -n"
-e "zpool get -o value -H feature@extensible_dataset $_RE_DATASET"
-e "zfs get -H syncoid:sync $_RE_DATASET"
-e "zfs get -Hpd 1 -t snapshot guid,creation $_RE_DATASET"
-e " *zfs send -R -w -nv?P $_RE_SNAPSHOT"
-e " *zfs send -R -w +$_RE_SNAPSHOT \| +lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
-e " *zfs send -R -w -nv?P +-I $_RE_SNAPSHOT $_RE_SNAPSHOT"
-e " *zfs send -R -w +-I $_RE_SNAPSHOT $_RE_SNAPSHOT \| lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
-e "command -v (lzop|mbuffer)"
)
## LOG non-whitelisted commands, execute whitelisted
echo "$SSH_ORIGINAL_COMMAND" |\
tee >(egrep -x -v "${_WHITELIST[@]}" \
| ts "non-whitelisted command issued: (client $SSH_CLIENT)" \
| logger -p local0.crit \
) |\
egrep -x "${_WHITELIST[@]}" | bash
@danboid
Copy link

danboid commented Nov 28, 2021

Hi @csarn

It's great to hear you've got syncoid to work without enabling root ssh logins, which is what I'm trying to achieve too.

I have tried to copy your config but I've been unable to get it to work yet:

root@asteria:~# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw zfsbackup@dionysus.cs.salford.ac.uk:lxdpool/containers/hermes astarray/backups/hermes
zfsbackup@dionysus.cs.salford.ac.uk: Permission denied (publickey).
zfsbackup@dionysus.cs.salford.ac.uk: Permission denied (publickey).
CRITICAL ERROR: ssh connection echo test failed for zfsbackup@dionysus.cs.salford.ac.uk with exit code 255 at /usr/sbin/syncoid line 1427.
root@asteria:~# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshkey=/root/.ssh/id_ecdsa zfsbackup@dionysus.cs.salford.ac.uk:lxdpool/containers/hermes astarray/backups/hermes
zfsbackup@dionysus.cs.salford.ac.uk: Permission denied (publickey).
zfsbackup@dionysus.cs.salford.ac.uk: Permission denied (publickey).
CRITICAL ERROR: ssh connection echo test failed for zfsbackup@dionysus.cs.salford.ac.uk with exit code 255 at /usr/sbin/syncoid line 1427.

In your example commands above, you run:

useradd zfsbackup --system

useradd doesn't create a home dir for new users by default but we need a zfsbackup home dir to store the authorized_keys file and the restrict_commands.sh script so why not run:

useradd -m zfsbackup --system

Instead to create the zfsbackup home dir at the same time?

It seems syncoid prefers to be run as root and so I created a SSH key as the root user of the destination machine and its that public key that I copied to the zfsbackup users authorized_keys on the remote machine.

I think I'm having problems because I'm running syncoid as root but the remote username is zfsbackup. I've tried specifying the path to my private key for syncoid but that hasn't worked. I have configured the .ssh directory and its files permissions correctly on both ends, I think.

Which user do you run your syncoid cron job as, if its not root? I shouldn't need to run any zfs allow commands on the destination machine if I'm running syncoid as root.

It would be great to get this process properly documented for sanoid or maybe the Arch wiki instead.

Thanks

@csarn
Copy link
Author

csarn commented Dec 22, 2021

Hi @danboid, you're right about the need to create the home directory. I added the switch to this gist.
About your use case:
I run it the same way you do, so the cronjob runs as root.
The zfsbackup user has to be created on the system that should be backed up. I clarified that in the README.md.
That also means that the "zfs allow" commands have to be issued on the machine that should be backed up.

I think your problem is the ssh connection itself, and has nothing to do with sanoid. Did you create the user on the remote system?
Also, if you add the "restrict" command script in zfsbackup's home dir, you have to make sure it is executable and on the PATH (or add the full path to the script in authorized_keys).

@rajil
Copy link

rajil commented Dec 27, 2021 •
edited

Thanks for the instructions. I am able to do a transfer without the restrict_commands.sh script. However, once i enable it i get an error,

# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshport=2222 --sshkey=/root/.ssh/mykey zfsbackup@masterserver.com:tank/stuff tank/stuff
Use of uninitialized value $sendsize in substitution (s///) at /usr/bin/syncoid line 1784.
Use of uninitialized value $sendsize in scalar chomp at /usr/bin/syncoid line 1788.
Use of uninitialized value $sendsize in pattern match (m//) at /usr/bin/syncoid line 1791.
Resuming interrupted zfs send/receive from tank/stuff to tank/stuff (~ UNKNOWN remaining):
lzop: <stdin>: not a lzop file
cannot receive: failed to read from stream
CRITICAL ERROR: ssh   -p 2222 -i /root/.ssh/mykey -S /tmp/syncoid-zfsbackup@masterserver.com-1640603651 zfsbackup@masterserver.com ' zfs send -w  -t **redacted** | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 0 |  zfs receive  -s -F 'tank/stuff' 2>&1 failed: 256 at /usr/bin/syncoid line 580.

Is the --sshport=2222 causing the shell script to fail?

@Mikesco3
Copy link

Thanks for the instructions. I am able to do a transfer without the restrict_commands.sh script. However, once i enable it i get an error,

# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshport=2222 --sshkey=/root/.ssh/mykey zfsbackup@masterserver.com:tank/stuff tank/stuff
Use of uninitialized value $sendsize in substitution (s///) at /usr/bin/syncoid line 1784.
Use of uninitialized value $sendsize in scalar chomp at /usr/bin/syncoid line 1788.
Use of uninitialized value $sendsize in pattern match (m//) at /usr/bin/syncoid line 1791.
Resuming interrupted zfs send/receive from tank/stuff to tank/stuff (~ UNKNOWN remaining):
lzop: <stdin>: not a lzop file
cannot receive: failed to read from stream
CRITICAL ERROR: ssh   -p 2222 -i /root/.ssh/mykey -S /tmp/syncoid-zfsbackup@masterserver.com-1640603651 zfsbackup@masterserver.com ' zfs send -w  -t **redacted** | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 0 |  zfs receive  -s -F 'tank/stuff' 2>&1 failed: 256 at /usr/bin/syncoid line 580.

Is the --sshport=2222 causing the shell script to fail?

if you created a user called zfsbackup shouldn't you be referencing ssh keys in the /home/zfsbackup/.ssh/mykey location?
because I can't help notice that you're referencing the root's ssh key, which the zfsbackup user isn't supposed to have access to.

@Mikesco3
Copy link

Mikesco3 commented Mar 22, 2022 •
edited

Sorry I'm kind of new to github, but I wanted to make a few suggestions

  1. On the readme.md file
    please add mkdir /home/zfsbackup/.ssh
    else the echo 'restrict,command="restrict_commands.sh" ssh-ed25519 ...' > /home/zfsbackup/.ssh/authorized_keys fails.

  2. On the restrict_commands.sh:
    There is no mention where to place this file, I ended up creating a .local/bin folder in the zfsbackup user's home path
    mkdir -p /home/zfsbackup/.local/bin
    and adding that path in line 2 of the restrict_commands.sh so that it reads
    export PATH=$PATH:$HOME/.local/bin:/usr/sbin
    instead of export PATH=/usr/sbin:$PATH

  3. Finally, if the restrict_commands.sh file fails with unknown command on line 26 error, it is because it is likely missing the ts command
    so in order to fix that I installed moreutils
    apt-get install moreutils

NOTE: just to be on the safe side it may be useful to ensure the user zfsbackup has permission to the folders we created on his home folder,
so to be safe run chown zfsbackup:zfsbackup /home/zfsbackup --recursive

@csarn
Copy link
Author

csarn commented Mar 23, 2022

Thanks for the instructions. I am able to do a transfer without the restrict_commands.sh script. However, once i enable it i get an error,

# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshport=2222 --sshkey=/root/.ssh/mykey zfsbackup@masterserver.com:tank/stuff tank/stuff
Use of uninitialized value $sendsize in substitution (s///) at /usr/bin/syncoid line 1784.
Use of uninitialized value $sendsize in scalar chomp at /usr/bin/syncoid line 1788.
Use of uninitialized value $sendsize in pattern match (m//) at /usr/bin/syncoid line 1791.
Resuming interrupted zfs send/receive from tank/stuff to tank/stuff (~ UNKNOWN remaining):
lzop: <stdin>: not a lzop file
cannot receive: failed to read from stream
CRITICAL ERROR: ssh   -p 2222 -i /root/.ssh/mykey -S /tmp/syncoid-zfsbackup@masterserver.com-1640603651 zfsbackup@masterserver.com ' zfs send -w  -t **redacted** | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 0 |  zfs receive  -s -F 'tank/stuff' 2>&1 failed: 256 at /usr/bin/syncoid line 580.

Is the --sshport=2222 causing the shell script to fail?

if you created a user called zfsbackup shouldn't you be referencing ssh keys in the /home/zfsbackup/.ssh/mykey location? because I can't help notice that you're referencing the root's ssh key, which the zfsbackup user isn't supposed to have access to.

Well no, the ssh key usage is correct. The backup job running on the backup server has to be run as root (it is not possible on linux to allow a non-root user to "zfs receive"). So root@backup has the private key, and logs into the server-to-be-backupped using the zfsbackup user account.

Thanks for your suggestions, I updated the readme accordingly. And I'll try to find out if running ssh on a different port will cause trouble.

@alembiq
Copy link

alembiq commented Apr 1, 2024

any luck? i'm stucked on the lzop error also :(

syncoid --sshkey=/var/lib/syncoid/kubera-2022-05-31 backup@10.10.10.10:zpool/nixos/etc tank/test --no-privilege-elevation --debug --sendoptions="w c " --sshoption=StrictHostKeyChecking=off
DEBUG: SSHCMD: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31
loginShellInit
DEBUG: checking availability of lzop on source...
DEBUG: checking availability of lzop on target...
DEBUG: checking availability of lzop on local machine...
DEBUG: checking availability of mbuffer on source...
DEBUG: checking availability of mbuffer on target...
DEBUG: checking availability of pv on local machine...
DEBUG: checking availability of zfs resume feature on source...
DEBUG: checking availability of zfs resume feature on target...
DEBUG: syncing source zpool/nixos/etc to target tank/test.
DEBUG: getting current value of syncoid:sync on zpool/nixos/etc...
ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs get -H syncoid:sync ''"'"'zpool/nixos/etc'"'"''
DEBUG: checking to see if tank/test on  is already in zfs receive using  ps -Ao args= ...
DEBUG: checking to see if target filesystem exists using "  zfs get -H name 'tank/test' 2>&1 |"...
DEBUG: getting list of snapshots on zpool/nixos/etc using ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs get -Hpd 1 -t snapshot guid,creation ''"'"'zpool/nixos/etc'"'"'' |...
DEBUG: creating sync snapshot using "ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs snapshot ''"'"'zpool/nixos/etc'"'"''@syncoid_kubera_2024-04-01:21:54:02-GMT02:00
"...
loginShellInit
DEBUG: target tank/test does not exist.  Finding oldest available snapshot on source zpool/nixos/etc ...
DEBUG: getting estimated transfer size from source -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 using "ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs send -w -c  -nvP ''"'"'zpool/nixos/etc@autosnap_2024-03-28_22:13:01_monthly'"'"'' 2>&1 |"...
DEBUG: sendsize = 782528
INFO: Sending oldest full snapshot zpool/nixos/etc@autosnap_2024-03-28_22:13:01_monthly (~ 764 KB) to new target filesystem:
DEBUG: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 ' zfs send -w -c  '"'"'zpool/nixos/etc'"'"'@'"'"'autosnap_2024-03-28_22:13:01_monthly'"'"' | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 782528 |  zfs receive  -s -F 'tank/test'
DEBUG: checking to see if tank/test on  is already in zfs receive using  ps -Ao args= ...
lzop: <stdin>: not a lzop file
mbuffer: error: outputThread: error writing to <stdout> at offset 0x10000: Broken pipe
0.00 B 0:00:00 [0.00 B/s] [>                                                                                                                                                                                                                 ]  0%
cannot receive: failed to read from stream
mbuffer: warning: error during output to <stdout>: Broken pipe
CRITICAL ERROR: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 ' zfs send -w -c  '"'"'zpool/nixos/etc'"'"'@'"'"'autosnap_2024-03-28_22:13:01_monthly'"'"' | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 782528 |  zfs receive  -s -F 'tank/test' failed: 256 at /usr/sbin/syncoid line 492.

@lediur
Copy link

lediur commented Apr 27, 2024 •
edited

I also encountered the above mysterious lzop: <stdin>: not a lzop file error. I initially tried to work around it by specifying --compress=none, but that resulted in a different cannot receive: failed to read from stream. Turns out this is a bit of a red herring.

/var/log/syslog on the remote host (source of the dataset) revealed that it was catching multiple "non-whitelisted" commands. It turns out that the two _RE_DATASET and _RE_SNAPSHOT regex in the restrict_commands.sh above don't account for capital letters in either the dataset or snapshot, and syncoid was trying to send snapshots that started with TEMP_.

Changing those two lines to:

_RE_DATASET=$'[\"\']+[A-Za-z0-9/_]+[\"\']+'
_RE_SNAPSHOT=$'[\"\']+[A-Za-z0-9/_]+[\"\']*@[\"\']*[A-Za-z0-9/_:-]+[\"\']+'

resolved my issue. I'm now happily pulling snapshots from my primary server to my backup server. @Mikesco3 harder to tell if this resolves your issue specifically, but @alembiq I do see a capital GMT in your log output so you might give it a shot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment