Skip to content

Instantly share code, notes, and snippets.

@Jip-Hop
Last active June 21, 2023 17:38
Show Gist options
  • Save Jip-Hop/af3b7a770dd483b07ac093c3b205323f to your computer and use it in GitHub Desktop.
Save Jip-Hop/af3b7a770dd483b07ac093c3b205323f to your computer and use it in GitHub Desktop.
Using Docker on TrueNAS SCALE (no Kubernetes)
#!/usr/bin/env bash
#
# Enable docker and docker-compose on TrueNAS SCALE (no Kubernetes)
#
# This script is a hack! Use it at your own risk!!
# Using this script to enable Docker is NOT SUPPORTED by ix-systems!
# You CANNOT use SCALE Apps while using this script!
#
# 1 Create a dedicated Docker zvol on one of your zpools: zfs create -V 100G data/_docker
# 2 Create an ext4 filesystem on your new zvol: mkfs.ext4 /dev/zvol/data/_docker
# 3 Create a directory where you can mount the ext4 filesystem: mkdir /mnt/data/_docker
# 4 Save this script somewhere else on your zpool, not in the Docker dataset
# 5 Edit line 24 and 25 of the script, set paths to the Docker directory and zvol you created
# 7 If this is NOT the first time you run this script (updating to new location/zvol):
# - Check the contents of /etc/fstab, and remove the old # docker-zvol-ext4 rule if it's there
# 6 You can now start Docker by running the script from the SCALE console
#
# For these changes to persist after SCALE reboots and upgrades, run the script at start-up
# Schedule this script to run via System Settings -> Advanced -> Init/Shutdown Scripts
# Click Add -> Type: Script and choose this script -> When: choose to run as Pre Init
## set the path to your docker directory and specify the zvol
docker_directory='/mnt/data/_docker'
docker_zvol='/dev/zvol/data/_docker'
## this part will add a new fstab rule, if the fstab file doesn't contain our special docker-zvol-ext4 comment
## if this is not the first time you run this script, our special comment is already there
## so if you ever change the docker_directory and docker_zvol paths in the future, also remove the old
## fstab rule and comment from /etc/fstab
grep -q 'docker-zvol-ext4' /etc/fstab || {
printf "# docker-zvol-ext4\n${docker_zvol} ${docker_directory} ext4 rw,relatime,stripe=4 0 0\n" >> /etc/fstab
mount "${docker_directory}" && echo 'Mounted docker directory'
}
## HEREDOC: docker/daemon.json
read -r -d '' JSON << END_JSON
{
"storage-driver": "overlay2",
"data-root": "${docker_directory}",
"exec-opts": [
"native.cgroupdriver=cgroupfs"
]
}
END_JSON
## path to docker daemon file
docker_daemon='/etc/docker/daemon.json'
if [ ${EUID} -ne 0 ]; then
echo "Please run this script as root or using sudo"
elif [ "$(systemctl is-enabled k3s)" == "enabled" ]; then
echo "You can not use this script while k3s is enabled"
elif [ "$(systemctl is-active k3s)" == "active" ]; then
echo "You can not use this script while k3s is active"
elif ! which docker &> /dev/null; then
echo "Docker executable not found"
elif ! chmod +x /usr/bin/docker-compose &> /dev/null; then
echo "Failed to make docker-compose executable"
elif ! install -d -m 755 -- /etc/docker &> /dev/null; then
echo "Failed to install directory: /etc/docker"
elif [ ! -d "${docker_directory}" ] ; then
echo "Directory not found: ${docker_directory}"
else
echo "Checking file: ${docker_daemon}"
if test "${JSON}" != "$(cat ${docker_daemon} 2> /dev/null)"; then
echo "Updating file: ${docker_daemon}"
jq -n "${JSON}" > ${docker_daemon}
if [ "$(systemctl is-active docker)" == "active" ]; then
echo "Restarting Docker"
systemctl restart docker
elif [ "$(systemctl is-enabled docker)" != "enabled" ]; then
echo "Enable and starting Docker"
systemctl enable --now docker
fi
fi
fi
@Ornias1993
Copy link

Ornias1993 commented Mar 6, 2022

Anyway, docker-compose on the host is being more actively disabled next release anyway

Curious, I haven't had much time to follow along...

Next release as in Bluefin release or is that coming in 22.02.1?

I thought I saw mention of them blocking compose while k3s is active but I'm not sure if that's true or if compose is just going to be blocked completely. I already have a few sanity checks in my "enable-docker" script. One of which is to check if k3s is running, and will exit my script without making changes if it is.

I personally made the initial PR, which was remade in another place of the code by iX Staff, to disable it. The alternative will be a considerably more safe way of deploying compose once done, but no news on that one yet.

No idea about the precise release, likely 22.02.1

@f00b4r0
Copy link

f00b4r0 commented Mar 6, 2022

@f00b4r0 It doesn't "need" to idle at 10-20% CPU. It MIGHT do that, but that's no guarantee.

It does for me and that's a deal breaker.

Don't take everything on the internet for granted or think everything on the internet applies to your personal setup.

Thanks for lecturing me. You're absolutely right: I don't. My report is from personal experience. I then asked a colleague who has k8 experience (I don't) and he confirmed "that's 'expected'".
Besides, for my use case having k8 on top of docker is just an extra maintenance overhead (on top of the cpu/energy overhead). Not needed, not wanted.

You might be aware that said context was not part of your post I responded on.

What context was missing in my initial sentence? I said I was trying out SCALE and I saw some behavior. Did it sound to you as if I was propagating unverified FUD?
If context was missing for you, how about being a little less assertive in your condescending reply?

About maintenance overhead: you have no control over it anyway, only maintanence access lays with iX Systems.

-EPARSE.

Anyway, docker-compose on the host is being more actively disabled next release anyway (and replaced by another alternative). So i'll leave it at that.

No problem. I don't need compose either. That's how simple my needs are.
I'm also capable of using apt-get (after making it chmod +x, because apparently removing the execution bit on system software is a good idea to fend off power users) to install whatever I please on what is a Debian-based system.

And if I don't like what I see in the next release, I'll be happy to move away and format the boot disk with a sensible distro + ZoL.

The one thing I find useful is the web interface because it is convenient, but at the end of the day I can do everything with a few scripts anyway (I already do replication that way after the 11.3 dashboard implementation trashed my backups - a.k.a. NAS-107487).

A system that constantly tries to outsmart its users is probably not for me.

@Ornias1993
Copy link

Ornias1993 commented Mar 6, 2022

If context was missing for you, how about being a little less assertive in your condescending reply?

Well, I don't really give a flying fritata about your opinion anyway.

And if I don't like what I see in the next release, I'll be happy to move away and format the boot disk with a sensible distro + ZoL.

Well, more freedom for users is not planned for BlueFin (even making somethings harder), so maybe you should already?

@tprelog
Copy link

tprelog commented Mar 6, 2022

A system that constantly tries to outsmart its users

I realize it can seem that way at first but that's not really the intention. Basically TrueNAS is built to be an appliance with a certain set of features. This is not intended a fully configurable host OS. These blocks are intended to prevent most users from making changes to their systems which can lead to breakage, and can not be supported by the developers.

@Jip-Hop
Copy link
Author

Jip-Hop commented Mar 6, 2022

And if I don't like what I see in the next release, I'll be happy to move away and format the boot disk with a sensible distro + ZoL.

Possible alternatives with a web interface are Ubuntu with the Cockpit plugins by 45drives or openmediavault 6 (not yet officially released). Both should provide NAS features and ZFS management in a web gui and come with more freedom, to install Docker for example.

@f00b4r0
Copy link

f00b4r0 commented Mar 6, 2022

And if I don't like what I see in the next release, I'll be happy to move away and format the boot disk with a sensible distro + ZoL.

Possible alternatives with a web interface are Ubuntu with the Cockpit plugins by 45drives or openmediavault 6 (not yet officially released). Both should provide NAS features and ZFS management in a web gui and come with more freedom, to install Docker for example.

@Jip-Hop
👍
Thanks. That's very helpful.
I suppose they also come with a community that's slightly less toxic than TrueNAS', as evidenced above :)

A system that constantly tries to outsmart its users

I realize it can seem that way at first but that's not really the intention. Basically TrueNAS is built to be an appliance with a certain set of features. This is not intended a fully configurable host OS. These blocks are intended to prevent most users from making changes to their systems which can lead to breakage, and can not be supported by the developers.

@tprelog I'm sorry, the argument falls on deaf ears. Either it's opensource and hackable by definition, or it's not.

The first line on SCALE homeage prominently reads:

"TrueNAS® SCALE is an Open Source Hyperconverged Infrastructure (HCI) solution." (emphasis mine).

Nothing stops iX from adding a disclaimer than any change made to their system comes with zero support. That's fine. Actively locking down users into a walled garden "because we know better" (or "because we don't want to be bothered by people tweaking our stuff") isn't.

@tprelog
Copy link

tprelog commented Mar 6, 2022

Either it's opensource

And the definition of open source reads

Open source is source code that is made freely available for possible modification and redistribution.

So you are free to fork the source code, make any changes you wish and compile / use the software as you see fit.

That does not mean the end product as provided by ix-system should be openly hackable. It's not a one size fits all solution and that's ok because we have plenty of other options out there.

Actively locking down users into a walled garden

That's the whole point of an appliance. I'm not trying to give you some justification as an excuse that I made up. I've used FreeNAS / TrueNAS for the last 5 years and it's always been this way. I'm really not trying to argue or be a dick. Honesty, it's just the way it is. Anybody who's used these products for any length of time will tell you the same 🤷

and hackable by definition, or it's not.

It is still hackable(ish) just not by definition. So we have to make our own scripts to "hack" what ix-systems has designed. I feel the same way as you about running a full kubernetes stack just for 2-3 extra services. It's not going to happen. No matter how great they make it, the applications that I run are not designed for kubernetes so they do not fully function when run using any official TrueNAS way.

Fortunately it's pretty easy to enable docker and use docker-compose on your own. Even the upcoming block looks easy to circumvent.

So I guess it just depends... A few minor changes could be easy to "hack" in and maintain yourself and may be worth the effort. If / when it becomes to much hassle using native docker-compose, I will be switching OS as well -- For 100% complete unbounded control, TrueNAS is simply not the product we are looking for.

@Jip-Hop
Copy link
Author

Jip-Hop commented Mar 6, 2022

Even the upcoming block looks easy to circumvent

Can you post a link/reference? Would be curious to see what to expect.

@Jip-Hop
Copy link
Author

Jip-Hop commented Mar 6, 2022

So we have to make our own scripts to "hack" what ix-systems has designed.

If you (or anyone) has useful scripts to enable Docker/Portainer on TrueNAS SCALE then please post them here. 😃

@tprelog
Copy link

tprelog commented Mar 6, 2022

Can you post a link/reference? Would be curious to see what to expect.

Hence it's time to disable the binary by default, to make clear it's not an included feature of TrueNAS SCALE, while still allowing advanced users to enable it on purpose.

If you (or anyone) has useful scripts to enable Docker/Portainer on TrueNAS SCALE then please post them here.

Here's mine -- Originally started as a fork from here but eventually refactored it. Not yet adjusted for the upcoming changes though

Updated to ensure docker-compose remains executable (unless it's completely removed) after the next SCALE release

https://gist.github.com/tprelog/7988dc6b196775f33929beb19f0090d7

@Jip-Hop
Copy link
Author

Jip-Hop commented Mar 7, 2022

Nice! Thanks for sharing 🙂

@reimnm
Copy link

reimnm commented Apr 28, 2022

In version 22.02.0.1, you just need,
1.unset app pool in app tab-> Settings
2.stop docker service, add two new default dataset to store docker root data and app volume,
one is for "data-root" parameter in /etc/docker/daemon.json
(just need add one parameter {"data-root": "/mnt/nvme/docker-data"} ,other parameters will same as other guides. [docker info] to check),
another one is "docker-config" which is to mount a path to your docker apps. you'd better to create a new dataset of docker-config(nvme->docker-config->netdata) rather than a folder
3.start docker service
4.you can remove previous default data-root files according to offical doc https://docs.docker.com/storage/storagedriver/zfs-driver/
Note: change dataset mountpoint to /var/lib/docker/ will not work, system will mount to /mnt/var/lib/docker/

@Jip-Hop
Copy link
Author

Jip-Hop commented Oct 15, 2022

Updated the script by @tprelog to use an ext4 formatted zvol since I am having issues with the ZFS graph driver.

Reference: an interesting discussion (without conclusion?) about optimizing zvols for Docker.

@Jip-Hop
Copy link
Author

Jip-Hop commented Oct 17, 2022

When this issue is resolved, we can use overlayfs (with zfs) instead of the zfs driver and there would be no need to use an ext4 formatted zvol. Luckily ixsystems is working on it.

@rkabrick
Copy link

100% a naive question however I'm trying to understand how this works. I have the script running on pre-init... Now am I able to proceed with installing TrueCharts apps in the UI? Or is UI app interface now irrelevant?

@Jip-Hop
Copy link
Author

Jip-Hop commented Oct 28, 2022

If you want to use this method then you're supposed to act as if TrueNAS Apps / TrueCharts doesn't exist. This is also mentioned in the comments of the script. It's for advanced users who need/want to use docker, docker-compose and possibly portainer for portability, performance or other reasons. If you're not sure why you're using this script then you'd probably be better off not using it and instead install Apps on TrueNAS via the official way: using the UI.

@rkabrick
Copy link

Yeah, I saw the comment I was just confused about if "Scale Apps" referred to their apps (not including the UI) or if the UI was decoupled from their app repo and pointed to native docker. I am looking to use portainer and was underwhelmed by the performance I got running through a VM) so I will proceed using this.

May I ask what your reason for changing Line 21 from Post-Init to Pre-Init?

@Ornias1993
Copy link

Yeah, I saw the comment I was just confused about if "Scale Apps" referred to their apps (not including the UI) or if the UI was decoupled from their app repo and pointed to native docker. I am looking to use portainer and was underwhelmed by the performance I got running through a VM) so I will proceed using this.

May I ask what your reason for changing Line 21 from Post-Init to Pre-Init?

SCALE Apps refers to the complete apps system for TrueNAS SCALE. GUI and backend.
It's fully kubernetes backed and does not 'point' to native docker.

@Jip-Hop
Copy link
Author

Jip-Hop commented Oct 29, 2022

May I ask what your reason for changing Line 21 from Post-Init to Pre-Init?

Well spotted! Since the script mounts the ext4 filesystem if it's not already in fstab it needs to run as early as possible.

@Ornias1993
Copy link

New zfs version has overlayfs support natively baked in btw :)

@Jip-Hop
Copy link
Author

Jip-Hop commented Oct 29, 2022

Yeah I just noticed! Super awesome! 😃Looking forward to using it on a stable SCALE release.

@Ornias1993
Copy link

Sources indicate it also fixes about 75% of the CPU load issues on SCALE Apps as well :)

@Jip-Hop
Copy link
Author

Jip-Hop commented Oct 29, 2022

Wonderful :) would be worth revisiting the Docker Compose TrueCharts app by that time. It really is comforting to know that one exists for when this Docker hack stops working.

@tprelog
Copy link

tprelog commented Nov 20, 2022

@Jip-Hop - I'm on RC 1 now - I only needed to add the "storage-driver": "overlay2", to the HEREDOC section of the script and everything seems to be working. I did not have to create, format, or even mount an extra zvol.

EDIT: Maybe worth noting, I started with a fresh docker-dataset after I changed the storage-driver.

@Jip-Hop
Copy link
Author

Jip-Hop commented Nov 20, 2022

Great! Thanks for testing and reporting. Didn't expect overlay2 support to be included so quickly. Then I know what I must do when SCALE 22.12.0 is released next month. Glad I can drop the ext4 workaround.😄

@Ornias1993
Copy link

From a docker level this should be the only required change.
Though this also means one of the primary reasons not to go with SCALE Apps (performance issues) are also solved for a large portion as well.

@Jip-Hop
Copy link
Author

Jip-Hop commented Dec 24, 2022

I'm back to using the script by tprelog now that I've updated to Bluefin. No need for the ext4 zvol :)

@Jip-Hop
Copy link
Author

Jip-Hop commented Jan 7, 2023

⚠️ In October 2023, TrueNAS SCALE Cobia will be released. At that time, ix-systems is making the switch to containerd and Docker will be removed.

The alternative way of running docker, based on systemd-nspawn is available for testing! 😄

@Jip-Hop
Copy link
Author

Jip-Hop commented Jan 9, 2023

@scepterus
Copy link

So we have to make our own scripts to "hack" what ix-systems has designed.

If you (or anyone) has useful scripts to enable Docker/Portainer on TrueNAS SCALE then please post them here. 😃

https://forum.level1techs.com/t/truenas-scale-native-docker-vm-access-to-host-guide/190882

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment