As TrueNAS 13 reached RELEASE milestone in May 2022 and TrueNAS 12 got abruptly EOL’d shortly after, the usual question stroke me: Should I update now or wait?
TrueNAS 13 offered the option to stay on CORE or switch to SCALE. I was attracted by the ability to run containers more easily with SCALE, however that meant switching from FreeBSD to Linux, and the whole point of sticking with TrueNAS over a simple Ubuntu server was for the (alleged) superiority of FreeBSD.
With this option settled — staying on CORE — I patiently waited for TrueNAS 13 to become more mature. As everyone who has run FreeNAS/TrueNAS for a few years can tell: upgrades are rarely smooth. Things often break and require lengthy operations and debugging. Time flew quickly and my TrueNAS 12.0-U8.1 was really getting old. As we are now at TrueNAS 13.0-U6, I thought this would be quite stable and the upgrade process should have been “fixed” of obvious bugs.
WRONG!
Step 1: I painstakingly proceeded to launch the upgrade to TrueNAS 13-U6 by providing the TrueNAS-13.0-U6-manual-update.tar file directly (otherwise, the download of the three separate files is horribly slow). After a reboot, all things looked OK. I unlocked my pools and checked my jails.
My wireguard jail is running. That’s nice. I use wireguard as a way to get into my network, which helps me manage my TrueNAS server and whatever is running on it, so it is pretty important that I keep my remote access working.
Step 2: Let’s check if jails need an update? Jails indeed don’t get updated. Some design choice I assume. How about I click on Update on each of them?
Uhhhhh.
No update available for the jail. It is running 12.2-RELEASE-p15 but there is no update. OK. I’m dumb, I must be missing something. Obviously there should be an update to bring it to 13.x, right?
Let’s open the shell and try pkg update and pkg upgrade. Some package updates for wireguard, OK, neat.
Step 3: I’m testing to connect to it just to verify nothing’s wrong.
Uhhhhhh.
Handshake for peer 1 (...) did not complete after 5 seconds, retrying (try 2)
Wireguard is not answering. WTF?!
Wireguard is running and it is listening on the correct port, which you can know by using sockstat.
Any other attempt at finding an issue in my network failed.
Step 4: Eventually, I checked the firewall rules in this jail, which is configured with ipfw, and tried to restart the firewall service using service ipfw restart. And….
ipfw: setsockopt(IP_FW_NAT44_XCONFIG): Invalid argument
Alright, a pretty error that is absolutely not helpful appears on the screen, along with other seemingly successful messages. Is it important? Who knows. I tried disabling the firewall, and I could connect to wireguard. So, the firewall is the issue.
Quick searches online pointed me to fake solutions like “rebuilding the ipfw binary should help” (yay!), as well as a post on TrueNAS forum suggesting that creating a fresh new jail with a base FreeBSD 13.1 would help (because yes of course, it is my hobby to recreate things that should already work). The post also suggested further network troubles, so this really filled me with joy…
Step 5: OK, so I have a jail based on FreeBSD 12.2 and the web UI tells me there is no update available (although I could at least update some packages with pkg update/pkg upgrade). How about I try to upgrade jails from the command line?
iocage upgrade -r 13.1 wireguard
After a while, I had to leave for work, spent the day without remote access, which I normally use daily, yay! Please run that within a screen otherwise you are doomed to never finish the process.
Some 14k updates later, and some warnings about files being changed/deleted, the update was complete!
Step 6: Let’s get into the jail and continue to update packages.
Uhhhhhh.
This time, wireguard packages are getting EOL’d. That means I need to re-learn how to install wireguard on TrueNAS due to FreeBSD 13.
=====
Message from wireguard-kmod-0.0.20220615_1:
--
At this time this code is new, unvetted, possibly buggy, and should be
considered "experimental". It might contain security issues. We gladly
welcome your testing and bug reports, but do keep in mind that this code
is new, so some caution should be exercised at the moment for using it
in mission critical environments.
--
===> NOTICE:
This port is deprecated; you may wish to reconsider installing it:
Only useful for FreeBSD 12 which is EoL soon.
It is scheduled to be removed on or after 2023-12-31.
I eventually learn that wireguard likes to run in the kernel, it is more efficient, but it is also less secure. It literally tells you that this is a piece of experimental code and it might have security vulnerabilities. OK, great. Why do I use FreeBSD again? I don’t want to trade some performance boost for security.
Step 7: So I will use wireguard-go, the Go user-land implementation. I get rid of wireguard-kmod: pkg remove wireguard wireguard-kmod. Make sure you don’t also remove wireguard-tools, otherwise you won’t be able to set it up as a service.
Just trying to restart the jail before I continue further and…
Uhhhhhh.
A new error that shouldn’t occur, but that apparently resolves itself if you try again!
Finally, it works!
Conclusion and rant: Upgrading your TrueNAS is always full of surprises and bugs. Some are due to how poorly integrated the upgrade process is. You would expect to get everything, jails included, updated, or at least a standard guide that tells you what to do when you upgrade to TrueNAS 13. I guess a simple jail with wireguard and a firewall isn’t a corner case. The worst part is that the error was silent: my wireguard jail was running and I couldn’t suspect it was not running well inside. In the past, that’s the jail that has been the least affected by TrueNAS upgrades. You would think that waiting 1yr+ to reach a very stable release would help? Well, no. Next, the way wireguard works differently on FreeBSD 13 added a layer of complexity.
Finally, TrueNAS website and documentation are really ugly. I cannot even find as of today (2023-11-28) a page that indicates which was the last v12.0 release and when v12 was EOL’d. As somebody pointed out on this forum post, “It would be great if that were written somewhere “official”.” And a power user with nearly 7k messages on the forum to reply to him: “In the link […] it quite clearly states that 13 is THE supported version. Which implies all others are not.” Because CLARITY is at its climax when IMPORTANT THINGS ARE NOT WRITTEN BUT ONLY IMPLIED. I wonder how such a(n) (eco)system survives.
Xavier de Carné 