TrueNAS - Revision history

M: /* Replace a bad disk in a pool */

2025-03-24T12:32:26Z

Replace a bad disk in a pool

← Older revision		Revision as of 12:32, 24 March 2025
Line 138:		Line 138:

	=== Replace a bad disk in a pool ===		=== Replace a bad disk in a pool ===

	~~OFFLINE it, open the case, replace it, add the new drive to the pool, and let it resilver. Keep your fingers crossed... but it should be okay as long as your drives are robustly vetted.~~


	* Watch TrueNAS for '''CRITICAL''' alerts that indicate a drive is failing its self-tests. You can check SMART results yourself to make sure it is actually bad.		* Watch TrueNAS for '''CRITICAL''' alerts that indicate a drive is failing its self-tests. You can check SMART results yourself to make sure it is actually bad.
Line 151:		Line 148:
	* Replace it; this will automatically start a resilver (buckle up for that, it takes FOREVER and will POUND your disks)		* Replace it; this will automatically start a resilver (buckle up for that, it takes FOREVER and will POUND your disks)
	Storage > Pools > badpool > baddrive > triple-dot-status > REPLACE		Storage > Pools > badpool > baddrive > triple-dot-status > REPLACE
	* If the resilver seems to go bad, or stop, or pause, or ANYTHING... you can try restarting TrueNAS. This actually clears a lot of error marking and may actually be a required step. Crazy TrueNAS.		* The resilver SHOULD be okay if you have vetted your drives well. If the resilver seems to go bad, or stop, or pause, or ANYTHING... you can try restarting TrueNAS. This actually clears a lot of error marking and may actually be a required step. Crazy TrueNAS.
	* Once resilver is done, you will probably need to clear pool errors.		* Once resilver is done, you will probably need to [[TrueNAS#Fix_any_pool_errors_ASAP\|clear pool errors]].
	* Run a full badblocks on the faulted drive and ensure it was the drive at fault (not connectors or something else)		* Run a full badblocks on the faulted drive and ensure it was the drive at fault (not connectors or something else)
	* Return the drive if under warranty; otherwise, RECYCLE IT		* Return the drive if under warranty; otherwise, RECYCLE IT

M: /* Replace a bad disk in a pool */

2025-03-24T12:31:09Z

Replace a bad disk in a pool

← Older revision		Revision as of 12:31, 24 March 2025
Line 138:		Line 138:

	=== Replace a bad disk in a pool ===		=== Replace a bad disk in a pool ===
	~~My 7-drive raidz arrays can only lose ONE drive before they go boom~~, ~~so you MUST~~ replace ~~bad disks immediately. raidz2 uses 2 drives, raidz3 uses three~~, ~~but SSD raidz you-can-lose-one-~~drive ~~is,~~ to me, ~~a sweet spot~~.
			OFFLINE it, open the case, replace it, add the new drive to the pool, and let it resilver. Keep your fingers crossed... but it should be okay as long as your drives are robustly vetted.


	* Watch TrueNAS for '''CRITICAL''' alerts that indicate a drive is failing its self-tests. You can check SMART results yourself to make sure it is actually bad.		* Watch TrueNAS for '''CRITICAL''' alerts that indicate a drive is failing its self-tests. You can check SMART results yourself to make sure it is actually bad.
Line 147:		Line 149:
	* Power down the whole fucking PROXMOX machine		* Power down the whole fucking PROXMOX machine
	* Pull it, and swap out bad drive for good		* Pull it, and swap out bad drive for good
	* Replace it		* Replace it; this will automatically start a resilver (buckle up for that, it takes FOREVER and will POUND your disks)
	Storage > Pools > badpool > baddrive > triple-dot-status > REPLACE		Storage > Pools > badpool > baddrive > triple-dot-status > REPLACE
			* If the resilver seems to go bad, or stop, or pause, or ANYTHING... you can try restarting TrueNAS. This actually clears a lot of error marking and may actually be a required step. Crazy TrueNAS.
			* Once resilver is done, you will probably need to clear pool errors.
	* Run a full badblocks on the faulted drive and ensure it was the drive at fault (not connectors or something else)		* Run a full badblocks on the faulted drive and ensure it was the drive at fault (not connectors or something else)
	* Return the drive if under warranty; otherwise, RECYCLE IT		* Return the drive if under warranty; otherwise, RECYCLE IT

			==== NOTE on raid level ====
			My 7-drive raidz arrays can only lose ONE drive before they go boom, so you MUST replace bad disks immediately. raidz2 uses 2 drives, raidz3 uses three, but SSD raidz you-can-lose-one-drive is, to me, a sweet spot.

	=== Remove a bad pool ===		=== Remove a bad pool ===

M: /* Maintenance */

2025-03-24T12:27:55Z

Maintenance

← Older revision		Revision as of 12:27, 24 March 2025
Line 87:		Line 87:
	First thing, [[TrueNAS#Burn_in_a_new_drive\|check the drive]]. Use smartmon tools, and if you see any issue there, better pull and replace it, and do a full badblocks test on the drive using the external SATA to USB mounter. Any problems, chuck it and replace it.		First thing, [[TrueNAS#Burn_in_a_new_drive\|check the drive]]. Use smartmon tools, and if you see any issue there, better pull and replace it, and do a full badblocks test on the drive using the external SATA to USB mounter. Any problems, chuck it and replace it.

	If you have a bad drive, you will need to ~~OFFLINE it, open the case,~~ replace it~~, add the new drive to the pool, and let it resilver. Keep your fingers crossed... but it should be okay as long as your drives are robustly vetted~~.		If you have a bad drive, you will need to [[TrueNAS#Replace_a_bad_disk_in_a_pool\|replace it]].

	Whether you replace the drive, or ensure it is good, you now have to clean the pool of the bad data (currently the bad data exists, and will persist!).		Whether you replace the drive, or ensure it is good, you now have to clean the pool of the bad data (currently the bad data exists, and will persist!).

M: /* Replace a bad disk in a raidz pool */

2025-03-24T12:26:49Z

Replace a bad disk in a raidz pool

← Older revision		Revision as of 12:26, 24 March 2025
Line 137:		Line 137:
	# I have a 1Gb switch, i guess that's all we get out of it?		# I have a 1Gb switch, i guess that's all we get out of it?

	=== Replace a bad disk in a ~~raidz~~ pool ===		=== Replace a bad disk in a pool ===
	My 7-drive raidz arrays can only lose ONE drive before they go boom, so you MUST replace bad disks immediately. raidz2 uses 2 drives, raidz3 uses three, but SSD raidz you-can-lose-one-drive is, to me, a sweet spot.		My 7-drive raidz arrays can only lose ONE drive before they go boom, so you MUST replace bad disks immediately. raidz2 uses 2 drives, raidz3 uses three, but SSD raidz you-can-lose-one-drive is, to me, a sweet spot.

	* Watch TrueNAS for '''CRITICAL''' alerts that indicate a drive is failing its self-tests.		* Watch TrueNAS for '''CRITICAL''' alerts that indicate a drive is failing its self-tests. You can check SMART results yourself to make sure it is actually bad.
	* Make note of its serial number.		* Make note of its serial number.
	* Find the drive in the pool, make note of its drive id (not needed but no harm).		* Find the drive in the pool, make note of its drive id (not needed but no harm).
Line 149:		Line 149:
	* Replace it		* Replace it
	Storage > Pools > badpool > baddrive > triple-dot-status > REPLACE		Storage > Pools > badpool > baddrive > triple-dot-status > REPLACE
			* Run a full badblocks on the faulted drive and ensure it was the drive at fault (not connectors or something else)
			* Return the drive if under warranty; otherwise, RECYCLE IT

	=== Remove a bad pool ===		=== Remove a bad pool ===

M: /* Maintenance */

2025-03-24T12:24:36Z

Maintenance

@@ Line 78: / Line 78: @@
 == Maintenance ==
 === Fix any pool errors ASAP ===
@@ Line 111: / Line 87: @@
 First thing, [[TrueNAS#Burn_in_a_new_drive|check the drive]].  Use smartmon tools, and if you see any issue there, better pull and replace it, and do a full badblocks test on the drive using the external SATA to USB mounter.  Any problems, chuck it and replace it.
-Once you are sure you have a good drive, you have to clean the pool of the bad data (currently the bad data exists, and will persist!).
+If you have a bad drive, you will need to OFFLINE it, open the case, replace it, add the new drive to the pool, and let it resilver.  Keep your fingers crossed... but it should be okay as long as your drives are robustly vetted.
 From CLI:
@@ Line 121: / Line 99: @@
 With any luck this will clear up pool errors.
 === Pool speed check ===

M: /* Fix any pool errors ASAP */

2025-03-24T12:15:51Z

Fix any pool errors ASAP

← Older revision		Revision as of 12:15, 24 March 2025
Line 109:		Line 109:
	It will be an error with the drive, the connectors to the drive, or a sun flare (seriously tho! like a power spike or VM termination that caused bad data).		It will be an error with the drive, the connectors to the drive, or a sun flare (seriously tho! like a power spike or VM termination that caused bad data).

	First thing, check the drive. Use smartmon tools, and if you see any issue there, better pull and replace it, and do a full badblocks test on the drive using the external SATA to USB mounter. Any problems, chuck it and replace it.		First thing, [[TrueNAS#Burn_in_a_new_drive\|check the drive]]. Use smartmon tools, and if you see any issue there, better pull and replace it, and do a full badblocks test on the drive using the external SATA to USB mounter. Any problems, chuck it and replace it.

	Once you are sure you have a good drive, you have to clean the pool of the bad data (currently the bad data exists, and will persist!).		Once you are sure you have a good drive, you have to clean the pool of the bad data (currently the bad data exists, and will persist!).

M: /* Maintenance */

2025-03-24T12:14:53Z

Maintenance

← Older revision		Revision as of 12:14, 24 March 2025
Line 102:		Line 102:

	Resilver happens when a drive issue requires the data to be rebalanced or redistributed. Buckle up for this one!		Resilver happens when a drive issue requires the data to be rebalanced or redistributed. Buckle up for this one!

			=== Fix any pool errors ASAP ===

			Once you get a pool error, you should immediately assess it so it doesn't corrupt data and make things worse.

			It will be an error with the drive, the connectors to the drive, or a sun flare (seriously tho! like a power spike or VM termination that caused bad data).

			First thing, check the drive. Use smartmon tools, and if you see any issue there, better pull and replace it, and do a full badblocks test on the drive using the external SATA to USB mounter. Any problems, chuck it and replace it.

			Once you are sure you have a good drive, you have to clean the pool of the bad data (currently the bad data exists, and will persist!).

			From CLI:
			sudo zpool status -v mine
			# rm all the files that are reported with errors - they are BAD
			sudo zpool clear mine
			# IMMEDIATELY after clearing errors, go into the UI and start a SCRUB!
			# Repeat as needed

			With any luck this will clear up pool errors.

	=== Pool speed check ===		=== Pool speed check ===

M: /* Upgrade from CORE to SCALE */

2025-01-13T22:33:51Z

Upgrade from CORE to SCALE

← Older revision		Revision as of 22:33, 13 January 2025
Line 165:		Line 165:
	** It started up flawlessy!		** It started up flawlessy!
	* I got advice to disallow root login		* I got advice to disallow root login
	** I went on a bender, adding a new user, moving home dir, and spent the time to get it configured as admin. See ~~[[hive]]~~ notes for details.		** I went on a bender, adding a new user, moving home dir, and spent the time to get it configured as admin. See #Users notes for details.
	* Re-set hostname		* Re-set hostname
	Network > Global Configuration > Settings > change hostname > Save		Network > Global Configuration > Settings > change hostname > Save

M: /* Overview */

2025-01-13T22:33:11Z

Overview

← Older revision		Revision as of 22:33, 13 January 2025
Line 1:		Line 1:
	== Overview ==		== Overview ==

			=== Users ===
			As of v24, TrueNAS sends alerts if you use root to log in to web UI. So I set up another user to login. I had to deal with these issues:
			* Even though everything was configured, I could not ssh until I added auxilary group "builtin_administrators" to the user.
			* Also, you should not disable password login as a password login user is required to log in to the web UI.
			* Home dirs are now properly located at
			/mnt/safe/safe-ds/software/apps/hive-home
			* I could not get connected to samba shares from bandit after ugrading bandit to ubuntu 24.04; finally got it working by checking the samba box on the user! wtf
			hive > Credentials Users > m Edit > check SMB User (and re-provide the same password)

	=== Pools ===		=== Pools ===

M at 15:41, 13 January 2025

2025-01-13T15:41:43Z

← Older revision		Revision as of 15:41, 13 January 2025
Line 156:		Line 156:
	** It started up flawlessy!		** It started up flawlessy!
	* I got advice to disallow root login		* I got advice to disallow root login
	** I went on a bender, adding a new user and spent the time to get it configured as admin.		** I went on a bender, adding a new user, moving home dir, and spent the time to get it configured as admin. See [[hive]] notes for details.
	** Home directories MUST be on a mounted pool, weird! ~~It is "by design"~~. ~~Mine are now here:~~		* Re-set hostname
	~~/mnt/safe/safe~~-~~ds/software/apps/hive-home/m~~
	* Change hostname ~~from freenas back to hive~~
	Network > Global Configuration > Settings > change hostname > Save		Network > Global Configuration > Settings > change hostname > Save