Melange history
GRIM DEATH
2024/03/23 We are harvesting the hive grim drives for use as melange drives for VM storate.
I changed the location of the hive System Dataset Pool from grim to safe.
I copied all grim data to safe (oops, i forgot SharedDownloads... it's empty now...)
I removed the 'grim' pool from FreeNAS.
Now I need to move the drives! I want to keep the PCI card as pass-thru, but the two grim drives are on it.
- make note of all hive drive assignments
- open melange
- remove both grim drives from the PCI passthru
- move the one mine drive that is on SATA from SATA to one of the PCI passthroughs
- move one safe drive from SATA to the other of the PCI passthroughs
- add both grim drives to SATA
- close and restart melange and see if you can reconnect everything
- the grim drives should now show up on melange, not passed through
- the safe and mine drives should show up passed through, but perhaps hive cannot associate them; if not, try to fix
- if not, RESET/KEEP GOING brother!
Let's go...
First, capture everything...
SATA DRIVES:
🌐 m@melange [~] sudo lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT DEVICE-ID(S)
sda 8:0 0 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FB01 /dev/disk/by-id/wwn-0x500a0751e5a2fb01
sdb 8:16 0 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FD56 /dev/disk/by-id/wwn-0x500a0751e5a2fd56
sdc 8:32 1 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D2 /dev/disk/by-id/wwn-0x500a0751e5a313d2
sdd 8:48 1 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D6 /dev/disk/by-id/wwn-0x500a0751e5a313d6
sde 8:64 1 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2117E59AAE1B /dev/disk/by-id/wwn-0x500a0751e59aae1b
sdf 8:80 1 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2121E5A2E131 /dev/disk/by-id/wwn-0x500a0751e5a2e131
sdg 8:96 1 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A3009A /dev/disk/by-id/wwn-0x500a0751e5a3009a
sdh 8:112 1 7.3T 0 disk /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5VUNJ0W706320H /dev/disk/by-id/wwn-0x5002538f33710e1d
nvme0n1 259:0 0 931.5G 0 disk /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N107994E /dev/disk/by-id/nvme-eui.002538510141169d
104 PASSTHRUS:
🌐 m@melange [~] sudo cat /etc/pve/qemu-server/104.conf boot: order=scsi0;net0 cores: 4 hostpci0: 0a:00.0,rombar=0 memory: 24576 name: hive net0: virtio=DA:E8:DA:81:EC:64,bridge=vmbr0,firewall=1 numa: 0 onboot: 1 ostype: l26 scsi0: local-lvm:vm-104-disk-0,size=25G scsi11: /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FD56,size=976762584K,backup=no,serial=2122E5A2FD56 scsi12: /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FB01,size=976762584K,backup=no,serial=2122E5A2FB01 scsi13: /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D2,size=976762584K,backup=no,serial=2122E5A313D2 scsi14: /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D6,size=976762584K,backup=no,serial=2122E5A313D6 scsi15: /dev/disk/by-id/ata-CT1000MX500SSD1_2117E59AAE1B,size=976762584K,backup=no,serial=2117E59AAE1B scsi16: /dev/disk/by-id/ata-CT1000MX500SSD1_2121E5A2E131,size=976762584K,backup=no,serial=2121E5A2E131 scsi17: /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A3009A,size=976762584K,backup=no,serial=2122E5A3009A scsi18: /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5VUNJ0W706320H,backup=0,serial=S5VUNJ0W706320H,size=7814026584K scsihw: virtio-scsi-pci smbios1: uuid=dc3d077c-0063-41fe-abde-a97674d14dc8 sockets: 1 startup: order=1,up=90 vmgenid: 1dcb048d-122f-4343-ad87-1a01ee8284a6
PRE-MOVE DRIVES:
- 7 1tb safe drives on SATA
- 1 8tb mine drive on SATA
- 2 4tb grim drives on PCI
- 6 8tb mine drives on PCI
PRE-MOVE hive:
da7 2122E5A3009A 931.51 GiB safe da8 S5VUNJ0W706320H 7.28 TiB mine da11 S5B0NW0NB01796J 3.64 TiB N/A da12 S4CXNF0M307721X 3.64 TiB N/A
POST-MOVE DRIVES:
- 6 1tb safe drives on SATA
- 2 8tb mine drives on SATA
- 1 1tb safe drive on SATA
- 7 8tb mine drives on PCI
STEPS THAT MAY NEED REVERSAL
Do i need to adjust hive or melange before opening case? I guess i could remove the grim sata passthru... AND the SAFE i'm going to move, too, it will no longer pass through (we will be using the PCI card passthru for it).
- shut down VMS, then hive (but not melange)
- remove two drives from SATA passthru, first is 1 from SAFE (moving to PCI card) and second is 1 from MINE
scsi16: /dev/disk/by-id/ata-CT1000MX500SSD1_2121E5A2E131,size=976762584K,backup=no,serial=2121E5A2E131 ^^^ SWITCHING TO SWAPPING THIS ONE, it is easier to access for swapping scsi17: /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A3009A,size=976762584K,backup=no,serial=2122E5A3009A ^^ ACTUALLY, NOT THIS ONE, leave it scsi18: /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5VUNJ0W706320H,backup=0,serial=S5VUNJ0W706320H,size=7814026584K
- shut down melange
- remove two 4GB drives from PCI: S5B0NW0NB01796J, S4CXNF0M307721X
- move the only 8TB from SATA to PCI: S5VUNJ0W706320H
- move 1 1TB from SATA to PCI: 2121E5A2E131 (not 2122E5A3009A)
- connect two 4GB drives to SATA: S5B0NW0NB01796J, S4CXNF0M307721X
WENT GREAT, hive recognized the move without me doing ANYTHING!!
Now we can set up the ex-grim 4TBs for VM usage, yay.
2023 July full upgrade
I need to install a Windows 11 VM. Also have Ubuntu 20.04 machines that should be moved to 22.04. Figured as good a reason as any for a full upgrade of everything.
- Upgrade bitpost first. Upon reboot, my $*(@ IP changed again. Fuck you google. Spent a while resetting that. Here are the notes (also in my Red Dead RP journal, keepin it real (real accessible when everything's down), lol!):
cast$ ssh bitpost # not bp or bitpost.com, so we get to the LAN resource sudo su - stronger_firewall_and_save # internet should now work # get new IP from whatsmyip # fix bitpost.com DNS at domains.google.com # WAIT for propagation.... might as well fix the other DNS records... sudo service dnsmasq restart ping bitpost.com # EVENTUALLY this will work! May need to repeat this AND previous step.
- Ask Tom to update E-S DNS to use new IP
- Upgrade abtdev1, then all Ubuntu boxes (glam is toughest), then positronic last, with this pattern:
mh-update-ubuntu # and reboot sudo do-release-upgrade # best to connect directly, but ssh worked fine too sudo shutdown -h now # to prep for melange reboot
- Upgrade hive's TrueNAS install, via https://hive CHECK FOR UPDATES, then shut it down
- Update and reboot melange PROXMOX install, via https://melange:8006 Datacenter > melange > Updates
- CHECK EVERYTHING
- proxmox samba share for backups
- samba shares
- at ptl to ensure it can get to positronic
- shitcutter and blogs and wiki and...
- I had a terrible time getting GLAM apache + PHP working again now that Ubuntu uses PHP 8.1; just needed to ENABLE THE MODULE, ffs:
a2enmod php8.1
6.3 > 7.0
Proxmox uses apt for upgrades. I followed this, for the most part.
- Update all VMS
- Shut down all VMS
- Fully update current version's apt packages - this took me from 6.3 to 6.4, a necessary first step.
sudo apt update sudo apt dist-upgrade
- Upgrade basic apt sources list from buster to bullseye
sudo sed -i 's/buster\/updates/bullseye-security/g;s/buster/bullseye/g' /etc/apt/sources.list # instructions discuss pve-enterprise but i needed to change pve-no-subscription instead - but same exact steps, otherwise # ie, leave this commented out, but might as well set to bullseye # /etc/apt/sources.list.d/pve-enterprise.list # and update this to bullseye # /etc/apt/sources.list.d/pve-no-subscription.list
- Perform the full upgrade to bullseye / pm 7
sudo apt update sudo apt dist-upgrade
- Reboot
Manual restart notes
NOTE: This shouldn't be a problem any more with newer staged order restart.
One time bandit samba shares don't mount (it comes up too fast perhaps?). So restart them then restart qbt nox:
mh-setup-samba-shares sudo service qbittorrent-nox restart
I did another round of `apt update && apt dist-upgrade` without stopping containers and it went fine (with bandit fixup still needed after reboot, tho).
sudo apt update sudo apt dist-upgrade ssh bandit mh-setup-samba-shares sudo service qbittorrent-nox restart
Add 7 1TB zraid
After adding 7 new 1 TB ssds:
🌐 m@melange [~] sudo lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT DEVICE-ID(S)
sdh 8:112 0 931.5G 0 disk /dev/disk/by-id/wwn-0x500a0751e5a2fb01 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FB01
sdi 8:128 0 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FD56 /dev/disk/by-id/wwn-0x500a0751e5a2fd56
sdj 8:144 1 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D2 /dev/disk/by-id/wwn-0x500a0751e5a313d2
sdk 8:160 1 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D6 /dev/disk/by-id/wwn-0x500a0751e5a313d6
sdl 8:176 1 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2117E59AAE1B /dev/disk/by-id/wwn-0x500a0751e59aae1b
sdm 8:192 1 931.5G 0 disk /dev/disk/by-id/wwn-0x500a0751e5a2e131 /dev/disk/by-id/ata-CT1000MX500SSD1_2121E5A2E131
sdn 8:208 1 931.5G 0 disk /dev/disk/by-id/wwn-0x500a0751e5a3009a /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A3009A
nvme0n1 259:0 0 931.5G 0 disk /dev/disk/by-id/nvme-eui.002538510141169d /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N107994E
Before adding 7 new 1 TB ssds:
🌐 m@melange [~] ls /dev/ autofs dm-8 i2c-7 net stdin tty28 tty5 ttyS12 ttyS6 vcsu1 block dm-9 i2c-8 null stdout tty29 tty50 ttyS13 ttyS7 vcsu2 btrfs-control dri i2c-9 nvme0 tty tty3 tty51 ttyS14 ttyS8 vcsu3 bus ecryptfs initctl nvme0n1 tty0 tty30 tty52 ttyS15 ttyS9 vcsu4 char fb0 input nvme0n1p1 tty1 tty31 tty53 ttyS16 udmabuf vcsu5 console fd kmsg nvme0n1p2 tty10 tty32 tty54 ttyS17 uhid vcsu6 core full kvm nvme0n1p3 tty11 tty33 tty55 ttyS18 uinput vfio cpu fuse lightnvm nvram tty12 tty34 tty56 ttyS19 urandom vga_arbiter cpu_dma_latency gpiochip0 log port tty13 tty35 tty57 ttyS2 userio vhci cuse hpet loop0 ppp tty14 tty36 tty58 ttyS20 vcs vhost-net disk hugepages loop1 pps0 tty15 tty37 tty59 ttyS21 vcs1 vhost-vsock dm-0 hwrng loop2 psaux tty16 tty38 tty6 ttyS22 vcs2 watchdog dm-1 i2c-0 loop3 ptmx tty17 tty39 tty60 ttyS23 vcs3 watchdog0 dm-10 i2c-1 loop4 ptp0 tty18 tty4 tty61 ttyS24 vcs4 zero dm-11 i2c-10 loop5 pts tty19 tty40 tty62 ttyS25 vcs5 zfs dm-12 i2c-11 loop6 pve tty2 tty41 tty63 ttyS26 vcs6 dm-13 i2c-12 loop7 random tty20 tty42 tty7 ttyS27 vcsa dm-14 i2c-13 loop-control rfkill tty21 tty43 tty8 ttyS28 vcsa1 dm-2 i2c-14 mapper rtc tty22 tty44 tty9 ttyS29 vcsa2 dm-3 i2c-2 mcelog rtc0 tty23 tty45 ttyprintk ttyS3 vcsa3 dm-4 i2c-3 mem shm tty24 tty46 ttyS0 ttyS30 vcsa4 dm-5 i2c-4 mpt2ctl snapshot tty25 tty47 ttyS1 ttyS31 vcsa5 dm-6 i2c-5 mpt3ctl snd tty26 tty48 ttyS10 ttyS4 vcsa6 dm-7 i2c-6 mqueue stderr tty27 tty49 ttyS11 ttyS5 vcsu
macOS USB passthru failed attempt
That doesn't work on macOS. Tried setting usb mapping via console, following this:
sudo qm monitor 111 qm> info usbhost qm> quit sudo qm set 111 -usb1 host=05ac:12a8
No luck, same result. Reading his remarks on USB forwarding, try resetting machine type:
machine: pc-q35-6.0 (instead of latest, which was 6.2 at time of writing) remove this from /etc/pve/qemu-server/111.conf: -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off
Hmm.. perhaps it is a conflict between Nick's usb keyboard config and my usb port selection... try plugging usb into another port and remapping...
No luck. FFS. Reset to 6.2 and see if we have any luck with hotplug line removed from config... Nope.
Keep trying permutations... nothing from googling indicates taht this shouldn't just FUCKING WORK...
Remove this and re-add the hotplug line, on the off chance it shouldn't be used with q35 v6.2:
-global nec-usb-xhci.msi=off
Nope, that jsut caused a problem with "Springboard", not working on this Mac, or some shit. Re-adding the line...
Well what now? Google more?
Update and reboot proxmox and retry... no luck.
Try changing from blue to light-blue port... the device is mapped so it should be passed through... nope.
Try this guy's approach to mount an EFI Disk
lsusb Bus 004 Device 009: ID 05ac:12a8 Apple, Inc. iPhone 5/5C/5S/6/SE ls -al /dev/bus/usb/004/009 crw-rw-r-- 1 root root 189, 392 Jul 22 16:10 /dev/bus/usb/004/009 sudo emacs /etc/pve/qemu-server/111.conf lxc.cgroup.devices.allow: c 189:* rwm lxc.mount.entry: /dev/bus/usb/004 dev/bus/usb/004 none bind,optional,create=dir
Nope.
Try mapping the port instead of device ID, from the Proxmox UI... Nope.
How can i check the apple side for any issues? straight up google for that, macOS not seeing a USB device.
System Information > USB > nada
hrmphhhh. Never got it working. RE-google next month maybe...
During original configuration, I added samba shares manually.
sudo emacs /etc/fstab # and paste samba stanza from another machine sudo emacs /root/samba_credentials sudo mkdir /spiceflow && sudo chmod 777 /spiceflow 🌐 m@melange [~] mkdir /spiceflow/bitpost 🌐 m@melange [~] mkdir /spiceflow/grim 🌐 m@melange [~] mkdir /spiceflow/mack 🌐 m@melange [~] mkdir /spiceflow/reservoir 🌐 m@melange [~] mkdir /spiceflow/sassy 🌐 m@melange [~] mkdir /spiceflow/safe
Now you can mount em up and hang em high!