Melange history

GRIM DEATH

NOTE on 2024/03/23 I removed the 'grim' pool from FreeNAS to free the drives for VM storage. The content was moved to 'mine'.

Now I need to move the drives! I want to keep the PCI card as pass-thru, but the two grim drives are on it.

• make note of all hive drive assignments
• open melange
  • remove both grim drives from the PCI passthru
  • move the one mine drive that is on SATA from SATA to one of the PCI passthroughs
  • move one safe drive from SATA to the other of the PCI passthroughs
  • add both grim drives to SATA
• close and restart melange and see if you can reconnect everything; if not, RESET/KEEP GOING brother!

I removed the passthru on the drive:

 🌐 m@melange  [~] sudo lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'

NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT DEVICE-ID(S)
sdh                            8:112  1   7.3T  0 disk   /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5VUNJ0W706320H /dev/disk/by-id/wwn-0x5002538f33710e1d

sudo cat /etc/pve/qemu-server/104.conf

sudo qm set 104 -delete ...

2023 July full upgrade

I need to install a Windows 11 VM. Also have Ubuntu 20.04 machines that should be moved to 22.04. Figured as good a reason as any for a full upgrade of everything.

• Upgrade bitpost first. Upon reboot, my $*(@ IP changed again. Fuck you google. Spent a while resetting that. Here are the notes (also in my Red Dead RP journal, keepin it real (real accessible when everything's down), lol!):

cast$ ssh bitpost # not bp or bitpost.com, so we get to the LAN resource
sudo su -
stronger_firewall_and_save # internet should now work
# get new IP from whatsmyip
# fix bitpost.com DNS at domains.google.com
# WAIT for propagation.... might as well fix the other DNS records...
sudo service dnsmasq restart
ping bitpost.com # EVENTUALLY this will work! May need to repeat this AND previous step.

• Ask Tom to update E-S DNS to use new IP
• Upgrade abtdev1, then all Ubuntu boxes (glam is toughest), then positronic last, with this pattern:

mh-update-ubuntu # and reboot
sudo do-release-upgrade # best to connect directly, but ssh worked fine too
sudo shutdown -h now # to prep for melange reboot

• Upgrade hive's TrueNAS install, via https://hive CHECK FOR UPDATES, then shut it down
• Update and reboot melange PROXMOX install, via https://melange:8006 Datacenter > melange > Updates
• CHECK EVERYTHING
  • proxmox samba share for backups
  • samba shares
  • at ptl to ensure it can get to positronic
  • shitcutter and blogs and wiki and...
  • I had a terrible time getting GLAM apache + PHP working again now that Ubuntu uses PHP 8.1; just needed to ENABLE THE MODULE, ffs:

a2enmod php8.1

6.3 > 7.0

Proxmox uses apt for upgrades. I followed this, for the most part.

• Update all VMS
• Shut down all VMS
• Fully update current version's apt packages - this took me from 6.3 to 6.4, a necessary first step.

sudo apt update
sudo apt dist-upgrade

• Upgrade basic apt sources list from buster to bullseye

sudo sed -i 's/buster\/updates/bullseye-security/g;s/buster/bullseye/g' /etc/apt/sources.list
# instructions discuss pve-enterprise but i needed to change pve-no-subscription instead - but same exact steps, otherwise
# ie, leave this commented out, but might as well set to bullseye
# /etc/apt/sources.list.d/pve-enterprise.list
# and update this to bullseye
# /etc/apt/sources.list.d/pve-no-subscription.list

• Perform the full upgrade to bullseye / pm 7

sudo apt update
sudo apt dist-upgrade

• Reboot

Manual restart notes

NOTE: This shouldn't be a problem any more with newer staged order restart.

One time bandit samba shares don't mount (it comes up too fast perhaps?). So restart them then restart qbt nox:

mh-setup-samba-shares
sudo service qbittorrent-nox restart

I did another round of `apt update && apt dist-upgrade` without stopping containers and it went fine (with bandit fixup still needed after reboot, tho).

sudo apt update
sudo apt dist-upgrade
ssh bandit
mh-setup-samba-shares
sudo service qbittorrent-nox restart

Add 7 1TB zraid

After adding 7 new 1 TB ssds:

 🌐 m@melange  [~] sudo lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT DEVICE-ID(S)
sdh                            8:112  0 931.5G  0 disk   /dev/disk/by-id/wwn-0x500a0751e5a2fb01 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FB01
sdi                            8:128  0 931.5G  0 disk   /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FD56 /dev/disk/by-id/wwn-0x500a0751e5a2fd56
sdj                            8:144  1 931.5G  0 disk   /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D2 /dev/disk/by-id/wwn-0x500a0751e5a313d2
sdk                            8:160  1 931.5G  0 disk   /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D6 /dev/disk/by-id/wwn-0x500a0751e5a313d6
sdl                            8:176  1 931.5G  0 disk   /dev/disk/by-id/ata-CT1000MX500SSD1_2117E59AAE1B /dev/disk/by-id/wwn-0x500a0751e59aae1b
sdm                            8:192  1 931.5G  0 disk   /dev/disk/by-id/wwn-0x500a0751e5a2e131 /dev/disk/by-id/ata-CT1000MX500SSD1_2121E5A2E131
sdn                            8:208  1 931.5G  0 disk   /dev/disk/by-id/wwn-0x500a0751e5a3009a /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A3009A
nvme0n1                      259:0    0 931.5G  0 disk   /dev/disk/by-id/nvme-eui.002538510141169d /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N107994E

Before adding 7 new 1 TB ssds:

 🌐 m@melange  [~] ls /dev/
autofs           dm-8       i2c-7         net        stdin   tty28  tty5       ttyS12  ttyS6    vcsu1
block            dm-9       i2c-8         null       stdout  tty29  tty50      ttyS13  ttyS7    vcsu2
btrfs-control    dri        i2c-9         nvme0      tty     tty3   tty51      ttyS14  ttyS8    vcsu3
bus              ecryptfs   initctl       nvme0n1    tty0    tty30  tty52      ttyS15  ttyS9    vcsu4
char             fb0        input         nvme0n1p1  tty1    tty31  tty53      ttyS16  udmabuf  vcsu5
console          fd         kmsg          nvme0n1p2  tty10   tty32  tty54      ttyS17  uhid     vcsu6
core             full       kvm           nvme0n1p3  tty11   tty33  tty55      ttyS18  uinput   vfio
cpu              fuse       lightnvm      nvram      tty12   tty34  tty56      ttyS19  urandom  vga_arbiter
cpu_dma_latency  gpiochip0  log           port       tty13   tty35  tty57      ttyS2   userio   vhci
cuse             hpet       loop0         ppp        tty14   tty36  tty58      ttyS20  vcs      vhost-net
disk             hugepages  loop1         pps0       tty15   tty37  tty59      ttyS21  vcs1     vhost-vsock
dm-0             hwrng      loop2         psaux      tty16   tty38  tty6       ttyS22  vcs2     watchdog
dm-1             i2c-0      loop3         ptmx       tty17   tty39  tty60      ttyS23  vcs3     watchdog0
dm-10            i2c-1      loop4         ptp0       tty18   tty4   tty61      ttyS24  vcs4     zero
dm-11            i2c-10     loop5         pts        tty19   tty40  tty62      ttyS25  vcs5     zfs
dm-12            i2c-11     loop6         pve        tty2    tty41  tty63      ttyS26  vcs6
dm-13            i2c-12     loop7         random     tty20   tty42  tty7       ttyS27  vcsa
dm-14            i2c-13     loop-control  rfkill     tty21   tty43  tty8       ttyS28  vcsa1
dm-2             i2c-14     mapper        rtc        tty22   tty44  tty9       ttyS29  vcsa2
dm-3             i2c-2      mcelog        rtc0       tty23   tty45  ttyprintk  ttyS3   vcsa3
dm-4             i2c-3      mem           shm        tty24   tty46  ttyS0      ttyS30  vcsa4
dm-5             i2c-4      mpt2ctl       snapshot   tty25   tty47  ttyS1      ttyS31  vcsa5
dm-6             i2c-5      mpt3ctl       snd        tty26   tty48  ttyS10     ttyS4   vcsa6
dm-7             i2c-6      mqueue        stderr     tty27   tty49  ttyS11     ttyS5   vcsu

macOS USB passthru failed attempt

That doesn't work on macOS. Tried setting usb mapping via console, following this:

sudo qm monitor 111
qm> info usbhost
qm> quit
sudo qm set 111 -usb1 host=05ac:12a8

No luck, same result. Reading his remarks on USB forwarding, try resetting machine type:

machine: pc-q35-6.0 (instead of latest, which was 6.2 at time of writing)
remove this from /etc/pve/qemu-server/111.conf: -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off

Hmm.. perhaps it is a conflict between Nick's usb keyboard config and my usb port selection... try plugging usb into another port and remapping...

No luck. FFS. Reset to 6.2 and see if we have any luck with hotplug line removed from config... Nope.

Keep trying permutations... nothing from googling indicates taht this shouldn't just FUCKING WORK...

Remove this and re-add the hotplug line, on the off chance it shouldn't be used with q35 v6.2:

-global nec-usb-xhci.msi=off

Nope, that jsut caused a problem with "Springboard", not working on this Mac, or some shit. Re-adding the line...

Well what now? Google more?

Update and reboot proxmox and retry... no luck.

Try changing from blue to light-blue port... the device is mapped so it should be passed through... nope.

Try this guy's approach to mount an EFI Disk

lsusb
  Bus 004 Device 009: ID 05ac:12a8 Apple, Inc. iPhone 5/5C/5S/6/SE
ls -al /dev/bus/usb/004/009
  crw-rw-r-- 1 root root 189, 392 Jul 22 16:10 /dev/bus/usb/004/009
sudo emacs /etc/pve/qemu-server/111.conf
  lxc.cgroup.devices.allow: c 189:* rwm
  lxc.mount.entry: /dev/bus/usb/004 dev/bus/usb/004 none bind,optional,create=dir

Nope.

Try mapping the port instead of device ID, from the Proxmox UI... Nope.

How can i check the apple side for any issues? straight up google for that, macOS not seeing a USB device.

System Information > USB > nada

hrmphhhh. Never got it working. RE-google next month maybe...

Add samba shares manually

During original configuration, I added samba shares manually.

sudo emacs /etc/fstab # and paste samba stanza from another machine
sudo emacs /root/samba_credentials
sudo mkdir /spiceflow && sudo chmod 777 /spiceflow
🌐 m@melange  [~] mkdir /spiceflow/bitpost
🌐 m@melange  [~] mkdir /spiceflow/grim
🌐 m@melange  [~] mkdir /spiceflow/mack
🌐 m@melange  [~] mkdir /spiceflow/reservoir
🌐 m@melange  [~] mkdir /spiceflow/sassy
🌐 m@melange  [~] mkdir /spiceflow/safe

Now you can mount em up and hang em high!

Old VMs

Mongolden