Melange
Running ProxMox and many VMs.
VMs
Hive - AbtDev1 - Bandit - Positronic - Morosoph - Glam - Matryoshka - Hoard
Usage
Copy media
On melange, you can plug microSSDs into the melange hub, and you can also reach samba shares.
So direct copy from shares to microSSD is possible without 1Gbit network bottleneck.
sudo su - mount /media rsync --progress /spiceflow/... /media/...
Turns out that's not much of a bottleneck yet... my best cards were only getting 100MB/s write speed... test more cards maybe! In the meantime, just use Nautilus on cast.
Reading should def be faster though, if you need to copy from microSSD to shares.
Configuration
Start order
Each VM has a "Start/Shutdown order" Option. We want some VMs to start first. To do that, we set them to a specific order number, with a delay, which will slot them to start before the default VMs. You can also specify an up value, allowing for a pause before starting the VMs in the next order. Nice.
We want hive to start first, so NAS drives can be connected on startup of other VMs.
We might want positronic to start before AbtDev1.... But to do that right, we need to set ALL other VMs to order 2, and AbtDev1 to order 3. It's not even really needed though - AbtDev1 won't be connecting to positronic until I go there and run a development build, and by then positronic should be well under way. So I'll go the much-easier route and leave all VMs other than hive to the default.
NOTE that bitpost is on bare metal, and needs positronic, so manage that manually on any restart of melange!
Hive: order=1,up=60 All other VMs: order=Any (the default)
Upgrade
Proxmox uses apt for upgrades.
6.3 > 7.0
I followed this, for the most part.
- Update all VMS
- Shut down all VMS
- Fully update current version's apt packages - this took me from 6.3 to 6.4, a necessary first step.
sudo apt update sudo apt dist-upgrade
- Upgrade basic apt sources list from buster to bullseye
sudo sed -i 's/buster\/updates/bullseye-security/g;s/buster/bullseye/g' /etc/apt/sources.list # instructions discuss pve-enterprise but i needed to change pve-no-subscription instead - but same exact steps, otherwise # ie, leave this commented out, but might as well set to bullseye # /etc/apt/sources.list.d/pve-enterprise.list # and update this to bullseye # /etc/apt/sources.list.d/pve-no-subscription.list
- Perform the full upgrade to bullseye / pm 7
sudo apt update sudo apt dist-upgrade
- Reboot
Melange VMs come back nicely. The only snag I caught was that bandit samba shares don't mount (it comes up too fast perhaps?). So restart them then restart qbt nox:
mh-setup-samba-shares sudo service qbittorrent-nox restart
Looking good!
apt dist-upgrade
I did another round of `apt update && apt dist-upgrade` without stopping containers and it went fine (with bandit fixup still needed after reboot, tho).
sudo apt update sudo apt dist-upgrade ssh bandit mh-setup-samba-shares sudo service qbittorrent-nox restart
Hardware
- LSI Broadcom SAS 9300-8i 8-port 12Gb/s SATA+SAS PCI-Express 3.0 Low Profile Host Bus Adapter: Pass through of 8 drive connections to hive to use for its storage
- 4 SAS connections on mobo, with a SAS-to-SATA cable; make sure to turn on "SATA mode" for U.2 in BIOS; all passed through to hive
- 4 SATA ssd connections on mobo; 3 are passed through to hive
- 1 TB M.2 nvme
/dev/mapper/pve-root / # /dev/nvme0n1p3 /boot/efi /dev/nvme0n1p2
Passthroughs
We pass the LSI PCI-E board (and its 8 ssd drives), the 4 SATA SSD drives, and the 4 U.2 SSD drives to Hive, for storage management.
LSI PCI-E board passthru
TODO (I don't remember)
SSD drives passthru
- Identify drives on melange
🌐 m@melange [~] sudo lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT DEVICE-ID(S)
sdh 8:112 0 931.5G 0 disk /dev/disk/by-id/wwn-0x500a0751e5a2fb01 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FB01
sdi 8:128 0 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FD56 /dev/disk/by-id/wwn-0x500a0751e5a2fd56
sdj 8:144 1 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D2 /dev/disk/by-id/wwn-0x500a0751e5a313d2
sdk 8:160 1 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D6 /dev/disk/by-id/wwn-0x500a0751e5a313d6
sdl 8:176 1 931.5G 0 disk /dev/disk/by-id/ata-CT1000MX500SSD1_2117E59AAE1B /dev/disk/by-id/wwn-0x500a0751e59aae1b
sdm 8:192 1 931.5G 0 disk /dev/disk/by-id/wwn-0x500a0751e5a2e131 /dev/disk/by-id/ata-CT1000MX500SSD1_2121E5A2E131
sdn 8:208 1 931.5G 0 disk /dev/disk/by-id/wwn-0x500a0751e5a3009a /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A3009A
nvme0n1 259:0 0 931.5G 0 disk /dev/disk/by-id/nvme-eui.002538510141169d /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N107994E
- Use qm set to pass them all to hive (VM 104)
sudo qm set 104 -scsi11 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FD56 sudo qm set 104 -scsi12 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FB01 sudo qm set 104 -scsi13 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D2 sudo qm set 104 -scsi14 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D6 sudo qm set 104 -scsi15 /dev/disk/by-id/ata-CT1000MX500SSD1_2117E59AAE1B sudo qm set 104 -scsi16 /dev/disk/by-id/ata-CT1000MX500SSD1_2121E5A2E131 sudo qm set 104 -scsi17 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A3009A # check sudo cat /etc/pve/qemu-server/104.conf
(restart hive)
USB
Plug something into melange and see what it sees:
lsusb Bus 004 Device 002: ID 05ac:12a8 Apple, Inc. iPhone 5/5C/5S/6/SE
Now we will pass that port through:
proxmox UI > Datacenter > Nodes > VMs > 111 (matcha) > Hardware > Add > USB > it will actually show up in a pick list!
Hell yeah so easy.
macOS
That doesn't work on macOS. Tried setting usb mapping via console, following this:
sudo qm monitor 111 qm> info usbhost qm> quit sudo qm set 111 -usb1 host=05ac:12a8
No luck, same result. Reading his remarks on USB forwarding, try resetting machine type:
machine: pc-q35-6.0 (instead of latest, which was 6.2 at time of writing) remove this from /etc/pve/qemu-server/111.conf: -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off
Hmm.. perhaps it is a conflict between Nick's usb keyboard config and my usb port selection... try plugging usb into another port and remapping...
No luck. FFS. Reset to 6.2 and see if we have any luck with hotplug line removed from config... Nope.
Keep trying permutations... nothing from googling indicates taht this shouldn't just FUCKING WORK...
Remove this and re-add the hotplug line, on the off chance it shouldn't be used with q35 v6.2:
-global nec-usb-xhci.msi=off
Nope, that jsut caused a problem with "Springboard", not working on this Mac, or some shit. Re-adding the line...
Well what now? Google more?
Update and reboot proxmox and retry... no luck.
Try changing from blue to light-blue port... the device is mapped so it should be passed through... nope.
Try this guy's approach to mount an EFI Disk
lsusb Bus 004 Device 009: ID 05ac:12a8 Apple, Inc. iPhone 5/5C/5S/6/SE ls -al /dev/bus/usb/004/009 crw-rw-r-- 1 root root 189, 392 Jul 22 16:10 /dev/bus/usb/004/009 sudo emacs /etc/pve/qemu-server/111.conf lxc.cgroup.devices.allow: c 189:* rwm lxc.mount.entry: /dev/bus/usb/004 dev/bus/usb/004 none bind,optional,create=dir
Nope.
Try mapping the port instead of device ID, from the Proxmox UI...
How can i check the apple side for any issues? straight up google for that, macOS not seeing a USB device.
Hardware history
Installation
sudo emacs /etc/fstab # and paste samba stanza from another machine sudo emacs /root/samba_credentials sudo mkdir /spiceflow && sudo chmod 777 /spiceflow 🌐 m@melange [~] mkdir /spiceflow/bitpost 🌐 m@melange [~] mkdir /spiceflow/grim 🌐 m@melange [~] mkdir /spiceflow/mack 🌐 m@melange [~] mkdir /spiceflow/reservoir 🌐 m@melange [~] mkdir /spiceflow/sassy 🌐 m@melange [~] mkdir /spiceflow/safe
Now you can mount em up and hang em high!