Melange: Difference between revisions

From Bitpost wiki
No edit summary
 
(20 intermediate revisions by the same user not shown)
Line 3: Line 3:
== VMs ==
== VMs ==


[[Hive]] - [[AbtDev1]] - [[Matcha]] - [[Bandit]] - [[Positronic]] - [[Morosoph]] - [[Glam]] - [[Matryoshka]] - [[Hoard]] - [[Tornados]]
[[AbtDev1]] - [[Archie]] - [[Bandit]] - [[Dev2]] - [[Glam]] - [[Hive]] - [[Matcha]] - [[Matryoshka]] - [[Morosoph]] - [[Positronic]] - [[Tornados]] - [[Windozer]]
 
old: [[Hoard]]
 
[[Add a new VM to melange]] - [[Clone a melange VM]]
 
=== VM Backups ===
Backups of all VMs (except hive data) are here:
Datacenter > Storage > bitpost-backup
That is a directly-configured samba link to bitpost.com's softraid.  It has enough raided storage to fit cascading backups for all VMs there.  Whoooop.
 
Proxmox UI manages ALL samba/cifs configuration.
 
NOTE: NEVER MOUNT /spiceflow/bitpost manually, as it conflicts with Proxmox samba link.  The Proxmox samba link should come up on melange reboot, and do everything it needs, itself.
 
You must configure the share as follows:
Datacenter > Storage > Add > Samba/CIFS
ID: bitpost-backup
Server: USE IP: 192.168.22.1
Username/pw: see private.txt
Share: backup
Nodes: (no restrictions/default)
Enable cheked
Content: VZDump
Domain: .  (a single dot)
Backup Retention:
    retention: keep-daily=1, keep-last=1, keep-monthly=1, keep-weekly=1, keep-yearly=1
 
It's been VERY troublesome to keep up and running, we will see if this new improved config (2025/01/01) does the trick.
 
==== Backup log ====
You can view the backup log to see exactly what was done.  Logs are kept next to the backups.  Eg:
[cast] cat /spiceflow/bitpost/dump/vzdump-qemu-101-2024_11_27-03_30_59.log
[melange] cat /mnt/pve/bitpost-backup/dump/vzdump-qemu-101-2024_11_27-03_30_59.log
 
Also, see melange activity in realtime with:
root@melange:~# journalctl -f


== Usage ==
== Usage ==
Line 12: Line 48:
So direct copy from shares to microSSD is possible without 1Gbit network bottleneck.
So direct copy from shares to microSSD is possible without 1Gbit network bottleneck.
  sudo su -
  sudo su -
  mount /media
  mount /dev/sdh1 /media
  rsync --progress /spiceflow/... /media/...
mount /spiceflow/safe
  rsync --progress /media/... /spiceflow/safe/...


Turns out that's not much of a bottleneck yet... my best cards were only getting 100MB/s write speed... test more cards maybe!  In the meantime, just use Nautilus on cast.
Turns out that's not much of a bottleneck yet... my best cards were only getting 100MB/s write speed... test more cards maybe!  In the meantime, just use Nautilus on cast.
Line 49: Line 86:
  VM > Options > Start/shutdown order: order=1,up=90
  VM > Options > Start/shutdown order: order=1,up=90


=== 6.3 > 7.0 ===
=== pve7to8 ===
This was pretty painless, following their [https://pve.proxmox.com/wiki/Upgrade_from_7_to_8 excellent guide]!


Proxmox uses apt for upgrades.
<pre>
I followed [https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0 this], for the most part.
Proxmox:  
* Update all VMS
use pve7to8, which vets upgrade
* Shut down all VMS
it advised to install:
* Fully update current version's apt packages - this took me from 6.3 to 6.4, a necessary first step.
apt install grub-efi-amd64 # for booting
sudo apt update
apt install chrony # for ntp time sync, better than shit systemd service which only runs on boot
sudo apt dist-upgrade
chrony is default in newer proxmox installs
* Upgrade basic apt sources list from buster to bullseye
sudo sed -i 's/buster\/updates/bullseye-security/g;s/buster/bullseye/g' /etc/apt/sources.list
# instructions discuss pve-enterprise but i needed to change pve-no-subscription instead - but same exact steps, otherwise
# ie, leave this commented out, but might as well set to bullseye
# /etc/apt/sources.list.d/pve-enterprise.list
# and update this to bullseye
# /etc/apt/sources.list.d/pve-no-subscription.list
* Perform the full upgrade to bullseye / pm 7
sudo apt update
sudo apt dist-upgrade
* Reboot


=== Troubleshoot ===
looks like we are ready!


NOTE: This shouldn't be a problem any more with newer staged order restart.
sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list
important!


One time bandit samba shares don't mount (it comes up too fast perhaps?).  So restart them then restart qbt nox:
here’s our source file, change from bullseye to bookworm:
mh-setup-samba-shares
nano /etc/apt/sources.list.d/pve-no-subscription.list
sudo service qbittorrent-nox restart


=== apt dist-upgrade ===
apt update
apt dist-upgrade # HERE WE GO


I did another round of `apt update && apt dist-upgrade` without stopping containers and it went fine (with bandit fixup still needed after reboot, tho).
sudoers, let’s take the new file, and re-add this to the end (...)


sudo apt update
</pre>
sudo apt dist-upgrade
ssh bandit
mh-setup-samba-shares
sudo service qbittorrent-nox restart


== Hardware ==
== Hardware ==
* Motherboard: [https://www.asus.com/motherboards-components/motherboards/workstation/pro-ws-x570-ace/ ASUS Pro WS X570-ACE]
** Storage: 2 M.2 slots, one free!
1 TB M.2 nvme
/dev/mapper/pve-root / # /dev/nvme0n1p3
/boot/efi /dev/nvme0n1p2
* GPU: Nvidia GK208 - GeForce GT 640
sudo lspci|grep VGA
VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 640 Rev. 2] (rev a1)


* [https://www.amazon.com/gp/product/B00DSURZYS/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 LSI Broadcom SAS 9300-8i 8-port 12Gb/s SATA+SAS PCI-Express 3.0 Low Profile Host Bus Adapter]: Pass through of 8 drive connections to hive to use for its storage
* [https://www.amazon.com/gp/product/B00DSURZYS/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 LSI Broadcom SAS 9300-8i 8-port 12Gb/s SATA+SAS PCI-Express 3.0 Low Profile Host Bus Adapter]: Pass through of 8 drive connections to hive to use for its storage
Line 94: Line 126:
* 4 SAS connections on mobo, with a SAS-to-SATA cable; make sure to turn on "SATA mode" for U.2 in BIOS; all passed through to hive
* 4 SAS connections on mobo, with a SAS-to-SATA cable; make sure to turn on "SATA mode" for U.2 in BIOS; all passed through to hive


* 4 SATA ssd connections on mobo; 3 are passed through to hive
* 4 SATA ssd connections on mobo, all passed through to hive
 
* 1 TB M.2 nvme
/dev/mapper/pve-root / # /dev/nvme0n1p3
/boot/efi /dev/nvme0n1p2


=== Memory ===
=== Memory ===
Line 128: Line 156:


==== LSI PCI-E board passthru ====
==== LSI PCI-E board passthru ====
TODO (I don't remember)
 
I didn't log this when I did it originally.  It must be in the 104.conf file tho... I'm guessing it is this:
 
🌐 m@melange  [~] sudo cat /etc/pve/qemu-server/104.conf
scsihw: virtio-scsi-pci


==== SSD drives passthru ====
==== SSD drives passthru ====
Line 170: Line 202:
Hell yeah so easy.
Hell yeah so easy.


===== macOS =====
== [[Melange history|History]] ==
 
That doesn't work on macOS.  Tried setting usb mapping via console, following this:
sudo qm monitor 111
qm> info usbhost
qm> quit
sudo qm set 111 -usb1 host=05ac:12a8
 
No luck, same result.  Reading his remarks on USB forwarding, try resetting machine type:
machine: pc-q35-6.0 (instead of latest, which was 6.2 at time of writing)
remove this from /etc/pve/qemu-server/111.conf: -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off
 
Hmm.. perhaps it is a conflict between Nick's usb keyboard config and my usb port selection... try plugging usb into another port and remapping...
 
No luck.  FFS.  Reset to 6.2 and see if we have any luck with hotplug line removed from config... Nope.
 
Keep trying permutations... nothing from googling indicates taht this shouldn't just FUCKING WORK...
 
Remove this and re-add the hotplug line, on the off chance it shouldn't be used with q35 v6.2:
-global nec-usb-xhci.msi=off
 
Nope, that jsut caused a problem with "Springboard", not working on this Mac, or some shit.  Re-adding the line...
 
Well what now?  Google more? 
 
Update and reboot proxmox and retry... no luck.
 
Try changing from blue to light-blue port... the device is mapped so it should be passed through... nope.
 
Try [https://raw.githubusercontent.com/vzamora/Proxmox-Cheatsheet/main/General%20Proxmox%20Setup%20and%20Notes/USB%20Passthrough%20Notes.md this guy's approach] to mount an EFI Disk
lsusb
  Bus 004 Device 009: ID 05ac:12a8 Apple, Inc. iPhone 5/5C/5S/6/SE
ls -al /dev/bus/usb/004/009
  crw-rw-r-- 1 root root 189, 392 Jul 22 16:10 /dev/bus/usb/004/009
sudo emacs /etc/pve/qemu-server/111.conf
  lxc.cgroup.devices.allow: c 189:* rwm
  lxc.mount.entry: /dev/bus/usb/004 dev/bus/usb/004 none bind,optional,create=dir
 
Nope.
 
Try mapping the port instead of device ID, from the Proxmox UI... Nope.
 
How can i check the apple side for any issues?  straight up google for that, macOS not seeing a USB device.
System Information > USB > nada
 
hrmphhhh.  Never got it working.  RE-google next month maybe...
 
=== [[Melange history|Hardware history]] ===
 
== Installation ==
=== Add samba shares manually ===
sudo emacs /etc/fstab # and paste samba stanza from another machine
sudo emacs /root/samba_credentials
sudo mkdir /spiceflow && sudo chmod 777 /spiceflow
🌐 m@melange  [~] mkdir /spiceflow/bitpost
🌐 m@melange  [~] mkdir /spiceflow/grim
🌐 m@melange  [~] mkdir /spiceflow/mack
🌐 m@melange  [~] mkdir /spiceflow/reservoir
🌐 m@melange  [~] mkdir /spiceflow/sassy
🌐 m@melange  [~] mkdir /spiceflow/safe
 
Now you can mount em up and hang em high!

Latest revision as of 23:43, 6 February 2025

Running ProxMox and many VMs.

VMs

AbtDev1 - Archie - Bandit - Dev2 - Glam - Hive - Matcha - Matryoshka - Morosoph - Positronic - Tornados - Windozer

old: Hoard

Add a new VM to melange - Clone a melange VM

VM Backups

Backups of all VMs (except hive data) are here: 

Datacenter > Storage > bitpost-backup

That is a directly-configured samba link to bitpost.com's softraid. It has enough raided storage to fit cascading backups for all VMs there. Whoooop.

Proxmox UI manages ALL samba/cifs configuration.

NOTE: NEVER MOUNT /spiceflow/bitpost manually, as it conflicts with Proxmox samba link. The Proxmox samba link should come up on melange reboot, and do everything it needs, itself.

You must configure the share as follows: 

Datacenter > Storage > Add > Samba/CIFS
ID: bitpost-backup
Server: USE IP: 192.168.22.1
Username/pw: see private.txt
Share: backup
Nodes: (no restrictions/default)
Enable cheked
Content: VZDump
Domain: .  (a single dot)
Backup Retention:
   retention: keep-daily=1, keep-last=1, keep-monthly=1, keep-weekly=1, keep-yearly=1

It's been VERY troublesome to keep up and running, we will see if this new improved config (2025/01/01) does the trick.

Backup log

You can view the backup log to see exactly what was done. Logs are kept next to the backups. Eg: 

[cast] cat /spiceflow/bitpost/dump/vzdump-qemu-101-2024_11_27-03_30_59.log 
[melange] cat /mnt/pve/bitpost-backup/dump/vzdump-qemu-101-2024_11_27-03_30_59.log

Also, see melange activity in realtime with: 

root@melange:~# journalctl -f

Usage

Copy media

On melange, you can plug microSSDs into the melange hub, and you can also reach samba shares.

So direct copy from shares to microSSD is possible without 1Gbit network bottleneck. 

sudo su -
mount /dev/sdh1 /media
mount /spiceflow/safe
rsync --progress /media/... /spiceflow/safe/...

Turns out that's not much of a bottleneck yet... my best cards were only getting 100MB/s write speed... test more cards maybe! In the meantime, just use Nautilus on cast.

Reading should def be faster though, if you need to copy from microSSD to shares.

Configuration

Start order

Each VM has a "Start/Shutdown order" Option. We want some VMs to start first. To do that, we set them to a specific order number, with a delay, which will slot them to start before the default VMs. You can also specify an up value, allowing for a pause before starting the VMs in the next order. Nice.

We want hive to start first, so NAS drives can be connected on startup of other VMs.

We might want positronic to start before AbtDev1.... But to do that right, we need to set ALL other VMs to order 2, and AbtDev1 to order 3. It's not even really needed though - AbtDev1 won't be connecting to positronic until I go there and run a development build, and by then positronic should be well under way. So I'll go the much-easier route and leave all VMs other than hive to the default.

NOTE that bitpost is on bare metal, and needs positronic, so manage that manually on any restart of melange! 

Hive: order=1,up=60
All other VMs: order=Any (the default)

Upgrade

Minor updates

Minor updates are so easy, eg for 7.3.6.

  • (optional) You don't NEED to, but it's a good time to upgrade and shutdown any VMs. It also went fine to do this while leaving all VMs up though!
  • Click:
Datacenter > melange > (melange bar) > Updates > Upgrade
  • Reboot

Reboot after upgrade

Melange VMs all come back nicely, because they come up staged well. See the order fo the VMs for details, particularly hive: 

VM > Options > Start/shutdown order: order=1,up=90

pve7to8

This was pretty painless, following their excellent guide! 

Proxmox: 
use pve7to8, which vets upgrade
it advised to install: 
apt install grub-efi-amd64 # for booting
apt install chrony # for ntp time sync, better than shit systemd service which only runs on boot
chrony is default in newer proxmox installs

looks like we are ready!

sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list
important!

here’s our source file, change from bullseye to bookworm:
nano /etc/apt/sources.list.d/pve-no-subscription.list

apt update
apt dist-upgrade # HERE WE GO

sudoers, let’s take the new file, and re-add this to the end (...)

Hardware

1 TB M.2 nvme 
/dev/mapper/pve-root / # /dev/nvme0n1p3
/boot/efi /dev/nvme0n1p2
  • GPU: Nvidia GK208 - GeForce GT 640
sudo lspci|grep VGA
VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 640 Rev. 2] (rev a1)
  • 4 SAS connections on mobo, with a SAS-to-SATA cable; make sure to turn on "SATA mode" for U.2 in BIOS; all passed through to hive
  • 4 SATA ssd connections on mobo, all passed through to hive

Memory

32GB x 4 sticks (maxed out)

128GB total, distributed as follows:

  • 34GB AbtDev1
  • 24GB Hive
  • 16GB Positronic
  • 12GB Matcha
  • 12GB Glam
  • 06GB Matryoshka
  • 06GB Bandit
  • 03GB Morosoph
  • 04GB Hoard (stopped)
  • 04GB Tornados (stopped)

Historic update after 64 > 128 upgrade Had ~83GB distributed, distribute +37GB, leaving ~8GB (OS provides ~125/128GB, ie it needs ~3GB)

  • 20GB AbtDev1 + 14 = 34816MB
  • 19GB Hive + 5 = 24576MB
  • 10GB Positronic +6 = 16384MB
  • 08GB Glam +4 = 12288MB
  • 08GB Matcha +4 = 12288MB
  • 02GB Bandit +4

Passthroughs

We pass the LSI PCI-E board (and its 8 ssd drives), the 4 SATA SSD drives, and the 4 U.2 SSD drives to Hive, for storage management.

LSI PCI-E board passthru

I didn't log this when I did it originally. It must be in the 104.conf file tho... I'm guessing it is this: 

🌐 m@melange  [~] sudo cat /etc/pve/qemu-server/104.conf
scsihw: virtio-scsi-pci

SSD drives passthru

  • Identify drives on melange
 🌐 m@melange  [~] sudo lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT DEVICE-ID(S)
sdh                            8:112  0 931.5G  0 disk   /dev/disk/by-id/wwn-0x500a0751e5a2fb01 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FB01
sdi                            8:128  0 931.5G  0 disk   /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FD56 /dev/disk/by-id/wwn-0x500a0751e5a2fd56
sdj                            8:144  1 931.5G  0 disk   /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D2 /dev/disk/by-id/wwn-0x500a0751e5a313d2
sdk                            8:160  1 931.5G  0 disk   /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D6 /dev/disk/by-id/wwn-0x500a0751e5a313d6
sdl                            8:176  1 931.5G  0 disk   /dev/disk/by-id/ata-CT1000MX500SSD1_2117E59AAE1B /dev/disk/by-id/wwn-0x500a0751e59aae1b
sdm                            8:192  1 931.5G  0 disk   /dev/disk/by-id/wwn-0x500a0751e5a2e131 /dev/disk/by-id/ata-CT1000MX500SSD1_2121E5A2E131
sdn                            8:208  1 931.5G  0 disk   /dev/disk/by-id/wwn-0x500a0751e5a3009a /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A3009A
nvme0n1                      259:0    0 931.5G  0 disk   /dev/disk/by-id/nvme-eui.002538510141169d /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N107994E
  • Use qm set to pass them all to hive (VM 104)
sudo qm set 104 -scsi11 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FD56
sudo qm set 104 -scsi12 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A2FB01
sudo qm set 104 -scsi13 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D2
sudo qm set 104 -scsi14 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A313D6
sudo qm set 104 -scsi15 /dev/disk/by-id/ata-CT1000MX500SSD1_2117E59AAE1B
sudo qm set 104 -scsi16 /dev/disk/by-id/ata-CT1000MX500SSD1_2121E5A2E131
sudo qm set 104 -scsi17 /dev/disk/by-id/ata-CT1000MX500SSD1_2122E5A3009A

# check
sudo cat /etc/pve/qemu-server/104.conf

(restart hive)

USB

Plug something into melange and see what it sees: 

lsusb
Bus 004 Device 002: ID 05ac:12a8 Apple, Inc. iPhone 5/5C/5S/6/SE

Now we will pass that port through: 

proxmox UI > Datacenter > Nodes > VMs > 111 (matcha) > Hardware > Add > USB > it will actually show up in a pick list!

Hell yeah so easy.

History

Retrieved from "https://bitpost.com/w/index.php?title=Melange&oldid=7668"