Maintenance reference

general tasks

Flash Drives - Scripts - Add a user Add a drive - Back up drive - Check disk speed Bash basics - OS X basics - DOS basics SSL certificate instructions udev: Assign unique names to your devices Automatically mount and unmount your devices udev: Autorun programs when connecting your devices master keyboard shortcuts map See .i3/config. Typically you can press ctrl-shift-K to open it in editor. The local file usually points to the common version, here: ~/development/config/common/home/m/.i3/config Fix things after IP changes Time Warner seems to keep giving the same IP back to my main linux router/server, as long as the NIC card doesn't change. It also happened when they upgraded my service (and started labeling my IP as dynamic to SpamHaus, fuckers...). If it changes, fix these things: mod evasive has a whitelist, it needs the actual IP, bleh... -rw-r--r-- 1 m m 1.1K Mar 28 22:13 /home/m/config/etc/apache2/modules.d/10_mod_evasive.conf Here is the list of things to do to the cablem modem to fully reset it: make sure you have connected the blue cable => onboard NIC of bitpost make sure you have connected the yellow wan cable => addon NIC in bitpost shut it all down restart modem, "wait 5 minutes" restart bitpost Ubuntu upgrade from command line To update ubuntu from an i3 terminal: sudo apt-get update # Fetches the list of available updates sudo apt-get upgrade # Strictly upgrades the current package sudo apt-get dist-upgrade # Installs updates (new ones) Disabling xscreensaver emacs ~/.xscreensaver timeout: 12:00:00 # the max timeout is 12 hours crontab -e # So... I have .xscreensaver set to max delay (12 hours) # and I call this every ten hours to restart the idle timer. * */10 * * * /usr/bin/xscreensaver-command --deactivate >/dev/null bash chmod dirs find /path/to/base/dir -type d -exec chmod g+x {} \; Test your drives - DO THIS OFTEN Look at available tests: smartctl -c /dev/sdf # to peek at available tests and how long they typically take to run Short test: smartctl -t short /dev/sdf watch -n 3 smartctl -l selftest /dev/sdf # you won't see anything until it finishes, takes about 1 minute Long test: smartctl -t long /dev/sdf watch -n 3 smartctl -l selftest /dev/sdf # takes about 5 minutes Mount an IMG file You have to peek at the file with fdisk: fdisk -lu /home/m/bitpost/download/torrents/OpenELEC-Generic.x86_64-5.0.8-efi.img From that you can tell the "units" and "Start" offsets: Device Start End Sectors Size Type /home/m/bitpost/download/torrents/OpenELEC-Generic.x86_64-5.0.8-efi.img1 2048 526336 524289 256M Microsoft basic data /home/m/bitpost/download/torrents/OpenELEC-Generic.x86_64-5.0.8-efi.img2 528384 593920 65537 32M Linux filesystem From there you can mount with a little math. Multiply the units by Start to get offset (512x2048=1048576), and use it to mount that partition: mount -t auto -o loop,offset=1048576 /home/m/bitpost/download/torrents/OpenELEC-Generic.x86_64-5.0.8-efi.img /media/m/iso aifol Chroot into gentoo from (say) ubuntu # These may already be done mount /mnt/gentoo cp -L /etc/resolv.conf /mnt/gentoo/etc/ mount -t proc none /mnt/gentoo/proc mount --rbind /dev /mnt/gentoo/dev chroot /mnt/gentoo /bin/bash # ** NOW IN CHROOT ** env-update source /etc/profile export PS1="(chroot) $PS1" # GET TO IT!  :-) Windows symbolic links Windows symlinks - needed for config files, svn overlay, etc. I'm setting up hard links from sublime install folder to shared config settings: mklink [/H /D /J] link_path Target_file/folder_path We had to use a soft link, hard won't go across drives. And you have to run the cmd window as admin. Don't specify type, and it will create a soft file link: mklink "C:\Users\Michael\AppData\Roaming\Sublime Text 3\Packages\User\Default (Windows).sublime-keymap" "D:\Michael's Data\development\config\common\home\m\.config\sublime-text-3\Packages\User\Default (Windows).sublime-keymap" mklink "C:\Users\Michael\AppData\Roaming\Sublime Text 3\Packages\User\Preferences.sublime-settings" "D:\Michael's Data\development\config\common\home\m\.config\sublime-text-3\Packages\User\Preferences.sublime-settings" Here is another example, setting up a link in an SVN repo folder to point to a git-controlled folder on another drive. SVN is OK with this, cool. (open ADMIN cmd prompt) cd C:\CODE\QA\ContinuousIntegration\Trunk mklink /J scripts "D:\Michael's Data\development\power_analytics\scripts" Junction created for scripts <<===>> D:\Michael's Data\development\power_analytics\scripts Windows route traffic to avoid VPN You can use "route" to assign paths to specific traffic. So bitpost.com requests go through eth0 instead of VPN, eg... C:\WINDOWS\system32>route add 174.109.14.112 192.168.50.1 Note that you can do this with [route] in linux too, syntax is a bit diff. Windows command prompt FULL SCREEN Type cmd in start search box and right-click on the cmd shortcut which appears in the results. Select Run CMD as administrator. Next, in the command prompt, type wmic and hit Enter. Now try to maximize it! Close it and again open it. It will open as a maximized window! You may have to ensure that the Quick Edit Mode in the Options tab is checked.

OS installation

Ubuntu 15.10 setup Ubuntu repo management To see what repos Ubuntu is currently using: cat /etc/apt/sources.list Set up a new Pi in 10 minutes The BEST thing to do is to copy the MicroSD from Carambola (marked with a black 'O'): open a terminal so we can watch the MicroSD /dev/sd{#} assignments sudo tail -f /var/log/syslog take the carambola MicroSD card out of the Pi and put it into an Anker hub (gets less hot than the small MicroSD sleeves - and it WILL get hot!) put another new MicroSD card into another Anker hub open another terminal dcfldd bs=4M if=/dev/sd{Letter of carambola} of=/dev/sd{Letter of new card} They will get HOT... I don't know how to make sure they don't get TOO hot yet... cross your fingers I guess... Drop the new card into the new Pi, boot, change /etc/hostname and reboot Raspberry Pi Raspbian setup Update gentoo kernel OpenELEC multi-boot install The easiest way is to add a new drive just for OpenElec and install OpenElec to it with the installer. But if you want to SHARE ONE DRIVE with other boots, DO NOT DO THAT :-) Do this instead: On an existing refind-booted system, set up two new ext4 partitions one about 2GB in size, labeled [SYSTEM], and marked as bootable the other with 10GB or more, labeled [STORAGE] prep the drives (no journal, ssd trim) download openelec and mount; there are a couple ways: get the img and install to a thumb get the img and mount (see below) Set up OE drives to mount in other OSes to /openelec-system and /openelec-storage /dev/disk/by-label/SYSTEM /openelec-system ext4 noatime 0 0 /dev/disk/by-label/STORAGE /openelec-storage ext4 noatime 0 0 copy target/KERNEL and target/SYSTEM to SYSTEM cp OpenELEC_img/target/KERNEL /openelec-system/ cp OpenELEC_img/target/SYSTEM /openelec-system/ set up UEFI boot subl /boot/efi/EFI/refind/refind.conf (and add this block) # MDM Trying this, from: http://openelec.tv/forum/64-installation/70783-how-to-efi-booting-openelec-on-new-pc-s-nuc-s # Only I had to change BOOT to SYSTEM. and quiet to debugging. menuentry OpenELEC { icon EFI/refind/icons/os_openelec.png volume SYSTEM ostype Linux loader KERNEL # options "boot=LABEL=SYSTEM disk=LABEL=STORAGE debugging" options "boot=LABEL=SYSTEM disk=LABEL=STORAGE quiet" } I THINK you need one or more of these too, not sure! cp target/boot/bootx64.efi /openelec-system/BOOT cp target/boot/bootx64.efi /boot/efi/EFI/ubuntu/ cp target/boot/bootx64.efi /boot/efi/EFI/ubuntu/ OpenELEC boot from thumb NOTE: I have the image already on a white stick with red lettering... anyway... Get the latest dev version (stable didn't work for me although this may change) dd it onto a thumb reboot and select to boot to the thumb in BIOS when the boot: line comes up, type "live" to get run a live Kodi rather than run the crufty old installer Windows 10 new setup Grep for Windows Create sublime symbolic links to preferences, typically here: C:\Users\Michael\AppData\Roaming\Sublime Text 3\Packages\User etc.

Cloud and AWS tasks
AWS clone create a snapshot of an instance create an instance of a snapshot

Tips and tricks

Maintenance of my music collection Ubuntu and shitty Seiki AMD Radeon driver sucks, won't go over 1080p; and Dune (AMD chipset) overscans so you can't see the edge. The only fix I found was to use xrandr. It kinda sucks though bc it's a resize solution that blurs all pixels. Cest la vie. xrandr --output HDMI-0 --set underscan on xrandr --output HDMI-0 --set "underscan hborder" 40 --set "underscan vborder" 22 Let's try the binary ATI driver. lspci -vvnn | grep VGA [...] Advanced Micro Devices, Inc. [AMD/ATI] RS780 [Radeon HD 3200] [1002:9610] nvidia For 4k video on an nvdia chipset (like wimpy), first get Ubuntu to use nvidia's driver: Go to System Settings-> Software & Updates Select the Additional Drivers tab Select the tested, proprietary NVIDIA binary driver REBOOT - the nvidia driver works at the kernel level so you won't be able to just restart X. fools WHY does ubuntu make you do all this? Next, here is the "magic super-secret" Seiki 4K goo that should go in xorg.conf (if needed). It's from: http://ubuntuforums.org/showthread.php?t=2164924 See /etc/X11/xorg.conf for the full details: # MDM added based on http://ubuntuforums.org/showthread.php?t=2164924 # Modeline "3840x2160" 307.00 3840 4016 4104 4400 2160 2168 2178 2250 +hsync +vsync # MDM 2015/02/12 A more recent one from: http://ubuntuforums.org/showthread.php?t=2164924&page=3&s=b53cb9ea1b751f4b9f2fe1aadfb6f43f Modeline "3840x2160" 296.70 3840 4016 4104 4400 2160 2168 2178 2250 +hsync +vsync Modeline "1920x1080" 297 1920 2008 2052 2200 1080 1084 1089 1125 +hsync +vsync To get a full output of what the hell X is doing: scite /var/log/Xorg.0.log & # Or create one if needed X -logverbose 6 :2 # this crashed my machine but did produce a log first scite /var/log/Xorg.2.log &

Boxes

BOX: bitpost drives My primary server at Peace Street I had to stuff the goddamned drives right into the case since the rosewill was giving me such shit. In hindsight, it might not have been the rosewill at all... I am considering restuffing the box with the drives... but for now, i have this: sda root sdb1 tvmind sdc1 film sdd & sde softraid sdf1 spicey sdg1 reservoir sdh1 plethora sdi1 win Before it was like this: ================================ BAYS of the bitpost.com Rosewill ================================ To figure this stuff out, it was easiest to mount/umount drives and/or just access the mounted files (and watch the bay lights, of course) Disk layout: 8 7 6 5 4 3 2 1 1) spicey-3g-mel 2) reservoir-3g-mel 3) tvmind-4g-mel 4) newraid1-4tb-bp WD Black installed 2014-10-05 5) plethora-3tb-bp i hope this was not being used - 3GB Seagate barracuda - mount says some ntfs looking thing with no partitions and unable to mount - so i reformatted ext4 6) win-pax09 some old windows drive with lots of game goodies; also has sdf2 swap partition (kill it some day) 7) film-6tb-bp WD Green installed 2014-10-05 8) newraid2-4tb-bp WD Black installed 2014-10-05 ================================ Before THAT, it was this... 8 bay SATA tower houses these drives: 2.0tb archive (sdb right?) 250gb landofthelost circa ABB timeframe /dev/disk/by-label/LandOfTheLost /dev/sdc1 1.5tb raid drive #1 /dev/disk/by-label/d-sp-raid -> ../../sdd1 2.0tb latest /dev/disk/by-label/d-sp-20newmovies -> ../../sde1 1.5tb raid drive #2 (needs label change maybe) There are also these drive partitions: / /dev/sda4 ubuntu root ub_root -> ../../sda5 /vista /dev/sda3 rtorrent keys rtorrent CAN actually work pretty well, IF you pay attention to these basic keys: enter - specify a torrent file to load - USE TAB COMPLETION ctrl-d - STOP torrent, smack that shit, delete it forever ctrl-s - START torrent - WARNING ONLY WORKS IF YOU DISABLE TERMINAL CAPTURE with... stty stop undef stty start undef ctrl-o - change download dir ctrl-k - close files - needed before ctrl-o, usually ctrl-r - recheck hash NOW GET rutorrent going!  :-) rtorrent fix torrents that didn't move into place properly * if the torrent is archived properly, use move_and_resymlink to move it: cd ~/download/torrents/archived/symlinks move_and_resymlink [symlink] [destdir] if things are a mess due to hard drive failure (etc), you should do this: use r(u)torrent to see where rtorrent thinks the files are copy the files, if available, to the final destination (under a .* dir in symlinks) create a symlink: ln -s .music/Fresh/ELO-Secret\ Messages ELO-Secret\ Messages close the torrent files (ctrl-k), move the destination (ctrl-o), start again (ctrl-s) Updating mediawiki installation cd /var/www/localhost/htdocs/mediawiki emacs LocalSettings_redirector.php (to hardcode each site) php maintenance/update.php (repeat for each site) emacs LocalSettings_redirector.php (to reset dynamic behavior) Creating a new ssh key pair for no-password access to a remote system Say you've got a new machine with user [m] and you want to connect to it, do this: install sshd, make sure it's running get on the new machine - ssh to the new machine by typing the password if you don't have direct access $ scp m@tdm:.ssh/id_rsa .ssh/ $ scp m@tdm:.ssh/authorized_keys .ssh/ wow I just had serious issues with basic ssh usage, so i'll put a summary on the wiki basically most misunderstandings stem from describing these two: client: machine that is trying to ssh into the server server: machine that the client user wants to get to most poor sots are just sitting on the client and they create a pair and push their public key to the server but in my case, more often, i want to create a pair on the server and push the private key to the multiple places i need to connect FROM when doing that, you have to push the public key into the server's authorized_keys and configure the client to juggle multiple private keys create a key pair: ssh suser@server ssh-keygen use defaults will create: .ssh/id_rsa (private key) .ssh/id_rsa.pub (public key) put the public key in place: cd .ssh cat id_rsa.pub >>authorized_keys put the private key on the client and configure: scp id_rsa cuser@client:.ssh/id_rsa_server ssh cuser@client em .ssh/config Host tdm thedigitalmachine.com Hostname thedigitalmachine.com IdentityFile ~/.ssh/id_rsa User m Host server Hostname server.com IdentityFile ~/.ssh/id_rsa_server User suser all is full of light bitpost LAN IP addresses All IP address management happens on bitpost, via dnsmasq, which, rocks. Set up machine names in /etc/hosts Set up IP leases by MAC address in top of: /etc/dnsmasq.conf Initially set up bitpost with DHCP as follows: emacs /etc/conf.d/net config_eth0="dhcp" config_eth1="192.168.22.1/24" Turn DHCP-assigned IP into static as follows: /etc/init.d/net.eth0 restart dhcpcd[3867]: eth0: carrier acquired dhcpcd[3867]: DUID 00:01:00:01:1a:c3:6f:da:e0:3f:49:a3:4b:d9 dhcpcd[3867]: eth0: IAID 49:a3:4b:d9 dhcpcd[3867]: eth0: rebinding lease of 66.57.227.83 dhcpcd[3867]: eth0: leased 66.57.227.83 for 86400 seconds dhcpcd[3867]: eth0: adding route to 66.57.226.0/23 dhcpcd[3867]: eth0: adding default route via 66.57.226.1 emacs /etc/conf.d/net config_eth0="66.57.227.83/23" routes_eth0="default via 66.57.226.1" /etc/init.d/net.eth0 restart We will monitor and log success here to see how it goes. 2014/05/05 set to static 66.57.227.83/23, gateway 66.57.226.1 view dnsmasq IP leases cat /var/lib/misc/dnsmasq.leases Updating mediawiki installation cd /var/www/localhost/htdocs/mediawiki emacs LocalSettings_redirector.php (to hardcode each site) php maintenance/update.php (repeat for each site) emacs LocalSettings_redirector.php (to reset dynamic behavior) BOX: melange My primary dev box 90% ubuntu for development 10% gaming machine, has steam, low end half-height video card that works fine for Risk of Rain etc. It has a mid-range video card, enough to drive 1080p x 2 for development. This used to be the primary Kodi box up until 10/2015, should still be current. It has the same high-end i5 or i7 of other 2015 boxes, fast. drives /dev/sda contains ubuntu, kodi, windows - using ubuntu 90% of the time /dev/sdb melange_data formatted as exfat used gparted to remove existing partitions used gdisk to create an EFI partition table and an 8300 "linux file system" mkfs.exfat /dev/sdb1 exfatlabel /dev/sdb1 melange-data emacs /etc/fstab /dev/disk/by-label/melange-data /melange_data exfat noatime 0 0 BOX: wallee My 2009 MacbookPro 5.3 multi-boot laptop (see blog for multi-boot howto) Ubuntu Install all hardware-specific packages in one shot: sudo apt-get install firmware-b43-installer xserver-xorg-input-mtrack Post-install MUST-DOs: broadcom "b43" wifi driver: sudo apt-get install firmware-b43-installer (reboot) synaptic driver (for trackpad) sudo apt-get install xserver-xorg-input-mtrack # use synclient to determine what works best su - synclient -l # lists all params synclient PalmDetect=1 # temporarily change settings # save what you like, here: root@wallee:~# emacs /etc/X11/xorg.conf.d/60-synaptics.conf Windows 10 Upgraded existing Windows 8.1 Hack the INF file of the nvidia driver to get it to install - runs PERFECTLY TODO add details Add bluetooth stack - or not - i think i ended up back at the default Windows keyboard driver to get function key working: /spiceflow/softraid/software/OS/windows/macbook_windows10_drivers/AppleKeyboardInstaller64.exe Reset 1020 cable modem so it grants a real IP address (not private) Resetting things sometimes has to be done in this precise order: cable modem -> blue cable -> **ONBOARD** bitpost NIC LAN switch -> yellow cable -> **ADDON CARD** bitpost NIC power off bitpost and cable modem power up cable modem wait 5 minutes power up bitpost verify that you can ping google.com, LAN works, etc. BOX: case To install ubuntu, I had to add [... --- nomodeset] to the boot line in the grub booter. Then I had to use a desktop 1080p monitor - the 4k Sony would not come up for some reason. BOX: Raspberry Pi 2 and 4k You can override settings in /flash/config.txt; here's how, from here: mount -o remount,rw /flash nano /flash/config.txt # Make your changes, then ctrl+x to save. mount -o remount,ro /flash reboot Settings for 4k (from here)... disable_overscan=1 hdmi_ignore_edid=0xa5000080 hdmi_cvt 3840 2160 30 framebuffer_width=3840 framebuffer_height=2160 hdmi_group=2 hdmi_mode=87 hdmi_pixel_freq_limit=400000000 max_framebuffer_width=3840 max_framebuffer_height=2160 I couldn't get 4K to stick in OpenELEC tho (except for the splash screen(!))... Settings->System->System->Video->Refresh->30Hz Settings->System->System->Video->Resolution->3840x2160 (it won't come up, just drops back to 1080p.. just as well, video at 30fps sucks...) And to overclock the Pi 2 (from here): # OVERCLOCKING TO 1.1 GHz, EXTREME # default is 700 # this will NOT void the warranty :-) arm_freq=1000 sdram_freq=500 core_freq=500 over_voltage=6 temp_limit=80 #Will throttle to default clock speed if hit. # # there is also the "turbo" thingee (that "voids warranty", hot!!) #force_turbo=1 BOX: tdm My primary server that does just about everything internet-related P8P67LE mobo Intel i5 quad core names that are hosted: thedigitalmachine.com thedigitalage.org BOX: dune Follow the ubuntu install. We also use a StarTech AC600 ac wifi adapter. The chipset is specified here. It's a Realtek - RTL8811AU. It is not supported with the vanilla 15.04 kernel, but it's easy to build a module from sources available on github: sudo apt-get update sudo apt-get install linux-headers-generic build-essential git git clone https://github.com/gnab/rtl8812au.git cd ~/rtl8812au make sudo make install sudo modprobe 8812au Then, run NetworkManager and nm-applet, and as soon as you plug the USB dongle in and remove the wired connection, you'll get a running wifi applet, YES. We also want to drive the crappy Seiki at 4K. A 4k mode shows up in our X probe, but ubuntu is not picking it...... yet... OLD gentoo NOTES My XMBC media center and samba-based storage server AMD post-kernel tasks: make sure you update /etc/conf.d/modules for any new modules added update-modules module-rebuild populate module-rebuild rebuild (if you feel brave) module-rebuild list (if you want to do it manually) # NOTE: NO MORE ATI-DRIVERS they no longer support my old-ass onboard HD 3000 graphics chip # But the open-source radeon driver does, oh yeah! emerge lirc # etc emerge -a1 $(qlist -IC x11-drivers/) update-modules dispatch-conf # NO! eselect opengl set ati # aticonfig --ovt opengl # "no supported adapters"? eselect opengl set xorg-x11 reboot BOX: wimpy My main development workstation P8P67LE mobo Intel i5 quad core website drives and partitions: sda HDD, OLD OSes, need to wipe and reformat to ntfs; sda2 ntfs 1.1TB old?? sdb sdb1 fat32 300MB wimpy-efi sdb2 ntfs 147GB windows10??? sdb3 fat32 287MB /boot/efi sdb4 linux-swap 7GB sdb5 ext4 wimpy-ubuntu-ssd 216GB sdb6 ext4 wimpy-gentoo-ssd 54GB sdb7 ext4 wimpy-openelec 19GB sdb9 fat32 "boot, esp" check in windows: wmic baseboard get product,Manufacturer,version,serialnumber boot is in /ubuntu/boot grub menu: /ubuntu/boot/grub/menu.lst post-kernel tasks: make sure you update /etc/conf.d/modules for any new modules added update-modules module-rebuild populate module-rebuild rebuild (if you feel brave) module-rebuild list (if you want to do it manually) emerge nvidia-drivers (etc) update-modules dispatch-conf eselect opengl set nvidia reboot if you break gentoo (or rather, if gentoo devs break gentoo, or open source hacks break gentoo, or whatever... but ultimately, you...)... boot into ubuntu su - ./chroot_to_gentoo (chroot) ./setup_chroot ** FIX IT ** to install a new kernel that ubuntu will serve up... build it in a chroot back in ubuntu, copy it: cp /mnt/gentoo/usr/src/linux/arch/x86_64/boot/bzImage /boot/linux-blahblbahba and update grub. you can change em both or just the second (and rerun grub-mkconfig >/boot/grub/grub.cfg): emacs /boot/grub/grub.cfg emacs /etc/grub.d/09_gentoo