Upgrading to Grub 2 on Xen VPS Guests

If you have a linux VPS server hosted at a cloud provider you may find that you’re stuck on legacy Grub, and in some cases, may find that update-grub doesn’t properly update when a new kernel is installed. Here you’ll find two solutions, one to get Grub Legacy updating properly, and the second on how to update to Grub 2. This was personally tested on Debian, but the principles should translate to most distros.

PART 1 -- Fixing Grub Legacy
The kernel updating issue is typically caused by Grub Legacy’s inability to realize it’s in a Xen guest environment. This can be solved by using to PvGrub to keep your /boot/grub/grub.conf updated (grub.conf is linked to /boot/grub/menu.lst).

1) Take a look in your /etc/fstab to find the root “/” partition for your VPS (if it lists a separate partition for /boot then use it instead). This will likely be something like “/dev/xvda1” or “/sda/xvda1”

2) Make a backup copy of your current /boot/grub/grub.conf

3) Install PVGrub (aka. pvgrub-legacy or pv-grub-menu in Deb). This will remove the grub-legacy package but keep grub-common:

4) Make sure your /etc/default/grub is configured how you want it, then run PvGrub:

5) PvGrub should now have updated your /boot/grub/grub.conf with entries for all your kernels, with a structure similar to the example below. Verify that the “root” line lists the root (or boot) partition from step 1. Also verify that the UUID listed in the “kernel” line matches the UUID for your root “/” partition (use “ls -al /dev/disk/by-uuid/” to list UUIDs). Finally, make sure the “default” line lists the kernel you want to boot.

6) Reboot and confirm you’re running the new kernel (uname -a).

7) If the system won’t reboot then use the recovery console provided by your host to log-in as root, manually mount your root (or boot) filesystem and copy your backup grub.conf back over the new one (or try to fix the errors in the new one).

Additional Resources:
Installing PvGrub on other distros (Arch, Cent, Fedora Gentoo, etc) by Rackspace
PvGrub on Amazon EC2 instances

PART 2 -- Upgrading to Grub 2 (aka. PvGrub 2, aka. Grub 2 Xen)
If, and only if, you have a fully working Grub Legacy that’s booting nicely, then you can proceed to upgrade to Grub 2. The kicker here is that in most cloud VPS environments, you cannot actually overwrite the existing Grub Legacy in the MBR/partition boot sector, so the approach here is to do a one-time tweak of the Grub Legacy install to chainload Grub 2. Once completed, the Grub Legacy config will remain static and all future kernel updates will be maintained using Grub 2 in the normal fashion (i.e. update-grub).

1) Again, make a backup copy of your currently working /boot/grub/grub.conf and /etc/default/grub files.

2) Remove any current grub packages installed on the system (this will only remove the packages, not the existing grub binaries in the boot sector):

3) Install Grub Xen package (Not grub-xen-host!)

4) Double check that it installed fully (should have been automagically by APT, but just in case):

or if appropriate:

Ignore the junk warnings as long as you see “No error reported”.

5) You should now have a file /boot/xen/pvboot-x86_64.elf (or pvboot-i386.elf)

6) Configure /etc/default/grub with your preferred settings, then run:

7) Verify that the Grub 2 config file (/boot/grub/grub.cfg, not grub.conf !!) was updated with your kernels and all looks in order (e.g. root partitions, etc).

8) Now setup Grub Legacy to chainload Grub 2. Edit /etc/grub/grub.conf (menu.lst). Set “default 0” and “timeout 1”, then add the following as the first entry in the kernels list (you may want to leave the other kernel entries as backup while testing). Make sure to adjust the “root” and “kernel” path as appropriate.

9) If you had “GRUB_DEFAULT=saved” in your /etc/default/grub then remember to run grub-set-default before rebooting. If you make any more changes to /etc/default/grub or the /etc/grub.d/ files, then re-run update-grub.

10) Reboot!

11) If everything has gone according to plan then your Grub Legacy config never needs touched again. If your VPS host actually supports direct loading of Grub 2 (currently rare), then it may even skip the Grub Legacy config and directly load your Grub 2. Either way, the net effect is essentially the same -- booting for all future kernel updates is now handled directly using your Grub 2 config. There should not be any packages updating your Grub Legacy grub.conf file (update-grub and update-grub2 are just linked commands, both for updating Grub 2’s /boot/grub/grub.cfg config).

Additional Resources/Credit:
Many thanks to Ian Campbell for this post.

Quality Linux Document Scanning with an HP LaserJet

Recently acquired an HP LaserJet Pro 200 all-in-one device which so far has been working decently with Ubuntu 14.10. Print and fax all work as expected. My only frustration has been that I scan a lot of text documents to PDF for archive purposes and was not able to get the same small filesize and clear quality scans under linux using Xsane that I was getting using the “Black/White” settings in the HP software on Windows. Under linux the scans would be either washed out (shades of gray text), or capture a lot of junk detail (paper folds, smudges, etc), whereas in Windows they would be a crisp black and white.

After spending too long playing with drivers, pdf tools, etc.. I found that the solution is just a matter of proper Xsane settings. The key is using the grayscale scan mode (not lineart mode) with extreme values for gamma and contrast.

These are the basics:

HP LaserJet Pro 200 color MFP M276nw
Ubuntu 14.50, Xsane 0.998

Steps for installing HP software:
* Install HPLIP packages from Ubuntu repositories (currently HP Linux Imaging and Printing System v3.14.6)
* Open the “HPLIP Toolbox” gui
* Use the “Diagnose HPLIP Driver” function in the gui to ensure all dependencies are installed
* Use the gui to find and setup the printer
* Once the printer is installed, use the “Install Required Plugin” option under that printer menu

Xsane settings:
Type: PDF
Scan Mode: Gray
Source Medium Type: Full Color Range
Scan Resolution: 300 (or better)

Gamma value: 0.30 (minimum)
Brightness: 0.0 (middle)
Contrast: 100.0 (maximum)

You can back off the extremes on the gamma and contrast a little to tweak, but this should provide a baseline scan nearly identical to the HP software presets on Windows.

Marvell 88E8056 – sky2 linux driver broken

Update 5/22/15: I’ve since found that lowering the MTU below 1500 with the sky2 driver will cause major packet loss when the interface is used in a bridged connection for a Virtualbox VM. I have since raised the MTU back to 1500 and haven’t had the issue reoccur with the current kernel (3.16.0-38-generic #52-Ubuntu SMP).

———

Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller

Symptom: During heavy network loads you get rx length errors and loss of network connectivity. Syslog will show similar messages to below.

Diagnosis: The sky2 driver remains broken for the Marvell 88E8056 on-board gigabit NICs.

Workaround: Set MTU permanetly to 1492 (default is likely set at 1500).

Check current MTU setting with:
# ip link list

Temporary MTU setting can be done with:
# ifconfig eth0 mtu 1492

Permanent MTU setting can be done via the GUI network manager settings or via edits to /etc/network/interfaces.

/var/log/syslog errors example:

Install Amarok 1.4 on a current Linux Desktop

As any long time Linux user knows, Amarok 1.4 was the best music player of all time.. and now thanks to the Trinity Desktop Environment folks, it still is! Using the Trinity Desktop packages it’s possible to get Amarok 1.4 back

Why? Because Amarok 2 is a travesty and needs to be taken out behind the barn. I’ve tried to tweak it and love it, but I just can’t. In my opinion, if the developers ever want to get back to Amarok dominance, they best thing they could do is focus every effort into getting 2 to look and function exactly as 1.4 did. Completely implement *all* the previous 1.4 feature set with the new engine, then, and only then, work on other enhancements.

Clementine on the other hand is a good fork effort, and if you want 95% of what Amarok 1.4 was, without any extra effort, then it should be just an apt-get away on Ubuntu, Mint, etc. (Though you must be able to learn to live with an orange wedge icon.)

Anyway, if you’re a troglodyte like me, and just want things to go back to how they were, then here it is. I’ve tested this on Kubuntu 11.x, 12.x, 14.04 and am currently using it 14.10 Plasma4 (see updates for 15.04 at the bottom of this post). These instructions should work for any of the Ubuntu derivatives, and a similar process should work for all the platforms that Trinity supports (Ubuntu, Debian, RedHat and Fedora).

* Add the Trinity repositories per the instructions here. For Ubuntu 14.10 (Utopic) that means add:
$ sudo nano /etc/apt/sources.list.d/trinity-desktop.list

Save the file.

* Add the Trinity GPG signing key:
$ sudo apt-key adv --keyserver keyserver.quickbuild.pearsoncomputing.net --recv-keys F5CFC95C

* Now install the amarok-trinity package. The Trinity folks highly recommend using aptitude instead of apt-get for dependency conflict reasons.
$ sudo apt-get install aptitude
$ sudo aptitude update
$ sudo aptitude install amarok-trinity

The last one should drag in all the necessary Trinity Desktop dependency packages with it. Assuming everything went as it should, you now have Trinity installed under “/opt/trinity”. The Amarok 1.4 executable is at “/opt/trinity/bin/amarok”.

* Now at this point you should be able to start Amarok by specifying the full path from the terminal, but you may get errors as the supporting Trinity files are not in your PATH. To add the Trinity Desktop to your user $PATH so that you can easily execute it in the future, add the following lines to the bottom of your user .profile (do this as your regular user, not root):
$ nano ~/.profile

* Now refresh your user PATH:
$ source ~/.profile

* Running the command below should now show “/opt/trinity/bin” at the end.
$ echo $PATH

* Now fire up Amarok 1.4 for a test drive (note that just “amarok” should still fire up Amarok 2.x if it’s installed):
$ amarokapp

Boom!

—--

(Optional) Adding a desktop icon for Amarok 1.4:

* Create and save a user desktop file with the following entries:
$ nano ~/Desktop/amarok14.desktop

If you have a plasma widget that displays your Desktop contents then right-click and select “Refresh View” to see your new icon.

—--

(Optional) Create a menu entry for Amarok 1.4:

* Copy the desktop file to your user menu:
$ cp ~/Desktop/amarok14.desktop ~/.local/share/applications/
The menu entry should now show up under Applications > Multimedia.

Note: If you decide to set Amarok 1.4 as your default audio player for some file types then you may want to edit the end of the Exec line in “~/.local/share/applications/amarok14.desktop” to “amarok --append %U” or “amarok --append %U --play” so that your existing playlist gets appended to instead of replaced when you open a music file straight from the file browser.

And the world is now as it should be.

—--

UPDATE (6/2015): Kubutnu 15.04 (Vivid) changes:

* As of this writing, Trinity is not fully supported in 15.04), but their Preliminary Stable Builds are working fine. Install in the same manner as above.

* I had to remove the “/usr/:” portion from “KDEDIRS=” path of the EXEC line for the desktop icon (above example has been amended).

* The system tray icon no longer works. This is a common issue for numerous KDE legacy apps with 15.04. If you previously had Amarok 1.4 installed and set to minimize to the system tray when you closed it then it may appear not to start now because you simply can’t see it. The fix is to first kill any running instances:
$ killall amarokapp

Then modify/add the entries below into the Amarok config file:
$ nano ~/.trinity/share/config/amarokrc

Until this is resolved, make sure you do not use the “Show tray icon” setting.

Multiple Linux Distro Installs on a LUKS Encrypted Harddrive

Goal:

Install multiple linux distros on one machine that use the same encrypted container.

Why:

I’m setting up a new laptop with a 1 TB drive and want to be able to natively boot multiple distributions of linux, but also want everything in the drive except the necessary /boot partition to be encrypted. Each distro will have it’s own LVM volume within the encrypted container but I want all the LVMs to live within the same encrypted container so that data can be shared between volumes and I can grow/shrink/rearrange the logical volumes as needed.

Also.. just because.

The Challenges:

1) Boot partitions -- In a typical multiboot setup I would install Grub for the “primary” OS to the harddrive boot sector and then install the Grub for the secondary distros into their respective partition boot sectors. The primary Grub menu seen at boot would then be configured to chainload the secondary distros. Unfortunately this doesn’t work as simply with the LUKS setup above as Grub can’t access the secondary bootloaders if they are sitting in an encrypted LUKS container.

2) Dumb Installers -- In trying to keep things simple, the installer programs for many distros have been streamlined to the point they cannot support non-standard LUKS/LVM setups such as this.

The Process:

WARNING: FOLLOWING THESE STEPS IS GUARANTEED TO DESTROY EVERY BYTE OF EXISTING DATA ON YOUR HARD DRIVE. I’m serious -- the very first few steps will likely make any existing data unrecoverable by the best NSA jock using the latest secret-alien-area51 gear with unlimited funding. DO NOT PROCEED if you have not backed up every piece of needed data on the installation disk and have physically disconnected any other drives. This is NOT an exact step-by-step guide for beginners and assumes you are already familiar with linux installation and command-line administration.

Preparing the harddrive.

* This writeup assumes the installation harddrive is /dev/sda, adjust accordingly.

* Download and burn a linux installation disk with a Live option (e.g. latest Debian, Ubuntu or Mint). Alternative installers could be adapted as well if you know your stuff.

* Boot into the live environment, open a Terminal and “sudo su -” to obtain a root prompt.

* Create a temporary full disk partition to properly prepare the disk for encryption by first filling it with random data (this is where all existing data goes bye-bye). Use “fdisk /dev/sda” to delete any exiting partitions then create a full disk /dev/sda1 partition using the whole disk with default values. After saving the new table, if you get the following error:

Then reboot before continuing or you will only overwrite whatever the old sda1 partition size was.

* The step of filling the new disk with random data is usually done with the agonizingly slow “dd if=/dev/urandom”. Thanks to these awesome guys we have a much quicker method available using a clever combo of /dev/zero and LUKS itself. (I now use this even for standard encrypted installs and can then skip the “wipe data” step provided by the installers.)

* Create a LUKS container using /dev/sda1:
# cryptsetup luksFormat /dev/sda1

* Open the LUKS container:
# cryptsetup luksOpen /dev/sda1 sda1_crypt
[re-enter password]

* Fill the container data using /dev/zero. LUKS will encrypt the /dev/zero input as it writes. For me this is roughly 8x faster than the /dev/urandom method and the result is just as good (or better):
# dd if=/dev/zero of=/dev/mapper/sda1_crypt bs=1M

* Once complete, close the crypto device.
# cryptsetup luksClose sda1_crypt

* Overwrite the small LUKS header space for extra security.
# dd if=/dev/urandom of=/dev/sda1 bs=512 count=2056

Create the new permanent partitions and LUKS container.

* Use fdisk to create a new partition layout. The following layout provides boot partitions for 3 linux distros and the rest of the disk as the encrypted container. If you want more distos, format more boot partitions accordingly. It is possible to use the same /boot for more than one distro, but it will make all kinds of ugliness when you run update-grub. Unfortunately Grub2 is not nearly as refined as the old Grub in customizing these things yet. Using separate boot partitions keeps things easy and clean.

# fdisk /dev/sda
new primary partition 1 = /dev/sda1, 512Mb, type 83 Linux (/boot for primary distro)
new primary partition 2 = /dev/sda2, 256Mb, type 83 Linux (/boot for alternate distro)
new primary partition 3 = /dev/sda3, 256Mb, type 83 Linux (/boot for alternate distro)
new extended partition = /dev/sda4, (disk remainder) type 5 extended
new partition 5 = /dev/sda5, (disk remainder) type 83 Linux (this is the encrypted partition)

Also, if you want to install non-encrypted OS’s (e.g. Windows) then limit the size of the encrypted partition and create additional extended partitions for those OS’s. Note that most non-encrypted OS’s do not need separate boot partitions because they can be chainloaded from the Grub that will be installed in the primary harddrive boot sector. Also note that some versions of Windows may have trouble with booting from extended partitions so you would need to modify your layout accordingly. (Personally, I prefer never run to Windows outside of a Virtualbox).

* Now create the new crypto device. Note that the password you use here will be required each time you boot.

# cryptsetup luksFormat /dev/sda5

* Open the new crypto partition:
# cryptsetup luksOpen /dev/sda5 sda5_crypt

* Create a new LVM Physical Volume (PV) from the crypto partition:
# pvcreate /dev/mapper/sda5_crypt

* Create a new LVM Volume Group (VG) using the PV:
# vgcreate [vg-name] /dev/mapper/sda5_crypt

* Create new LVM Logical Volumes (LVs) for swap and root (/). The swap LV will be common for all the distros, but each distro will have it’s own root LV. It may be helpful to name the root LV based on distro (e.g. “kubuntu-root”). You can delete and create new LVs if you swap out distros later. You can also start small as it’s easy to grow the LV later using lvextend (and most filesystems with resize2fs) if you need more space.
# lvcreate -L[8G] -n [swap-lv-name] [vg-name]
# lvcreate -L[50G] -n [root1-lv-name] [vg-name]

* Check everything with “lvdisplay”. If it all looks good then leave the terminal window open and move to the GUI installer.

Installing the initial (primary) Linux distro

* Click on the Live CD’s gui installer icon (probably right on the desktop) and follow the instructions. When you get to the disk setup selections, select the Manual option. Use the screens to configure:
/dev/sda1 as format ext2, mount /boot
/dev/mapper/[vg-name]-[swap-lv-name] as swap area
/dev/mapper/[vg-name]-[root1-lv-name] as format ext4, mount /

All the other available partitions should be listed as “Do not use”.

* If/when you are presented with a Grub install choice, then install this grub instance to the harddrive boot sector “/dev/sda”.

* Complete the gui install, but once completed DO NOT REBOOT YET, instead click the “Continue Testing” option. LUKS must be configured before you reboot.

Enabling LUKS on the Linux distro

* Once the installer completes, you need to chroot from the Live CD environment into your newly installed distro so you can configure LUKS before rebooting.

* From the terminal window you were using previously (as root):
# mkdir /mnt/newroot
# mount /dev/mapper/[vg-name]-[root1-lv-name] /mnt/newroot
# mount -o bind /proc /mnt/newroot/proc
# mount -o bind /dev /mnt/newroot/dev
# mount -o bind /dev/pts /mnt/newroot/dev/pts
# mount -o bind /sys /mnt/newroot/sys
# cd /mnt/newroot
# chroot /mnt/newroot

You are now in the newly installed distro’s environment (the previous /mnt/newroot is now “/”).

* Mount the /boot partition for this distro (e.g. /dev/sda1 for the primary distro)
# mount /dev/sdaX /boot

* Get the UUID of the crypto partition you created previously with fdisk:
# blkid /dev/sda5

* Take the UUID given above (without the quotes) and create the file “/etc/crypttab” with the following single line. (This file will be the same for all linux distros within the LUKS container).
# nano /etc/crypttab

* Check /etc/fstab to ensure everything looks okay. It should list mounts for swap, /boot and /.

* You may want to add the following line to /etc/default/grub to keep your grub menu clean. This will restrict Grub to only creating menu entries for items in this distro’s /boot partition instead of searching the entire disk and creating menu entries for everything it finds.
GRUB_DISABLE_OS_PROBER=true

* Ensure that the “cryptsetup” and “cryptsetup-bin” packages are installed on the system (should be included on most modern distros) as well as initramfs-tools. For Debian based systems:
# dpkg -l | grep cryptsetup
# dpkg -l | grep initramfs

* Update the kernel boot images and grub menu:
# update-initramfs -u
# update-grub

# Take a look in /boot to ensure the initrd and vmlinuz images exist. Also take a peek at /boot/grub/grub.cfg to review your menu entries.

* Optional: If you want to save time and keep a tidy Grub menu then you can setup menu entries for your other distros now. They obviously won’t work until you install those distros, but it’ll save the step of having to boot back into the primary to update grub again after each install. To do this, add the “GRUB_DISABLE_OS_PROBER” entry mentioned above. Next, manually create the “chainload” entries for the other boot partitions by adding the following to the end of the 40_custom file.

# nano /etc/grub.d/40_custom

Once completed, run “update-grub” again and recheck your “/boot/grub/grub.cfg”. You should see these entries near the bottom.

* Time to take the plunge!
# reboot

* Make sure you eject the Live CD. Once the system reboots you should be presented with your Grub menu and shortly afterwards the boot process will pause and prompt for your LUKS password. Once entered you should boot into the normal user login prompt. Congrats!

Installing the additional Linux distros

* For installing the additional distros it’s easy to create the root LV for those while still logged into your primary distro. In a new terminal window:
$ sudo lvcreate -L[50G] -n [root2-lv-name] [vg-name]
$ sudo lvcreate -L[50G] -n [root3-lv-name] [vg-name]

There is no need to create a secondary swap LV. They’ll all use the same one.

* Reboot into the Live CD for the new distro. Once booted, open a terminal window and get to a root prompt like before.

* Now open the LUKS container:
# cryptsetup luksOpen /dev/sda5 sda5_crypt
[enter pass]

* Once open, activate the existing LVM LVs:
# lvscan

* If they didn’t automatically active then run:
# lvchange -a y

* If you didn’t already create the root LV for the new distro then do so now:
# lvcreate -L[50G] -n [root2-lv-name] [vg-name]

* Once again, leave the terminal window open and click on the Live CD’s gui installer and follow the instructions. When you get to the disk setup selections, select the Manual option. Use the screens to configure (use /dev/sda2 for the second distro /dev/sda3 for the third):
/dev/sda2 as format ext2, mount /boot
/dev/mapper/[vgname]-[swap-lv-name] as swap area
/dev/mapper/[vgname]-[root2-lv-name] as format ext4, mount /

DOUBLE CHECK that you are using one of the empty /boot partitions and root LV before proceeding or you will overwrite your existing install. Aside from swap, all the partitions for your other existing installs should be marked “Do not use”.

* Continue the install, when prompted for the grub installation choice, this time do not choose /dev/sda, but instead use the partition you defined as /boot (e.g. /dev/sda2). Grub may whine about being installed in a partition, but it works fine (at least for me so far.. knock on wood.)

* Complete the gui install, but again, once completed DO NOT REBOOT YET. Click the “Continue Testing” option.

* Follow the exact same steps as last time to chroot. Make sure you use the new root LV when mounting /mnt/newroot and the proper partition when mounting /boot.

* Use the same blkid process to create the /etc/crypttab file (it should be identical to the one you created last time).

* For these additional distros definitely disable the OS prober in /etc/default/grub. You do not need to add anything to 40_custom this time.
GRUB_DISABLE_OS_PROBER=true

* Again, ensure that the “cryptsetup” and “cryptsetup-bin” packages are installed on this system (should be included on most modern distros) as well as initramfs-tools.

* And again, update the kernel boot images and grub menu for this distro:
# update-initramfs -u
# update-grub

* Reboot!
# reboot

At startup you should see the same Grub menu as before (from the grub installed in /dev/sda), but this time select the “Linux /dev/sda2 chainload” menu option. This will then chainload you to a second Grub menu for that distro. The beauty in this is each distro controls it’s own grub entries and only messes with it’s own initramfs images. You can shorten up the timeouts in the respective /etc/default/grub configs to keep the boot sequence quick. You may also edit the chainload entries in the primary distro’s /etc/grub.d/40_custom file to something more descriptive (maybe “My ub3r l33t h4x0r Kali Install” ;).

Cheers!

PS -- Comments and corrections are invited. I wrote this from memory, while consuming IPAs, a few weeks after the fact so I’m positive it’s full of holes.. hopefully not ones that leave you hanging, but Dragons There May Be.

VirtualBox Tips

Lessons learned the hard way while going virtual with VB.

Host box -- Quad Xeon cores: Ubuntu 12.04 LTS, VirtualBox 4.3.6 (91406) PUEL, phpVirtualBox management gui.

—--
Issue:
Getting error when trying to install a 64bit linux guest: “This kernel requires an x86-64 CPU, but only detected an i686 CPU….”

The host is 64 bit, chipset supports VT and Bios settings are enabled. The guest container has IO APIC, PAE/NX, VT-x/AMD-V and Nested Paging enabled. I can select 64 bit templates for Type and Version… everything should be working for 64 bit.. but doesn’t.

Solution: There’s a bug in the gui on some systems that prevents 64 bit from being properly set on the Guest. Workaround is to create the Guest container in the gui as normal, then use the CLI on the host box to manually set the Guest to 64 bit:

—--
Issue:
Could not get linux Guests to connect to unused physical NICs on Host using Bridged Adapter setting in Guest.

Solution: Turns out the Guest are not able to activate (aka. turn on) the host NIC. NICs should be activated on the Host, but not configured.

Add the following for each NIC (replace “X”) that should be made available for a Guest bridge:

/etc/network/interfaces (on Host)

Credit: http://lastoctet.com/2009/04/12/bring-up-interface-in-debian-ubuntu-with/

—--
Issue:
Install VirtualBoxGuestAdditions_XX.iso in a VirtualBox linux guest (for VirtualBox PUEL version only, not OSE).

Solution:
1) add .iso file on Host as an IDE cdrom in the Guest vbox container settings then boot guest
2) apt-get install linux-headers-[yourkernelversion] build-essential dkms
3) edit Guest /etc/fstab and add:

4) sudo mount /media/cdrom && cd /media/cdrom
5) sudo ./VBoxLinuxAdditions.run -- -nox11
6) If you don’t have X11 on the guest you can ignore “Installing the Window System drivers …fail! (Could not find the X.Org or XFree86 Window System.)” error
7) “sudo lsmod” and confirm “vboxsf” and “vboxguest” modules are loaded

—--
Issue:
Adding VirtualBox Shared Folders into /etc/fstab would cause linux Guest boot failure “/sbin/mount.vboxsf: mounting failed with the error: No such device“. User then had the option to Skip or mount Manually.

The problem is that /sbin/mountall is trying to mount the vboxsf filesystems listed in /etc/fstab before the Virtualbox Guest Additions service has started and loaded the vboxsf module. From searching it sounds like they fixed this somehow in virtualbox-ose, but not in the PUEL version, and I didn’t like the “rc.local” workaround I was finding online. I tried adding the modules to /etc/modules, but it didn’t help.

Solution: What finally worked for me was to build the modules into the boot image using initramfs so they are already loaded at boot with Grub. Again, this is only for the PUEL version:

1) add the Shared Folders in the VBox gui but do not select “Auto-mount” (Auto-mount would mount them under /media/sf_[name] instead of using your fstab entry)

2) add the folders to the Guest /etc/fstab as:

3) run “mount -a” to mount the shares, then run “mount” to see if they are successfully mounted. If not, then you have other issues (probably need to install Virtualbox Guest Additions on Guest vm).

4) install initramfs-tools package

5) add the following two lines in /etc/initramfs-tools/modules

6) run “update-initramfs -u” as root (or sudo)

7) reboot the Guest vm

—--
Issue:
eth0 could not be found on the new Guest VM after cloning an existing Guest.

Solution:
1) Edit file on Guest VM using console: /etc/udev/rules.d/70-persistent-net.rules

2) Delete the entry with the existing eth0

3) Change the eth1 entry to eth0

4) Save file and reboot. Networking will still fail at boot.

5) Use “ifconfig -a” to confirm that new eth0 was created

6) Use Guest console (CLI) to manually setup networking in /etc/network/interfaces

7) Restart networking “/etc/init.d/networking restart” and verify new IP assignment with ifconfig

—--
Issue:
SSH server keys need to be regenerated for security after cloning a Guest VM.

Solution:
1) use RDP console to login
2) “service ssh stop”
3) “rm -rf /etc/ssh/ssh_host_*”
4) “ssh-keygen -A”
5) “service ssh start”

—--

Solving Firefox Error with phpVirtualBox

If you haven’t tried phpVirtualBox, you should. This is a great tool that allows you to manage a headless VirtualBox server via a full featured web gui. In my case, one that’s running VirtualBox on an Ubutntu 12.04 LTS host server.

The phpVirtualBox author deserves praise, honor and a harem of beautiful servants of his choosing for providing this wonderful opensource tool.

Unfortunately, there was trouble in my paradise. I had phpvirtualbox up and running, and could access it successfully using Opera, Chrome, Internet Explorer (gasp) and even Firefox on Windows, but not via Firefox (v26) on my core linux workstation.

So.. in a attempt to save others the hours of puzzling tinkering with Apache configs, content-types, mod_deflate experiments, staring at LiveHeaders and Firebug; here’s the simple answer:

You need to enable DOM Storage in your FF browser to use phpVirtualBox.

Step 1: Type “about:config” into your FF address bar and hit Enter.

Step 2: Search for “dom.storage”

Step 3: Set the “dom.storage.enabled” value to “true”

Step 4: Close the config browser tab and refresh your phpvirtualbox page.

Step 5: You’re welcome.

Cheers.

Ubuntu with Grub2 + LUKS encrypted LVM root + hidden USB keyfile

STANDARD DISCLAIMER APPLIES. USE AT YOUR OWN RISK. This is what worked for me. Your system may be different. IF THINGS GO WRONG YOU’LL BE STUCK DOING MANUAL SYSTEM RECOVERY. This is a high-level guide, not an exact step-by-step and assumes you are familiar with linux administration and command line.

Started with a fresh install of Ubuntu Server (Ubuntu Lucid 10.04.3 LTS) and wanted the same USB keyfile setup as used previously in the links below, but needed to adapt for Grub2.
Debian Lenny + LUKS encrypted root + hidden USB keyfile
Debian Lenny + LUKS encrypted root + hidden USB keyfile (part 2)

I’m too embarrassed to publish how long it took me to figure this out. Let’s just say it’s working now:

1) System setup using the manual partitioner to create crypt container partition and LVM partitions. Use the same process as described here to create and test the usb key and the udev rule file.

2) The keyscript has changed (including modifying the keyscript filename to remove the hyphens, update-initramfs would not copy in the script with hyphens in the filename). Create the following keyscript as “/usr/local/sbin/unlockusbkey.sh”

Make the script executable:

3) Create/edit the following files (edit sda2_crypt and the UUID for your system):

/etc/crypttab

/etc/initramfs-tools/conf.d/cryptroot

/etc/initramfs-tools/modules

4) Create this file (and make executable) to ensure your custom udev rule gets copied into the initrd image when running update-initramfs.

/etc/initramfs-tools/hooks/udevusbkey

5) Edit your Grub2 config. The key line is “GRUB_CMDLINE_LINUX_DEFAULT”. Notice the initrd keyscript location is different than the previous setups (using cryptsetup 2:1.1.0~rc2-1ubuntu13 and initramfs-tools 0.92bubuntu78).

/etc/default/grub

6) Run “update-grub” and check the generated /boot/grub/grub.cfg file to verify the cryptopts kernel options were added.

7) Update the initrd image for your current kernel (or just run “update-initramfs -u” to update all kernels):

8) If you want to verify that everything has copied correctly, you can unpack your current initrd.img to the tmp directory and look through the extracted files. The keyscript should have been copied into the “lib/cryptsetup/scripts” directory and and udev rule into “lib/udev/rules.d/”. The grub keyscript line above should match the location of the keyscript in the initrd image.

9) If all looks in order then reboot. With the USB stick inserted the system should boot all the way to the login prompt. Without the USB stick it should prompt you for your crypt password to mount the root filesystem.

If something goes wrong you’ll likely get dropped to the “(initramfs)” prompt. From here you can manually unlock the crypt partition using “cryptsetup luksOpen /dev/sda2 sda2_crypt” and entering your crypt password. If this unlocks successfully then typing “exit” should drop you back into the automated boot sequence.

————-

Steps for creating & adding additional crypt disks:

1) Use fdisk to create the new partition (using /dev/sdx1 for example) with Type 83, Linux

2) Optional -- good idea to check for badblocks and fill partition with pseudo-random data:
badblocks -c 10240 -s -w -t random -v /dev/sdx1

Or, for truly paranoid types (verrry slow):
dd if=/dev/urandom of=/dev/sdx1

3) Create new luks partition
# cryptsetup --verify-passphrase --verbose --hash=sha256 --cipher=aes-cbc-essiv:sha256 --key-size=256 luksFormat /dev/sdx1
# cryptsetup luksOpen /dev/sdx1 sdx1_crypt

4) create LVM
# pvcreate /dev/mapper/sdx1_crypt
# vgcreate vgname /dev/mapper/sdx1_crypt
# lvcreate -LXXG -n lvname vgname

5) format new LVM partition (example uses ext4 fs)
# mkfs.ext4 /dev/vgname/lvname

6) mount new LV (add to /etc/fstab to make persistent)
# mount /dev/mapper/vgX-nameX /mnt/nameX

7) find the uuid of the encrypted physical partition
# ls -al /dev/disk/by-uuid
uuid-goes-here -> ../../sdx1

or

cryptsetup luksDump /dev/sdx1

8) add new line to /etc/crypttab
sdx1_crypt /dev/disk/by-uuid/uuid-goes-here none luks,keyscript=/usr/local/sbin/unlockusbkey.sh

9) Add the USB key to the new crypt partition:
cryptsetup luksAddKey /dev/sdx1 /root/luks-secret.key --key-slot 1

10) Update the initrd for good measure:
update-initramfs -u

11) profit.

Untangle unravels

After some minor hacking earlier in the year I’ve been okay with Untangle’s performance, until this week. A couple days ago I noticed Untangle wanted to do some updating. I went in and backed up the configs and a few key files, checked the appropriate files were listed as protected in the admin interface and let ‘er rip via the web interface. Untangle goes about it’s business then reboots…. BAM… no love, boot errors. Shite.

After tossing together a USB stick with SystemRescueCD to investigate it becomes clear that Untangle replaced my /boot/grub/menu.lst with a fresh copy. It didn’t just update and add a kernel entry, it just replaced the file altogether. All my grub kopt options for booting the encrypted LVM volumes = gone. Mother fu**er.

Yes, I read the disclaimer. I know customizing virtually anything on Untangle voids the warranty, no user-serviceable parts inside, this tag not to be removed except by consumer, parental guidance is advised, call before you dig, breaking seal constitutes acceptance of agreement, blah, blah, blah.. So why does this piss me off??

1) /boot/grub/menu.lst and /boot/grub/grub.conf (just for good measure) were both listed as “protected” in the admin interface. According to Untangle that means they should not be touched. I took that at face value.. you say you’re not going to touch something, then don’t.. EVER. Give me an error the update can’t be completed if necessary, but don’t just ignore a file is listed as protected and update it anyway.

2) Even if it’s not listed as protected, why would you ever update a Grub config by replacing the file? Grub has a proven process for updating kernel entries that every major distro has used for years. If Untangle used the standard process then it would have picked up my kopt parameters and added them to updated kernel entries and all would have been well with the world. But just replacing the config file carte blanche? Poor form.

3) To add insult to injury -- there wasn’t even a kernel change, so there was ZERO reason for updating the file. Untangle did it just for fun.

So yeah.. I can handle a system with a few unique hooks and a closed internal vm, but pissing on standard practices for updating the underlying core Debian distro.. dealbreaker.

Untangle is out, done, later.. thanks for the good times, wish you well.

Video Encoding for Online HD

Goal: To emulate the YouTube 720p HD video encoding settings to use as a baseline for encoding my own HD vids for use with a player on my own site. YouTube’s 720p encoding seems to be a good balance between filesize and quality. The various built-in render profiles I tested in kdenlive (v0.8) did not seem to have the same efficiency.

NOTE: This is not a guide to the best settings for uploading video to YouTube, this a guide for emulating the encoding process YouTube uses in creating HD video.

YouTube testing process:
1) Uploaded a 441M mp4 (960×720 H.264 8000k) to YouTube. Original source mp4 was from GoPro HD at 960p (1280×960) then spliced, edited and scaled to 960×720 using kdenlive.

2) Downloaded the “720p HD” mp4 file that YouTube created from rendering my video.

3) Used mediainfo to analyze the downloaded 170.4M mp4 file:

Recreating:
1) After much tinkering I’ve found the following kdenlive options seem to provide nearly identical results. Use these to create a custom render profile:

2) The resulting rendered video (using the same 441M source file) is nearly identical in size (170.5M for the kdenlive file vs 170.4M for the YouTube file) and in visual quality.

3) The mp4 rendered by kdenlive is still not quite ready for online streaming as the index remains at the video’s end (google “moov atom” for more info). This means the video will not start playing in an online player like JW Player until the entire file is downloaded into buffer. To fix this, I used a handy tool called MP4Box (apt-get install gpac) to interleave the index info through the video file. Note that this will operate directly on the file you specify so you may want to create a copy of the original first.

Thoughts:
* YouTube is either using some other encoder than ffmpeg or has hacked it to write custom metadata, but again, the video output is nearly identical.
* I’m not sure why YouTube’s encoder uses variable frame rates. The source file is a constant frame rate and ffmpeg retains this by default.
* The same thing can be accomplished using ffmpeg from the CLI without the kdenlive front end; just convert the parameters from the render profile to the ffmpeg switch format.

References:
* Kdenlive -- http://www.kdenlive.org/
* FFmpeg -- “man ffmpeg” and “ffmpeg -h”
* FFmpeg Docs -- http://ffmpeg.org/ffmpeg-doc.html
* FFmpeg / x264 mapping -- http://sites.google.com/site/linuxencoding/x264-ffmpeg-mapping
* FFmpeg preset profiles -- http://juliensimon.blogspot.com/2009/01/howto-ffmpeg-x264-presets.html
* MP4Box -- http://gpac.wp.mines-telecom.fr/mp4box/