Install Amarok 1.4 on a current Linux Desktop

Posted: December 19th, 2014 | Author: | Filed under: computing | Tags: , , | 5 Comments »


As any long time Linux user knows, Amarok 1.4 was the best music player of all time.. and now thanks to the Trinity Desktop Environment folks, it still is! Using the Trinity Desktop packages it’s possible to get Amarok 1.4 back

Why? Because Amarok 2 is a travesty and needs to be taken out behind the barn. I’ve tried to tweak it and love it, but I just can’t. In my opinion, if the developers ever want to get back to Amarok dominance, they best thing they could do is focus every effort into getting 2 to look and function exactly as 1.4 did. Completely implement *all* the previous 1.4 feature set with the new engine, then, and only then, work on other enhancements.

Clementine on the other hand is a good fork effort, and if you want 95% of what Amarok 1.4 was, without any extra effort, then it should be just an apt-get away on Ubuntu, Mint, etc. (Though you must be able to learn to live with an orange wedge icon.)

Anyway, if you’re a troglodyte like me, and just want things to go back to how they were, then here it is. I’ve tested this on Kubuntu 11.x, 12.x, 14.04 and am currently using it 14.10 Plasma4 (see updates for 15.04 at the bottom of this post). These instructions should work for any of the Ubuntu derivatives, and a similar process should work for all the platforms that Trinity supports (Ubuntu, Debian, RedHat and Fedora).

* Add the Trinity repositories per the instructions here. For Ubuntu 14.10 (Utopic) that means add:
$ sudo nano /etc/apt/sources.list.d/trinity-desktop.list

deb utopic main
deb-src utopic main
deb utopic main
deb-src utopic main

Save the file.

* Add the Trinity GPG signing key:
$ sudo apt-key adv --keyserver --recv-keys F5CFC95C

* Now install the amarok-trinity package. The Trinity folks highly recommend using aptitude instead of apt-get for dependency conflict reasons.
$ sudo apt-get install aptitude
$ sudo aptitude update
$ sudo aptitude install amarok-trinity

The last one should drag in all the necessary Trinity Desktop dependency packages with it. Assuming everything went as it should, you now have Trinity installed under “/opt/trinity”. The Amarok 1.4 executable is at “/opt/trinity/bin/amarok”.

* Now at this point you should be able to start Amarok by specifying the full path from the terminal, but you may get errors as the supporting Trinity files are not in your PATH. To add the Trinity Desktop to your user $PATH so that you can easily execute it in the future, add the following lines to the bottom of your user .profile (do this as your regular user, not root):
$ nano ~/.profile

# added for trinity/amarok14
# set PATH so it includes trinity desktop
if [ -d "/opt/trinity/bin" ] ; then

* Now refresh your user PATH:
$ source ~/.profile

* Running the command below should now show “/opt/trinity/bin” at the end.
$ echo $PATH

* Now fire up Amarok 1.4 for a test drive (note that just “amarok” should still fire up Amarok 2.x if it’s installed):
$ amarokapp



(Optional) Adding a desktop icon for Amarok 1.4:

* Create and save a user desktop file with the following entries:
$ nano ~/Desktop/amarok14.desktop

#!/usr/bin/env xdg-open
[Desktop Entry]
Name=Amarok 1.4
Name[en_US]=Amarok 1.4
GenericName=Audio Player
GenericName[en_US]=Audio Player
Comment=Amarok 1.4 from Trinity Desktop
Comment[en_US]=Amarok 1.4 from Trinity Desktop
Exec=PATH=/opt/trinity/bin:$PATH KDEDIRS=/opt/trinity/ KDEHOME=$HOME/.trinity XDG_DATA_DIRS=/opt/trinity/share/:/usr/share/ MANPATH=/opt/trinity/share/man amarok %U

If you have a plasma widget that displays your Desktop contents then right-click and select “Refresh View” to see your new icon.


(Optional) Create a menu entry for Amarok 1.4:

* Copy the desktop file to your user menu:
$ cp ~/Desktop/amarok14.desktop ~/.local/share/applications/
The menu entry should now show up under Applications > Multimedia.

Note: If you decide to set Amarok 1.4 as your default audio player for some file types then you may want to edit the end of the Exec line in “~/.local/share/applications/amarok14.desktop” to “amarok --append %U” or “amarok --append %U --play” so that your existing playlist gets appended to instead of replaced when you open a music file straight from the file browser.

And the world is now as it should be.


UPDATE (6/2015): Kubutnu 15.04 (Vivid) changes:

* As of this writing, Trinity is not fully supported in 15.04), but their Preliminary Stable Builds are working fine. Install in the same manner as above.

* I had to remove the “/usr/:” portion from “KDEDIRS=” path of the EXEC line for the desktop icon (above example has been amended).

* The system tray icon no longer works. This is a common issue for numerous KDE legacy apps with 15.04. If you previously had Amarok 1.4 installed and set to minimize to the system tray when you closed it then it may appear not to start now because you simply can’t see it. The fix is to first kill any running instances:
$ killall amarokapp

Then modify/add the entries below into the Amarok config file:
$ nano ~/.trinity/share/config/amarokrc


[General Options]
Show Tray Icon=false

Until this is resolved, make sure you do not use the “Show tray icon” setting.

Multiple Linux Distro Installs on a LUKS Encrypted Harddrive

Posted: December 18th, 2014 | Author: | Filed under: computing | Tags: , , , , | 13 Comments »


Install multiple linux distros on one machine that use the same encrypted container.


I’m setting up a new laptop with a 1 TB drive and want to be able to natively boot multiple distributions of linux, but also want everything in the drive except the necessary /boot partition to be encrypted. Each distro will have it’s own LVM volume within the encrypted container but I want all the LVMs to live within the same encrypted container so that data can be shared between volumes and I can grow/shrink/rearrange the logical volumes as needed.

Also.. just because.

The Challenges:

1) Boot partitions -- In a typical multiboot setup I would install Grub for the “primary” OS to the harddrive boot sector and then install the Grub for the secondary distros into their respective partition boot sectors. The primary Grub menu seen at boot would then be configured to chainload the secondary distros. Unfortunately this doesn’t work as simply with the LUKS setup above as Grub can’t access the secondary bootloaders if they are sitting in an encrypted LUKS container.

2) Dumb Installers -- In trying to keep things simple, the installer programs for many distros have been streamlined to the point they cannot support non-standard LUKS/LVM setups such as this.

The Process:

WARNING: FOLLOWING THESE STEPS IS GUARANTEED TO DESTROY EVERY BYTE OF EXISTING DATA ON YOUR HARD DRIVE. I’m serious -- the very first few steps will likely make any existing data unrecoverable by the best NSA jock using the latest secret-alien-area51 gear with unlimited funding. DO NOT PROCEED if you have not backed up every piece of needed data on the installation disk and have physically disconnected any other drives. This is NOT an exact step-by-step guide for beginners and assumes you are already familiar with linux installation and command-line administration.

Preparing the harddrive.

* This writeup assumes the installation harddrive is /dev/sda, adjust accordingly.

* Download and burn a linux installation disk with a Live option (e.g. latest Debian, Ubuntu or Mint). Alternative installers could be adapted as well if you know your stuff.

* Boot into the live environment, open a Terminal and “sudo su -” to obtain a root prompt.

* Create a temporary full disk partition to properly prepare the disk for encryption by first filling it with random data (this is where all existing data goes bye-bye). Use “fdisk /dev/sda” to delete any exiting partitions then create a full disk /dev/sda1 partition using the whole disk with default values. After saving the new table, if you get the following error:

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Then reboot before continuing or you will only overwrite whatever the old sda1 partition size was.

* The step of filling the new disk with random data is usually done with the agonizingly slow “dd if=/dev/urandom”. Thanks to these awesome guys we have a much quicker method available using a clever combo of /dev/zero and LUKS itself. (I now use this even for standard encrypted installs and can then skip the “wipe data” step provided by the installers.)

* Create a LUKS container using /dev/sda1:
# cryptsetup luksFormat /dev/sda1

This will overwrite data on /dev/sda1 irrevocably.
Are you sure? (Type uppercase yes): [YES]
Enter LUKS passphrase: [enter a random one-time-use password]
Verify passphrase: [re-enter password]

* Open the LUKS container:
# cryptsetup luksOpen /dev/sda1 sda1_crypt
[re-enter password]

* Fill the container data using /dev/zero. LUKS will encrypt the /dev/zero input as it writes. For me this is roughly 8x faster than the /dev/urandom method and the result is just as good (or better):
# dd if=/dev/zero of=/dev/mapper/sda1_crypt bs=1M

* Once complete, close the crypto device.
# cryptsetup luksClose sda1_crypt

* Overwrite the small LUKS header space for extra security.
# dd if=/dev/urandom of=/dev/sda1 bs=512 count=2056

Create the new permanent partitions and LUKS container.

* Use fdisk to create a new partition layout. The following layout provides boot partitions for 3 linux distros and the rest of the disk as the encrypted container. If you want more distos, format more boot partitions accordingly. It is possible to use the same /boot for more than one distro, but it will make all kinds of ugliness when you run update-grub. Unfortunately Grub2 is not nearly as refined as the old Grub in customizing these things yet. Using separate boot partitions keeps things easy and clean.

# fdisk /dev/sda
new primary partition 1 = /dev/sda1, 512Mb, type 83 Linux (/boot for primary distro)
new primary partition 2 = /dev/sda2, 256Mb, type 83 Linux (/boot for alternate distro)
new primary partition 3 = /dev/sda3, 256Mb, type 83 Linux (/boot for alternate distro)
new extended partition = /dev/sda4, (disk remainder) type 5 extended
new partition 5 = /dev/sda5, (disk remainder) type 83 Linux (this is the encrypted partition)

Also, if you want to install non-encrypted OS’s (e.g. Windows) then limit the size of the encrypted partition and create additional extended partitions for those OS’s. Note that most non-encrypted OS’s do not need separate boot partitions because they can be chainloaded from the Grub that will be installed in the primary harddrive boot sector. Also note that some versions of Windows may have trouble with booting from extended partitions so you would need to modify your layout accordingly. (Personally, I prefer never run to Windows outside of a Virtualbox).

* Now create the new crypto device. Note that the password you use here will be required each time you boot.

# cryptsetup luksFormat /dev/sda5

This will overwrite data on /dev/sda5 irrevocably.
Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: [new password]
Verify passphrase: [re-enter password]

* Open the new crypto partition:
# cryptsetup luksOpen /dev/sda5 sda5_crypt

* Create a new LVM Physical Volume (PV) from the crypto partition:
# pvcreate /dev/mapper/sda5_crypt

* Create a new LVM Volume Group (VG) using the PV:
# vgcreate [vg-name] /dev/mapper/sda5_crypt

* Create new LVM Logical Volumes (LVs) for swap and root (/). The swap LV will be common for all the distros, but each distro will have it’s own root LV. It may be helpful to name the root LV based on distro (e.g. “kubuntu-root”). You can delete and create new LVs if you swap out distros later. You can also start small as it’s easy to grow the LV later using lvextend (and most filesystems with resize2fs) if you need more space.
# lvcreate -L[8G] -n [swap-lv-name] [vg-name]
# lvcreate -L[50G] -n [root1-lv-name] [vg-name]

* Check everything with “lvdisplay”. If it all looks good then leave the terminal window open and move to the GUI installer.

Installing the initial (primary) Linux distro

* Click on the Live CD’s gui installer icon (probably right on the desktop) and follow the instructions. When you get to the disk setup selections, select the Manual option. Use the screens to configure:
/dev/sda1 as format ext2, mount /boot
/dev/mapper/[vg-name]-[swap-lv-name] as swap area
/dev/mapper/[vg-name]-[root1-lv-name] as format ext4, mount /

All the other available partitions should be listed as “Do not use”.

* If/when you are presented with a Grub install choice, then install this grub instance to the harddrive boot sector “/dev/sda”.

* Complete the gui install, but once completed DO NOT REBOOT YET, instead click the “Continue Testing” option. LUKS must be configured before you reboot.

Enabling LUKS on the Linux distro

* Once the installer completes, you need to chroot from the Live CD environment into your newly installed distro so you can configure LUKS before rebooting.

* From the terminal window you were using previously (as root):
# mkdir /mnt/newroot
# mount /dev/mapper/[vg-name]-[root1-lv-name] /mnt/newroot
# mount -o bind /proc /mnt/newroot/proc
# mount -o bind /dev /mnt/newroot/dev
# mount -o bind /dev/pts /mnt/newroot/dev/pts
# mount -o bind /sys /mnt/newroot/sys
# cd /mnt/newroot
# chroot /mnt/newroot

You are now in the newly installed distro’s environment (the previous /mnt/newroot is now “/”).

* Mount the /boot partition for this distro (e.g. /dev/sda1 for the primary distro)
# mount /dev/sdaX /boot

* Get the UUID of the crypto partition you created previously with fdisk:
# blkid /dev/sda5

* Take the UUID given above (without the quotes) and create the file “/etc/crypttab” with the following single line. (This file will be the same for all linux distros within the LUKS container).
# nano /etc/crypttab

sda5_crypt UUID=your-blkid-goes-here none luks

* Check /etc/fstab to ensure everything looks okay. It should list mounts for swap, /boot and /.

* You may want to add the following line to /etc/default/grub to keep your grub menu clean. This will restrict Grub to only creating menu entries for items in this distro’s /boot partition instead of searching the entire disk and creating menu entries for everything it finds.

* Ensure that the “cryptsetup” and “cryptsetup-bin” packages are installed on the system (should be included on most modern distros) as well as initramfs-tools. For Debian based systems:
# dpkg -l | grep cryptsetup
# dpkg -l | grep initramfs

* Update the kernel boot images and grub menu:
# update-initramfs -u
# update-grub

# Take a look in /boot to ensure the initrd and vmlinuz images exist. Also take a peek at /boot/grub/grub.cfg to review your menu entries.

* Optional: If you want to save time and keep a tidy Grub menu then you can setup menu entries for your other distros now. They obviously won’t work until you install those distros, but it’ll save the step of having to boot back into the primary to update grub again after each install. To do this, add the “GRUB_DISABLE_OS_PROBER” entry mentioned above. Next, manually create the “chainload” entries for the other boot partitions by adding the following to the end of the 40_custom file.

# nano /etc/grub.d/40_custom

menuentry "Linux /dev/sda2 chainload" {
set root=(hd0,2)
chainload +1

menuentry "Linux /dev/sda3 chainload" {
set root=(hd0,3)
chainload +1

Once completed, run “update-grub” again and recheck your “/boot/grub/grub.cfg”. You should see these entries near the bottom.

* Time to take the plunge!
# reboot

* Make sure you eject the Live CD. Once the system reboots you should be presented with your Grub menu and shortly afterwards the boot process will pause and prompt for your LUKS password. Once entered you should boot into the normal user login prompt. Congrats!

Installing the additional Linux distros

* For installing the additional distros it’s easy to create the root LV for those while still logged into your primary distro. In a new terminal window:
$ sudo lvcreate -L[50G] -n [root2-lv-name] [vg-name]
$ sudo lvcreate -L[50G] -n [root3-lv-name] [vg-name]

There is no need to create a secondary swap LV. They’ll all use the same one.

* Reboot into the Live CD for the new distro. Once booted, open a terminal window and get to a root prompt like before.

* Now open the LUKS container:
# cryptsetup luksOpen /dev/sda5 sda5_crypt
[enter pass]

* Once open, activate the existing LVM LVs:
# lvscan

* If they didn’t automatically active then run:
# lvchange -a y

* If you didn’t already create the root LV for the new distro then do so now:
# lvcreate -L[50G] -n [root2-lv-name] [vg-name]

* Once again, leave the terminal window open and click on the Live CD’s gui installer and follow the instructions. When you get to the disk setup selections, select the Manual option. Use the screens to configure (use /dev/sda2 for the second distro /dev/sda3 for the third):
/dev/sda2 as format ext2, mount /boot
/dev/mapper/[vgname]-[swap-lv-name] as swap area
/dev/mapper/[vgname]-[root2-lv-name] as format ext4, mount /

DOUBLE CHECK that you are using one of the empty /boot partitions and root LV before proceeding or you will overwrite your existing install. Aside from swap, all the partitions for your other existing installs should be marked “Do not use”.

* Continue the install, when prompted for the grub installation choice, this time do not choose /dev/sda, but instead use the partition you defined as /boot (e.g. /dev/sda2). Grub may whine about being installed in a partition, but it works fine (at least for me so far.. knock on wood.)

* Complete the gui install, but again, once completed DO NOT REBOOT YET. Click the “Continue Testing” option.

* Follow the exact same steps as last time to chroot. Make sure you use the new root LV when mounting /mnt/newroot and the proper partition when mounting /boot.

* Use the same blkid process to create the /etc/crypttab file (it should be identical to the one you created last time).

* For these additional distros definitely disable the OS prober in /etc/default/grub. You do not need to add anything to 40_custom this time.

* Again, ensure that the “cryptsetup” and “cryptsetup-bin” packages are installed on this system (should be included on most modern distros) as well as initramfs-tools.

* And again, update the kernel boot images and grub menu for this distro:
# update-initramfs -u
# update-grub

* Reboot!
# reboot

At startup you should see the same Grub menu as before (from the grub installed in /dev/sda), but this time select the “Linux /dev/sda2 chainload” menu option. This will then chainload you to a second Grub menu for that distro. The beauty in this is each distro controls it’s own grub entries and only messes with it’s own initramfs images. You can shorten up the timeouts in the respective /etc/default/grub configs to keep the boot sequence quick. You may also edit the chainload entries in the primary distro’s /etc/grub.d/40_custom file to something more descriptive (maybe “My ub3r l33t h4x0r Kali Install” ;).


PS -- Comments and corrections are invited. I wrote this from memory, while consuming IPAs, a few weeks after the fact so I’m positive it’s full of holes.. hopefully not ones that leave you hanging, but Dragons There May Be.

Video Encoding for Online HD

Posted: May 21st, 2011 | Author: | Filed under: computing | Tags: , , , , | No Comments »

Goal: To emulate the YouTube 720p HD video encoding settings to use as a baseline for encoding my own HD vids for use with a player on my own site. YouTube’s 720p encoding seems to be a good balance between filesize and quality. The various built-in render profiles I tested in kdenlive (v0.8) did not seem to have the same efficiency.

NOTE: This is not a guide to the best settings for uploading video to YouTube, this a guide for emulating the encoding process YouTube uses in creating HD video.

YouTube testing process:
1) Uploaded a 441M mp4 (960×720 H.264 8000k) to YouTube. Original source mp4 was from GoPro HD at 960p (1280×960) then spliced, edited and scaled to 960×720 using kdenlive.

2) Downloaded the “720p HD” mp4 file that YouTube created from rendering my video.

3) Used mediainfo to analyze the downloaded 170.4M mp4 file:

$ mediainfo videofile.mp4
Complete name : videofile.mp4
Format : MPEG-4
Format profile : Base Media / Version 2
Codec ID : mp42
File size : 170 MiB
Duration : 7mn 34s
Overall bit rate : 3 144 Kbps
Encoded date :
Tagged date :
gsst :
gstd :
gssd :
gshh :

ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L3.1
Format settings, CABAC : Yes
Format settings, ReFrames : 1 frame
Format settings, GOP : M=1, N=60
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 7mn 34s
Bit rate mode : Variable
Bit rate : 2 986 Kbps
Maximum bit rate : 4 670 Kbps
Width : 960 pixels
Height : 720 pixels
Display aspect ratio : 4:3
Frame rate mode : Variable
Frame rate : 30.000 fps
Minimum frame rate : 29.412 fps
Maximum frame rate : 55.556 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.144
Stream size : 162 MiB (95%)
Tagged date :

ID : 2
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : LC
Codec ID : 40
Duration : 7mn 34s
Bit rate mode : Variable
Bit rate : 152 Kbps
Maximum bit rate : 178 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 44.1 KHz
Compression mode : Lossy
Stream size : 8.24 MiB (5%)
Title : (C) 2007 Google Inc. v08.13.2007.
Encoded date :
Tagged date :

1) After much tinkering I’ve found the following kdenlive options seem to provide nearly identical results. Use these to create a custom render profile:

Profile Name: Custom Web 960x720 3k (4:3) 2pass
Extension: mp4
Parameters: f=mp4 acodec=aac ab=160k ar=44100 vcodec=libx264 vpre=hq minrate=0 maxrate=5000k b=3000k coder=1 refs=1 g=60 s=920x720 aspect=%dar mbd=2 trellis=1 mv4=1 subq=7 qmin=10 qcomp=0.6 qdiff=4 qmax=51 pass=2

2) The resulting rendered video (using the same 441M source file) is nearly identical in size (170.5M for the kdenlive file vs 170.4M for the YouTube file) and in visual quality.

$ mediainfo kdenlivevideo.mp4
Complete name : kdenlivevideo.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : isom
File size : 170 MiB
Duration : 7mn 34s
Overall bit rate : 3 146 Kbps
Writing application : Lavf52.64.2

ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L3.1
Format settings, CABAC : Yes
Format settings, ReFrames : 2 frames
Format settings, GOP : M=1, N=60
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 7mn 34s
Bit rate mode : Variable
Bit rate : 3 000 Kbps
Width : 920 pixels
Height : 720 pixels
Display aspect ratio : 4:3
Frame rate mode : Constant
Frame rate : 30.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.151
Stream size : 162 MiB (95%)
Writing library : x264 core 106 Ubuntu_2:0.106.1741-3
Encoding settings : cabac=1 / ref=1 / deblock=1:0:0 / analyse=0x3:0x113 / me=umh / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / constrained_intra=0 / bframes=3 / b_pyramid=0 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=60 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=2pass / mbtree=1 / bitrate=3000 / ratetol=1.3 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.41 / aq=1:1.00

ID : 2
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : LC
Codec ID : 40
Duration : 7mn 34s
Bit rate mode : Variable
Bit rate : 144 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 44.1 KHz
Compression mode : Lossy
Stream size : 7.70 MiB (5%)

3) The mp4 rendered by kdenlive is still not quite ready for online streaming as the index remains at the video’s end (google “moov atom” for more info). This means the video will not start playing in an online player like JW Player until the entire file is downloaded into buffer. To fix this, I used a handy tool called MP4Box (apt-get install gpac) to interleave the index info through the video file. Note that this will operate directly on the file you specify so you may want to create a copy of the original first.

$ MP4Box -inter 500 kdenlivevideo.mp4

* YouTube is either using some other encoder than ffmpeg or has hacked it to write custom metadata, but again, the video output is nearly identical.
* I’m not sure why YouTube’s encoder uses variable frame rates. The source file is a constant frame rate and ffmpeg retains this by default.
* The same thing can be accomplished using ffmpeg from the CLI without the kdenlive front end; just convert the parameters from the render profile to the ffmpeg switch format.

* Kdenlive --
* FFmpeg -- “man ffmpeg” and “ffmpeg -h”
* FFmpeg Docs --
* FFmpeg / x264 mapping --
* FFmpeg preset profiles --
* MP4Box --

Tangling with Untangle

Posted: October 18th, 2010 | Author: | Filed under: computing | Tags: , | 8 Comments »

This is a work in progress as I build up an Untangle box for personal use. Using the Lite (open source) version 7.4 (EDIT 11.15.2010 -- now running 8.0.0) on a Xeon server with 4 physical interfaces.

DO NOT follow any of this verbatim. This is a jumble of notes for my personal use that I hope may help a few other folks trying for a similar setup.

Install from USB disk:
Use manual partitioning to setup full disk encryption (sda2_crypt) with LVM for separate / and /home. ext2 /boot, xfs for root & home

Enable & lockdown ssh:
* edit /etc/ssh/sshd.conf with my preferred config
* mv /etc/ssh/sshd_not_to_be_run /etc/ssh/sshd_not_to_be_run.bak
* restart ssh

Use the terminal from the gui to create root password and a non-root user
Lockdown ssh from root login and setup key login for non-root user (disable direct root login and restrict to specific user account using AllowUsers, also enable MaxStartups 10:30:60).

Add USB key for LUKS

Install GRUB password and backup menu.lst against future Untangle updates
Add grub boot option for verbose, non-splash boot and make default selection
# altoptions=(text boot) vga=791
Add /boot/grub/menu.lst as a file override in the gui.

/etc/default/bootlogd change BOOTLOGD_ENABLE=Yes

Edit /etc/apt/sources.list to uncomment debian main. (re-comment and apt-get update when done installing)

Install xfsprogs
Install/configure smartmontools
Install mlocate
Install memtest86+
Install/configure lm-sensors & i2c-tools

Secure the Untangle kiosk (X does not belong on servers but Untangle depends on it.. oh well.)
Instructions for locking down physical access to the Untangle kiosk:

First, follow jdelagarza’s excellent post on setting up xscreensaver and test that it’s working:

Unfortunately this can still be easily defeated using ctrl+alt key combos to restart X.
1) build a good xorg.conf (see example below) so Xserver uses /etc/X11/xorg.conf instead of falling back to /etc/X11/xorg-untangle-vesa.conf (which is useless to modify directly as it automatically overwritten by /home/kiosk/.bashrc)

2) add a ServerFlags section with the following options:
“DontZap” to disable ctrl+alt+bksp and ctrl+alt+del
“DontVTSwitch” to disable ctrl+alt+F[1-12] (especially want to disable F10 which also kills X)
“DontZoom” to disable using ctrl+alt+[+/-] to crash X

3) reboot/restart X
My current /etc/X11/xorg.conf

# xorg.conf (X.Org X Window System server configuration file)
# If you have edited this file but would like it to be automatically updated
# again, run the following command:
# sudo dpkg-reconfigure -phigh xserver-xorg

Section "ServerFlags"
Option "DontZap" "true"
Option "DontVTSwitch" "true"
Option "DontZoom" "true"

Section "InputDevice"
Identifier "Generic Keyboard"
Driver "kbd"
Option "XkbRules" "xorg"
Option "XkbModel" "pc104"
Option "XkbLayout" "us"

Section "InputDevice"
Identifier "Configured Mouse"
Driver "mouse"

Section "Device"
Identifier "Configured Video Device"
Driver "vesa"

Section "Monitor"
Identifier "Configured Monitor"
VertRefresh 60

Section "Screen"
Identifier "Default Screen"
Monitor "Configured Monitor"
Device "Configured Video Device"
DefaultDepth 24
SubSection "Display"
Depth 24
Virtual 1024 768
Modes "1024x768"

You can verify which xorg.conf file the Xserver is using with this command:
# cat /var/log/Xorg.0.log | grep “Using config file”

See this forum post for comments:

Reconfiguring packet filter to allow DHCP on other interfaces

Hmmm.. trying to research how to serve DHCP on interfaces other than the Internal and DMZ but still be able to use packet filter to block unwanted interfaces. The following post explains how we can config dnsmasq to serve on multiple interfaces but the only suggestion for configuring the packet filter is to turn off DHCP filtering for all interfaces, which leaves the External interface exposed:

I didn’t like this, and after a bit of searching it appears I’m not alone:

So.. as a long time linux user it seemed only proper to crack open my Untagle box and do a bit of digging.

It appears that the following ruby script is responsible for doing most the work of translating the packet filter gui into actual iptables rules:


When you apply changes in the gui the script then rebuilds the iptables rules in this file:


By studying this file you can see that when you apply the built-in “Allow DHCP Requests from the internal interface.” rule it creates the following iptables rule:

iptables -t filter -I INPUT 1 -p udp -m mark --mark 2/2 -m multiport --destination-ports 67 -j RETURN

The “Allow DMZ..” rule does the same except that the “--mark 2/2” portion changes for each interface.

I then tried making an “Allow DHCP” rule of my own for my eth3 using the gui which creates this actual iptable rule:

iptables -t mangle -A firewall-rules -p udp -m multiport --destination-ports 67 -m mark --mark 8/8 -j RETURN

Now we’re onto something… I then checked the “Block all DHCP Requests to the local DHCP Server.” rule and found it creates:

iptables -t filter -A INPUT -p udp -m multiport --destination-port 67 -j DROP

Bingo! The difference is that all the user rules are created by appending rules to the end of “firewall-rules” rule set of the “mangle” table whereas the built-in set of Allow rules are inserting rules into the top of the input chain (INPUT 1) in the default “filter” table. In short this means that when creating custom user rules they’re getting added after the the built-in “Block all DHCP..” rule so the DHCP packets are being dropped before they ever get to the user created Allow rules (vs the built-in Allow rules that get inserted before the built-in “Block all DHCP..” rule).

I was able to test this by manually adding an iptable rule for my eth3 interface that uses the insert vs the append method and it was successful in allowing DHCP (but this would get overwritten by untangle in the next rule refresh):

iptables -t filter -I INPUT 1 -p udp -m mark --mark 8/8 -m multiport --destination-ports 67 -j RETURN

So.. now understanding the issue. The simplest solution I found was to just abandon the built-in rules and do it all through a couple user rules.

1) Uncheck: “Block all DHCP Requests to the local DHCP Server.”, “Allow DHCP Requests from the DMZ interface.”, “Allow DHCP Requests from the internal interface.”

2) Create a user rule to accept on any of the interfaces you do want DCHP:
Action: Pass, Protocol: UDP, Destination Port: 67, Source Interface: Internal, eth3

3) Create a rule to Drop on all the interfaces:
Action: Drop, Protocol: UDP, Destination Port: 67, Source Interface: all (even the ones you checked in the previous Pass rule)

Make sure the Pass rule is ordered above the Drop rule. Assuming your DHCP is configured correctly you should now have DHCP access on any interfaces you have checked within the Pass rule.

Mapping the Untangle Packet Filter rules.

The interfaces on this box are identified by Untangle as follows ( the “--mark #/#” is basically another identifier for the different interfaces within iptables):
eth0 = External (--mark 1/1)
eth1 = Internal (--mark 2/2 and --mark 258/258)
eth2 = DMZ (--mark 4/4)
eth3 = eth3 (--mark 8/8)
VPN (--mark 128/128)
All Interfaces (--mark 256/256)

First line = the built-in rule description listed in the Untangle gui.
Second/third lines = actual rule(s) created in /etc/untangle-net-alpaca/iptables-rules.d/400-firewall by checking the gui description.


Allow DHCP Requests from the internal interface.
${IPTABLES} -t filter -I INPUT 1 -p udp -m mark --mark 2/2 -m multiport --destination-ports 67 -j RETURN

Allow DHCP Requests from the DMZ interface.
${IPTABLES} -t filter -I INPUT 1 -p udp -m mark --mark 4/4 -m multiport --destination-ports 67 -j RETURN

Block all DHCP Requests to the local DHCP Server.
${IPTABLES} -t filter -A INPUT -p udp -m multiport --destination-port 67 -j DROP

Prefer Local DHCP Traffic from non-internal interfaces.
${IPTABLES} -t mangle -A FORWARD -p udp -m multiport --destination-ports 67,68 -m physdev --physdev-is-bridged --physdev-out eth1 -j DROP
${IPTABLES} -t mangle -A FORWARD -p udp -m multiport --destination-ports 67,68 -m physdev --physdev-is-bridged --physdev-in eth1 -j DROP

Accept DHCP traffic to the local DHCP client.
${IPTABLES} -t filter -I INPUT 1 -p udp -m multiport --destination-ports 68 -j RETURN

Accept DNS traffic from the Internal and VPN interfaces to the local DNS Server.
${IPTABLES} -t mangle -A firewall-rules -p udp -m multiport --destination-ports 53 -m mark --mark 258/258 -j RETURN
${IPTABLES} -t mangle -A firewall-rules -p udp -m multiport --destination-ports 53 -m mark --mark 384/384 -j RETURN

Accept DNS traffic to the local DNS Server from all interfaces.
${IPTABLES} -t mangle -A firewall-rules -p udp -m multiport --destination-ports 53 -m mark --mark 256/256 -j RETURN

Accept SNMP traffic from the Internal interface.
${IPTABLES} -t mangle -A firewall-rules -p udp -m multiport --destination-ports 161 -m mark --mark 258/258 -j RETURN

Accept SNMP traffic from all interfaces.
${IPTABLES} -t mangle -A firewall-rules -p udp -m multiport --destination-ports 161 -m mark --mark 256/256 -j RETURN

Block OpenVPN traffic from the internal interface.
${IPTABLES} -t mangle -A firewall-rules -p udp -m multiport --destination-ports 1194 -m mark --mark 258/258 -j alpaca-pfi-drop

Accept OpenVPN traffic from all interfaces.
${IPTABLES} -t mangle -A firewall-rules -p udp -m multiport --destination-ports 1194 -m mark --mark 256/256 -j RETURN

Accept SSH traffic from all interfaces.
${IPTABLES} -t mangle -A firewall-rules -p tcp -m multiport --destination-ports 22 -m mark --mark 256/256 -j RETURN

Allow Ping on all interfaces.
${IPTABLES} -t mangle -A firewall-rules -p icmp -m mark --mark 256/256 -j RETURN

Block all local traffic.
(added to 700-nat-firewall)
${IPTABLES} -t mangle -A firewall-rules -j alpaca-pfi-drop

Accept incoming VPN traffic when running as a VPN client.

Route VPN traffic that would go through the Bridge.

Samba + SWAT + GOsa setup:
# apt-get install samba=2:3.2.5-4lenny11 samba-common=2:3.2.5-4lenny11
# apt-get install swat smbfs samba-docs
Create separate Packet Filter rules to allow 135,139,445 TCP and 137,138 UDP from select interfaces
Setup via SWAT (localhost 901)

uncomment apt deb sources and add deb unstable
# apt-get install gosa (to install dependent packages)
# apt-get install -t unstable (to install newest gosa with fcgi bug fix)
# apt-get install gosa-schema gosa-plugin-ldapmanager gosa-plugin-samba smbldap-tools libnss-ldap libpam-ldap
# apt-get remove lighttpd (conflicts with apache2)

Use /usr/share/doc/gosa/slapd.conf-example/slapd.conf.gz for /etc/ldap/slapd.conf and customize

pulling gosa off and using webmin instead.
# add webmin repo
deb sarge contrib

Other good references:

Untangle -- useful files:

* Configure untangle modules
* Add LUKS disk encryption for second SATA drive and for external eSATA backup drives
* Setup filesharing via Samba
* OpenVPN
* Bootp
* Enable sadc in /etc/default/sysstat ?

*** This post is still under construction ***

Debian Lenny + LUKS encrypted root + hidden USB keyfile

Posted: May 22nd, 2010 | Author: | Filed under: computing | Tags: , , | 16 Comments »

The setup:

* Recently built up a new 1U Xeon quad core to take over home server duties.

* Installed a fresh copy of Debian Lenny using a USB stick install and the debian-504-amd64-netinst.iso image.

* Setup and partitioned disks manually with the Debian installer:

/dev/sda1 as /boot
/dev/sda2 as encrypted LUKS partition sda2_crypt
sda2_crypt as an LVM physical volume (technically /dev/dm-0)
/dev/mapper/vg0-swap as swap
/dev/mapper/vg0-root as root

* Finished standard install, configured system basics and tested working LUKS setup using normal console password entry during boot.

* Converted system to Proxmox VE running proxmox-ve-2.6.24 kernel. (Shouldn’t be relevant to this how-to).

The Challenge:

All is working well, but since this will be a headless server sitting down in the utility room it’s going to be a PITA to have to physically enter the LUKS password at each reboot. I could setup a remote LUKS passphrase over ssh, but really I want the machine to be able to survive a reboot without my intervention to get it running. So, for my needs a USB key sounds like the ticket.

I found a few nice how-to’s via the Google, but I wanted a few tweaks so I ended up using a blend of the following:
* The passwordless disk encryption in Debian Etch how-to is an excellent guide and provided much of what I needed.
* The Unlocking a luks volume with a USB key how-to doesn’t work for encrypting root, but I liked the idea of hiding the key between the MBR and first partition of the USB stick (yes, I know, security through obscurity is bad.. blah, blah).
* I found a nice udev config in section 2.1 of this post.
* Found the solution of adding kopts parameters to grub from somewhere else I can’t seem to find again.

Getting it done:

This assumes you already have a working LUKS setup using console password entry.

Creating the key

1) Insert your usb stick and use dmesg to identify the device file. We’ll assume /dev/sdx.

2) Fill the entire usb stick with random data (this will erase all data on the usb stick). If you’re extra paranoid and have lots of spare time use /dev/random instead.

# dd if=/dev/urandom of=/dev/sdx bs=1

3) If you still want the usb stick to be usable for storing data you’ll need to recreate the partition table and filesystem. Use fdisk to create a new single partition and mark the partition as type “W95 FAT32” then use mkfs.vfat to format the new partition.

4) Extract 4096 bits of random data off the usb stick that will become the new keyfile from in between the MBR and the first partition.

# dd if=/dev/sdx of=/root/luks-secret.key bs=512 skip=4 count=8

5) Add the key to your LUKS encrypted partition in key slot 1 (your current LUKS password should already be in slot 0). You’ll be prompted for you current password when running this.

# cryptsetup luksAddKey /dev/sda2 /root/luks-secret.key --key-slot 1

Your usb stick is now ready to go.

6) If you don’t want to keep a backup copy of this key on your filesystem then use shred.

# shred --remove --zero /root/luks-secret.key

Creating a udev rule

This will make your specific usb stick available at /dev/usbkey when inserted. Other usb sticks will be ignored even if they contained the same keyfile.

1) Run the following command to get the necessary information about the usb stick.

# udevadm info -a -p $(udevadm info -q path -n /dev/sdx)

2) In the output, look for the section that contains SUBSYSTEMS==”usb”, DRIVERS==”usb”, ATTR{manufacturer}, ATTR{product} and ATTR{serial}. Use this information for creating the rule in the next step.

3) Create a file in “/etc/udev/rules.d/99-unlock-luks.rules” that contains the following (all on one line):

SUBSYSTEMS=="usb", DRIVERS=="usb", ATTRS{manufacturer}=="XYZ Corporation", ATTRS{product}=="Flash Thingy", ATTRS{serial}=="0123456789abc", SYMLINK+="usbkey%n"

4) Reload udev rules with:

# udevadm control --reload-rules

5 ) Test that /dev/usbkey is created when the usb stick is inserted.

Create the keyscript

This will be the shell script responsible for reading the keyfile from the usb stick and passing it via cat to cryptsetup when called upon during boot. If the keyfile is not available or valid then it will revert to asking for your normal LUKS password on the console. The nice part about this keyscript is that by reading the keyfile directly using dd we get to skip worrying about drivers for mounting the usb stick’s filesystem.

1) Create a keyscript containing the following as “/usr/local/sbin/”


# flag tracking key-file availability

# check and modprobe the USB driver if not already loaded
cat /proc/modules | busybox grep usb_storage >/dev/null 2>&1
if [ $USBLOAD -gt 0 ]; then
modprobe usb_storage >/dev/null 2>&1

# give the system time to settle and open the USB device
sleep 10

# check for the specifc /dev/usbkey device created by udev using /etc/udev/rules.d/99-unlock-luks.rules
if [ -b /dev/usbkey ]; then
# if device exists then output the keyfile from the usb key (hidden key is 4096 bytes long starting at 2048 bytes)
dd if=/dev/usbkey bs=512 skip=4 count=8 | cat

if [ $OPENED -ne $TRUE ]; then
echo "FAILED to get USB key file ..." >&2
/lib/cryptsetup/askpass "Try LUKS password: "
echo "Success loading key file. Moving on." >&2

sleep 2

2) Make the script executable:

# chmod a+x /usr/local/sbin/

Configure cryptsetup & initramfs-tools

1) Install initramfs-tools package.

2) Add the keyscript parameter to /etc/crypttab

sda2_crypt /dev/sda2 none luks,keyscript=/usr/local/sbin/

3) Add the following modules to /etc/initramfs-tools/modules

aes-x86_64 (or aes-i586 if running 32-bit)

4) Add the following to /etc/initramfs-tools/conf.d/cryptroot


5) Ensure “MODULES=most” and “BUSYBOX=y” are set in /etc/initramfs-tools/initramfs.conf

Update grub config & initrd images

Now we need to build a new boot initrd.img that contains the scripts and modules we configured above.
1) First, create a “safe” backup copy of your current initrd image.

# cd /boot

Copy your current initrd version.
# cp initrd.img-2.6.XX-X-amd64 initrd.img-2.6.XX-X-amd64-safe

2) Edit /boot/grub/menu.lst and create a duplicate of your current boot option that utilizes the “safe” initrd.img.

title Debian GNU/Linux, kernel 2.6.XX-X-amd64
root (hd0,0)
kernel /vmlinuz-2.6.XX-X-amd64 root=/dev/mapper/vg0-root ro
initrd /initrd.img-2.6.XX-X-amd64

title Debian GNU/Linux, kernel 2.6.XX-X-amd64-safe
root (hd0,0)
kernel /vmlinuz-2.6.XX-X-amd64 root=/dev/mapper/vg0-root ro
initrd /initrd.img-2.6.XX-X-amd64-safe

3) For some reason I still haven’t figured out (and haven’t spent much time further researching), I needed to add the “cryptopts” to my kernel boot options to make everything work. Otherwise, instead of the init scripts mounting root, I would get errors similar to “LVM driver is detected but LVM is not configured” during boot.
Again, edit /boot/grub/menu.lst and add the “cryptopts” parameters to your current kopts line.

# kopt=root=/dev/mapper/vg0-root ro cryptopts=target=sda2_crypt,source=/dev/sda2,lvm=vg0-root,keyscript=/keyscripts/

4) Run “update-grub”. This will update all your boot kernels and the kopts line from the previous step will ensure it’s added to new kernels as well.

title Debian GNU/Linux, kernel 2.6.XX-X-amd64
root (hd0,0)
kernel /vmlinuz-2.6.XX-X-amd64 root=/dev/mapper/vg0-root ro cryptopts=target=sda2_crypt,source=/dev/sda2,lvm=vg0-root,keyscript=/keyscripts/
initrd /initrd.img-2.6.XX-X-amd64

5) Now that you have a backup “safe” initrd.img, it’s time to update your current one to include the scripts and modules configured in the previous steps so that they are available at boot. Thanks to the initramfs-tools package, this is as simple as:

# update-initramfs -u -k 2.6.XX-X-amd64

6) If you want to verify that everything has copied correctly, you can unpack your current initrd.img to the tmp directory and look through the extracted files. The keyscript should have been copied into the “keyscripts” folder.

# cd /tmp
# zcat /boot/initrd.img-2.6.XX-X-amd64 | cpio -iv
# ls -al keyscripts/


1) Reboot with the usb stick installed and select the boot option that uses the new initrd.img you created. The system should boot all the way to the login prompt.
2) Reboot without the usb stick and it should stop at the prompt for your LUKS password.
3) If you run into issues you can reboot with your “safe” kernel.


In my opinion there are a couple advantages to this setup:
1) The udev script approach makes it a tiny bit more difficult for someone to use an alternate usb stick even if they had the keyfile.
2) The dd method of “hiding” the keyfile in random deadspace means that even if someone got their hands on your usb stick they wouldn’t know they had a keyfile.


So.. what’s the point to encrypting if you’re just going to leave the key sitting in the machine? For me, the drive encryption isn’t about protecting against the NSA or uber-l33t hackers. I just want to ensure if the hardware gets ripped off during a break-in that the data is secure. My home setup will allow me to physically secure the usb key nearby and run a usb extension cable from the key to the server. I’ll take my chances that the average criminal would just unplug any cables when taking the machine and regardless, it would be extremely difficult to physically get to the secured usb stick even if they knew to look for it. Of course if you value security over the convenience of unattended reboots then just pull usb stick when not in use.


Update 10/14/2010 -- corrected several typos.
Update 11/24/2010 -- added part 2:
Update 01/02/2011 -- Updated version for Ubuntu/Grub2: