Monthly Archives: March 2009

Deploy Archlinux Chroot onto VPS

Update 7/20/2010: I updated this script to be more Lxc friendly. And I also made small patch to modify inittab rc.sysinit rc.shutdown for lxc. If you are not use using dhcpd, you will still need to modify /etc/rc.conf to setup default route.

Download the new script here, and Lxc patch.


Most VPS providers do not have archlinux image or allow changing root device like Linode does. Even though I am comfortable dealing with debian or ubuntu, but tiny difference between them are still annoying over the time. So I decide to install a mini chroot enviroment onto all of them to normalize linux enviroment.

If you want to use a ubuntu or debian chroot, you probably should read DebootstrapChroot. My method here only applys to Archlinux.

These scripts are only for Linux newbies like myself, who are lazy to type all that many commands every time. If you are a Linux guru or sysadmin, you may find this method trivial, insecure or laughable.

Prepare your local system

I assume you already have at least one working Archlinux system installed. First you need to install some necessary tools. If you do not have an archlinux installed, you may skip to last section of post and test the one I built.

pacman -Sy devtools lzma cpio

Devtools includes mkarchroot which is a script bootstrip a mini root similar to debootstrap. If you just run “mkarchroot miniroot base”, it can make you a working mini archlinux. But the default installation is huge about 500MB. You probably do not want all of them inside a VPS enviroment.

Lzma, Cpio are my choice of packaging, you can also use zip, tar, gzip or bzip2, and modify other parts of my script accordingly.

Make a working chroot

The first script is to make a compact mini root and compress it to a single file.
You can either download (outdated) or copy/paste following lines to a file name miniarch

#!/bin/bash
# 2009 Copyright Yejun Yang (yejunx AT gmail DOT com)
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
# http://creativecommons.org/licenses/by-nc-sa/3.0/us/

PACKS="sed gawk coreutils filesystem texinfo grep pacman 
       module-init-tools wget curl net-tools procps nano tar cpio zip 
       gzip bzip2 lzma psmisc initscripts iputils dnsutils iproute2 
       less dash which"

if [[ $1 == i686 ]]; then
  ARCH=i686
else
  ARCH=x86_64
fi

ROOT=mini_$ARCH

cat < pacman.conf
[options]
HoldPkg     = pacman glibc
SyncFirst   = pacman

[core]
Server = ftp://mirror.cs.vt.edu/pub/ArchLinux/$repo/os/$ARCH
Server = http://archlinux.mirrors.uk2.net/$repo/os/$ARCH
Include = /etc/pacman.d/mirrorlist
[extra]
Server = ftp://mirror.cs.vt.edu/pub/ArchLinux/$repo/os/$ARCH
Server = http://archlinux.mirrors.uk2.net/$repo/os/$ARCH
Include = /etc/pacman.d/mirrorlist
[community]
Server = ftp://mirror.cs.vt.edu/pub/ArchLinux/$repo/os/$ARCH
Server = http://archlinux.mirrors.uk2.net/$repo/os/$ARCH
Include = /etc/pacman.d/mirrorlist
EOF

mkarchroot -C pacman.conf $ROOT $PACKS

chmod 666 $ROOT/dev/null
mknod -m 666 $ROOT/dev/random c 1 8
mknod -m 666 $ROOT/dev/urandom c 1 9
mknod -m 600 $ROOT/dev/console c 5 1
mkdir -m 755 $ROOT/dev/pts
mkdir -m 1777 $ROOT/dev/shm

echo nameserver 4.2.2.1 > $ROOT/etc/resolv.conf
echo nameserver 4.2.2.2 >> $ROOT/etc/resolv.conf

find $ROOT -depth -print | cpio -ov | lzma -5 > $ROOT.cpio.lzma

Modify PACKS= to packages you want to be installed.

You should also modify the Server= to whichever fast for you. I used rankmirrors to find out the fastest server.

Run this script

./miniarch

or

./miniarch i686

or both.
It will make a minimal working chroot for Archlinux under currect directory and pack them into a single file mini_x86_64.cpio.lzma or mini_i686.cpio.lzma. These two file should be around 40MB if everything worked correctly.

Copy these .lzma file to your webserver root. Now you can safely delete the working directory

Deploy to VPS

You can download the files you just made to your vps and unpack them. But I made simple script to do that.

You can download or copy/paste following line to a file name deploy

#!/bin/bash
# 2009 Copyright Yejun Yang (yejunx AT gmail DOT com)
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
# http://creativecommons.org/licenses/by-nc-sa/3.0/us/

ARCH=$(uname -m)

if [[ $ARCH != x86_64 ]]; then
ARCH=i686
fi

URL=http://YOURWEBSERVER/

if [ -e /var/chroot/mini_${ARCH} ]; then
  echo "**** /var/chroot/mini_${ARCH} already exists. "
  echo "**** You have to remove previous deployment."
  exit 1
fi

mkdir -p /var/chroot
cd /var/chroot
echo "Start downloading ${URL}mini_${ARCH}.cpio.lzma , be patient ..."
wget -q -O - ${URL}mini_${ARCH}.cpio.lzma | lzma -d | cpio -idv

deploy_success () {
cat <

Change URL= to your own webserver.

Before you running this script on your target machine. Make sure lzma, wget and cpio are installed. If you are using ubuntu, you can run

sudo aptitude update
sudo aptitude install lzma wget cpio

Running this script will deploy a mini chrootable archlinux in to /var/chroot/mini_i686 or /var/chroot/mini_x86_64. The unpacked size will be around 200MB.

To simplify this process, you can copy this file to webserver as well.

wget -q -O - http://yourwebsite/deploy |sudo bash

done.

For lazy people or testing only

If you are so lazy to make your own archlinux mini root or you don't have a working archlinux, you may test my prebuilt mini root by running following line, you will still need lzma, cpio and wget on your target machine.

wget -q -O - http://bit.ly/iZzq |sudo bash

Disclaimer

I DO NOT guarantee the correctness of these script and my prebuilt chroot. Be caution running any command with sudo. You may not hold me responsible for anything happened to your system.

Updates:
April 5, 2009: changed /bin/sh to /bin/bash

Plugin Image Optimizer

This plugin will reduce uploaded image size in wordpress.
You will also need to install Optipng , Jhead and unsafe mode php.

These tools will only strip meta information of your images, therefore the result should be lossless.

You can download this plugin here.

A Funny Saiyuki Episode

A Beautiful Assassin is really funny, stupid and romantic Saiyuki episode.

You promised me you’d follow me to the end of the earth.

<a href="http://www.joost.com/37nir0s/t/Saiyuki-Ep-5-A-Beautiful-Assassin">Saiyuki Ep. 5: A Beautiful Assassin</a>

Akama in Stargate SG-1

When I was watching Stargate SG-1 season 1 episode Fire and Water, I think I found the archetype of Akama in WoW.

The description of male Draenei from WoWWiki:

Draenei males have tendrils coming out of their chin and a fan-like forehead plate which rises and overlaps other forehead plates behind it.

CDN Latency Compared

I compared some puplar commercial CDN’s global latency by using just-ping.

The raw test result you can downloaded from here.

Result Summary:

.

Network

Average

Median

.

Akamai

9.67

2.9

.

Aol CDN

9.22

4

.

Panthercdn

62.36

12.7

.

LimeLight

58.04

13.2

.

Mosso Cloud Files

56.81

13.5

.

Amazon Cloudfront

62.82

18.6

.

Google Homepage

53.53

23.15

.

Cachefly

54.57

28.2

.

Google Ajax Library

54.96

28.5

.

Homemade CDN

76.31

29

.

Yahoo Homepage

82.77

38.4

.

Google App Engine

76.03

42.8

.

US East

130.11

96.9

.

SimpleCDN

142.84

100.8

.

US West

156.32

165.4

Chart:

CDN Latency Comparison

Notes:

  1. Akamai
  2. Aol CDN is served by Akamai
  3. Panther Express
  4. Limelight Networks
  5. Mosso is served by Limelight
  6. Amazon CloudFront Images on this page are served by CloudFront
  7. Google
  8. CacheFly
  9. Google AJAX Libraries
  10. This page is served by my own homemade CDN, you can test speed here
  11. Yahoo
  12. Google App Engine
  13. Single location ip in New Jersey USA listed here for comparison purpose
  14. SimpleCDN
  15. Single location ip in California USA listed here for comparison purpose

Disclamer: I am not affiliated to any company mentioned above. Nor do I endorse the accuracy of these results.

My Google Voice

My grandcentral account is finally ready for Google Voice.

Here is my shiny Google Voice Widget. It will be redirect to voice mail.

Undo email

Google added undo send to their Gmail service.

It reminds me Naruto Please, Mr. Postman! episode. In which Naruto and Jiraiya trying to get back a manuscript from some hardcore postman.

Cloud Server

I just ordered a new cloud server from mosso.

From the server IP address 67-23-8-*.slicehost.net, apparently it is a rebranded slicehost.

To fully utilize my vps, I also build a home-grow GeoDNS server. So if you visit my blog from different locations, you could possibly get 4 different IPs.

Simple way to disable direcotry index in webserver

This script will disable webserver’s directory index for currect directory and all subdirectories.

IFS=$'n'
for i in $( find . -type d); do
    if [[ ! -f "$i/index.html" && ! -f "$i/index.php" ]]; then
        touch "$i/index.html"
    fi
done

Cloud files uploading script

This is a very simple linux bash script to help uploading files to mosso cloud files.

Mosso uploading script: Mosso.sh
Updates: Here is a Mac OSX Version (provided by Bryan Rehbein)

You will also need curl .

After you sign up mosso cloud files service, first you need to login your mosso cloud files control panel then navigate to Your Account / API Access, where you will generate your API key. Edit the begin of mosso.sh

API_KEY=YOURAPIKEYHERE 
USER=yourusername 
CONTAINER=bucket

Now change to the directory where you want to upload files. e.g. a wordpress installation:

cd /srv/http/wordpress
bash mosso.sh wp-includes/js/jquery
Uploading wp-includes/js/jquery/jquery.table-hotkeys.js .... done.
Uploading wp-includes/js/jquery/interface.js .... done.
Uploading wp-includes/js/jquery/ui.core.js .... done.
Uploading wp-includes/js/jquery/jquery.color.js .... done.
Uploading wp-includes/js/jquery/ui.tabs.js .... done.
Uploading wp-includes/js/jquery/ui.resizable.js .... done.
Uploading wp-includes/js/jquery/jquery.hotkeys.js .... done.
Uploading wp-includes/js/jquery/ui.sortable.js .... done.
Uploading wp-includes/js/jquery/ui.dialog.js .... done.
Uploading wp-includes/js/jquery/jquery.js .... done.
Uploading wp-includes/js/jquery/ui.draggable.js .... done.
Uploading wp-includes/js/jquery/jquery.form.js .... done.
Uploading wp-includes/js/jquery/suggest.js .... done.
Uploading wp-includes/js/jquery/jquery.schedule.js .... done.
HTTP/1.1 202 Accepted
Date: Sat, 14 Mar 2009 00:06:41 GMT
Server: Apache
X-CDN-URI: http://cdn.cloudfiles.mosso.com/c12345
Content-Length: 0
Content-Type: text/plain; charset=UTF-8

Write down the url on the line begn with X-CDN-URI. This is your mosso CDN url. You can also find this url inside your mosso control panel.

If you want to upload entire current directory, you can execute this script without argument.