Monthly Archives: February 2009 - Page 2

Nginx vs Cherokee

I did a micro benchmark for quick comparison between nginx and cherokee on Linode 540 VPS. Result may differ on a dedicate server. static file(100MB)

Cherokee
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software:        Cherokee/0.99.0
Server Hostname:        localhost
Server Port:            80

Document Path:          /100mb.test
Document Length:        104857600 bytes

Concurrency Level:      20
Time taken for tests:   30.696 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      20971557200 bytes
HTML transferred:       20971520000 bytes
Requests per second:    6.52 [#/sec] (mean)
Time per request:       3069.623 [ms] (mean)
Time per request:       153.481 [ms] (mean, across all concurrent requests)
Transfer rate:          667184.09 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.2      1       1
Processing:  3004 3069  73.9   3048    3228
Waiting:        0    1   0.2      1       2
Total:       3004 3069  73.9   3048    3228

Percentage of the requests served within a certain time (ms)
  50%   3048
  66%   3068
  75%   3071
  80%   3191
  90%   3228
  95%   3228
  98%   3228
  99%   3228
 100%   3228 (longest request)
Nginx
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software:        nginx/0.6.35
Server Hostname:        localhost
Server Port:            80

Document Path:          /100mb.test
Document Length:        104857600 bytes

Concurrency Level:      20
Time taken for tests:   30.543 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      20971571000 bytes
HTML transferred:       20971520000 bytes
Requests per second:    6.55 [#/sec] (mean)
Time per request:       3054.277 [ms] (mean)
Time per request:       152.714 [ms] (mean, across all concurrent requests)
Transfer rate:          670536.80 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.2      0       1
Processing:  3006 3054  32.6   3047    3106
Waiting:        0    1   0.4      1       3
Total:       3006 3054  32.6   3048    3107
ERROR: The median and mean for the initial connection time are more than twice the standard
       deviation apart. These results are NOT reliable.

Percentage of the requests served within a certain time (ms)
  50%   3048
  66%   3069
  75%   3087
  80%   3098
  90%   3107
  95%   3107
  98%   3107
  99%   3107
 100%   3107 (longest request)

Small html file (468 bytes)

Nginx
Server Software:        nginx/0.6.35
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        468 bytes

Concurrency Level:      200
Time taken for tests:   0.123 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      737100 bytes
HTML transferred:       491400 bytes
Requests per second:    8150.03 [#/sec] (mean)
Time per request:       24.540 [ms] (mean)
Time per request:       0.123 [ms] (mean, across all concurrent requests)
Transfer rate:          5866.59 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        3    8   1.7      9      11
Processing:     6   14   3.3     14      24
Waiting:        3   11   3.3     12      23
Total:         14   22   2.6     22      33

Percentage of the requests served within a certain time (ms)
  50%     22
  66%     23
  75%     23
  80%     24
  90%     25
  95%     26
  98%     29
  99%     30
 100%     33 (longest request)
Cherokee
Server Software:        Cherokee/0.99.0
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        468 bytes

Concurrency Level:      200
Time taken for tests:   0.139 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      732224 bytes
HTML transferred:       509184 bytes
Requests per second:    7203.83 [#/sec] (mean)
Time per request:       27.763 [ms] (mean)
Time per request:       0.139 [ms] (mean, across all concurrent requests)
Transfer rate:          5151.19 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        6   11   2.5     12      16
Processing:     7   14   3.9     14      27
Waiting:        3   10   3.9      9      19
Total:         16   25   3.5     25      33

Percentage of the requests served within a certain time (ms)
  50%     25
  66%     27
  75%     27
  80%     28
  90%     29
  95%     31
  98%     32
  99%     32
 100%     33 (longest request)
Varnish
Server Software:        ----------
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        468 bytes

Concurrency Level:      200
Time taken for tests:   0.141 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      879491 bytes
HTML transferred:       528372 bytes
Requests per second:    7067.44 [#/sec] (mean)
Time per request:       28.299 [ms] (mean)
Time per request:       0.141 [ms] (mean, across all concurrent requests)
Transfer rate:          6070.07 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        6   11   2.5     11      16
Processing:     6   14   3.5     13      24
Waiting:        3   10   3.5      9      21
Total:         17   25   3.4     25      38

Percentage of the requests served within a certain time (ms)
  50%     25
  66%     26
  75%     26
  80%     27
  90%     29
  95%     31
  98%     33
  99%     34
 100%     38 (longest request)

Small image file (23k bytes)

Nginx
Server Software:        nginx/0.6.35
Server Hostname:        localhost
Server Port:            80

Document Path:          /cherokee-logo.png
Document Length:        23619 bytes

Concurrency Level:      200
Time taken for tests:   0.139 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      24624128 bytes
HTML transferred:       24379632 bytes
Requests per second:    7171.34 [#/sec] (mean)
Time per request:       27.889 [ms] (mean)
Time per request:       0.139 [ms] (mean, across all concurrent requests)
Transfer rate:          172449.16 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    5   3.6      4      16
Processing:     0   20   6.1     23      29
Waiting:        0   15   5.7     17      26
Total:          6   25   5.5     26      45

Percentage of the requests served within a certain time (ms)
  50%     26
  66%     27
  75%     27
  80%     27
  90%     28
  95%     35
  98%     41
  99%     43
 100%     45 (longest request)
Cherokee
Server Software:        Cherokee/0.99.0
Server Hostname:        localhost
Server Port:            80

Document Path:          /cherokee-logo.png
Document Length:        23619 bytes

Concurrency Level:      200
Time taken for tests:   0.180 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      26518338 bytes
HTML transferred:       26287947 bytes
Requests per second:    5564.92 [#/sec] (mean)
Time per request:       35.939 [ms] (mean)
Time per request:       0.180 [ms] (mean, across all concurrent requests)
Transfer rate:          144113.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4   11   3.4     11      18
Processing:    11   21   5.0     21      36
Waiting:        3    8   3.7      7      24
Total:         21   32   4.7     33      43

Percentage of the requests served within a certain time (ms)
  50%     33
  66%     34
  75%     35
  80%     36
  90%     38
  95%     40
  98%     41
  99%     43
 100%     43 (longest request)
Varnish
Server Software:        --------------
Server Hostname:        localhost
Server Port:            80

Document Path:          /cherokee-logo.png
Document Length:        23619 bytes

Concurrency Level:      200
Time taken for tests:   0.154 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      23933000 bytes
HTML transferred:       23619000 bytes
Requests per second:    6498.44 [#/sec] (mean)
Time per request:       30.777 [ms] (mean)
Time per request:       0.154 [ms] (mean, across all concurrent requests)
Transfer rate:          151882.08 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4   11   3.0     11      17
Processing:    10   18   4.5     17      33
Waiting:        2    6   2.1      5      12
Total:         15   29   4.6     29      48

Percentage of the requests served within a certain time (ms)
  50%     29
  66%     29
  75%     30
  80%     31
  90%     33
  95%     36
  98%     41
  99%     46
 100%     48 (longest request)

Nginx is the fastest to serve small static file on my VPS. No significant difference in serving larging file.

Speedup Mysql and Webserver with Intel Compiler and tcmalloc

After reading some recent benchmark reguarding tcmalloc  performance on mysql. I decide to rebuild my whole webhosting stack with it.

ICC is intel’s c++ compiler, which has faster performance is also memtioned on mysql website.

Most distros should already has google performance tools prepackaged. Installation of ICC is slightly more complicated, you can download it directly from intel’s website which is free download for non-commercial use. Archlinux and Gentoo both have packaged installer. On ubuntu/debian system you probably also need build-essential and apt-build to rebuilt packages. On archlinux you will need base-devel and abs.

For most packages, the following bash script can be used before configuration/make step.  Don’t ommit the dot on first line and change the path of iccvars.sh to your installation directory.

 . /opt/intel/Compiler/11.0/081/bin/iccvars.sh intel64
CC=icc
CFLAGS="-xHOST -O3 -no-prec-div "
LD=xild
AR=xiar
CXX=icpc
CXXFLAGS="-xHOST -O3 -no-prec-div "
LDFLAGS=-ltcmalloc_minimal
export CC CFLAGS LD AR CXX CXXFLAGS LDFLAGS

These setting seems safe for all packages. Here is a summary of package specific cflags setting.

  Mysql Cherokee Nginx Varnish PHP Memcached
-static No No No No N/A No
-ipo No No Yes No N/A Yes
LDFLAGS=-ltcmalloc_minimal Yes Yes Yes Yes Yes Yes
configure option –disable-shared –with-mysqld-libs=-ltcmalloc_minimal None None –disable-jemalloc Failed with ICC None

This might disappoint you. But the rebuilt software stacks show no improvement whatsoever in my benchmark.

IPv6 support still lacking

I was playing HE.net‘s IPv6 certification test last night. And I got stucked on last step, adding an AAAA tld glue record for my domain. Unfortuantely the domain registrar I am using, 1&1 AG,  does not support AAAA record in their system.

Here is the email I got this afternoon,

Thank you for contacting us. Unfortunately, our system does not support AAAA record. Our system only supports modifying the DNS, A-record & Mx-Records as well as CNAME. If you have any further questions please do not hesitate to contact us.

— Sincerely,
xxxxxxx xxxxxxx
Technical Support
1&1 Internet

The test from he.net is not easy. Here is the list of softwares and services I used last night through trial and error.

  1. The first test is browsing an IPv6 website from he.net. I tried many methods. On my local Windows machine, I can ping -6 ipv6 address, I can copy and paste ipv6 into browser. But browser just refused to connect to remote host by ipv6 hostname.  I tried to build ipv6 enabled Lynx on a linux machine, which also failed. Because none of above worked, I had to cheat on this step by using http://ipv6.he.net.ipv4.sixxs.org/.   
  2. Run a webserver on IPv6 address. Very simple, I added an ipv6 address into my webserver, Cherokee, and everything worked as they should.  
  3. Run a mail server on IPv6 address. Postfix, the mail server I am currently using, supports IPv6 natively. Simply add inet_protocols = all into main.cf, postfix will start listerning on IPv6 socket. First email unfortunately failed due to dkim-milter refused connection from ipv6 address. After disabling, 2nd email passed through.  
  4. Have a RDNS record for my MX record. I am using ipv6 patched tinydns. NS and SOA record for my /48 prefix to tinydns are required to get response for PTR record. It took me a while to figure that out. Patched tinydns has a utility add-host6 to handle record for both forward and reverse dns but ns and soa need to be added manually.  
  5. Run DNS server natively on IPv6. Even though tinydns is listerning on IPv6 address and my local dig show correct response. But he.net did not accept it for unknown reason. So I installed Bind9, and everything just worked in less than 5 minutes.  
  6. Add an AAAA glue record to TLD nameserver. 1&1 AG does not support it as I said at beginning of the post. I am transferring my domain to GKG.net, hope they works, and I will update this post once I finish this step.  Here is a list of domain registrars supporting ipv6 glue.

So here is my test result.  

I am using dnsmadeeasy for my DNS, but they do not support IPv6 in any means. My VPS providor, Linode LLC, supports AAAA record in nameserver, but no ipv6 PTR record and no native IPv6 connection, no ipv6 dns servers.

How to use SimpleCDN with WordPress

Update 2/24/2009: This method is deprecated. I wrote a plugin to rewite urls.
Update again: SimpleCDN is slower comparing to CloudFront, I moved all my link to CF. If you have problem copying files to S3, you can read this post.

WordPress is my favorite bloging software, with many builtin features and powerful plugins. WordPress.com, the blogging platform by automattic, has been using CDN for a long time. However there is no public available solution to make self hosted WordPress to use a CDN as well. Fortunately the php code itself is very clean to read, so it took me about 5 minutes to find where I should modify to fully integrate my new CDN address.

Before you proceed, please read my previous posts on how to use CNAME with SimpleCDN and how to process javascript and css files for WordPress. Make sure your SimpleCDN’s CNAME and bucket working properly. I used a mirror bucket which points to my blog address, that is probably the simplest way to mirror your static assets. And use ‘-s1′ pre-url to insure client side cache working, because SimpleCDN does not use 304 not modified http header at all. I used bucket name ‘s.mudy.info’ for my own blog.

Once your DNS full propagated, open your favorite text editor open following 3 files.

wp-includes/class.wp-scripts.php‘: near line 75, change

$src = $this->base_url . $src;

to

$src = 'http://yourcdnbucketname' . $src;

There is no trailing slash after your bucketname. If you want to make sure client browser always use cached version, you can also comments out the following line.

// $src = add_query_arg('ver', $ver, $src);

But this change may break WordPress upgrade.

wp-include/class.wp-styles.php‘: near line 79, do the same change as above.

wp-include/theme.php‘: near line 504, change

return apply_filters('theme_root_uri', content_url('themes'), get_option('siteurl'));

to

return 'http://yourcdnbucketname/wp-content/themes';

That’s it. Now clean your cache and test your blog.

Optimize WordPress javascript placement

YSlow rule 6: Put JS at the bottom. This is a very simple rule, but it has noticeable improvement on actual page loading time. Because javascript in head will block both page elements loading and rendering. The placement of javascript in WordPress 2.7.1 is hard coded to be placed in html head, because most ajax plugin would expect javascript library to be fully loaded before page load. If you use WordPress without any fancy ajax related plugins, it is very possible to load javascript at bottom of every page without breaking any blog function. I also tested it with Lightbox 2 without any noticeable problems. The measurable page loading time difference is near half a second when browser cache is empty. Before you proceed to make any change to WordPress php files, I would recommend you to do a full backup of database and related php files. Now open ‘wp-includes/default-filters.php‘, search wp_print_scripts. Near line 184, you should find,

add_action('wp_head', 'wp_print_scripts');

Change this to

add_action('wp_footer', 'wp_print_scripts');

Clear your WordPress page cache and test your WordPress. Ideally one could write a plugin hooking into filter ‘print_scripts_array’ to remove plugins from html head conditionally. But I have not found one yet. There are a lot other methods to safely speedup javascript loading in html header. Unfortunately they are not very easily implemented and cross different browsers. Steve, author of Yslow had a post and talk explaining the whole situation.

One line script to minify js css directory using yuicompressor

The default installation of WordPress use full version of css and javascript library. In order to minify all of these files, I wrote this bash script to minify all of them. There will be no progress bar while minifying, so be patient.

cd wordpress
find -H . -type f -writable  ( -name *.css -o -name *.js )  
-exec sh -c "yuicompressor {} -o /tmp/yui.tmp && mv /tmp/yui.tmp {}" ;

This will only work under POSIX system.
Findutils should be included in most popular Linux distros.
Yuicompressor can be downloaded from here.

SimpleCDN CNAME Confusion

I am using SimpleCDN to serve some static scripts and images of WordPress on my website.

Not like AWS CloudFront, Amazon’s CDN service, which only allows expiration date carried from S3 file meta. SimpleCDN supports fine tuning Expiration, P3P and gzip encoding through something called Pre-URL. Which look like these

http://s.mudy.info.simplecdn.net/

http://s.mudy.info-s1.simplecdn.net/

http://s.mudy.info-e7.simplecdn.net/

Here ‘-s1′ means special 1 with expire and gzip; ‘-e7′ means 10 years expire without gzip. And a lot more with different combo effects.

SimpleCDN also supports CNAME to any of these urls. Because there is not a lot document on how to use simplecdn. I mistakenly CNAMEed ‘s.mudy.info’ to the first URL

s.mudy.info.simplecdn.net

Run my website through Yslow, I got a lot F score due to lack of expire headers and non-gzip on static files. I realize I have to use these different Pre-URLs in order to get http headers, so I substitute ‘s.mudy.info’ in WordPress with ‘s.mudy.info-s1.simplecdn.net’ or ‘s.mudy.info-e7.simplecdn.net’. Originally very simple urls’ become long and urgly.

Due to some DNS setting error, I accidentally pinged 2 different SimpleCDN pre-urls. The results are 2 different IPs. I reallized I made another mistake today, these pre-urls are supposed to be used as CNAME to get different http headers and gzip encoding, because these hostname all has their own IPs. A quick testing I CNAME ‘s.mudy.info’ to

s.mudy.info-s1.simplecdn.net

which gives nicely gzipped files with correct expiration settings.

This design is what I initially did not expect. Because traditional CDN usually use HTTP hostname to distinguish objects, so I normally would think I have to use different hostnames. However SimpleCDN use anycast , so they use same IP block on on all edge servers, which make it possible to use different IP addresses to distinguish requests.

Using tinydns wildcards with cautions

Tinydns support wildcards. However I just made a mistake make my own website inaccessible.

I was pointing all subdomains to a single host including this website.

C*.mudy.info:nw.mudy.info:1200

It worked perfect fine, until today I encounter a website which requires [email protected] to verify ownership. I know it should just working without any further settings, however to be nice I added an additional mx record to my Tinydns to explicitly indicates the mail host of blog.mudy.info

@blog.mudy.info::nw.mudy.info.:5:1200

After that my website become inaccessible. Apparently tinydns will stop looking into data once a matching name found, even though the record type does not match and there are better matching wildcards.

To fix this, I added another explicit A record.

+blog.mudy.info:66.246.138.44:1200

CNAME record will work as well.

Correct way to password protect Cherokee webserver

I have played Cherokee server on my personal blog a couple of days. I used it for some private file. Some directories need to be password protected. Initially I set it up in this way.

Incorrect password protection for Cherokee webserver

Incorrect password protection for Cherokee webserver

Then I tested against some static files, it obviously worked. However a couple days later, I realize the php script inside this directory was not protected at all including index.php. A rather easy fix, I added the same auth method to php handler as well. However as I add more stuff into my private directory, some of them require individual handler to work correct, so I added same http authentication method to all of them. It is really a pain to maintain such a long auth list, suddenly I realized I must have done this in a wrong way.

After digging into the cherokee document and cookbook, I find this simple solution to protect a whole directory.

  1. Add a directory rule which match the directory you want to protect.
  2. Set the handler to None in Handler tab
  3. Set authentication method in Security tab.
  4. Move this rule to the top and uncheck final.
Correct way to password protect cherokee webserver.

Correct way to password protect cherokee webserver.

That’s it.

How to do http redirect on varnish

There is no built in function to do redirecting in Varnish.  After googling around, here is the result.

Using error page of varnish to return an object with location header, since obj is not available within vcl_recv.

sub vcl_recv {
if (req.http.host ~ "^(www.)?mudy.info$")
{
error 302;
}
}
sub vcl_error {
if (obj.status == 302 && req.http.host ~ "^(www.)?mudy.info$")
{
set obj.http.Location = "http://blog.mudy.info/";
}
}

I am unsure whether there is a simpler way to verify the request, so I test the same condition again inside vcl_error.

If you want request url appended to redirection, you can use this instead.

set obj.http.Location = "http://blog.mudy.info" req.url;