因为石头的服务器上面放了多个网站,有5,6个域名,加上子域名的话有将近10个,除了有一个域名备案过其他都没有备案,终于被和谐了,服务器IP被封了80号端口,所有网站惨遭毒手,目前除了shitou's blog和十人族 - Tenerer.com已经被石头放到国外服务器,其他网站(MIX BOY, 塌客, mochichat)等石头的子项目都暂时无法访问...
下面是大家最熟悉和亲切的段子

太阳给草打电话
太阳:喂,草你吗,我日。
草:我草,你谁啊?
太阳:我日啊
草:我草,你到底谁啊
太阳:我日啊,你草吧
草:TMD,你到底是谁啊,我草
太阳:我日,我日啊
草:我草.
这时,
太阳的妈妈接过电话:我日他妈呀,你是草吧,草你妈呢
蛋定,蛋定.....
身在天朝,迟早要还的...
mongodb对存储数据的格式要求比较严格,比如在对Integer字段做匹配查询时如果把参数写成"1", 那么是匹配不到的,特别是用惯MySQL等关系数据库的同学更要注意了,
在mongodb的console中对时间范围进行匹配:
>var start = new Date(2010, 10, 1, 0, 0, 0);
>var end = new Date(2010, 10, 1, 10, 0, 0);
>db.table.distince("username", {created: {$gte: start, $lte: end}});
上面是统计在置顶时间段内唯一的用户名数
参考文档:
server: 192.168.1.8, Ubuntu, NFS, MFS, 8G内存, 双核Intel(R) Core(TM) i3 CPU 2.93GHz, MFS挂载目录为/mnt/mfsdir1, 是独立的磁盘块(dd建立)
client: 192.168.1.106, Mac OS X, Nginx, Nginx访问目录中分别挂载192.168.1.8的/mnt/mfsdir1(MFS分区)和NFS共享目录/mnt/nfs(属于/分区)
NFS和MFS的配置均为默认的, 这里要注意的是因为测试平台是Mac OS, 所以在NFS共享文件配置中需要加入参数insecure, 否则Mac OS会报opeartion not permite
测试的文件为389K的图片文件
#ab -c 100 -n 1000 http://192.168.1.106/filesystem_benckmark/nfs/big.jpg
Server Software: nginx/0.7.65
Server Hostname: 192.168.1.106
Server Port: 80
Document Path: /filesystem_benckmark/nfs/big.jpg
Document Length: 398102 bytes
Concurrency Level: 100
Time taken for tests: 1.510 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 398317000 bytes
HTML transferred: 398102000 bytes
Requests per second: 662.46 [#/sec] (mean)
Time per request: 150.952 [ms] (mean)
Time per request: 1.510 [ms] (mean, across all concurrent requests)
Transfer rate: 257684.67 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.1 0 10
Processing: 12 147 43.8 148 326
Waiting: 1 62 29.6 59 159
Total: 12 147 44.1 148 329
Percentage of the requests served within a certain time (ms)
50% 148
66% 164
75% 172
80% 177
90% 200
95% 228
98% 249
99% 264
100% 329 (longest request)
#ab -c 100 -n 1000 http://192.168.1.106/filesystem_benckmark/mfs/big.jpg
Server Software: nginx/0.7.65
Server Hostname: 192.168.1.106
Server Port: 80
Document Path: /filesystem_benckmark/mfs/big.jpg
Document Length: 398102 bytes
Concurrency Level: 100
Time taken for tests: 0.721 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 401711927 bytes
HTML transferred: 401494992 bytes
Requests per second: 1386.30 [#/sec] (mean)
Time per request: 72.135 [ms] (mean)
Time per request: 0.721 [ms] (mean, across all concurrent requests)
Transfer rate: 543840.75 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 1.4 2 13
Processing: 31 68 11.8 68 96
Waiting: 1 11 8.5 9 33
Total: 33 70 11.8 70 97
Percentage of the requests served within a certain time (ms)
50% 70
66% 73
75% 75
80% 76
90% 87
95% 91
98% 95
99% 96
100% 97 (longest request)
从以上测试可以看出MooseFS在读性能上还是完胜NFS的, 为NFS的200%
测试的文件为5.6M的二进制文件
#ab -c 10 -n 100 http://192.168.1.106/filesystem_benckmark/nfs/production-20100326.log.tar.gz
Concurrency Level: 10
Time taken for tests: 2.156 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 591628100 bytes
HTML transferred: 591605100 bytes
Requests per second: 46.38 [#/sec] (mean)
Time per request: 215.603 [ms] (mean)
Time per request: 21.560 [ms] (mean, across all concurrent requests)
Transfer rate: 267974.85 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.7 0 5
Processing: 17 211 110.5 215 480
Waiting: 1 12 10.4 9 64
Total: 17 211 110.6 215 481
Percentage of the requests served within a certain time (ms)
50% 215
66% 261
75% 287
80% 305
90% 340
95% 387
98% 474
99% 481
100% 481 (longest request)
#ab -c 10 -n 100 http://192.168.1.106/filesystem_benckmark/mfs/production-20100326.log.tar.gz
Concurrency Level: 10
Time taken for tests: 0.671 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 591628100 bytes
HTML transferred: 591605100 bytes
Requests per second: 149.06 [#/sec] (mean)
Time per request: 67.089 [ms] (mean)
Time per request: 6.709 [ms] (mean, across all concurrent requests)
Transfer rate: 861190.96 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 50 65 9.8 63 93
Waiting: 0 2 1.0 1 4
Total: 50 65 9.8 63 93
WARNING: The median and mean for the waiting time are not within a normal deviation
These results are probably not that reliable.
Percentage of the requests served within a certain time (ms)
50% 63
66% 68
75% 71
80% 73
90% 76
95% 88
98% 91
99% 93
100% 93 (longest request)
在大文件的读取上MFS更是表现明显, 性能为NFS的300%
写测试这里使用了dd方法, 写入数据大小为100M
#time dd if=/dev/zero of=sometestfile bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes transferred in 8.829279 secs (11597776 bytes/sec)
8.91 real 0.10 user 3.39 sys
#time dd if=/dev/zero of=sometestfile bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes transferred in 6.531960 secs (15676765 bytes/sec)
8.81 real 0.12 user 2.56 sys
可以看出MFS的写性能达到了百兆网卡的全速(14Mb/s), NFS为11Mb/s, 在时间上也可以看出MFS的优势, 再加上MooseFS的分布式存储和容错功能, 的确非常不错
对于单目录下存在大量文件时单个文件查找和操作仍需要测试,待续
Mochichat的新版本上线了,Erlang代码核心全部重写了,这次前端使用了Rails, 数据库采用的MongoDB
架构:

目前可能还存在bug, 而且页面也比较简单,还需要改善
测试地址: 摸我
当有一些消耗cpu和时间的动作需要执行时,我们就可以使用Gearman来转移负载,尤其对于Web中图片裁剪缩放等操作非常有用,Gearman是一个分布式的运算工具, 支持并行,负载均衡,夸语言,容错等特点,具体介绍看这里
Gearman的设计

Gearman架构

从上面图中可以看出来Gearman的流程为
Client -> Job Server -> Worker
Client为客户端调用,
Job Server为调度器,即gearmand进程
Worker为运算进程
Client和Worker是独立的,客户端锁采用的语言和Worker没有任何关系,比如客户端可以使用ruby,Worker可以采用C来实现
上面这种架构下,如果任何一个worker down掉都不会影响整个系统的使用,
如果把client也与每个Job Server相连,那个任何一个Job Server down掉也不会影响整个系统的运行
Gearman提供有丰富的API支持,包括目前几乎所有的主流语言,当然也包括Ruby(gearman-ruby gem)
随gearmand发布的源码包中也附带了基于shell的client和worker命令行调用工具
Gearman需要依赖libevent和libevent的头文件, 因此在需要事先安装libevent的源码包,如果是redhat,debian或者ubuntu就需要安装libevent, libevent-devel(debian, ubuntu为libevent-dev)包,当然源码包还是最方便的,这里的采用的是Mac OS X 10.6,安装采用的是源码包
然后就可以直接编译安装gearmand了,这里采用的是gearmand-0.13.tar.gz
首先启动gearmand服务(job server)
#sbin/gearmand -u root -vv
或者后台运行-d,-vv打印日志启动
gearmand默认端口号为4730
shell下的调用
然后启动worker
#bin/gearman -w -f wc -- wc -l &
-w为指定worker, -f为方法名称, wc -l为方法内容
上面会启动一个worker进程
worker默认会连接localhost:4730端口,也可以自己指定多组Job Server,参数查看--help
client调用:
#bin/gearman -f wc < /etc/passwd
26
-f wc为制定调用的worker方法,/etc/passwd为参数
client默认也是连接localhost:4730,也可以自己指定多组Job Server,参数查看--help
通过以上shell的测试,Gearman的使用已经基本清晰了,下面结合Ruby我们来实战一下
首先安装Gearman的ruby API
#gem install gearman-ruby
worker.rb
require 'rubygems'
require 'gearman'
Gearman::Util.debug = true
servers = ['localhost:4730']
w = Gearman::Worker.new(servers)
w.add_ability('sum') do |args, job|
num = args
num.to_i * 100
end
w.work
client.rb
require 'rubygems'
require 'gearman'
Gearman::Util.debug = true
servers = ['localhost:4730']
client = Gearman::Client.new(servers)
taskset = Gearman::Taskset.new
task = Gearman::Task.new('sum', 2000)
task.on_complete {|d| puts "TASK 1: #{d}" }
taskset << task
client.run(taskset)
测试
#ruby worker.rb
另一终端
#ruby client.rb
就可以看到结果了,也可以结合worker和job server的日志输出来观察
下面再来看一看怎样把gearman-ruby和Rails结合
在environment.rb中
require 'gearman'
#需要首先启动gearmand, port: 4730
#然后启动worker
Gearman::Util.debug = true
GearmanClient = Gearman::Client.new(['localhost:4730', '192.168.1.109:4730'])
module GearmanJob
def self.do(call_name, args, task_hash_args = {}, &block)
#taskset = Gearman::Taskset.new
task_hash_args = { :background => false }.merge(task_hash_args)
#gearman-ruby doesn't support asyc call now.
task = Gearman::Task.new(call_name, args, task_hash_args)
task.on_complete { |d| yield d } #if block_given? and !task_hash_args[:background]
#task.on_status {|d| puts "Status: #{d}"} if task_hash_args[:background]
#taskset << task
GearmanClient.run(task)
end
end
在控制器中
def call
GearmanJob.do(params[:call_name], params[:id]) do |data|
render :inline => "output: #{data}
"
return
end
end
目前gearman-ruby还不支持异步调用,虽然可以使用:background => true参数,但是action还是会处于等待状态,只是不能得到worker的返回值,shitou和gearman-ruby的作者已经发邮件请教了这个问题,作者说会仔细研究下的,只有等待gearman-ruby作者的最新进展了
不过话说回来,如果真的需要后台运行任务的话可以直接使用starling, workling等工具,然后再结合gearman-ruby
参考资料
http://gearman.org/index.php?id=download
还有一个类似的工具Resque, 看这里
ttp://github.com/blog/542-introducing-resque

just DO NOT support IE