环境:

server: 192.168.1.8, Ubuntu, NFS, MFS, 8G内存, 双核Intel(R) Core(TM) i3 CPU 2.93GHz, MFS挂载目录为/mnt/mfsdir1, 是独立的磁盘块(dd建立)

client: 192.168.1.106, Mac OS X, Nginx, Nginx访问目录中分别挂载192.168.1.8的/mnt/mfsdir1(MFS分区)和NFS共享目录/mnt/nfs(属于/分区)

NFS和MFS的配置均为默认的, 这里要注意的是因为测试平台是Mac OS, 所以在NFS共享文件配置中需要加入参数insecure, 否则Mac OS会报opeartion not permite

小文件read测试

测试的文件为389K的图片文件

NFS测试结果:

#ab -c 100 -n 1000 http://192.168.1.106/filesystem_benckmark/nfs/big.jpg

Server Software:        nginx/0.7.65
Server Hostname:        192.168.1.106
Server Port:            80

Document Path:          /filesystem_benckmark/nfs/big.jpg
Document Length:        398102 bytes

Concurrency Level:      100
Time taken for tests:   1.510 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0

Total transferred:      398317000 bytes
HTML transferred:       398102000 bytes
Requests per second:    662.46 [#/sec] (mean)
Time per request:       150.952 [ms] (mean)
Time per request:       1.510 [ms] (mean, across all concurrent requests)
Transfer rate:          257684.67 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.1      0      10
Processing:    12  147  43.8    148     326
Waiting:        1   62  29.6     59     159
Total:         12  147  44.1    148     329

Percentage of the requests served within a certain time (ms)
  50%    148
  66%    164
  75%    172
  80%    177
  90%    200
  95%    228
  98%    249
  99%    264
 100%    329 (longest request)

MFS测试结果:

#ab -c 100 -n 1000 http://192.168.1.106/filesystem_benckmark/mfs/big.jpg

Server Software:        nginx/0.7.65
Server Hostname:        192.168.1.106
Server Port:            80

Document Path:          /filesystem_benckmark/mfs/big.jpg
Document Length:        398102 bytes

Concurrency Level:      100
Time taken for tests:   0.721 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0

Total transferred:      401711927 bytes
HTML transferred:       401494992 bytes
Requests per second:    1386.30 [#/sec] (mean)
Time per request:       72.135 [ms] (mean)
Time per request:       0.721 [ms] (mean, across all concurrent requests)
Transfer rate:          543840.75 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   1.4      2      13
Processing:    31   68  11.8     68      96
Waiting:        1   11   8.5      9      33
Total:         33   70  11.8     70      97

Percentage of the requests served within a certain time (ms)
  50%     70
  66%     73
  75%     75
  80%     76
  90%     87
  95%     91
  98%     95
  99%     96
 100%     97 (longest request)

从以上测试可以看出MooseFS在读性能上还是完胜NFS的, 为NFS的200%

大文件read测试

测试的文件为5.6M的二进制文件

NFS测试结果:

#ab -c 10 -n 100 http://192.168.1.106/filesystem_benckmark/nfs/production-20100326.log.tar.gz

Concurrency Level:      10
Time taken for tests:   2.156 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      591628100 bytes
HTML transferred:       591605100 bytes
Requests per second:    46.38 [#/sec] (mean)
Time per request:       215.603 [ms] (mean)
Time per request:       21.560 [ms] (mean, across all concurrent requests)
Transfer rate:          267974.85 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.7      0       5
Processing:    17  211 110.5    215     480
Waiting:        1   12  10.4      9      64
Total:         17  211 110.6    215     481

Percentage of the requests served within a certain time (ms)
  50%    215
  66%    261
  75%    287
  80%    305
  90%    340
  95%    387
  98%    474
  99%    481
 100%    481 (longest request)

MFS测试结果:

#ab -c 10 -n 100 http://192.168.1.106/filesystem_benckmark/mfs/production-20100326.log.tar.gz

Concurrency Level:      10
Time taken for tests:   0.671 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      591628100 bytes
HTML transferred:       591605100 bytes
Requests per second:    149.06 [#/sec] (mean)
Time per request:       67.089 [ms] (mean)
Time per request:       6.709 [ms] (mean, across all concurrent requests)
Transfer rate:          861190.96 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    50   65   9.8     63      93
Waiting:        0    2   1.0      1       4
Total:         50   65   9.8     63      93
WARNING: The median and mean for the waiting time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%     63
  66%     68
  75%     71
  80%     73
  90%     76
  95%     88
  98%     91
  99%     93
 100%     93 (longest request)

在大文件的读取上MFS更是表现明显, 性能为NFS的300%

write测试

写测试这里使用了dd方法, 写入数据大小为100M

NFS write

#time dd if=/dev/zero of=sometestfile bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes transferred in 8.829279 secs (11597776 bytes/sec)
        8.91 real         0.10 user         3.39 sys

MFS write

#time dd if=/dev/zero of=sometestfile bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes transferred in 6.531960 secs (15676765 bytes/sec)
        8.81 real         0.12 user         2.56 sys

可以看出MFS的写性能达到了百兆网卡的全速(14Mb/s), NFS为11Mb/s, 在时间上也可以看出MFS的优势, 再加上MooseFS的分布式存储和容错功能, 的确非常不错

对于单目录下存在大量文件时单个文件查找和操作仍需要测试，待续

find平时用于查找文件是不可缺少的，find的用法也比较复杂，平时我们用的最多的就是:

find ./ -name "*filename*"

了， 其他一些复杂的用法平时都不怎么用到，这里列下shitou经常用的参数吧:

#找出当前目录下所有以production开头.log结尾的文件

find ./ -name "production*.log" 

#如果当前目录下还有其他子文件夹，但你不想递归查找子文件夹下的目录时使用

find ./ -maxdepth 1 -name "production*.log" 

#如果碰巧当前目录下有个匹配到"production*.log"的文件夹，而你只想要regular(一般)文件时

find ./ -maxdepth 1 -name "production*.log" -type f

#找出当前目录下包含有shitou字符的文件

find ./ -maxdepth 1 -type f | xargs fgrep 'shitou'



#好的，我们继续

#如果你想找最近7天创建的文件时

find ./ -maxdepth 1 -name "production*.log" -type -ctime -7

#如果你想找7天前创建的文件时

find ./ -maxdepth 1 -name "production*.log" -type -ctime +7



#下面来个正真非业余研究的

#如果你想找一个时间段内创建的文件时怎么办呢，find只支持-cnewer参数，即找到比某个文件创建时间更新的文件，

#不过利用这个就够了

#首先我们创建两个文件，并且指定创建时间就为我们要寻找时间段的起至， 用于我们作为基准比较使用

touch tmp -t 201005130000

touch tmp1 -t 201005200000

#上面给定时间戳时要按照“年年年年月月日日时时分分”方式指定，好了现在就可以了

find ./ -maxdepth 1 -name "production-*log" -type f -cnewer tmp -and ! -cnewer tmp1

上面的-ctime为create time的意思，还有-atime(access time)，-mtime(modify time)

-type参数还可以有d(目录)，f(一版文件)，等等

另外还有-group查找指定用户组的，-perm查找指定权限的(0644这样的)，-size指定大小，-regex使用正则表达式

find后还可以跟管道(|)，用xargs命令把查找到的文件用作后续命令的参数传递进来，看上面的例子

如来神掌打完收工

参考资料

Finding Files

awk不支持多维数组， 在官方的文档中所说的多维数组并不是传统语言所说的多维数组，awk的多维数组是如:

awk '{
     if (max_nf < NF)
          max_nf = NF
     max_nr = NR
     for (x = 1; x <= NF; x++)
          vector[x, NR] = $x
}

END {
     for (x = 1; x <= max_nf; x++) {
          for (y = max_nr; y >= 1; --y)
               printf("%s ", vector[x, y])
          printf("\n")
     }
}'

在某些情况下需要使用传统多维数组的方式对数据进行统计，生成例如

arr[key] = [field1, filed2, filed3]

这样的数据结构，虽然awk默认不支持这样的数据结构，但是我们可以自己以另一种方式实现，比如上面的数据结构可以变成这样:

arr[key] = "field1@field2@field3"; split(arr[key], arr2, "@")

就是以字符串分割的方式，然后再用split还原, 下面是一个例子，统计下面文件中第一个field重复行的后面字段进行相加，并添加几个filed实现比例统计, 文件是这样的, file.txt:

1270022377 17 0 0 0 0.00 0.00 0.00
1270022399 9 0 0 0 0.00 0.00 0.00
1270022377 17 0 0 0 0.00 0.00 0.00
1270538893 16 8 2 0 50.00 12.50 0.00
1270022399 9 0 0 0 0.00 0.00 0.00
1270538893 16 8 2 0 50.00 12.50 0.00

shell:

awk 'BEGIN{print("Ad View Hit Reg Pay Hit(%) Reg(%) Pay(%)");} \
    1{if($1 in a) { \
        split(a[$1], b, "@"); \
        a[$1]=sprintf("%s@%s@%s@%s@%s", $1, int($2)+int(b[2]), int($3)+int(b[3]), int($4)+int(b[4]), int($5)+int(b[5])); } \
      else { \
        a[$1]=sprintf("%s@%s@%s@%s@%s", $1, int($2), int($3), int($4), int($5))} \
     }; \
     END{for(line in a) { \
            split(a[line], c, "@"); \
            printf("%s %s %s %s %s %.2f %.2f %.2f\n", c[1], c[2], c[3], c[4], c[5], \
                int(c[3])/int(c[2])*100, \
                int(c[4])/int(c[2])*100, \
                int(c[5])/int(c[2])*100); \
        }}' file.txt

结果:

Ad View Hit Reg Pay Hit(%) Reg(%) Pay(%)
1270538893 32 16 4 0 50.00 12.50 0.00
1270022377 34 0 0 0 0.00 0.00 0.00
1270022399 18 0 0 0 0.00 0.00 0.00

监控服务器存活的脚本，用ruby写的，需要安装有action_mailer gem,

在前面的日志中有说明怎样连接gmail来发送邮件的

require 'rubygems'
require 'action_mailer'
require 'ping'
require 'fileutils'
require 'tlsmail' #need install tlsmail gem to support TLS connect
Net::SMTP.enable_tls(OpenSSL::SSL::VERIFY_NONE)

#发送通知from账户设置
ActionMailer::Base.smtp_settings = {
  :address => 'smtp.gmail.com',
  :port => '25',
  :domain => 'gmail.com',
  :user_name => 'username',
  :password => 'passwd',
  :authentication => :login
}

#recp: 收到通知的邮箱地址，可以填多个地址，逗号分隔
#sub: 邮件主题
#bo: 邮件内容
class SimpleMailer < ActionMailer::Base
  def simple_message(recp, sub, bo)
    from 'sender@gmail.com'
    recipients recp
    subject sub
    body bo
  end
end

class ServerMonitor
  def initialize
    #config
    @hit = 5  #最大失败次数，达到后才会发送邮件，避免误报
    @tmp = "/tmp/server_monitor_tmp"  #计数器地址
    @recp = 'mytake6@gmail.com, std8545@yahoo.com.cn'  #收信的邮箱地址
    @host = ['www.ccok.me', 'www.google.com']  #需要监控的主机
    
    #do not touch these
    @hosts = Struct.new(:host, :status)
    @servers, @msg = [], []
    @host.each { |h| @servers << @hosts.new(h, true) }
    
    FileUtils.mkdir(@tmp) unless File.exists? @tmp
  end
  
  def run
    @servers.each do |s| 
      num = Ping.pingecho(s.host, 5, 80) ? 0 : rf(s.host) + 1
      File.open(dest(s.host), 'w') { |f| f.puts num }
    end

    @servers.each do |e|
      if rf(e.host) == @hit
        e.status = false
        File.open(dest(e.host), 'w') { |f| f.puts 0 }
      end
    end
    
    send_mail
  end
  
  def rf file
    return 0 unless File.exist?(dest(file))
    File.open(dest(file), 'r').readlines.to_s.chomp.to_i
  end
  
  def dest host
    "#{@tmp}/#{host}"
  end
  
  def send_mail
    @servers.each { |m| @msg << m.host unless m.status }
    SimpleMailer.deliver_simple_message @recp, 
        "server monitor", 
        @msg.join(', ') + " is Down!!! " unless @msg.empty?
  end
end

#go
ServerMonitor.new.run

可以放到crontab中每2分钟运行一次

在Debian中使用apt-get安装软件包时经常会提示让你插入netinst的光盘：

Media change: please insert the disc labeled

当没有时就无法进行安装了， 这时可以打开文件/etc/apt/sources.list文件，注释掉cdrom那一行，然后再执行apt-get update更新下deb仓库， 这样以后再使用apt-get安装时就不会再搜寻cdrom了

有时我们会进入一个很深层次的目录，同时需要进入其他目录，这时来回切换就显得非常繁琐，这时你可以使用：

#cd -

或者history, 然后! 数字 来执行历史命令， 当然也可以用向上箭头翻出历史命令， 不过当你不是一条命令进入上一目录时就用cd - 是最方便的了