Arch-03-19- NoSQL实践

虽然对新技术甚为谨慎,但目前大家所做的伪 sns 的 nosql 似乎也让一帮同事兴奋不已,没办法,试试吧。

不讨论nosql 的长短,直接实施吧,关键在于 nosql 表的设计和与 postgrsql 间数据的关系组织。

一、用 YCSB 测试候选的数据库

(1)候选:hbase,voldemort,mongodb

(2)ycsb 环境:centos5.4,12G,4x4核 服务器

(3)下载安装  ycsb 略过

(4)Hbase-0.9

    . 插入数据 100w

java -cp build/ycsb.jar:db/hbase/lib/* com.yahoo.ycsb.Client -load -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=family -p recordcount=10000000 -s > load.dat
10 sec: 73825 operations; 7380.29 current ops/sec; [INSERT AverageLatency(ms)=0.11]
 20 sec: 127145 operations; 5329.87 current ops/sec; [INSERT AverageLatency(ms)=0.16]
 30 sec: 217374 operations; 9020.19 current ops/sec; [INSERT AverageLatency(ms)=0.09]
...
 平均 insert 8k/sec
java -cp build/ycsb.jar:db/hbase/lib/* com.yahoo.ycsb.Client -t -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=family -p operationcount=1000000 -s -threads 10 -target 100 > transactions.dat
 

(5)Voldemort-0.9

    初始化插入 500w,0.7k/s

YCSB Client 0.1
Command line: -load -db com.yahoo.ycsb.db.VoldemortClient -p bootstrap_urls=tcp://localhost:6666 -P workloads/workloada -s -threads 10
[OVERALL], RunTime(ms), 655195.0
[OVERALL], Throughput(ops/sec), 7631.315867795084
[INSERT], Operations, 5000000
[INSERT], AverageLatency(ms), 1.2568772
[INSERT], MinLatency(ms), 0
[INSERT], MaxLatency(ms), 2289
[INSERT], 95thPercentileLatency(ms), 1
[INSERT], 99thPercentileLatency(ms), 3
[INSERT], Return=0, 5000000

    写/读各50w 次,写 0.83k/s,R1.2k/s

YCSB Client 0.1
Command line: -t -db com.yahoo.ycsb.db.VoldemortClient -p bootstrap_urls=tcp://localhost:6666 -P workloads/workloada -s
[OVERALL], RunTime(ms), 1026399.0
[OVERALL], Throughput(ops/sec), 974.2799827357587
[UPDATE], Operations, 501020
[UPDATE], AverageLatency(ms), 1.2046704722366373
[UPDATE], MinLatency(ms), 0
[UPDATE], MaxLatency(ms), 1887
[UPDATE], 95thPercentileLatency(ms), 5
[UPDATE], 99thPercentileLatency(ms), 8
[UPDATE], Return=0, 501020

....


[READ], Operations, 498980
[READ], AverageLatency(ms), 0.8289450478977113
[READ], MinLatency(ms), 0
[READ], MaxLatency(ms), 359
[READ], 95thPercentileLatency(ms), 5
[READ], 99thPercentileLatency(ms), 7
[READ], Return=0, 498980
 

(6)mongodb

    初始化插入 500w,0.1k/s

YCSB Client 0.1
Command line: -load -db com.yahoo.ycsb.db.MongoDbClient -p mongodb.database=mytestdbt100 -p mongodb.url=mongodb://localhost:27017 -P workloads/workloada -s -threads 100
new database url = localhost:27017/mytestdbt100
new database url = localhost:27017/mytestdbt100
...100行

mongo connection created with localhost:27017/mytestdbt100
mongo connection created with localhost:27017/mytestdbt100
...100行

[OVERALL], RunTime(ms), 4664799.0
[OVERALL], Throughput(ops/sec), 1071.857544129983
[INSERT], Operations, 5000000
[INSERT], AverageLatency(ms), 93.0971558
[INSERT], MinLatency(ms), 0
[INSERT], MaxLatency(ms), 34143
[INSERT], 95thPercentileLatency(ms), 220
[INSERT], Return=0, 5000000
 

    写/读各50w 次,写 0.13k/s,R3.7k/s

YCSB Client 0.1
Command line: -t -db com.yahoo.ycsb.db.MongoDbClient -p mongodb.database=mytestdb100 -p mongodb.url=mongodb://localhost:27017 -P workloads/workloada -s -threads 100
new database url = localhost:27017/mytestdb100
new database url = localhost:27017/mytestdb100
...100

mongo connection created with localhost:27017/mytestdb100
mongo connection created with localhost:27017/mytestdb100
...100

[OVERALL], RunTime(ms), 40395.0
[OVERALL], Throughput(ops/sec), 24755.539051862856
[UPDATE], Operations, 500052
[UPDATE], AverageLatency(ms), 7.622983209746186
[UPDATE], MinLatency(ms), 0
[UPDATE], MaxLatency(ms), 1344
[UPDATE], 95thPercentileLatency(ms), 18
[UPDATE], 99thPercentileLatency(ms), 27
[UPDATE], Return=1, 500052
...

READ], Operations, 499948
[READ], AverageLatency(ms), 0.27114819941273893
[READ], MinLatency(ms), 0
[READ], MaxLatency(ms), 1308
[READ], 95thPercentileLatency(ms), 1
[READ], 99thPercentileLatency(ms), 1
[READ], Return=1, 499948
...
 

(7)MongoDB Java 框架选择

    . 参考 http://blog.csdn.net/fengzanfeng/article/details/6634433

安装

Mongodb: http://be-evil.org/install-mongodb-on-centos.html

然而关于 nosql 的讨论是没有休止的:

为什么说很多NoSQL的Benchmark是扯淡? 

解读NoSQL数据库的四大家族 

微博架构与平台安全演讲稿

新浪微博在技术上是如何架构的-微博架构与平台安全

http://www.360doc.com/content/11/0710/04/1542811_132636642.shtml

相关推荐