-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathlearn_ycsb.html
1 lines (1 loc) · 18.9 KB
/
learn_ycsb.html
1
<!doctype html><html lang="zh-CN" class="night"><head><meta charset="utf-8"><meta content="width=device-width,initial-scale=1,maximum-scale=4,user-scalable=0" name="viewport"><title>Ede's Blog</title><meta name="description" content="Try to be a qualified programmer"><meta property="og:type" content="website"><meta property="og:description" content="Try to be a qualified programmer"><meta property="og:title" content="Ede's Blog"><meta property="og:site_name" content="Ede's Blog"><meta property="og:url" content="https://ede.ink"><meta property="og:image" content="https://edeity.oss-cn-shenzhen.aliyuncs.com/public/edeity_o.png"><link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"><link rel="mainfest" href="/mainfest.json"><link rel="stylesheet" href="/public/css/common.css"><link rel="stylesheet" href="//at.alicdn.com/t/font_707055_4b9og9sc5lx.css"><script>!function(){var e=-1!==window.location.search.indexOf("theme=night")||"night"===window.localStorage.getItem("edeity-theme_theme"),t=-1!==window.location.search.indexOf("theme=light")||"light"===window.localStorage.getItem("edeity-theme_theme");(new Date).getHours();var n=document.querySelector("html");e?n.classList.add("night"):t?n.classList.remove("night"):n.classList.add("night")}(),document.addEventListener("DOMContentLoaded",function(){null!==document.querySelector("ol.toc")&&(document.querySelector("#nav-bar").style.cssText="display: block")})</script><script async src="https://www.googletagmanager.com/gtag/js?id=G-M3J9QSEE2Z"></script><script>function gtag(){dataLayer.push(arguments)}window.dataLayer=window.dataLayer||[],gtag("js",new Date),gtag("config","G-M3J9QSEE2Z")</script><meta name="generator" content="Hexo 5.0.0"></head><body><div class="loading"></div><div id="switch" data-switch="{"toc":true,"use_pwa":false}"></div><header class="fullscreen"><div class="toolbar"><i class="iconfont icon-menu"></i></div><h1><a href="/">Ede's Blog</a></h1><div class="head-link"><a class="btn waves" href="/"><span><i class="iconfont icon-home">Home </i></span></a><a class="btn waves" href="/about/index.html"><span><i class="iconfont icon-me">About </i></span></a><a class="btn waves" target="_blank" rel="noopener" href="https://github.com/edeink"><span><i class="iconfont icon-github">Github</i></span></a></div></header><div class="some-link"><a class="btn" id="light-or-not"><i class="iconfont icon-light"></i> </a><a style="display:none" class="btn" id="up-to-top"><i class="iconfont icon-up"></i></a></div><div id="nav-bar" style="display:none"><div class="toc"><ol class="toc"><li class="toc-item toc-level-2"><a class="toc-link" href="#%E5%89%8D%E8%A8%80"><span class="toc-number">1.</span> <span class="toc-text">前言</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#%E5%B7%A5%E5%85%B7%E7%AE%80%E4%BB%8B"><span class="toc-number">2.</span> <span class="toc-text">工具简介</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#%E4%BD%BF%E7%94%A8%E8%BF%87%E7%A8%8B"><span class="toc-number">3.</span> <span class="toc-text">使用过程</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#%E5%AE%89%E8%A3%85%E5%8F%8A%E4%BD%BF%E7%94%A8"><span class="toc-number">3.1.</span> <span class="toc-text">安装及使用</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#%E8%A7%A3%E8%AF%BB%E5%8F%82%E6%95%B0"><span class="toc-number">3.2.</span> <span class="toc-text">解读参数</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#%E5%8F%82%E6%95%B0%E5%90%AB%E4%B9%89"><span class="toc-number">3.3.</span> <span class="toc-text">参数含义</span></a><ol class="toc-child"><li class="toc-item toc-level-4"><a class="toc-link" href="#%E5%85%B6%E4%BB%96%E7%BB%86%E8%8A%82"><span class="toc-number">3.3.1.</span> <span class="toc-text">其他细节</span></a></li></ol></li></ol></li><li class="toc-item toc-level-2"><a class="toc-link" href="#%E4%BB%A3%E7%A0%81%E6%A6%82%E8%BF%B0"><span class="toc-number">4.</span> <span class="toc-text">代码概述</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#%E5%B0%BE%E5%A3%B0"><span class="toc-number">5.</span> <span class="toc-text">尾声</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#%E9%99%84%E5%BD%95"><span class="toc-number">6.</span> <span class="toc-text">附录</span></a></li></ol></div></div><main id="content-main" class="section"><div class="list-item"><h1 class="post-title"><a id="Yahoo! Cloud Serving Benchmark" class="article-link" href="">Yahoo! Cloud Serving Benchmark</a></h1><div class="post-meta"><time class="meta published">Jun 3, 2015</time></div><div class="article"><div class="post-excerpt markdown-body"><h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>该工具是大二唯一看过完整源码的程序</p><ul><li><a target="_blank" rel="noopener" href="https://github.com/brianfrankcooper/YCSB">github代码</a></li><li><a target="_blank" rel="noopener" href="https://b9f6702a-a-62cb3a1a-s-sites.googlegroups.com/site/brianfrankcooper/home/publications/ycsb.pdf?attachauth=ANoY7crUx8yzbi4_2j79JbiZSn1tNQYHofw3OX-vksjCvIsxYAICyl1t6EgPaoCiS9xf5_cxle0BWnxVTm53pooXHKA7yhL4wMpwT4uqgRyeJzcugUsedHvhNtJh6HKknTn2UI79Y9ZeiEfIv69hKr6UllFtQ963sfAINGIBY5qNbCE-PkkMewILE91mHQZFHl4Np-LYYruxkc0hMKiHbGCt4rKReCYKS0HYudE_EwTh1eM7Z3fP7tU=&attredirects=0">Papers</a></li></ul><p>仅在此总结为一篇文章；</p><h2 id="工具简介"><a href="#工具简介" class="headerlink" title="工具简介"></a>工具简介</h2><blockquote><p>Yahoo! Cloud Serving Benchmark (YCSB) 。是 Yahoo 公司的一个用来对云服务进行基础测试的工具。</p></blockquote><p>使用方法：</p><p><code>./bin/ycsb load redis -P workloada/workload -p redis.host=localhost</code></p><h2 id="使用过程"><a href="#使用过程" class="headerlink" title="使用过程"></a>使用过程</h2><ul><li>操作系统：ubuntu</li><li>以测试redis数据库的性能为例：</li></ul><h3 id="安装及使用"><a href="#安装及使用" class="headerlink" title="安装及使用"></a>安装及使用</h3><ul><li>安装redis</li></ul><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">wget http://download.redis.io/releases/redis-3.0.0.tar.gz</span><br><span class="line">tar xzf redis-3.0.0.tar.gz</span><br><span class="line"><span class="built_in">cd</span> redis-3.0.0</span><br><span class="line">make</span><br><span class="line">./src/redis-server <span class="comment"># 此处若缺少tcl，可能报错，参考附录安装tcl</span></span><br></pre></td></tr></table></figure><ul><li>安装java</li></ul><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">java -version <span class="comment"># 假如已安装，则不需再次安装</span></span><br><span class="line">sudo apt-get install default-jre</span><br></pre></td></tr></table></figure><ul><li>安装并运行YCSB</li></ul><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">wget https://github.com/downloads/brianfrankcooper/YCSB/ycsb-0.1.4.tar.gz</span><br><span class="line">tar xfvz ycsb-0.1.4</span><br><span class="line"><span class="built_in">cd</span> ycsb-0.1.4</span><br><span class="line">./bin/ycsb load redis -p redis.host=localhost -P workload/workloada > test_redis.txt</span><br></pre></td></tr></table></figure><h3 id="解读参数"><a href="#解读参数" class="headerlink" title="解读参数"></a>解读参数</h3><figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line">YCSB Client <span class="number">0.1</span></span><br><span class="line">Command line: -db com.yahoo.ycsb.db.RedisClient -P workloads/workloada -p redis.host=localhost -load</span><br><span class="line">【OVERALL】, RunTime(ms), <span class="number">782.0</span> <span class="comment">//运行总时间为782毫秒</span></span><br><span class="line">【OVERALL】, Throughput(ops/sec), <span class="number">1278.772378516624</span> <span class="comment">//throughput 吞吐量（ops：操作数/sec ：秒） 即每秒进行约1278次操作</span></span><br><span class="line">【INSERT】, Operations, <span class="number">1000</span> <span class="comment">//一共执行了1000条操作</span></span><br><span class="line">【INSERT, AverageLatency(us), <span class="number">589.714</span> <span class="comment">//平均每条【INSERT】插入操作耗时约 589us</span></span><br><span class="line">【INSERT】, MinLatency(us), <span class="number">146</span> <span class="comment">//【INSERT】单次操作最少耗时</span></span><br><span class="line">【INSERT】, MaxLatency(us), <span class="number">19760</span> <span class="comment">//【INSERT】单次操作最大耗时</span></span><br><span class="line">【INSERT】, <span class="number">95</span>thPercentileLatency(ms), <span class="number">2</span> <span class="comment">// 执行到95%时，该次操作延迟</span></span><br><span class="line">【INSERT】, <span class="number">99</span>thPercentileLatency(ms), <span class="number">5</span> <span class="comment">// 执行到99%时，该次操作延迟</span></span><br><span class="line">【INSERT】, Return=<span class="number">0</span>, <span class="number">1000</span> <span class="comment">// 0~1000条操作均成功执行</span></span><br><span class="line">【INSERT】, <span class="number">0</span>, <span class="number">901</span> <span class="comment">// 下面1000条【INSERT】记录代表了平均延迟分布（单位ms），如该记录就表示有901条【INSERT】操作在0~1ms之间</span></span><br><span class="line">【INSERT】, <span class="number">1</span>, <span class="number">36</span> <span class="comment">// 36条操作在1~2ms之间…</span></span><br><span class="line">【INSERT】, <span class="number">2</span>, <span class="number">21</span></span><br><span class="line">【INSERT】, <span class="number">3</span>, <span class="number">19</span></span><br><span class="line">【INSERT】, <span class="number">4</span>, <span class="number">8</span></span><br><span class="line">【INSERT】, <span class="number">5</span>, <span class="number">7</span></span><br><span class="line">【INSERT】, <span class="number">6</span>, <span class="number">1</span></span><br><span class="line">【INSERT】, <span class="number">7</span>, <span class="number">3</span></span><br><span class="line">【INSERT】, <span class="number">8</span>, <span class="number">2</span></span><br><span class="line">【INSERT】, <span class="number">9</span>, <span class="number">0</span></span><br><span class="line"> …………</span><br><span class="line">【INSERT】, <span class="number">996</span>, <span class="number">0</span></span><br><span class="line">【INSERT】, <span class="number">997</span>, <span class="number">0</span></span><br><span class="line">【INSERT】, <span class="number">998</span>, <span class="number">0</span></span><br><span class="line">【INSERT】, <span class="number">999</span>, <span class="number">0</span></span><br><span class="line">【INSERT】, ><span class="number">1000</span>, <span class="number">0</span></span><br></pre></td></tr></table></figure><h3 id="参数含义"><a href="#参数含义" class="headerlink" title="参数含义"></a>参数含义</h3><p>参数 <code>-P workload/workloada</code>用于加载配置文件，如</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">./bin/ycsb load redis -p redis.host=localhost -P workload/workloada > test_redis.txt</span><br></pre></td></tr></table></figure><p>workloada文件解析如下：</p><figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line">recordcount=<span class="number">1000</span> <span class="comment">// 执行操作记录数</span></span><br><span class="line">operationcount=<span class="number">1000</span> <span class="comment">// 执行操作的记录数</span></span><br><span class="line">workload=com.yahoo.ycsb.workloads.CoreWorkload <span class="comment">// workload具体类</span></span><br><span class="line">readallfields=<span class="literal">true</span> <span class="comment">// 是否读取所有字段</span></span><br><span class="line">readproportion=<span class="number">0.5</span> <span class="comment">// read操作占比：50%</span></span><br><span class="line">updateproportion=<span class="number">0.5</span> <span class="comment">// update操作占比：50%</span></span><br><span class="line">scanproportion=<span class="number">0</span> <span class="comment">// scan操作占比：0</span></span><br><span class="line">insertproportion=<span class="number">0</span> <span class="comment">// insert操作占比：0</span></span><br><span class="line">requestdistribution=zipfian <span class="comment">// 采用zipfian分布</span></span><br></pre></td></tr></table></figure><h4 id="其他细节"><a href="#其他细节" class="headerlink" title="其他细节"></a>其他细节</h4><ul><li>recordcount 和 operationcount</li></ul><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># workload配置</span></span><br><span class="line">recordcount = 2000</span><br><span class="line">operationcount = 3000</span><br><span class="line"><span class="comment"># shell</span></span><br><span class="line">./bin/ycsb load … <span class="comment"># 将执行2000条操作</span></span><br><span class="line">./bin/ycsb run … <span class="comment"># 将执行3000条操作</span></span><br></pre></td></tr></table></figure><ul><li><p>load 和 run</p><ul><li><p><code>load</code>执行的是【INSERT操作】</p></li><li><p><code>run</code>执行的是配置文件中的操作，如</p></li></ul><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">readproportion=0.5</span><br><span class="line">updateproportion=0.5</span><br></pre></td></tr></table></figure><p>将执行50%的【READ】和50%的【UPDATE】</p></li><li><p>概率分布</p><ul><li>zipfian distribution：<a target="_blank" rel="noopener" href="https://baike.baidu.com/item/Zipf%E5%AE%9A%E5%BE%8B/1577540?fr=aladdin">zipfian分布</a></li><li>skewed distribution：<a target="_blank" rel="noopener" href="https://baike.baidu.com/item/%E5%81%8F%E6%80%81%E5%88%86%E5%B8%83/445413?fr=aladdin">偏态分布</a></li></ul></li></ul><h2 id="代码概述"><a href="#代码概述" class="headerlink" title="代码概述"></a>代码概述</h2><p>命令<code>./bin/ycsb run redis -p redis.host=localhost -P workload/workloada</code>在代码中会经过两个阶段：</p><ol><li>读取命令参数：<ol><li>分析终端中的命令参数</li><li>读取分析-P worklod/workload* 文件中的参数</li></ol></li><li>根据（一）中的参数进行相应的数据库测试操作<ol><li>根据YCSB目录下诸如<code>./hbase</code>；<code>./redis</code>；<code>./mongodb</code>等文件夹中对应的 <code>*Client.java</code>连接相应的数据库，以及该类文件提供的<code>insert</code>，<code>delete</code>，<code>update</code>， <code>scan</code>四种基本方法对数据库进行操作</li><li>上述四种操作的执行嵌套在<code>DBWrapper.java</code>中，DBWrapper实际上是一个包装类，将对上述执行操作进行额外的统计（如<code>每条操作所需的时间</code>）</li><li><code>DBWrapper</code>收集到的每次操作的数据最终将在<code>com.yahoo.ycsb.measurements</code> 以及<code>com.yahoo.ycsb.measurements.exporter</code> package中汇总和格式化：（如<code>test_redis.txt</code>中显示的格式）<ul><li>值得注意，在<code>com.yahoo.ycsb.generator package</code>所有类的作用，是因为<code>非关系数据库</code>是<code>key-value</code>的模式，我们将通过该generator生成可用且不重复的key值</li></ul></li></ol></li></ol><h2 id="尾声"><a href="#尾声" class="headerlink" title="尾声"></a>尾声</h2><p>据说<code>YCSB</code>的一大优点在于其扩展性，然而编写代码仍是件很烦躁的事情（对不熟悉<code>JAVA</code>的同学更是如此），tips：参考*Client.java文件，实现对特定数据库的连接和<code>CRUD</code>基本操作，即可生成想要的结果。</p><h2 id="附录"><a href="#附录" class="headerlink" title="附录"></a>附录</h2><ul><li>安装tcl8.5或更改版本</li></ul><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">wget http://downloads.sourceforge.net/tcl/tcl8.6.4-src.tar.gz</span><br><span class="line">tar -xf ../tcl8.6.4-html.tar.gz –strip-components=1</span><br><span class="line"><span class="comment"># 下列配置源自tcl官网，复制运行即可</span></span><br><span class="line"><span class="built_in">export</span> SRCDIR=<span class="built_in">pwd</span> &&</span><br><span class="line"><span class="built_in">cd</span> unix &&</span><br><span class="line">./configure –prefix=/usr \</span><br><span class="line">–mandir=/usr/share/man \</span><br><span class="line">([ (uname -m) = x86_64 ] && <span class="built_in">echo</span> –<span class="built_in">enable</span>-64bit) &&</span><br><span class="line">make &&</span><br><span class="line">sed -e “s<span class="comment">#SRCDIR/unix#/usr/lib#” \</span></span><br><span class="line">-e “s<span class="comment">#SRCDIR#/usr/include#” \</span></span><br><span class="line">-i tclConfig.sh &&</span><br><span class="line">sed -e “s<span class="comment">#SRCDIR/unix/pkgs/tdbc1.0.3#/usr/lib/tdbc1.0.3#” \</span></span><br><span class="line">-e “s<span class="comment">#SRCDIR/pkgs/tdbc1.0.3/generic#/usr/include#” \</span></span><br><span class="line">-e “s<span class="comment">#SRCDIR/pkgs/tdbc1.0.3/library#/usr/lib/tcl8.6#” \</span></span><br><span class="line">-e “s<span class="comment">#SRCDIR/pkgs/tdbc1.0.3#/usr/include#” \</span></span><br><span class="line">-i pkgs/tdbc1.0.3/tdbcConfig.sh &&</span><br><span class="line">sed -e “s<span class="comment">#SRCDIR/unix/pkgs/itcl4.0.3#/usr/lib/itcl4.0.3#” \</span></span><br><span class="line">-e “s<span class="comment">#SRCDIR/pkgs/itcl4.0.3/generic#/usr/include#” \</span></span><br><span class="line">-e “s<span class="comment">#SRCDIR/pkgs/itcl4.0.3#/usr/include#” \</span></span><br><span class="line">-i pkgs/itcl4.0.3/itclConfig.sh &&</span><br><span class="line"><span class="built_in">unset</span> SRCDIR</span><br></pre></td></tr></table></figure></div></div></div><div class="more section"><div class="pre"><a class="article-link" href="/use_ali_cloud.html"><i class="iconfont icon-right"></i> <span>阿里云搭建wordpress记</span></a></div><div class="next"><a class="article-link" href="/perfect_code.html">《程序员修炼之道》读后感 <i class="iconfont icon-right"></i></a></div></div></main></body><footer class="section fullscreen"><div class="footer-desc">Edeink © 2015-2022 · Powered by Hexo</div></footer><script src="https://cdn.bootcdn.net/ajax/libs/jquery/3.6.0/jquery.js"></script><script src="/public/js/init.js"></script></html>