-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathatom.xml
461 lines (274 loc) · 624 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>yangcl's</title>
<subtitle>悟已往之不谏，知来者之可追。</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="https://yangchenglong11.github.io/"/>
<updated>2018-07-10T09:33:54.702Z</updated>
<id>https://yangchenglong11.github.io/</id>
<author>
<name>Yang Chenglong</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>transaction</title>
<link href="https://yangchenglong11.github.io/2018/04/25/transaction/"/>
<id>https://yangchenglong11.github.io/2018/04/25/transaction/</id>
<published>2018-04-25T06:27:58.000Z</published>
<updated>2018-07-10T09:33:54.702Z</updated>
<content type="html"><![CDATA[<p>这篇文章详细的介绍以太坊中的交易。<br><a id="more"></a></p><h2 id="1-基本概念"><a href="#1-基本概念" class="headerlink" title="1. 基本概念"></a>1. 基本概念</h2><h3 id="1-1-常用数据类型-哈希值和地址"><a href="#1-1-常用数据类型-哈希值和地址" class="headerlink" title="1.1 常用数据类型 哈希值和地址"></a>1.1 常用数据类型 哈希值和地址</h3><p>两个最常用的自定义数据类型<strong>common.Hash用来表示哈希值，common.Address</strong>表示地址</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"># /commons/types.<span class="keyword">go</span> </div><div class="line"><span class="keyword">const</span> ( </div><div class="line"> HashLength = <span class="number">32</span> </div><div class="line"> AddressLength = <span class="number">20</span> </div><div class="line">) </div><div class="line"><span class="keyword">type</span> Hash [HashLength]<span class="keyword">byte</span> </div><div class="line"><span class="keyword">type</span> Address [AddressLength]<span class="keyword">byte</span></div></pre></td></tr></table></figure><p>在Ethereum 代码里，所有用到的哈希值，都使用该Hash类型，长度为32bytes，即256 bits；Ethereum 中所有跟帐号(Account)相关的信息，比如交易转帐的转出帐号(地址)和转入帐号(地址)，都会用该Address类型表示，长度20bytes。</p><h3 id="1-2-汽油-Gas-和以太币-Ether"><a href="#1-2-汽油-Gas-和以太币-Ether" class="headerlink" title="1.2 汽油(Gas)和以太币(Ether)"></a>1.2 汽油(Gas)和以太币(Ether)</h3><p>Gas, 是Ethereum里对所有活动进行消耗资源计量的单位。这里的活动是泛化的概念，包括但不限于：转帐，合约的创建，合约指令的执行，执行中内存的扩展等等。所以Gas可以想象成现实中的汽油或者燃气。</p><p>Ether, 是Ethereum世界中使用的数字货币，也就是常说的以太币。如果某个帐号，Address A想要发起一个交易，比如一次简单的转帐，即向 Address B 发送一笔金额H，那么Address A 本身拥有的Ether，除了转帐的数额H之外，还要有额外一笔金额用以支付交易所耗费的Gas。</p><p>如果可以实现Gas和Ether之间的换算，那么Ethereum系统里所有的活动，都可以用Ether来计量。这样，Ether就有了点一般等价物，也就是货币的样子。</p><h3 id="1-3-区块是交易的集合"><a href="#1-3-区块是交易的集合" class="headerlink" title="1.3 区块是交易的集合"></a>1.3 区块是交易的集合</h3><p>区块(Block)是Ethereum的核心结构体之一。在整个区块链(BlockChain)中，一个个Block是以<strong>单向链表</strong>的形式相互关联起来的。Block中带有一个Header(指针), Header结构体带有Block的所有属性信息，其中的ParentHash 表示该区块的父区块哈希值， 亦即<strong>Block之间关联起来的前向指针</strong>。只不过要想得到父区块(parentBlock)对象，直接解析这个ParentHash是不够的， 而是要将ParentHash同其他字符串([]byte)组合成合适的key([]byte), 去kv数据库里查询相应的value才能解析得到。 Block和Header的部分成员变量定义如下：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"># /core/types/block.<span class="keyword">go</span> </div><div class="line"><span class="keyword">type</span> Block <span class="keyword">struct</span> { </div><div class="line"> header *Header </div><div class="line"> transactions Transactions <span class="comment">// type Transactions []*Transaction </span></div><div class="line"> ... </div><div class="line">} </div><div class="line"><span class="keyword">type</span> Header <span class="keyword">struct</span> { </div><div class="line"> ParentHash common.Hash </div><div class="line"> Number *big.Int </div><div class="line"> ... </div><div class="line">}</div></pre></td></tr></table></figure><p>Header的整型成员Number表示该区块在整个区块链(BlockChain)中所处的位置，每一个区块相对于它的父区块，其Number值是+1。这样，整个区块链会存在一个原始区块，即创世块(GenesisBlock)， 它的Number是0，由系统通过配置文件生成而不必去额外挖掘(mine)。Block和BlockChain的实现细节，之后会有更详细的讨论。</p><p>Block中还有一个Tranction(指针)数组，这是我们这里关注的。<strong>Transaction(简称tx)，是Ethereum里标示一次交易的结构体</strong>， 它的成员变量包括转帐金额，转入方地址等等信息。Transaction的完整声明如下：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line"># /core/types/transaction.<span class="keyword">go</span> </div><div class="line"><span class="keyword">type</span> Transaction <span class="keyword">struct</span> { </div><div class="line"> data txdata </div><div class="line"> hash, size, from atomic.Value <span class="comment">// for cache </span></div><div class="line">} </div><div class="line"><span class="keyword">type</span> txdata <span class="keyword">struct</span> { </div><div class="line"> AccountNonce <span class="keyword">uint64</span> </div><div class="line"> Price *big.Int </div><div class="line"> GasLimit *big.Int </div><div class="line"> Recipient *common.Address </div><div class="line"> Amount *big.Int </div><div class="line"> Payload []<span class="keyword">byte</span> </div><div class="line"> V, R, S *big.Int <span class="comment">// for signature </span></div><div class="line"> Hash *common.Hash <span class="comment">// for marshaling </span></div><div class="line">}</div></pre></td></tr></table></figure><p>每个tx都声明了自己的(Gas)Price 和 GasLimit。 Price指的是单位Gas消耗所折抵的Ether多少，它的高低意味着执行这个tx有多么昂贵。GasLimit 是该tx执行过程中所允许消耗资源的总上限，通过这个值，我们可以防止某个tx执行中出现恶意占用资源的问题，这也是Ethereum中有关安全保护的策略之一。拥有独立的Price和GasLimit, 也意味着每个tx之间都是相互独立的。</p><p><strong>转帐转入方地址Recipient可能为空(nil)</strong>，这时在后续执行tx过程中，Ethereum 需要创建一个地址来完成这笔转帐。Payload是重要的数据成员，它既可以作为所创建合约的指令数组，其中每一个byte作为一个单独的虚拟机指令；也可以作为数据数组，由合约指令进行操作。合约由以太坊虚拟机(Ethereum Virtual Machine, EVM)创建并执行。</p><p>你在这里可能会有个疑问，为何交易的定义里没有声明转帐的转出方地址？ tx 的转帐转出方地址确实没有如转入方一样被显式的声明出来，而是被<strong>加密</strong>隐藏起来了，在Ethereum里这个转出方地址是机密，不能直接暴露。这个对tx加密的环节，在Ethereum里被称为签名(sign), 关于它的实现细节容后再述。</p><h1 id="2-交易的执行"><a href="#2-交易的执行" class="headerlink" title="2. 交易的执行"></a>2. 交易的执行</h1><p>Block 类型的基本目的之一，就是为了执行交易。狭义的交易可能仅仅是一笔转帐，而广义的交易同时还会支持许多其他的意图。Ethereum 中采用的是广义交易概念。按照其架构设计，交易的执行可大致分为<strong>内外两层结构</strong>：<strong>第一层是虚拟机外</strong>，包括执行前将Transaction类型转化成Message，创建虚拟机(EVM)对象，计算一些Gas消耗，以及执行交易完毕后创建收据(Receipt)对象并返回等；<strong>第二层是虚拟机内</strong>，包括执行转帐，和创建合约并执行合约的指令数组。</p><h3 id="2-1-虚拟机外"><a href="#2-1-虚拟机外" class="headerlink" title="2.1 虚拟机外"></a>2.1 虚拟机外</h3><h4 id="2-1-1-入口和返回值"><a href="#2-1-1-入口和返回值" class="headerlink" title="2.1.1 入口和返回值"></a>2.1.1 入口和返回值</h4><p>执行tx的入口函数是StateProcessor的Process()函数，其实现代码如下：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line"># /core/state_processor.<span class="keyword">go</span> </div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(p *StateProcessor)</span> <span class="title">Process</span><span class="params">(block *types.Block, statedb *state.StateDB, cfg vm.Config)</span> <span class="params">(types.Receipts, []*types.Log, <span class="keyword">uint64</span>, error)</span></span> {</div><div class="line"><span class="keyword">var</span> (</div><div class="line">receipts types.Receipts</div><div class="line">usedGas = <span class="built_in">new</span>(<span class="keyword">uint64</span>)</div><div class="line">header = block.Header()</div><div class="line">allLogs []*types.Log</div><div class="line">gp = <span class="built_in">new</span>(GasPool).AddGas(block.GasLimit())</div><div class="line">)</div><div class="line">......</div><div class="line"><span class="comment">// Iterate over and process the individual transactions</span></div><div class="line"><span class="keyword">for</span> i, tx := <span class="keyword">range</span> block.Transactions() {</div><div class="line">statedb.Prepare(tx.Hash(), block.Hash(), i)</div><div class="line">receipt, _, err := ApplyTransaction(p.config, p.bc, <span class="literal">nil</span>, gp, statedb, header, tx, usedGas, cfg)</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, <span class="literal">nil</span>, <span class="number">0</span>, err</div><div class="line">}</div><div class="line">receipts = <span class="built_in">append</span>(receipts, receipt)</div><div class="line">allLogs = <span class="built_in">append</span>(allLogs, receipt.Logs...)</div><div class="line">}</div><div class="line"><span class="comment">// Finalize the block, applying any consensus engine specific extras (e.g. block rewards)</span></div><div class="line">p.engine.Finalize(p.bc, header, statedb, block.Transactions(), block.Uncles(), receipts)</div><div class="line"></div><div class="line"><span class="keyword">return</span> receipts, allLogs, *usedGas, <span class="literal">nil</span></div><div class="line">}</div></pre></td></tr></table></figure><p>GasPool 类型其实就是big.Int。在一个Block的处理过程(即其所有tx的执行过程)中，GasPool 的值能够告诉你，剩下还有多少Gas可以使用。在每一个tx执行过程中，Ethereum 还设计了偿退(refund)环节，所偿退的Gas数量也会加到这个GasPool里。</p><p>Process()函数的核心是一个for循环，它将Block里的所有tx逐个遍历执行。具体的执行函数叫ApplyTransaction()，它每次执行tx, 会返回一个收据(Receipt)对象。Receipt结构体的声明如下：</p><p><img src="/2018/04/25/transaction/receipt.png" alt="img"></p><p>Receipt 中有一个Log类型的数组，其中每一个Log对象记录了Tx中一小步的操作。所以，每一个tx的执行结果<strong>，</strong>由<strong>一个Receipt对象</strong>来表示；更详细的内容，由<strong>一组Log对象</strong>来记录。这个Log数组很重要，比如在不同Ethereum节点(Node)的相互同步过程中，待同步区块的Log数组有助于验证同步中收到的block是否正确和完整，所以会被单独同步(传输)。</p><p>Receipt的PostState保存了创建该 Receipt 对象时，整个Block内所有“帐户”的当时状态。Ethereum 里用stateObject 来表示一个账户 Account，这个账户可转帐(transfer value), 可执行tx, 它的唯一标示符是一个Address类型变量。 这个Receipt.PostState 就是当时所在 Block 里所有 stateObject 对象的 RLP Hash 值。</p><p>Bloom 类型是一个 Ethereum 内部实现的一个 256bit长 Bloom Filter。它可用来快速验证一个新收到的对象是否处于一个已知的大量对象集合之中。这里Receipt的Bloom，被用以验证某个给定的Log是否处于Receipt已有的Log数组中。</p><h4 id="2-1-2-消耗Gas，亦奖励Gas"><a href="#2-1-2-消耗Gas，亦奖励Gas" class="headerlink" title="2.1.2 消耗Gas，亦奖励Gas"></a>2.1.2 消耗Gas，亦奖励Gas</h4><p>我们来看下StateProcessor.ApplyTransaction()的具体实现，它的基本流程如下图：</p><p><img src="/2018/04/25/transaction/Apply_Transaction.png" alt="img"></p><p>ApplyTransaction()首先根据输入参数分别封装出一个Message对象和一个EVM对象，然后加上一个传入的GasPool类型变量，由TransitionDb()函数完成tx的执行，待TransitionDb()返回之后，创建一个收据Receipt对象，最后返回该Recetip对象，以及整个tx执行过程所消耗Gas数量。</p><p>GasPool对象是在一个Block执行开始时创建，并在该Block内所有tx的执行过程中共享，对于一个tx的执行可视为“全局”存储对象； Message由此次待执行的tx对象转化而来，并携带了解析出的tx的(转帐)转出方地址，属于待处理的数据对象；EVM 作为Ethereum世界里的虚拟机(Virtual Machine)，作为此次tx的实际执行者，完成转帐和合约(Contract)的相关操作。</p><p>我们来细看下TransitioDb()的执行过程(/core/state_transition.go)。假设有StateTransition对象st, 其成员变量initialGas表示初始可用Gas数量，gas表示即时可用Gas数量，初始值均为0，于是st.TransitionDb() 可由以下步骤展开：</p><ol><li>购买Gas。首先从交易的(转帐)转出方账户扣除一笔Ether，费用等于tx.data.GasLimit * tx.data.Price；同时 st.initialGas = st.gas = tx.data.GasLimit；然后(GasPool) gp -= st.gas。</li><li>计算tx的固有Gas消耗 - intrinsicGas。它分为两个部分，每一个tx预设的消耗量，这个消耗量还因tx是否含有(转帐)转入方地址而略有不同；以及针对tx.data.Payload的Gas消耗，Payload类型是[]byte，关于它的固有消耗依赖于[]byte中非0字节和0字节的长度。最终，st.gas -= intrinsicGas</li><li>EVM执行。如果交易的(转帐)转入方地址(tx.data.Recipient)为空，调用EVM的Create()函数；否则，调用Call()函数。无论哪个函数返回后，更新st.gas。</li><li>计算本次执行交易的实际Gas消耗： requiredGas = st.initialGas - st.gas</li><li><strong>偿退Gas</strong>。它包括两个部分：首先将剩余st.gas 折算成Ether，归还给交易的(转帐)转出方账户；然后，基于实际消耗量requiredGas，系统提供一定的<strong>补偿</strong>，数量为refundGas。refundGas 所折算的Ether会被立即加在(转帐)转出方账户上，同时st.gas += refundGas，gp += st.gas，即剩余的Gas加上系统补偿的Gas，被一起归并进GasPool，供之后的交易执行使用。</li><li><strong>奖励所属区块的挖掘者</strong>：系统给所属区块的作者，亦即挖掘者账户，增加一笔金额，数额等于 st.data=Price * (st.initialGas - st.gas)。注意，这里的st.gas在步骤5中被加上了refundGas, 所以这笔奖励金所对应的Gas，其数量小于该交易实际消耗量requiredGas。</li></ol><p>由上可见，除了步骤3中EVM 函数的执行，其他每个步骤都在围绕着Gas消耗量作文章(EVM 虚拟机的运行原理容后再述)。到这里，大家可以对Gas在以太坊系统里的作用有个初步概念，Gas就是Ethereum系统中的血液。</p><p>由于步骤6的<strong>奖励机制</strong>的存在<strong>，</strong>才会吸引社会上的矿工(miner)去卖力“挖矿”(mining)。越大的运算能力带来越多的的区块(交易)产出，矿工也就能通过该奖励机制赚取越多的以太币。</p><h4 id="2-1-3-交易的数字签名"><a href="#2-1-3-交易的数字签名" class="headerlink" title="2.1.3 交易的数字签名"></a>2.1.3 交易的数字签名</h4><p>Ethereum 中每个交易(transaction，tx)对象在被放进block时，都是经过<strong>数字签名</strong>的，这样可以在后续传输和处理中随时验证tx是否经过篡改。Ethereum 采用的数字签名是<strong>椭圆曲线数字签名算法</strong>(Elliptic Cure Digital Signature Algorithm,<a href="https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm" target="_blank" rel="external">ECDSA</a>)。ECDSA 相比于基于大质数分解的RSA数字签名算法，可以在提供相同安全级别(in bits)的同时，仅需更短的公钥(public key)。这里需要特别留意的是，<strong>tx的转帐转出方地址，就是对该tx对象作ECDSA签名计算时所用的公钥publicKey</strong>。</p><p>Ethereum中的数字签名计算过程所生成的签名(signature), 是一个长度为65bytes的字节数组，它被截成三段放进tx中，前32bytes赋值给成员变量R, 再32bytes赋值给S，末1byte赋给V，当然由于R、S、V声明的类型都是*big.Int, 上述赋值存在[]byte -> big.Int的类型转换。</p><p><img src="/2018/04/25/transaction/signature.png" alt="img"></p><p>当需要恢复出tx对象的转帐转出方地址时(比如在需要执行该交易时)，Ethereum 会先从tx的signature中恢复出公钥，再将公钥转化成一个common.Address类型的地址，signature由tx对象的三个成员变量R,S,V转化成字节数组[]byte后拼接得到。</p><p>Ethereum 对此定义了一个接口Signer, 用来执行挂载签名，恢复公钥，对tx对象做哈希等操作。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// core/types/transaction_signing.go </span></div><div class="line"><span class="keyword">type</span> Signer innterface { </div><div class="line"> Sender(tx *Transaction) (common.Address, error) </div><div class="line"> SignatureValues(tx *Transaction, sig []<span class="keyword">byte</span>) (r, s, v *big.Int, err error) </div><div class="line"> Hash(tx *Transaction) common.Hash </div><div class="line"> Equal(Signer) <span class="keyword">bool</span> </div><div class="line">}</div></pre></td></tr></table></figure><p>生成数字签名的函数叫SignTx()，它会先调用其他函数生成signature, 然后调用tx.WithSignature()将signature分段赋值给tx的成员变量R,S,V。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">SignTx</span><span class="params">(tx Transaction, s Signer, prv *ecdsa.PrivateKey)</span> <span class="params">(Transaction, error)</span></span></div></pre></td></tr></table></figure><p>恢复出转出方地址的函数叫Sender()， 参数包括一个Signer, 一个Transaction，代码如下：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">Sender</span><span class="params">(signer Signer, tx *Transaction)</span> <span class="params">(common.Address, error)</span></span> { </div><div class="line"> <span class="keyword">if</span> sc := tx.from().Load(); sc != null { </div><div class="line"> sigCache := sc.(sigCache)<span class="comment">// cache exists, </span></div><div class="line"> <span class="keyword">if</span> sigCache.signer.Equal(signer) { </div><div class="line"> <span class="keyword">return</span> sigCache.from, <span class="literal">nil</span> </div><div class="line"> } </div><div class="line"> } </div><div class="line"> addr, err := signer.Sender(tx) </div><div class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> { </div><div class="line"> <span class="keyword">return</span> common.Address{}, err </div><div class="line"> } </div><div class="line"> tx.from.Store(sigCache{signer: signer, from: addr}) <span class="comment">// cache it </span></div><div class="line"> <span class="keyword">return</span> addr, <span class="literal">nil</span> </div><div class="line">}</div></pre></td></tr></table></figure><p>Sender()函数体中，signer.Sender()会从本次数字签名的签名字符串(signature)中恢复出公钥，并转化为tx的转出方地址。</p><p>在上文提到的ApplyTransaction()实现中，Transaction对象需要首先被转化成Message接口，用到的AsMessage()函数即调用了此处的Sender()。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// core/types/transaction.go </span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(tx *Transaction)</span> <span class="title">AsMessage</span><span class="params">(s Signer)</span> <span class="params">(Message,error)</span></span> { </div><div class="line"> msg := Message{ </div><div class="line"> price: <span class="built_in">new</span>(big.Int).Set(tx.data.price) </div><div class="line"> gasLimit: <span class="built_in">new</span>(big.Int).Set(tx.data.GasLimit) </div><div class="line"> ... </div><div class="line"> } </div><div class="line"> <span class="keyword">var</span> err error </div><div class="line"> msg.from, err = Sender(s, tx) </div><div class="line"> <span class="keyword">return</span> msg, err </div><div class="line">}</div></pre></td></tr></table></figure><p> 在Transaction对象tx的转帐转出方地址被解析出以后，tx 就被完全转换成了Message类型，可以提供给虚拟机EVM执行了。</p><h3 id="2-2-虚拟机内"><a href="#2-2-虚拟机内" class="headerlink" title="2.2 虚拟机内"></a>2.2 虚拟机内</h3><p>每个交易(Transaction)带有两部分内容需要执行：</p><ol><li>转帐，由转出方地址向转入方地址转帐一笔以太币Ether; </li><li>携带的[]byte类型成员变量Payload，其每一个byte都对应了一个单独虚拟机指令。</li></ol><p>这些内容都是由EVM(Ethereum Virtual Machine)对象来完成的。EVM 结构体是Ethereum虚拟机机制的核心，它以及与之协同的struct的关系如下：</p><p><img src="/2018/04/25/transaction/evm.png" alt="img"></p><p>其中Context结构体分别携带了Transaction的信息(GasPrice, GasLimit)，Block的信息(Number, Difficulty)，以及转帐函数等，提供给EVM；StateDB 接口是针对state.StateDB 结构体设计的本地行为接口，可为EVM提供statedb的相关操作； Interpreter结构体作为解释器，用来解释执行EVM中合约(Contract)的指令(Code)。</p><h4 id="2-2-1-完成转帐"><a href="#2-2-1-完成转帐" class="headerlink" title="2.2.1 完成转帐"></a>2.2.1 完成转帐</h4><p>交易的转帐操作由Context对象中的TransferFunc类型函数来实现，类似的函数类型，还有CanTransferFunc和GetHashFunc。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// core/vm/evm.go </span></div><div class="line"><span class="keyword">type</span> { </div><div class="line"> CanTransferFunc <span class="function"><span class="keyword">func</span><span class="params">(StateDB, common.Address, *big.Int)</span> </span></div><div class="line"><span class="function"> <span class="title">TransferFunc</span> <span class="title">func</span><span class="params">(StateDB, common.Address, common.Address, *big.Int)</span> </span></div><div class="line"><span class="function"> <span class="title">GetHashFunc</span> <span class="title">func</span><span class="params">(<span class="keyword">uint64</span>)</span> <span class="title">common</span>.<span class="title">Hash</span> </span></div><div class="line"><span class="function">}</span></div></pre></td></tr></table></figure><p>这三个类型的函数变量CanTransfer, Transfer, GetHash，在Context初始化时从外部传入，目前使用的均是一个本地实现：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// core/evm.go </span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">NewEVMContext</span><span class="params">(msg Message, header *Header, chain ChainContext, author *Address)</span></span>{ </div><div class="line"> <span class="keyword">return</span> vm.Context { </div><div class="line"> CanTransfer: CanTransfer, </div><div class="line"> Transfer: Transfer, </div><div class="line"> GetHash： GetHash(header, chain), </div><div class="line"> ... </div><div class="line"> } </div><div class="line">} </div><div class="line"> </div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">CanTransfer</span><span class="params">(db vm.StateDB, addr common.Address, amount *big.Int)</span></span> { </div><div class="line"> <span class="keyword">return</span> db.GetBalance(addr).Cmp(amount) >= <span class="number">0</span> </div><div class="line">} </div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">Transfer</span><span class="params">(db vm.StateDB, sender, recipient common.Address, amount *big.Int)</span></span> { </div><div class="line"> db.SubBalance(sender, amount) </div><div class="line"> db.AddBalance(recipient, amount) </div><div class="line">}</div></pre></td></tr></table></figure><p>可见目前的转帐函数Transfer()的逻辑非常简单，转帐的转出账户减掉一笔以太币，转入账户加上一笔以太币。由于EVM调用的Transfer()函数实现完全由Context提供，所以，假设如果基于Ethereum平台开发，需要设计一种全新的“转帐”模式，那么只需写一个新的Transfer()函数实现，在Context初始化时赋值即可。</p><p>有人或许会问，这里Transfer()函数中对转出和转入账户的操作会立即生效么？万一两步操作之间有错误发生怎么办？答案是不会立即生效。StateDB 并不是真正的数据库，只是一行为类似数据库的结构体。它在内部以Trie的数据结构来管理各个基于地址的账户，可以理解成一个cache；当该账户的信息有变化时，变化先存储在Trie中。仅当整个Block要被插入到BlockChain时，StateDB 里缓存的所有账户的所有改动，才会被真正的提交到底层数据库。</p><h4 id="2-2-2-合约的创建和赋值"><a href="#2-2-2-合约的创建和赋值" class="headerlink" title="2.2.2 合约的创建和赋值"></a>2.2.2 合约的创建和赋值</h4><p>合约(Contract)是EVM用来执行(虚拟机)指令的结构体。先来看下Contract的定义：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// core/vm/contract.go </span></div><div class="line"><span class="keyword">type</span> ContractRef <span class="keyword">interface</span> { </div><div class="line"> Address() common.Address </div><div class="line">} </div><div class="line"><span class="keyword">type</span> Contract <span class="keyword">struct</span> { </div><div class="line"> CallerAddress common.Address </div><div class="line"> caller ContractRef </div><div class="line"> self ContractRef </div><div class="line"> </div><div class="line"> jumpdests destinations </div><div class="line"> Code []<span class="keyword">byte</span> </div><div class="line"> CodeHash common.Hash </div><div class="line"> CodeAddr *Address </div><div class="line"> Input []<span class="keyword">byte</span> </div><div class="line"> Gas <span class="keyword">uint64</span> </div><div class="line"> value *big.Int </div><div class="line"> Args []<span class="keyword">byte</span> </div><div class="line"> DelegateCall <span class="keyword">bool</span> </div><div class="line">}</div></pre></td></tr></table></figure><p>在这些成员变量里，caller是转帐转出方地址(账户)，self是转入方地址，不过它们的类型都用接口ContractRef来表示；Code是指令数组，其中每一个byte都对应于一个预定义的虚拟机指令；CodeHash 是Code的RLP哈希值；Input是数据数组，是指令所操作的数据集合；Args 是参数。</p><p>有意思的是self这个变量，为什么转入方地址要被命名成self呢？ Contract实现了ContractRef接口，返回的恰恰就是这个self地址。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(c *Contract)</span> <span class="title">Address</span><span class="params">()</span> <span class="title">common</span>.<span class="title">Address</span></span> { </div><div class="line"> <span class="keyword">return</span> c.self.Address() </div><div class="line">}</div></pre></td></tr></table></figure><p>所以当Contract对象作为一个ContractRef接口出现时，它返回的地址就是它的self地址。那什么时候Contract会被类型转换成ContractRef呢？当Contract A调用另一个Contract B时，A就会作为B的caller成员变量出现。<strong>Contract可以调用Contract</strong>，这就为系统在业务上的潜在扩展，提供了空间。</p><p>创建一个Contract对象时，<strong>重点关注对self的初始化，以及对Code, CodeAddr 和Input的赋值</strong>。 </p><p>另外，StateDB 提供方法SetCode()，可以将指令数组Code存储在某个stateObject对象中; 方法GetCode()，可以从某个stateObject对象中读取已有的指令数组Code。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(self *StateDB)</span> <span class="title">SetCode</span><span class="params">(addr common.Address, code []<span class="keyword">byte</span>)</span> </span></div><div class="line"><span class="function"><span class="title">func</span> <span class="params">(self *StateDB)</span> <span class="title">GetCode</span><span class="params">(addr common.Address)</span> <span class="title">code</span> []<span class="title">byte</span></span></div></pre></td></tr></table></figure><p>刚刚说过，<strong>stateObject 是Ethereum里用来管理一个账户所有信息修改的结构体</strong>，它以一个Address类型变量为唯一标示符。StateDB 在内部用一个巨大的map结构来管理这些stateObject对象。所有账户信息-包括Ether余额，指令数组Code, 该账户发起合约次数nonce等-它们发生的所有变化，会首先缓存到StateDB里的某个stateObject里，然后在合适的时候，被StateDB一起提交到底层数据库。注意，<strong>一个Contract所对应的stateObject的地址，是Contract的self地址，也就是转帐的转入方地址</strong>。</p><p>EVM 目前有五个函数可以创建并执行Contract，按照作用和调用方式，可以分成两类:</p><ul><li>Create(), Call(): 二者均在StateProcessor的ApplyTransaction()被调用以<strong>执行单个交易</strong>，并且都有<strong>调用转帐函数完成转帐</strong>。</li></ul><ul><li>CallCode(), DelegateCall(), StaticCall()：三者由于分别对应于不同的虚拟机指令(1 byte)操作，<strong>不会用以执行单个交易</strong>，也都<strong>不能处理转帐</strong>。</li></ul><p>考虑到与执行交易的相关性，这里着重探讨Create()和Call()。先来看Call()，它用来处理(转帐)转入方地址不为空的情况：</p><p><img src="/2018/04/25/transaction/evm_call.png" alt="img"></p><p>Call()函数的逻辑可以简单分为以上6步。其中步骤(3)调用了转帐函数Transfer()，转入账户caller, 转出账户addr；步骤(4)创建一个Contract对象，并初始化其成员变量caller, self(addr), value和gas； 步骤(5)赋值Contract对象的Code, CodeHash, CodeAddr成员变量；步骤(6) 调用run()函数执行该合约的指令，最后Call()函数返回。相关代码可见：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// core/vm/evm.go </span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(evm *EVM)</span> <span class="title">Call</span><span class="params">(caller ContractRef, addr common.Address, input []<span class="keyword">byte</span>, gas <span class="keyword">uint64</span>, value *big.Int)</span> <span class="params">(ret []<span class="keyword">byte</span>, leftGas *big.Int, error)</span></span>{ </div><div class="line"> ... </div><div class="line"> <span class="keyword">var</span> snapshot = evm.StateDB.Snapshot() </div><div class="line"> contract.SetCallCode(&addr, evm.StateDB.GetCodeHash(addr), evm.StateDB.GetCode(addr)) </div><div class="line"> ret, err = run(evm, snapshot, contract, input) </div><div class="line"> <span class="keyword">return</span> ret, contract.Gas, err </div><div class="line">}</div></pre></td></tr></table></figure><p>因为此时(转帐)转入地址不为空，所以直接将入参addr初始化Contract对象的self地址，并可从StateDB中(其实是以addr标识的账户stateObject对象)读取出相关的Code和CodeHash并赋值给contract的成员变量。注意，此时转入方地址参数addr同时亦被赋值予contract.CodeAddr。</p><p>再来看看EVM.Create()，它用来处理(转帐)转入方地址为空的情况。</p><p><img src="/2018/04/25/transaction/evm_create.png" alt="img"></p><p>与Call()相比，Create()因为没有Address类型的入参addr，其流程有几处明显不同：</p><ul><li>步骤(3)中创建一个新地址contractAddr，作为(转帐)转入方地址，亦作为Contract的self地址；</li><li>步骤(6)由于contracrAddr刚刚新建，db中尚无与该地址相关的Code信息，所以会将类型为[]byte的参数code，赋值给Contract对象的Code成员；</li><li>步骤(8)将本次执行合约的返回结果，作为contractAddr所对应账户(stateObject对象)的Code储存起来，以备下次调用。</li></ul><p>还有一点，Call()有一个入参input类型为[]byte，而Create()有一个入参code类型同样为[]byte，没有入参input，它们之间有无关系？其实，它们来源都是Transaction对象tx的成员变量Payload，调用EVM.Create()或Call()的入口在StateTransition.TransitionDb()中，当tx.Recipent为空时，tx.data.Payload 被当作所创建Contract的Code；当tx.Recipient 不为空时，tx.data.Payload 被当作Contract的Input。</p><h4 id="2-2-3-预编译的合约"><a href="#2-2-3-预编译的合约" class="headerlink" title="2.2.3 预编译的合约"></a>2.2.3 预编译的合约</h4><p>EVM中执行合约(指令)的函数是run()，其实现代码如下：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// core/vm/evm.go </span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">run</span><span class="params">(evm *EVM, snapshot <span class="keyword">int</span>, contract *Contract, input []<span class="keyword">byte</span>)</span> <span class="params">([]<span class="keyword">byte</span>, error)</span></span> { </div><div class="line"> <span class="keyword">if</span> contract.CodeAddr != <span class="literal">nil</span> { </div><div class="line"> precompiles := PrecompiledContractsHomestead </div><div class="line"> ... </div><div class="line"> <span class="keyword">if</span> p := precompiles[*contract.CodeAddr]; p != <span class="literal">nil</span> { </div><div class="line"> <span class="keyword">return</span> RunPrecompiledContract(p, input, contract) </div><div class="line"> } </div><div class="line"> } </div><div class="line"> <span class="keyword">return</span> evm.interpreter.Run(snapshot, contract, input) </div><div class="line">}</div></pre></td></tr></table></figure><p>可见如果待执行的Contract对象恰好属于一组预编译的合约集合-此时以指令地址CodeAddr为匹配项-那么它可以直接运行；没有经过预编译的Contract，才会由Interpreter解释执行。这里的”预编译”，可理解为不需要编译(解释)指令(Code)。预编译的合约，其逻辑全部固定且已知，所以执行中不再需要Code，仅需Input即可。</p><p>在代码实现中，预编译合约只需实现两个方法Required()和Run()即可，这两方法仅需一个入参input。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// core/vm/contracts.go </span></div><div class="line"><span class="keyword">type</span> PrecompiledContract <span class="keyword">interface</span> { </div><div class="line"> RequiredGas(input []<span class="keyword">byte</span>) <span class="keyword">uint64</span> </div><div class="line"> Run(input []<span class="keyword">byte</span>) ([]<span class="keyword">byte</span>, error) </div><div class="line">} </div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">RunPrecompiledContract</span><span class="params">(p PrecompiledContract, input []<span class="keyword">byte</span>, contract *Contract)</span> <span class="params">(ret []<span class="keyword">byte</span>, err error)</span></span> { </div><div class="line"> gas := p.RequiredGas(input) </div><div class="line"> <span class="keyword">if</span> contract.UseGas(gas) { </div><div class="line"> <span class="keyword">return</span> p.Run(input) </div><div class="line"> } </div><div class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, ErrOutOfGas </div><div class="line">}</div></pre></td></tr></table></figure><p>目前，Ethereuem 代码中已经加入了多个预编译合约，功能覆盖了包括椭圆曲线密钥恢复，SHA-3(256bits)哈希算法，RIPEMD-160加密算法等等。相信基于自身业务的需求，二次开发者完全可以加入自己的预编译合约，大大加快合约的执行速度。</p><h4 id="2-2-4-解释器执行合约的指令"><a href="#2-2-4-解释器执行合约的指令" class="headerlink" title="2.2.4 解释器执行合约的指令"></a>2.2.4 解释器执行合约的指令</h4><p>解释器Interpreter用来执行(非预编译的)合约指令。它的结构体UML关系图如下所示：</p><p><img src="/2018/04/25/transaction/Interpreter.png" alt="img"></p><p>Interpreter结构体通过一个Config类型的成员变量，间接持有一个包括256个operation对象在内的数组JumpTable。operation是做什么的呢？<strong>每个operation对象正对应一个已定义的虚拟机指令，它所含有的四个函数变量execute, gasCost, validateStack, memorySize 提供了这个虚拟机指令所代表的所有操作</strong>。每个指令长度1byte，Contract对象的成员变量Code类型为[]byte，就是这些虚拟机指令的任意集合。operation对象的函数操作，主要会用到Stack，Memory, IntPool 这几个自定义的数据结构。</p><p>这样一来，Interpreter的Run()函数就很好理解了，其核心流程就是逐个byte遍历入参Contract对象的Code变量，将其解释为一个已知的operation，然后依次调用该operation对象的四个函数，流程示意图如下：</p><p><img src="/2018/04/25/transaction/Interpreter_run.png" alt="img"></p><p>operation在操作过程中，会需要几个数据结构： Stack，实现了标准容器 -栈的行为；Memory，一个字节数组，可表示线性排列的任意数据;还有一个intPool，提供对big.Int数据的存储和读取。</p><p>已定义的operation，种类很丰富，包括：</p><ul><li>算术运算：ADD，MUL，SUB，DIV，SDIV，MOD，SMOD，EXP…；</li><li>逻辑运算：LT，GT，EQ，ISZERO，AND，XOR，OR，NOT…；</li><li>业务功能：SHA3，ADDRESS，BALANCE，ORIGIN，CALLER，GASPRICE，LOG1，LOG2…等等</li></ul><p>需要特别注意的是LOGn指令操作，它用来创建n个Log对象，这里n最大是4。还记得Log在何时被用到么？每个交易(Transaction,tx)执行完成后，会创建一个Receipt对象用来记录这个交易的执行结果。Receipt携带一个Log数组，用来记录tx操作过程中的所有变动细节，而这些<strong>Log，正是通过合适的LOGn指令-即合约指令数组(Contract.Code)中的单个byte，在其对应的operation里被创建出来的</strong>。每个新创建的Log对象被缓存在StateDB中的相对应的stateObject里，待需要时从StateDB中读取。</p><h2 id="3-小结"><a href="#3-小结" class="headerlink" title="3. 小结"></a>3. 小结</h2><p>以太坊的出现大大晚于比特币，虽然明显受到比特币系统的启发，但在整个功能定位和设计架构上却做了很多更广更深的思考和尝试。以太坊更像是一个经济活动平台，而并不局限一种去中心化数字代币的产生，分发和流转。</p><ul><li><strong>Gas是Ethereum系统的血液。</strong>一切资源，活动，交互的开销，都以Gas为计量单元。如果定义了一个GasPrice，那么所有的Gas消耗亦可等价于以太币Ether。</li><li><strong>Block是Transaction的集合。</strong>Block在插入BlockChain前，需要将所有Transaction逐个执行。Transaction的执行会消耗发起方的Ether，但系统在其执行完成时，会给予其作者（挖掘出这个Block的账户）一笔补偿，这笔补偿是“矿工”赚取收入的来源之一。</li><li>Ethereum 定义了自己的<strong>虚拟机EVM</strong>, 它与<strong>合约(Contract)</strong>机制相结合，能够在提供非常丰富的操作的同时，又能很好的控制存储空间和运行速度。Contract由Transaction转化得到。</li><li>Ethereum 里的哈希函数，用的是<strong>SHA-3，256 bits</strong>；数据(数组)的序列化，用的是<strong>RLP编码</strong>，所以所有对象，数组的哈希算法，实际用的RLP + SHA-3。数字签名算法，使用了<strong>椭圆曲线数字签名算法(ECDSA)</strong>。</li></ul>]]></content>
<summary type="html">
<p>这篇文章详细的介绍以太坊中的交易。<br></p>
</summary>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/categories/BlockChain/"/>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/tags/BlockChain/"/>
</entry>
<entry>
<title>PoA</title>
<link href="https://yangchenglong11.github.io/2018/04/10/PoA/"/>
<id>https://yangchenglong11.github.io/2018/04/10/PoA/</id>
<published>2018-04-10T09:24:07.000Z</published>
<updated>2018-07-10T09:27:13.506Z</updated>
<content type="html"><![CDATA[<h2 id="1-PoA-理论介绍"><a href="#1-PoA-理论介绍" class="headerlink" title="1. PoA 理论介绍"></a>1. PoA 理论介绍</h2><h3 id="1-1-以太坊中-PoA-产生的背景"><a href="#1-1-以太坊中-PoA-产生的背景" class="headerlink" title="1.1 以太坊中 PoA 产生的背景"></a>1.1 以太坊中 PoA 产生的背景</h3><p>如果你想用以太坊搭建一个联盟/私有链, 并要求该链交易成本更低甚至没有, 交易延时更低,并发更高, 还拥有完全的控制权(意味着被攻击概率更低). 目前以太坊采用 PoW 或后续的 casper 能否满足要求?<br><a id="more"></a></p><ul><li>首先, PoW 存在 <strong>51% 攻击问题</strong>, 恶意挖矿者超过全网算力的 51% 后基本上就能完全控制整个网络. 由于链无法被更改, 已上链的数据也无法更改, 但恶意挖矿者也可以做一些 DoS 攻击阻止合法交易上链,考虑到具有相同创世块的旷工都能加入你的网络, 潜在的安全隐患会长期存在.</li><li>其次, PoW <strong>大量的电力资源消耗</strong>也是需要作为后续成本考虑. PoS 可以解决部分 PoW 问题, 比如节约电力,在一定程度上保护了51％的攻击(恶意旷工会被惩罚), 但从控制权和安全考虑还有欠缺, 因为 PoS 还是允许任何符合条件的旷工加入。</li></ul><p>在已经运行的测试网络 Ropsten 中, 由于 PoW 设定的难度较低,恶意旷工滥用较低的 PoW 难度并将 gaslimit 扩大到90亿（正常是470万），发送大量的交易瘫痪了整个网络。而在此之前，攻击者也尝试了多次非常长的重组(reorgs)，导致不同客户端之间的分叉，甚至不同的版本。</p><p>这些攻击的根本原因是 PoW 网络的安全性依赖于背后的算力。而从零开始重新启动一个新的 testnet 将不会解决任何问题，因为攻击者可以一次又一次地进行相同的攻击。 Parity 团队决定采取紧急解决方案，回滚大量的块，并设置不允许 gaslimit 超过某一阈值的软分叉规则。</p><p>虽然 Parity 的解决方案可能在短期内有效, 但是这不是优雅的：Ethereum 本身应该具有动态 gaslimit 限制; 也不可移植：其他客户端需要自己实现新的软分叉逻辑; 并与同步模式不兼容, 也不支持轻客户端; 尽管并不完美，但是Parity 的解决方案仍然可行。 一个更长期的替代解决方案是使用 PoA 共识,相对简单并容易实现.</p><h3 id="1-2-PoA-的特点"><a href="#1-2-PoA-的特点" class="headerlink" title="1.2. PoA 的特点"></a>1.2. PoA 的特点</h3><ul><li>PoA 是依靠预设好的授权节点(signers)，负责产生 block.</li><li>可以由已授权的 signer 选举(投票超过50%)加入新的 signer。</li><li>即使存在恶意 signer,他最多只能攻击连续块(数量是 <code>(SIGNER_COUNT / 2) + 1)</code> 中的1个,期间可以由其他signer 投票踢出该恶意 signer。</li><li>可指定产生 block 的时间。</li></ul><h3 id="1-3-PoA需要解决的问题"><a href="#1-3-PoA需要解决的问题" class="headerlink" title="1.3. PoA需要解决的问题"></a>1.3. PoA需要解决的问题</h3><ol><li>如何控制挖矿频率,即出块时间</li><li>如何验证某个块的有效性</li><li>如何动态调整授权签名者(signers)列表,并全网动态同步</li><li>如何在 signers 之间分配挖矿的负载或者叫做挖矿的机会</li></ol><p>对应的解决办法如下:</p><ol><li>协议规定采用固定的 block 出块时间, 区块头中的时间戳间隔为 15s</li><li>先看看 block 同步的方法,从中来分析 PoA 中验证 block 的解决办法</li></ol><p>有两种同步 blockchain 的方法</p><ol><li>经典方法是从创世块开始挨个执行所有交易。 这是经过验证的，但是在 Ethereum 的复杂网络中，计算量非常大。</li><li>另一个是仅下载区块头并验证其有效性，之后可以从网络下载任意的近期状态对最近的区块头进行检查。</li></ol><p>显然第二种方法更好. 由于 PoA 方案的块可以仅由可信任的签名者来创建, 因此，客户端看到的每个块（或块头）可以与可信任签名者列表进行匹配。 要验证该块的有效性就必须得到该块对应的签名者列表, 如果签名者在该列表中带包该块有效. 这里的挑战是如何维护并及时更改的授权签名者列表？ 存储在智能合约中?不可行, 因为在快速轻量级同步期间无法访问状态。</p><p>因此, <strong>授权签名者列表必须完全包含在块头中</strong> 。那么需要改变块头的结构, 引入新的字段来满足投票机制吗? 这也不可行：改变这样的核心数据结构将是开发者的噩梦。</p><p>所以授权签名者名单必须完全符合当前的数据模型, 不能改变区块头中的字段，而是 <strong>复用当前可用的字段: Extra 字段. </strong></p><p><strong>Extra</strong> 是可变长数组, 对它的修改是 <code>非侵入</code> 操作, 比如 RLP,hash 操作都支持可变长数据. Extra 中包含所有签名者列表和当前节点的签名者对该区块头的签名数据(可以恢复出来签名者的地址).</p><ol><li>更新一个动态的签名者列表的方法是复用区块头中的 <strong>Coinbase 和 Nonce 字段</strong> ，以创建投票方案：</li></ol><ul><li>常规的块中这两个字段置为0</li><li>如果签名者希望对授权签名者列表进行更改，则将：<ul><li><strong>Coinbase</strong> 设置为被投票的签名者</li><li>将 <strong>Nonce</strong> 设置为 <strong>0</strong> 或 <strong>0xff … f</strong> 投票,代表 <strong>添加或移除</strong></li><li>任何同步的客户端都可以在块处理过程中“统计”投票，并通过投票结果来维护授权签名者列表。</li></ul></li></ul><p>为了避免一个无限的时间来统计投票，我们设置一个投票窗口, 为一个 epoch,长度是30000个block。每个 epoch的起始清空所有历史的投票, 并作为签名者列表的检查点. 这允许客户端仅基于检查点哈希进行同步，而不必重播在链路上完成的所有投票。</p><ol><li>目前的方案是在所有 signer 之间轮询出块, 并通过算法保证同一个 signer 只能签名 <code>(SIGNER_COUNT / 2) + 1)</code> 个 block 中第一个.</li></ol><p>综上, PoA 的工作流程如下:</p><ol><li>在创世块中指定一组初始授权的 signers, <strong>所有地址</strong> 保存在创世块 Extra 字段中</li><li>启动挖矿后, 该组 signers 开始对生成的 block 进行 <strong>签名并广播</strong></li><li><strong>签名结果</strong> 保存在区块头的 Extra 字段中</li><li>Extra 中更新当前高度已授权的 <strong>所有 signers 的地址</strong> ,因为有新加入或踢出的 signer</li><li>每一高度都有一个 signer 处于 IN-TURN 状态, 其他 signer 处于 OUT-OF-TURN 状态, IN-TURN 的 signer 签名的 block 会<strong>立即广播</strong> , OUT-OF-TURN 的 signer 签名的 block 会 <strong>延时</strong>一点随机时间后再广播, 保证 IN-TURN的签名 block 有更高的优先级上链</li><li>如果需要加入一个新的 signer, signer 通过 API 接口发起一个 proposal, 该 proposal 通过复用区块头 <strong>Coinbase (新 signer 地址)和 Nonce(“0xffffffffffffffff”)</strong> 字段广播给其他节点. 所有已授权的 signers 对该新的 signer 进行”加入”投票, 如果赞成票超过 signers 总数的50%, 表示同意加入</li><li>如果需要踢出一个旧的 signer, 所有已授权的 signers 对该旧的 signer 进行”踢出”投票, 如果赞成票超过signers 总数的50%, 表示同意踢出</li></ol><h4 id="1-4-signer-对区块头进行签名"><a href="#1-4-signer-对区块头进行签名" class="headerlink" title="1.4 signer 对区块头进行签名"></a>1.4 signer 对区块头进行签名</h4><ol><li>Extra 的长度至少65字节以上(签名结果是65字节,即 R, S, V, V是0或1)</li><li>对 blockHeader中所有字段除了 Extra 的 <strong>后65字节</strong> 外进行 <strong>RLP 编码</strong></li><li>对编码后的数据进行 <code>Keccak256</code> <strong>hash</strong></li><li>签名后的数据(65字节)保存到 Extra 的 <strong>后65字节</strong> 中</li></ol><h4 id="1-5-授权策略"><a href="#1-5-授权策略" class="headerlink" title="1.5 授权策略"></a>1.5 授权策略</h4><p>以下建议的策略将减少网络流量和分叉</p><ul><li>如果签名者被允许签署一个块（在授权列表中，但最近没有签名）。<ul><li>计算下一个块的最优签名时间（父块时间+ BLOCK_PERIOD）。</li><li>如果签名人是 in-turn，立即进行签名和广播block。</li><li>如果签名者是 out-of-turn，延迟 <code>rand(SIGNER_COUNT * 500ms)</code> 后再签名并广播</li></ul></li></ul><h4 id="1-6-级联投票"><a href="#1-6-级联投票" class="headerlink" title="1.6 级联投票"></a>1.6 级联投票</h4><p>当移除一个授权的签名者时,可能会导致其他移除前的投票成立. 例: ABCD 4 个 signer, AB加入E,此时不成立(没有超过50%), 如果 ABC 移除 D, 会自动导致加入 E 的投票成立(2/3的投票比例)</p><h4 id="1-7-投票策略"><a href="#1-7-投票策略" class="headerlink" title="1.7 投票策略"></a>1.7 投票策略</h4><p>因为 blockchain 可能会小范围重组(small reorgs), 常规的投票机制(cast-and-forget, 投票和忘记)可能不是最佳的，因为包含单个投票的 block 可能不会在最终的链上,会因为已有最新的 block 而被抛弃。</p><p>一个简单但有效的办法是对 signers 配置”提议( proposal )”.例如 “add 0x…”, “drop 0x…”, 有多个并发的提议时, 签名代码”随机”选择一个提议注入到该签名者签名的 block 中,这样多个并发的提议和重组( reorgs )都可以保存在链上.</p><p>该列表可能在一定数量的 block/epoch 之后过期，提案通过并不意味着它不会被重新调用，因此在提议通过时不应立即丢弃。</p><ul><li>加入和踢除新的 signer 的投票都是立即生效的,参与下一次投票计数</li><li>加入和踢除都需要 <strong>超过当前 signer 总数的50%</strong> 的 signer 进行投票</li><li>可以踢除自己(也需要超过50%投票)</li><li>可以并行投票(A,B交叉对C,D进行投票), 只要最终投票数操作50%</li><li>进入一个新的 epoch, 所有之前的 pending 投票都作废, 重新开始统计投票</li></ul><h4 id="1-8-投票场景举例"><a href="#1-8-投票场景举例" class="headerlink" title="1.8 投票场景举例"></a>1.8 投票场景举例</h4><ul><li>ABC, C踢除B, A踢除C, B踢除C, A踢除B, 结果是剩下 AB</li><li>ABCD, AB先分别踢除CD, C踢除D, 即使C投给自己留下的票, 结果是剩下 AB</li><li>ABCDE, ABC先分别加入F(成功,ABCDEF), BCDE踢除F(成功,ABCDE), DE 加入F(失败,ABCDE), BCD踢除A(成功, BCDE), B加入F(此时BDE加入F,满足超过50%投票), 结果是剩下 BCDEF</li></ul><h3 id="1-9-PoA中的攻击及防御"><a href="#1-9-PoA中的攻击及防御" class="headerlink" title="1.9 PoA中的攻击及防御"></a>1.9 PoA中的攻击及防御</h3><ul><li>恶意签名者(Malicious signer). 恶意用户被添加到签名者列表中，或签名者密钥/机器遭到入侵. 解决方案是，N个授权签名人的列表，任一签名者只能对每K个 block 签名其中的1个。这样尽量减少损害，其余的矿工可以投票踢出恶意用户。</li><li>审查签名者(Censoring signer). 如果一个签名者（或一组签名者）试图检查 block 中其他 signer 的提议(特别是投票踢出他们), 为了解决这个问题，我们将签名者的允许的挖矿频率限制在1/(N/2)。如果他不想被踢出出去, 就必须控制超过50%的signers.</li><li>“垃圾邮件”签名者(Spamming signer). 这些 signer 在每个他们签名的 block 中都注入一个新的投票提议.由于节点需要统计所有投票以创建授权签名者列表, 久而久之之后会产生大量垃圾的无用的投票, 导致系统运行变慢.通过 epoch 的机制,每次进入新的epoch都会丢弃旧的投票</li><li>并发块(Concurrent blocks). 如果授权签名者的数量为N，我们允许每个签名者签名是1/K，那么在任何时候，至少N-K个签名者都可以成功签名一个 block。为了避免这些 block 竞争( <strong>分叉</strong> )，每个签名者生成一个新 block 时都会加一点随机延时。这确保了很难发生分叉。</li></ul><h2 id="2-PoA共识引擎算法实现分析"><a href="#2-PoA共识引擎算法实现分析" class="headerlink" title="2. PoA共识引擎算法实现分析"></a>2. PoA共识引擎算法实现分析</h2><p>节点：普通的以太坊节点，没有区块生成的权利。</p><p>矿工：具有区块生成权利的以太坊节点</p><p>委员会：所有矿工的集合</p><h3 id="投票方法"><a href="#投票方法" class="headerlink" title="投票方法"></a>投票方法</h3><p>所有投票都是在委员生成新区块的过程中完成，具体流程如下：</p><ul><li>1）委员生成新区块时，先为该区块初始化一个 header。（prepare 方法，consensus/clique/clique.go）</li><li>2）从 proposals 中随机获取一个投票，将被投票的节点地址写入 header.coinbase，将提名是添加还是删除写入 header.Nonce (添加：0xffffffffffffffff 删除：0)，若该委员生成的这个区块最终被写入区块链，则header 中的投票也被写入区块链。（ prepare 方法，consensus/clique/clique.go）</li><li>3）委员在生成新区块时，会创建新的 snapshot，新的 snapshot 是由上一 checkponitinterval 时间点存储到数据库中的快照加入当前时间点和 checkpointinterval 时间点之间所有的 headers 数据组成。添加 header 过程中，若该 header 的 number 是 Epoch 时间点，则会将 snap 中的 Votes 和 Tally 两个集合清零。（apply方法，consensus/clique/snapshot.go）</li><li>4）新的 snapshot 添加 header 过程中，会检查每一个 header 中存储的投票，若该投票 snap.Votes 中已经存在，则将 snap.Votes 和 snap.Tally 两个集合的该投票删除。（apply 方法，consensus/clique/snapshot.）将每一个 header 中有效的提名写入新snapshot的snap.Votes和snap.Tally集合。（apply方法，consensus/clique/snapshot.go）</li><li>5）判断snap.Tally集合中某个被提名的节点，提名的次数是否大于snap.Signers的1/2,即是否有超过一半的委员对该节点进行投票，若超过，则投票成功，该节点会被添加到委员会或者从委员会中删除。（apply方法，consensus/clique/snapshot.go）</li></ul><p><strong>注释：snapshot 快照中的记录的委员会，即 Signers 集合，初始化时来源于创世块 header 中的 Extra</strong></p><h3 id="clique-中一些概念和定义"><a href="#clique-中一些概念和定义" class="headerlink" title="clique 中一些概念和定义"></a>clique 中一些概念和定义</h3><ul><li><strong>EPOCH_LENGTH</strong> : epoch 长度是30000个 block, 每次进入新的epoch,前面的投票都被清空,重新开始记录,这里的投票是指加入或移除signer</li><li><strong>CheckpointInterval</strong>：为常量1024（consensus/clique/clique.go中定义），即每当区块链的高度为1024的整数倍时，到达checkpointInterval时间点。</li><li><strong>BLOCK_PERIOD</strong> : 出块时间, 默认是15s</li><li><strong>UNCLE_HASH</strong> : 总是 <code>Keccak256(RLP([]))</code> ,因为没有uncle</li><li><strong>SIGNER_COUNT</strong> : 每个block都有一个signers的数量</li><li><strong>SIGNER_LIMIT</strong> : 等于 <code>(SIGNER_COUNT / 2) + 1</code><ul><li>每个singer只能签名连续SIGNER_LIMIT个block中的1个, 比如有5个signer:ABCDE, 对4个block进行签名, 不允许签名者为ABAC, 因为A在连续3个block中签名了2次</li></ul></li><li><strong>NONCE_AUTH</strong> : 表示投票类型是加入新的signer; 值= <code>0xffffffffffffffff</code></li><li><strong>NONCE_DROP</strong> : 表示投票类型是踢除旧的的signer; 值= <code>0x0000000000000000</code></li><li><strong>EXTRA_VANITY</strong> : 代表block头中Extra字段中的保留字段长度: 32字节</li><li><strong>EXTRA_SEAL</strong> : 代表block头中Extra字段中的存储签名数据的长度: 65字节</li><li><strong>IN-TURN/OUT-OF-TURN</strong> : 每个block都有一个in-turn的signer, 其他signers是out-of-turn, in-turn的signer的权重大一些, 出块的时间会快一点, 这样可以保证该高度的block被in-turn的signer挖到的概率很大.</li></ul><p>clique中最重要的两个数据结构:</p><ul><li>共识引擎的结构:</li></ul><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Clique <span class="keyword">struct</span> {</div><div class="line">config *params.CliqueConfig <span class="comment">// 系统配置参数</span></div><div class="line">db ethdb.Database <span class="comment">// 数据库: 用于存取检查点快照</span></div><div class="line">recents *lru.ARCCache <span class="comment">//保存最近block的快照, 加速reorgs</span></div><div class="line">signatures *lru.ARCCache <span class="comment">//保存最近block的签名, 加速挖矿</span></div><div class="line">proposals <span class="keyword">map</span>[common.Address]<span class="keyword">bool</span> <span class="comment">//当前signer提出的proposals列表</span></div><div class="line">signer common.Address <span class="comment">// signer地址</span></div><div class="line">signFn SignerFn <span class="comment">// 签名函数</span></div><div class="line">lock sync.RWMutex <span class="comment">// 读写锁</span></div><div class="line">}</div></pre></td></tr></table></figure><p>用户通过RPC接口，调用Propose(address common.Address, auth bool)方法（consensus/clique/api.go）,进行投票，address表示要投票的节点的地址，auth表示要从将该地址加入委员会，还是从委员会中删除。</p><p>Propose 方法将 address 和 auth 两个输入参数写入到 clique.proposals 集合中。</p><p>任何一个委员会的委员，可以在任意时刻进行投票，投票包括两种，即加入委员会和从委员会中删除。</p><ul><li>snapshot的结构:</li></ul><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Snapshot <span class="keyword">struct</span> {</div><div class="line">config *params.CliqueConfig <span class="comment">// 系统配置参数</span></div><div class="line">sigcache *lru.ARCCache <span class="comment">// 保存最近block的签名缓存,加速ecrecover</span></div><div class="line">Number <span class="keyword">uint64</span> <span class="comment">// 创建快照时的block号,即生成快照时的区块链高度</span></div><div class="line">Hash common.Hash <span class="comment">// 创建快照时的block hash</span></div><div class="line">Signers <span class="keyword">map</span>[common.Address]<span class="keyword">struct</span>{} <span class="comment">// 此刻的授权的signers</span></div><div class="line">Recents <span class="keyword">map</span>[<span class="keyword">uint64</span>]common.Address <span class="comment">// 最近的一组signers, key=blockNumber</span></div><div class="line">Votes []*Vote <span class="comment">// 按时间顺序排列的投票列表</span></div><div class="line">Tally <span class="keyword">map</span>[common.Address]Tally <span class="comment">// 当前的投票计数，以避免重新计算，其中的Tally是该节点被投票的次数</span></div><div class="line">}</div></pre></td></tr></table></figure><p>Snapshot是一个快照，矿工程序在区块链高度为CheckpointInterval的整数倍时，会对当前相关数据和状态形成快照，并存储到数据库中。</p><p>除了这两个结构, 对block头的部分字段进行了复用定义, ethereum的block头定义:</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Header <span class="keyword">struct</span> {</div><div class="line">ParentHash common.Hash</div><div class="line">UncleHash common.Hash</div><div class="line">Coinbase common.Address</div><div class="line">Root common.Hash</div><div class="line">TxHash common.Hash</div><div class="line">ReceiptHash common.Hash</div><div class="line">Bloom Bloom</div><div class="line">Difficulty *big.Int</div><div class="line">Number *big.Int</div><div class="line">GasLimit *big.Int</div><div class="line">GasUsed *big.Int</div><div class="line">Time *big.Int</div><div class="line">Extra []<span class="keyword">byte</span></div><div class="line">MixDigest common.Hash</div><div class="line">Nonce BlockNonce</div><div class="line">}</div></pre></td></tr></table></figure><ul><li>创世块中的Extra字段包括:<ul><li>32字节的前缀(extraVanity)</li><li>所有signer的地址</li><li>65字节的后缀(extraSeal): 保存signer的签名</li></ul></li><li>其他block的Extra字段只包括extraVanity和extraSeal</li><li>Time字段表示产生block的时间间隔是:blockPeriod(15s)</li><li>Nonce字段表示进行一个投票: 添加( nonceAuthVote: <code>0xffffffffffffffff</code> )或者移除( nonceDropVote: <code>0x0000000000000000</code> )一个signer</li><li>Coinbase字段存放被投票的地址<ul><li>举个栗子: signerA的一个投票:加入signerB, 那么Coinbase存放B的地址</li></ul></li><li>Difficulty字段的值: 2-是 <strong>本block的签名者</strong> (in turn), 1- <strong>非本block的签名者</strong> (out of turn)</li></ul><p>POA共识算法中，委员会中的每一个矿工都会持续的生成新的区块，对于同一个Number的区块，不通的矿工生成该块时优先级不同。</p><p>优先级计算方法:</p><ul><li>Number:要生成的区块的块号</li><li>Signers：snapshot中记录的委员会集合，并根据矿工的地址进行了升序排列</li><li>Offset：矿工在Signers集合中的位置</li><li><p>若：(number % uint64(len(signers))) == uint64(offset)，则优先级最高，header. Difficulty =2;否则,header.Difficulty = 1</p><p>总结如下：</p></li></ul><table><thead><tr><th>序号</th><th>字段</th><th>POW</th><th>POA</th></tr></thead><tbody><tr><td>1</td><td>Coinbase</td><td>挖矿奖励地址</td><td>被提名为矿工的节点地址</td></tr><tr><td>2</td><td>Nonce</td><td>随机数</td><td>提名分类，添加或删除</td></tr><tr><td>3</td><td>Extra</td><td>其他数据</td><td>在Epoch时间点，存储当前委员会集合Signers</td></tr><tr><td>4</td><td>Difficulty</td><td>挖矿难度</td><td>优先级</td></tr><tr><td>5</td><td>Time</td><td>产生block的时间</td><td>产生block的时间间隔</td></tr></tbody></table><p>下面对比较重要的函数详细分析实现流程</p><h4 id="共识引擎-clique-的初始化"><a href="#共识引擎-clique-的初始化" class="headerlink" title="共识引擎 clique 的初始化"></a>共识引擎 clique 的初始化</h4><p>在 <code>Ethereum.StartMining</code> 中,如果Ethereum.engine配置为clique.Clique, 根据当前节点的矿工地址(默认是acounts[0]), 配置clique的 <strong>签名者</strong> : <code>clique.Authorize(eb, wallet.SignHash)</code> ,其中 <strong>签名函数</strong> 是SignHash,对给定的hash进行签名.</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// New creates a Clique proof-of-authority consensus engine with the initial</span></div><div class="line"><span class="comment">// signers set to the ones provided by the user.</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">New</span><span class="params">(config *params.CliqueConfig, db ethdb.Database)</span> *<span class="title">Clique</span></span> {</div><div class="line"><span class="comment">// Set any missing consensus parameters to their defaults</span></div><div class="line">conf := *config</div><div class="line"><span class="keyword">if</span> conf.Epoch == <span class="number">0</span> {</div><div class="line">conf.Epoch = epochLength</div><div class="line">}</div><div class="line"><span class="comment">// Allocate the snapshot caches and create the engine</span></div><div class="line">recents, _ := lru.NewARC(inmemorySnapshots)</div><div class="line">signatures, _ := lru.NewARC(inmemorySignatures)</div><div class="line"></div><div class="line"><span class="keyword">return</span> &Clique{</div><div class="line">config: &conf,</div><div class="line">db: db,</div><div class="line">recents: recents,</div><div class="line">signatures: signatures,</div><div class="line">proposals: <span class="built_in">make</span>(<span class="keyword">map</span>[common.Address]<span class="keyword">bool</span>),</div><div class="line">}</div><div class="line">}</div></pre></td></tr></table></figure><h4 id="Clique-Prepare-chain-header"><a href="#Clique-Prepare-chain-header" class="headerlink" title="Clique.Prepare(chain , header)"></a>Clique.Prepare(chain , header)</h4><p>Prepare是共识引擎接口之一. 该函数配置header中共识相关的参数(Cionbase, Difficulty, Extra, MixDigest, Time)</p><ul><li>对于非epoch的block( <code>number % Epoch != 0</code> ):</li></ul><ol><li>得到Clique.proposals中的投票数据(例:A加入C, B踢除D)</li><li>根据snapshot的signers分析投票数否有效(例: C原先没有在signers中, 加入投票有效, D原先在signers中,踢除投票有效)</li><li>从被投票的地址列表(C,D)中, <strong>随机选择一个地址</strong> ,作为该header的Coinbase,设置Nonce为加入( <code>0xffffffffffffffff</code> )或者踢除( <code>0x0000000000000000</code> )</li><li><code>Clique.signer</code> 如果是本轮的签名者(in-turn), 设置header.Difficulty = diffInTurn(2), 否则就是diffNoTurn(1)</li><li>配置header.Extra的数据为[ <code>extraVanity</code> + <code>snap中的全部signers</code> + <code>extraSeal</code> ]</li><li>MixDigest需要配置为nil</li><li>配置时间戳:Time为父块的时间+15s</li></ol><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// Prepare implements consensus.Engine, preparing all the consensus fields of the</span></div><div class="line"><span class="comment">// header for running the transactions on top.</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(c *Clique)</span> <span class="title">Prepare</span><span class="params">(chain consensus.ChainReader, header *types.Header)</span> <span class="title">error</span></span> {</div><div class="line"><span class="comment">// If the block isn't a checkpoint, cast a random vote (good enough for now)</span></div><div class="line">header.Coinbase = common.Address{}</div><div class="line">header.Nonce = types.BlockNonce{}</div><div class="line"></div><div class="line">number := header.Number.Uint64()</div><div class="line"><span class="comment">// Assemble the voting snapshot to check which votes make sense</span></div><div class="line">snap, err := c.snapshot(chain, number<span class="number">-1</span>, header.ParentHash, <span class="literal">nil</span>)</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> err</div><div class="line">}</div><div class="line"><span class="keyword">if</span> number%c.config.Epoch != <span class="number">0</span> {</div><div class="line">c.lock.RLock()</div><div class="line"></div><div class="line"><span class="comment">// Gather all the proposals that make sense voting on</span></div><div class="line">addresses := <span class="built_in">make</span>([]common.Address, <span class="number">0</span>, <span class="built_in">len</span>(c.proposals))</div><div class="line"><span class="keyword">for</span> address, authorize := <span class="keyword">range</span> c.proposals {</div><div class="line"><span class="keyword">if</span> snap.validVote(address, authorize) {</div><div class="line">addresses = <span class="built_in">append</span>(addresses, address)</div><div class="line">}</div><div class="line">}</div><div class="line"><span class="comment">// If there's pending proposals, cast a vote on them</span></div><div class="line"><span class="keyword">if</span> <span class="built_in">len</span>(addresses) > <span class="number">0</span> {</div><div class="line">header.Coinbase = addresses[rand.Intn(<span class="built_in">len</span>(addresses))]</div><div class="line"><span class="keyword">if</span> c.proposals[header.Coinbase] {</div><div class="line"><span class="built_in">copy</span>(header.Nonce[:], nonceAuthVote)</div><div class="line">} <span class="keyword">else</span> {</div><div class="line"><span class="built_in">copy</span>(header.Nonce[:], nonceDropVote)</div><div class="line">}</div><div class="line">}</div><div class="line">c.lock.RUnlock()</div><div class="line">}</div><div class="line"><span class="comment">// Set the correct difficulty</span></div><div class="line">header.Difficulty = CalcDifficulty(snap, c.signer)</div><div class="line"></div><div class="line"><span class="comment">// Ensure the extra data has all it's components</span></div><div class="line"><span class="keyword">if</span> <span class="built_in">len</span>(header.Extra) < extraVanity {</div><div class="line">header.Extra = <span class="built_in">append</span>(header.Extra, bytes.Repeat([]<span class="keyword">byte</span>{<span class="number">0x00</span>}, extraVanity-<span class="built_in">len</span>(header.Extra))...)</div><div class="line">}</div><div class="line">header.Extra = header.Extra[:extraVanity]</div><div class="line"></div><div class="line"><span class="keyword">if</span> number%c.config.Epoch == <span class="number">0</span> {</div><div class="line"><span class="keyword">for</span> _, signer := <span class="keyword">range</span> snap.signers() {</div><div class="line">header.Extra = <span class="built_in">append</span>(header.Extra, signer[:]...)</div><div class="line">}</div><div class="line">}</div><div class="line">header.Extra = <span class="built_in">append</span>(header.Extra, <span class="built_in">make</span>([]<span class="keyword">byte</span>, extraSeal)...)</div><div class="line"></div><div class="line"><span class="comment">// Mix digest is reserved for now, set to empty</span></div><div class="line">header.MixDigest = common.Hash{}</div><div class="line"></div><div class="line"><span class="comment">// Ensure the timestamp has the correct delay</span></div><div class="line">parent := chain.GetHeader(header.ParentHash, number<span class="number">-1</span>)</div><div class="line"><span class="keyword">if</span> parent == <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> consensus.ErrUnknownAncestor</div><div class="line">}</div><div class="line">header.Time = <span class="built_in">new</span>(big.Int).Add(parent.Time, <span class="built_in">new</span>(big.Int).SetUint64(c.config.Period))</div><div class="line"><span class="keyword">if</span> header.Time.Int64() < time.Now().Unix() {</div><div class="line">header.Time = big.NewInt(time.Now().Unix())</div><div class="line">}</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span></div><div class="line">}</div></pre></td></tr></table></figure><h4 id="获取给定时间点的一个快照-Clique-snapshot"><a href="#获取给定时间点的一个快照-Clique-snapshot" class="headerlink" title="获取给定时间点的一个快照 Clique.snapshot"></a>获取给定时间点的一个快照 Clique.snapshot</h4><ul><li><p>先查找 Clique.recents 中是否有缓存, 有的话就返回该 snapshot</p></li><li><p>在查找持久化存储中是否有缓存, 有的话就返回该 snapshot</p></li><li><p>如果是创世块</p><ol><li>从 Extra 中取出所有的 signers</li><li><code>newSnapshot(Clique.config, Clique.signatures, 0, genesis.Hash(), signers)</code></li></ol><ul><li>signatures 是最近的签名快照</li><li>signers 是所有的初始 signers</li></ul><p>把 snapshot 加入到 Clique.recents 中, 并持久化到 db 中</p></li><li><p>其他普通块</p><ul><li>沿着父块 hash 一直往回找是否有 snapshot, 如果没找到就记录该区块头</li><li>如果找到最近的 snapshot, 将前面记录的 headers 都 <code>applay</code> 到该 snapshot 上</li><li>保存该最新的 snapshot 到缓存 Clique.recents 中, 并持久化到 db 中</li></ul></li></ul><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// snapshot retrieves the authorization snapshot at a given point in time.</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(c *Clique)</span> <span class="title">snapshot</span><span class="params">(chain consensus.ChainReader, number <span class="keyword">uint64</span>, hash common.Hash, parents []*types.Header)</span> <span class="params">(*Snapshot, error)</span></span> {</div><div class="line"><span class="comment">// Search for a snapshot in memory or on disk for checkpoints</span></div><div class="line"><span class="keyword">var</span> (</div><div class="line">headers []*types.Header</div><div class="line">snap *Snapshot</div><div class="line">)</div><div class="line"><span class="keyword">for</span> snap == <span class="literal">nil</span> {</div><div class="line"><span class="comment">// If an in-memory snapshot was found, use that</span></div><div class="line"><span class="keyword">if</span> s, ok := c.recents.Get(hash); ok {</div><div class="line">snap = s.(*Snapshot)</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line"><span class="comment">// If an on-disk checkpoint snapshot can be found, use that</span></div><div class="line"><span class="keyword">if</span> number%checkpointInterval == <span class="number">0</span> {</div><div class="line"><span class="keyword">if</span> s, err := loadSnapshot(c.config, c.signatures, c.db, hash); err == <span class="literal">nil</span> {</div><div class="line">log.Trace(<span class="string">"Loaded voting snapshot form disk"</span>, <span class="string">"number"</span>, number, <span class="string">"hash"</span>, hash)</div><div class="line">snap = s</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line">}</div><div class="line"><span class="comment">// If we're at block zero, make a snapshot</span></div><div class="line"><span class="keyword">if</span> number == <span class="number">0</span> {</div><div class="line">genesis := chain.GetHeaderByNumber(<span class="number">0</span>)</div><div class="line"><span class="keyword">if</span> err := c.VerifyHeader(chain, genesis, <span class="literal">false</span>); err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, err</div><div class="line">}</div><div class="line">signers := <span class="built_in">make</span>([]common.Address, (<span class="built_in">len</span>(genesis.Extra)-extraVanity-extraSeal)/common.AddressLength)</div><div class="line"><span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="built_in">len</span>(signers); i++ {</div><div class="line"><span class="built_in">copy</span>(signers[i][:], genesis.Extra[extraVanity+i*common.AddressLength:])</div><div class="line">}</div><div class="line">snap = newSnapshot(c.config, c.signatures, <span class="number">0</span>, genesis.Hash(), signers)</div><div class="line"><span class="keyword">if</span> err := snap.store(c.db); err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, err</div><div class="line">}</div><div class="line">log.Trace(<span class="string">"Stored genesis voting snapshot to disk"</span>)</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line"><span class="comment">// No snapshot for this header, gather the header and move backward</span></div><div class="line"><span class="keyword">var</span> header *types.Header</div><div class="line"><span class="keyword">if</span> <span class="built_in">len</span>(parents) > <span class="number">0</span> {</div><div class="line"><span class="comment">// If we have explicit parents, pick from there (enforced)</span></div><div class="line">header = parents[<span class="built_in">len</span>(parents)<span class="number">-1</span>]</div><div class="line"><span class="keyword">if</span> header.Hash() != hash || header.Number.Uint64() != number {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, consensus.ErrUnknownAncestor</div><div class="line">}</div><div class="line">parents = parents[:<span class="built_in">len</span>(parents)<span class="number">-1</span>]</div><div class="line">} <span class="keyword">else</span> {</div><div class="line"><span class="comment">// No explicit parents (or no more left), reach out to the database</span></div><div class="line">header = chain.GetHeader(hash, number)</div><div class="line"><span class="keyword">if</span> header == <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, consensus.ErrUnknownAncestor</div><div class="line">}</div><div class="line">}</div><div class="line">headers = <span class="built_in">append</span>(headers, header)</div><div class="line">number, hash = number<span class="number">-1</span>, header.ParentHash</div><div class="line">}</div><div class="line"><span class="comment">// Previous snapshot found, apply any pending headers on top of it</span></div><div class="line"><span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="built_in">len</span>(headers)/<span class="number">2</span>; i++ {</div><div class="line">headers[i], headers[<span class="built_in">len</span>(headers)<span class="number">-1</span>-i] = headers[<span class="built_in">len</span>(headers)<span class="number">-1</span>-i], headers[i]</div><div class="line">}</div><div class="line">snap, err := snap.apply(headers)</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, err</div><div class="line">}</div><div class="line">c.recents.Add(snap.Hash, snap)</div><div class="line"></div><div class="line"><span class="comment">// If we've generated a new checkpoint snapshot, save to disk</span></div><div class="line"><span class="keyword">if</span> snap.Number%checkpointInterval == <span class="number">0</span> && <span class="built_in">len</span>(headers) > <span class="number">0</span> {</div><div class="line"><span class="keyword">if</span> err = snap.store(c.db); err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, err</div><div class="line">}</div><div class="line">log.Trace(<span class="string">"Stored voting snapshot to disk"</span>, <span class="string">"number"</span>, snap.Number, <span class="string">"hash"</span>, snap.Hash)</div><div class="line">}</div><div class="line"><span class="keyword">return</span> snap, err</div><div class="line">}</div></pre></td></tr></table></figure><h4 id="Snapshot-apply-headers"><a href="#Snapshot-apply-headers" class="headerlink" title="Snapshot.apply(headers)"></a>Snapshot.apply(headers)</h4><p>创建一个新的授权signers的快照, 将从上一个snapshot开始的区块头中的proposals更新到最新的snapshot上</p><ol><li>对入参headers进行完整性检查: 因为可能传入多个区块头, <strong>block号必须连续</strong></li><li>遍历所有的header, 如果block号刚好处于epoch的起始(number%Epoch == 0),将snapshot中的Votes和Tally复位( <strong>丢弃历史全部数据</strong> )</li><li>对于每一个header,从签名中恢复得到 <strong>signer</strong></li><li>如果该signer在snap.Recents中, 说明 <strong>最近已经有过签名</strong> , 不允许再次签名, 直接 <strong>返回</strong> 结束</li><li>记录该signer最近已经有过签名，且是该block的签名者: <code>snap.Recents[number] = signer</code></li><li>统计header.Coinbase的投票数,如果超过signers总数的50%，执行加入或移除操作</li><li>删除snap.Recents中的一个signer记录: key=number- (uint64(len(snap.Signers)/2 + 1)), 表示释放该signer,下次可以对block进行签名了</li><li>清空被移除的Coinbase的投票</li><li>移除snap.Votes中该Conibase的所有投票记录</li><li>移除snap.Tally中该Conibase的所有投票数记录</li></ol><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// apply creates a new authorization snapshot by applying the given headers to</span></div><div class="line"><span class="comment">// the original one.</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Snapshot)</span> <span class="title">apply</span><span class="params">(headers []*types.Header)</span> <span class="params">(*Snapshot, error)</span></span> {</div><div class="line"><span class="comment">// Allow passing in no headers for cleaner code</span></div><div class="line"><span class="keyword">if</span> <span class="built_in">len</span>(headers) == <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> s, <span class="literal">nil</span></div><div class="line">}</div><div class="line"><span class="comment">// Sanity check that the headers can be applied</span></div><div class="line"><span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="built_in">len</span>(headers)<span class="number">-1</span>; i++ {</div><div class="line"><span class="keyword">if</span> headers[i+<span class="number">1</span>].Number.Uint64() != headers[i].Number.Uint64()+<span class="number">1</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, errInvalidVotingChain</div><div class="line">}</div><div class="line">}</div><div class="line"><span class="keyword">if</span> headers[<span class="number">0</span>].Number.Uint64() != s.Number+<span class="number">1</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, errInvalidVotingChain</div><div class="line">}</div><div class="line"><span class="comment">// Iterate through the headers and create a new snapshot</span></div><div class="line">snap := s.<span class="built_in">copy</span>()</div><div class="line"></div><div class="line"><span class="keyword">for</span> _, header := <span class="keyword">range</span> headers {</div><div class="line"><span class="comment">// Remove any votes on checkpoint blocks</span></div><div class="line">number := header.Number.Uint64()</div><div class="line"><span class="keyword">if</span> number%s.config.Epoch == <span class="number">0</span> {</div><div class="line">snap.Votes = <span class="literal">nil</span></div><div class="line">snap.Tally = <span class="built_in">make</span>(<span class="keyword">map</span>[common.Address]Tally)</div><div class="line">}</div><div class="line"><span class="comment">// Delete the oldest signer from the recent list to allow it signing again</span></div><div class="line"><span class="keyword">if</span> limit := <span class="keyword">uint64</span>(<span class="built_in">len</span>(snap.Signers)/<span class="number">2</span> + <span class="number">1</span>); number >= limit {</div><div class="line"><span class="built_in">delete</span>(snap.Recents, number-limit)</div><div class="line">}</div><div class="line"><span class="comment">// Resolve the authorization key and check against signers</span></div><div class="line">signer, err := ecrecover(header, s.sigcache)</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, err</div><div class="line">}</div><div class="line"><span class="keyword">if</span> _, ok := snap.Signers[signer]; !ok {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, errUnauthorized</div><div class="line">}</div><div class="line"><span class="keyword">for</span> _, recent := <span class="keyword">range</span> snap.Recents {</div><div class="line"><span class="keyword">if</span> recent == signer {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, errUnauthorized</div><div class="line">}</div><div class="line">}</div><div class="line">snap.Recents[number] = signer</div><div class="line"></div><div class="line"><span class="comment">// Header authorized, discard any previous votes from the signer</span></div><div class="line"><span class="keyword">for</span> i, vote := <span class="keyword">range</span> snap.Votes {</div><div class="line"><span class="keyword">if</span> vote.Signer == signer && vote.Address == header.Coinbase {</div><div class="line"><span class="comment">// Uncast the vote from the cached tally</span></div><div class="line">snap.uncast(vote.Address, vote.Authorize)</div><div class="line"></div><div class="line"><span class="comment">// Uncast the vote from the chronological list</span></div><div class="line">snap.Votes = <span class="built_in">append</span>(snap.Votes[:i], snap.Votes[i+<span class="number">1</span>:]...)</div><div class="line"><span class="keyword">break</span> <span class="comment">// only one vote allowed</span></div><div class="line">}</div><div class="line">}</div><div class="line"><span class="comment">// Tally up the new vote from the signer</span></div><div class="line"><span class="keyword">var</span> authorize <span class="keyword">bool</span></div><div class="line"><span class="keyword">switch</span> {</div><div class="line"><span class="keyword">case</span> bytes.Equal(header.Nonce[:], nonceAuthVote):</div><div class="line">authorize = <span class="literal">true</span></div><div class="line"><span class="keyword">case</span> bytes.Equal(header.Nonce[:], nonceDropVote):</div><div class="line">authorize = <span class="literal">false</span></div><div class="line"><span class="keyword">default</span>:</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, errInvalidVote</div><div class="line">}</div><div class="line"><span class="keyword">if</span> snap.cast(header.Coinbase, authorize) {</div><div class="line">snap.Votes = <span class="built_in">append</span>(snap.Votes, &Vote{</div><div class="line">Signer: signer,</div><div class="line">Block: number,</div><div class="line">Address: header.Coinbase,</div><div class="line">Authorize: authorize,</div><div class="line">})</div><div class="line">}</div><div class="line"><span class="comment">// If the vote passed, update the list of signers</span></div><div class="line"><span class="keyword">if</span> tally := snap.Tally[header.Coinbase]; tally.Votes > <span class="built_in">len</span>(snap.Signers)/<span class="number">2</span> {</div><div class="line"><span class="keyword">if</span> tally.Authorize {</div><div class="line">snap.Signers[header.Coinbase] = <span class="keyword">struct</span>{}{}</div><div class="line">} <span class="keyword">else</span> {</div><div class="line"><span class="built_in">delete</span>(snap.Signers, header.Coinbase)</div><div class="line"></div><div class="line"><span class="comment">// Signer list shrunk, delete any leftover recent caches</span></div><div class="line"><span class="keyword">if</span> limit := <span class="keyword">uint64</span>(<span class="built_in">len</span>(snap.Signers)/<span class="number">2</span> + <span class="number">1</span>); number >= limit {</div><div class="line"><span class="built_in">delete</span>(snap.Recents, number-limit)</div><div class="line">}</div><div class="line"><span class="comment">// Discard any previous votes the deauthorized signer cast</span></div><div class="line"><span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="built_in">len</span>(snap.Votes); i++ {</div><div class="line"><span class="keyword">if</span> snap.Votes[i].Signer == header.Coinbase {</div><div class="line"><span class="comment">// Uncast the vote from the cached tally</span></div><div class="line">snap.uncast(snap.Votes[i].Address, snap.Votes[i].Authorize)</div><div class="line"></div><div class="line"><span class="comment">// Uncast the vote from the chronological list</span></div><div class="line">snap.Votes = <span class="built_in">append</span>(snap.Votes[:i], snap.Votes[i+<span class="number">1</span>:]...)</div><div class="line"></div><div class="line">i--</div><div class="line">}</div><div class="line">}</div><div class="line">}</div><div class="line"><span class="comment">// Discard any previous votes around the just changed account</span></div><div class="line"><span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="built_in">len</span>(snap.Votes); i++ {</div><div class="line"><span class="keyword">if</span> snap.Votes[i].Address == header.Coinbase {</div><div class="line">snap.Votes = <span class="built_in">append</span>(snap.Votes[:i], snap.Votes[i+<span class="number">1</span>:]...)</div><div class="line">i--</div><div class="line">}</div><div class="line">}</div><div class="line"><span class="built_in">delete</span>(snap.Tally, header.Coinbase)</div><div class="line">}</div><div class="line">}</div><div class="line">snap.Number += <span class="keyword">uint64</span>(<span class="built_in">len</span>(headers))</div><div class="line">snap.Hash = headers[<span class="built_in">len</span>(headers)<span class="number">-1</span>].Hash()</div><div class="line"></div><div class="line"><span class="keyword">return</span> snap, <span class="literal">nil</span></div><div class="line">}</div></pre></td></tr></table></figure><h4 id="Clique-Seal-chain-block-stop"><a href="#Clique-Seal-chain-block-stop" class="headerlink" title="Clique.Seal(chain, block , stop)"></a>Clique.Seal(chain, block , stop)</h4><p>Seal也是共识引擎接口之一. 该函数用clique.signer对block的进行签名. 在pow算法中, 该函数进行hash运算来解”难题”.</p><ul><li>如果signer没有在snapshot的signers中,不允许对block进行签名</li><li>如果不是本block的签名者,延时一定的时间(随机)后再签名, 如果是本block的签名者, 立即签名.</li><li>签名结果放在Extra的extraSeal的65字节中</li></ul><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// Seal implements consensus.Engine, attempting to create a sealed block using</span></div><div class="line"><span class="comment">// the local signing credentials.</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(c *Clique)</span> <span class="title">Seal</span><span class="params">(chain consensus.ChainReader, block *types.Block, stop <-<span class="keyword">chan</span> <span class="keyword">struct</span>{})</span> <span class="params">(*types.Block, error)</span></span> {</div><div class="line">header := block.Header()</div><div class="line"></div><div class="line"><span class="comment">// Sealing the genesis block is not supported</span></div><div class="line">number := header.Number.Uint64()</div><div class="line"><span class="keyword">if</span> number == <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, errUnknownBlock</div><div class="line">}</div><div class="line"><span class="comment">// For 0-period chains, refuse to seal empty blocks (no reward but would spin sealing)</span></div><div class="line"><span class="keyword">if</span> c.config.Period == <span class="number">0</span> && <span class="built_in">len</span>(block.Transactions()) == <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, errWaitTransactions</div><div class="line">}</div><div class="line"><span class="comment">// Don't hold the signer fields for the entire sealing procedure</span></div><div class="line">c.lock.RLock()</div><div class="line">signer, signFn := c.signer, c.signFn</div><div class="line">c.lock.RUnlock()</div><div class="line"></div><div class="line"><span class="comment">// Bail out if we're unauthorized to sign a block</span></div><div class="line">snap, err := c.snapshot(chain, number<span class="number">-1</span>, header.ParentHash, <span class="literal">nil</span>)</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, err</div><div class="line">}</div><div class="line"><span class="keyword">if</span> _, authorized := snap.Signers[signer]; !authorized {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, errUnauthorized</div><div class="line">}</div><div class="line"><span class="comment">// If we're amongst the recent signers, wait for the next block</span></div><div class="line"><span class="keyword">for</span> seen, recent := <span class="keyword">range</span> snap.Recents {</div><div class="line"><span class="keyword">if</span> recent == signer {</div><div class="line"><span class="comment">// Signer is among recents, only wait if the current block doesn't shift it out</span></div><div class="line"><span class="keyword">if</span> limit := <span class="keyword">uint64</span>(<span class="built_in">len</span>(snap.Signers)/<span class="number">2</span> + <span class="number">1</span>); number < limit || seen > number-limit {</div><div class="line">log.Info(<span class="string">"Signed recently, must wait for others"</span>)</div><div class="line"><-stop</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, <span class="literal">nil</span></div><div class="line">}</div><div class="line">}</div><div class="line">}</div><div class="line"><span class="comment">// Sweet, the protocol permits us to sign the block, wait for our time</span></div><div class="line">delay := time.Unix(header.Time.Int64(), <span class="number">0</span>).Sub(time.Now()) <span class="comment">// nolint: gosimple</span></div><div class="line"><span class="keyword">if</span> header.Difficulty.Cmp(diffNoTurn) == <span class="number">0</span> {</div><div class="line"><span class="comment">// It's not our turn explicitly to sign, delay it a bit</span></div><div class="line">wiggle := time.Duration(<span class="built_in">len</span>(snap.Signers)/<span class="number">2</span>+<span class="number">1</span>) * wiggleTime</div><div class="line">delay += time.Duration(rand.Int63n(<span class="keyword">int64</span>(wiggle)))</div><div class="line"></div><div class="line">log.Trace(<span class="string">"Out-of-turn signing requested"</span>, <span class="string">"wiggle"</span>, common.PrettyDuration(wiggle))</div><div class="line">}</div><div class="line">log.Trace(<span class="string">"Waiting for slot to sign and propagate"</span>, <span class="string">"delay"</span>, common.PrettyDuration(delay))</div><div class="line"></div><div class="line"><span class="keyword">select</span> {</div><div class="line"><span class="keyword">case</span> <-stop:</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, <span class="literal">nil</span></div><div class="line"><span class="keyword">case</span> <-time.After(delay):</div><div class="line">}</div><div class="line"><span class="comment">// Sign all the things!</span></div><div class="line">sighash, err := signFn(accounts.Account{Address: signer}, sigHash(header).Bytes())</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, err</div><div class="line">}</div><div class="line"><span class="built_in">copy</span>(header.Extra[<span class="built_in">len</span>(header.Extra)-extraSeal:], sighash)</div><div class="line"></div><div class="line"><span class="keyword">return</span> block.WithSeal(header), <span class="literal">nil</span></div><div class="line">}</div></pre></td></tr></table></figure><h4 id="Clique-VerifySeal-chain-header"><a href="#Clique-VerifySeal-chain-header" class="headerlink" title="Clique.VerifySeal(chain, header)"></a>Clique.VerifySeal(chain, header)</h4><p>VerifySeal也是共识引擎接口之一.</p><ol><li>从header的签名中恢复账户地址,改地址要求在snapshot的signers中</li><li>检查header中的Difficulty是否匹配(in turn或out of turn)</li></ol><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// VerifySeal implements consensus.Engine, checking whether the signature contained</span></div><div class="line"><span class="comment">// in the header satisfies the consensus protocol requirements.</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(c *Clique)</span> <span class="title">VerifySeal</span><span class="params">(chain consensus.ChainReader, header *types.Header)</span> <span class="title">error</span></span> {</div><div class="line"><span class="keyword">return</span> c.verifySeal(chain, header, <span class="literal">nil</span>)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// verifySeal checks whether the signature contained in the header satisfies the</span></div><div class="line"><span class="comment">// consensus protocol requirements. The method accepts an optional list of parent</span></div><div class="line"><span class="comment">// headers that aren't yet part of the local blockchain to generate the snapshots</span></div><div class="line"><span class="comment">// from.</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(c *Clique)</span> <span class="title">verifySeal</span><span class="params">(chain consensus.ChainReader, header *types.Header, parents []*types.Header)</span> <span class="title">error</span></span> {</div><div class="line"><span class="comment">// Verifying the genesis block is not supported</span></div><div class="line">number := header.Number.Uint64()</div><div class="line"><span class="keyword">if</span> number == <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> errUnknownBlock</div><div class="line">}</div><div class="line"><span class="comment">// Retrieve the snapshot needed to verify this header and cache it</span></div><div class="line">snap, err := c.snapshot(chain, number<span class="number">-1</span>, header.ParentHash, parents)</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> err</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Resolve the authorization key and check against signers</span></div><div class="line">signer, err := ecrecover(header, c.signatures)</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> err</div><div class="line">}</div><div class="line"><span class="keyword">if</span> _, ok := snap.Signers[signer]; !ok {</div><div class="line"><span class="keyword">return</span> errUnauthorized</div><div class="line">}</div><div class="line"><span class="keyword">for</span> seen, recent := <span class="keyword">range</span> snap.Recents {</div><div class="line"><span class="keyword">if</span> recent == signer {</div><div class="line"><span class="comment">// Signer is among recents, only fail if the current block doesn't shift it out</span></div><div class="line"><span class="keyword">if</span> limit := <span class="keyword">uint64</span>(<span class="built_in">len</span>(snap.Signers)/<span class="number">2</span> + <span class="number">1</span>); seen > number-limit {</div><div class="line"><span class="keyword">return</span> errUnauthorized</div><div class="line">}</div><div class="line">}</div><div class="line">}</div><div class="line"><span class="comment">// Ensure that the difficulty corresponds to the turn-ness of the signer</span></div><div class="line">inturn := snap.inturn(header.Number.Uint64(), signer)</div><div class="line"><span class="keyword">if</span> inturn && header.Difficulty.Cmp(diffInTurn) != <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> errInvalidDifficulty</div><div class="line">}</div><div class="line"><span class="keyword">if</span> !inturn && header.Difficulty.Cmp(diffNoTurn) != <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> errInvalidDifficulty</div><div class="line">}</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span></div><div class="line">}</div></pre></td></tr></table></figure><h4 id="Clique-Finalize"><a href="#Clique-Finalize" class="headerlink" title="Clique.Finalize"></a>Clique.Finalize</h4><p>Finalize也是共识引擎接口之一. 该函数生成一个block, 没有叔块处理,也没有奖励机制</p><ol><li><code>header.Root</code> : 状态根保持原状</li><li><code>header.UncleHash</code> : 为nil</li><li><code>types.NewBlock(header, txs, nil, receipts)</code> : 封装并返回最终的block</li></ol><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// Finalize implements consensus.Engine, ensuring no uncles are set, nor block</span></div><div class="line"><span class="comment">// rewards given, and returns the final block.</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(c *Clique)</span> <span class="title">Finalize</span><span class="params">(chain consensus.ChainReader, header *types.Header, state *state.StateDB, txs []*types.Transaction, uncles []*types.Header, receipts []*types.Receipt)</span> <span class="params">(*types.Block, error)</span></span> {</div><div class="line"><span class="comment">// No block rewards in PoA, so the state remains as is and uncles are dropped</span></div><div class="line">header.Root = state.IntermediateRoot(chain.Config().IsEIP158(header.Number))</div><div class="line">header.UncleHash = types.CalcUncleHash(<span class="literal">nil</span>)</div><div class="line"></div><div class="line"><span class="comment">// Assemble and return the final block for sealing</span></div><div class="line"><span class="keyword">return</span> types.NewBlock(header, txs, <span class="literal">nil</span>, receipts), <span class="literal">nil</span></div><div class="line">}</div></pre></td></tr></table></figure><h4 id="API-Propose-addr-auth"><a href="#API-Propose-addr-auth" class="headerlink" title="API.Propose(addr, auth)"></a>API.Propose(addr, auth)</h4><p>添加一个proposal: 调用者对addr的投票, auth表示加入还是踢出</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// Propose injects a new authorization proposal that the signer will attempt to</span></div><div class="line"><span class="comment">// push through.</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(api *API)</span> <span class="title">Propose</span><span class="params">(address common.Address, auth <span class="keyword">bool</span>)</span></span> {</div><div class="line">api.clique.lock.Lock()</div><div class="line"><span class="keyword">defer</span> api.clique.lock.Unlock()</div><div class="line"></div><div class="line">api.clique.proposals[address] = auth</div><div class="line">}</div></pre></td></tr></table></figure><h4 id="API-Discard-addr"><a href="#API-Discard-addr" class="headerlink" title="API.Discard(addr)"></a>API.Discard(addr)</h4><p>删除一个proposal</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// Discard drops a currently running proposal, stopping the signer from casting</span></div><div class="line"><span class="comment">// further votes (either for or against).</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(api *API)</span> <span class="title">Discard</span><span class="params">(address common.Address)</span></span> {</div><div class="line">api.clique.lock.Lock()</div><div class="line"><span class="keyword">defer</span> api.clique.lock.Unlock()</div><div class="line"></div><div class="line"><span class="built_in">delete</span>(api.clique.proposals, address)</div><div class="line">}</div></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h2 id="1-PoA-理论介绍"><a href="#1-PoA-理论介绍" class="headerlink" title="1. PoA 理论介绍"></a>1. PoA 理论介绍</h2><h3 id="1-1-以太坊中-PoA-产生的背景"><a href="#1-1-以太坊中-PoA-产生的背景" class="headerlink" title="1.1 以太坊中 PoA 产生的背景"></a>1.1 以太坊中 PoA 产生的背景</h3><p>如果你想用以太坊搭建一个联盟/私有链, 并要求该链交易成本更低甚至没有, 交易延时更低,并发更高, 还拥有完全的控制权(意味着被攻击概率更低). 目前以太坊采用 PoW 或后续的 casper 能否满足要求?<br></p>
</summary>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/categories/BlockChain/"/>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/tags/BlockChain/"/>
</entry>
<entry>
<title>mine</title>
<link href="https://yangchenglong11.github.io/2018/03/26/mine/"/>
<id>https://yangchenglong11.github.io/2018/03/26/mine/</id>
<published>2018-03-26T09:16:21.000Z</published>
<updated>2018-07-10T09:23:29.403Z</updated>
<content type="html"><![CDATA[<p>前几篇分别介绍了以太坊的基本概念，基本环节-交易，区块、区块链的存储方式等，这篇打算介绍一下“挖矿“得到新区块的整个过程，然后下一篇讲下不同共识算法的实现细节。<br><a id="more"></a></p><h2 id="总览"><a href="#总览" class="headerlink" title="总览"></a>总览</h2><p>在Ethereum 代码中，名为miner的包(package)负责向外提供一个“挖矿”得到的新区块，其主要结构体的UML关系图如下图所示：</p><p><img src="/2018/03/26/mine/miner.png" alt="img"></p><p>处于入口的类是Miner，它作为公共类型，向外暴露mine功能；它有一个worker类型的成员变量，负责管理mine过程；worker内部有一组Agent接口类型对象，这个接口有两个实现:CpuAgent和RemoteAgent。这里使用的是CpuAgent，该Agent会完成一个块的出块工作，同级的多个Agent是竞争关系，最终通过共识算法完成出一个块的工作。Work结构体主要用以携带数据，被视为挖掘一个区块时所需的数据环境。</p><p>主要的数据传输发生在worker和它的Agent(们)之间：在合适的时候，worker把一个Work对象发送给每个Agent，然后任何一个Agent完成mine时，将一个经过授权确认的Block加上那个更新过的Work，组成一个Result对象发送回worker。</p><p>调用方worker内部声明了一个Agent数组，但目前只有一个实现类CpuAgent的对象会被加到该数组。CpuAgent通过全局的<<engine>>对象，借助共识算法完成最终的区块授权。</engine></p><p>另外，unconfirmedBlocks 也挺特别，它会以unconfirmedBlock的形式存储最近一些本地挖掘出的区块。在一段时间之后，根据区块的Number和Hash，再确定这些区块是否已经被收纳进主干链(canonical chain)里，以输出Log的方式来告知用户。</p><p>对于一个新区块被挖掘出的过程，代码实现上基本分为两个环节：一是组装出一个新区块，这个区块的数据基本完整，包括成员Header的部分属性，以及交易列表txs，和叔区块组uncles[]，并且所有交易已经执行完毕，所有收据(Receipt)也已收集完毕，这部分主要由worker完成；二是填补该区块剩余的成员属性，比如Header.Difficulty等，并完成授权，这些工作是由Agent调用<engine>接口实现体，利用共识算法来完成的。</engine></p><h2 id="新区块的组装流程"><a href="#新区块的组装流程" class="headerlink" title="新区块的组装流程"></a>新区块的组装流程</h2><p>挖掘新区块的流程入口在Miner里，具体入口在Miner结构体的创建函数(避免称之为‘构造函数’)里。</p><p><img src="/2018/03/26/mine/new_miner.png" alt="img"></p><h3 id="Miner的函数"><a href="#Miner的函数" class="headerlink" title="Miner的函数"></a>Miner的函数</h3><p>####New()</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">New</span><span class="params">(eth Backend, config *params.ChainConfig, mux *event.TypeMux, engine consensus.Engine)</span> *<span class="title">Miner</span></span> {</div><div class="line"> miner := &Miner{</div><div class="line"> eth: eth,</div><div class="line"> mux: mux,</div><div class="line"> engine: engine,</div><div class="line"> worker: newWorker(config, engine, common.Address{}, eth, mux),</div><div class="line"> canStart: <span class="number">1</span>,</div><div class="line"> }</div><div class="line"> miner.Register(NewCpuAgent(eth.BlockChain(), engine))</div><div class="line"> <span class="keyword">go</span> miner.update()</div><div class="line"></div><div class="line"> <span class="keyword">return</span> miner</div><div class="line">}</div></pre></td></tr></table></figure><p>在New()里，针对新对象miner的各个成员变量初始化完成后，会紧跟着创建worker对象，然后将Agent对象登记给worker，最后用一个单独线程去运行miner.Update()函数；</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">newWorker</span><span class="params">(config *params.ChainConfig, engine consensus.Engine, coinbase common.Address, eth Backend, mux *event.TypeMux)</span> *<span class="title">worker</span></span> {</div><div class="line"> worker := &worker{</div><div class="line"> config: config,</div><div class="line"> engine: engine,</div><div class="line"> eth: eth,</div><div class="line"> mux: mux,</div><div class="line"> txCh: <span class="built_in">make</span>(<span class="keyword">chan</span> core.TxPreEvent, txChanSize),<span class="comment">// TxPreEvent事件是TxPool发出的事件，代表一个新交易tx加入到了交易池中，这时候如果work空闲会将该笔交易收进work.txs，准备下一次打包进块。</span></div><div class="line"> chainHeadCh: <span class="built_in">make</span>(<span class="keyword">chan</span> core.ChainHeadEvent, chainHeadChanSize),<span class="comment">// ChainHeadEvent事件，代表已经有一个块作为链头，此时work.update函数会监听到这个事件，则会继续挖新的区块。</span></div><div class="line"> chainSideCh: <span class="built_in">make</span>(<span class="keyword">chan</span> core.ChainSideEvent, chainSideChanSize),<span class="comment">// ChainSideEvent事件，代表有一个新块作为链的旁支，会被放到possibleUncles数组中，可能称为叔块。</span></div><div class="line"> chainDb: eth.ChainDb(),<span class="comment">// 区块链数据库</span></div><div class="line"> recv: <span class="built_in">make</span>(<span class="keyword">chan</span> *Result, resultQueueSize),</div><div class="line"> chain: eth.BlockChain(), <span class="comment">// 链</span></div><div class="line"> proc: eth.BlockChain().Validator(),</div><div class="line"> possibleUncles: <span class="built_in">make</span>(<span class="keyword">map</span>[common.Hash]*types.Block),<span class="comment">// 存放可能称为下一个块的叔块数组</span></div><div class="line"> coinbase: coinbase,</div><div class="line"> agents: <span class="built_in">make</span>(<span class="keyword">map</span>[Agent]<span class="keyword">struct</span>{}),</div><div class="line"> unconfirmed: newUnconfirmedBlocks(eth.BlockChain(), miningLogAtDepth),<span class="comment">// 返回一个数据结构，包括追踪当前未被确认的区块。</span></div><div class="line"> }</div><div class="line"> <span class="comment">// 注册TxPreEvent事件到tx pool交易池</span></div><div class="line"> worker.txSub = eth.TxPool().SubscribeTxPreEvent(worker.txCh)</div><div class="line"> <span class="comment">// 注册事件到blockchain</span></div><div class="line"> worker.chainHeadSub = eth.BlockChain().SubscribeChainHeadEvent(worker.chainHeadCh)</div><div class="line"> worker.chainSideSub = eth.BlockChain().SubscribeChainSideEvent(worker.chainSideCh)</div><div class="line"> <span class="keyword">go</span> worker.update()</div><div class="line"></div><div class="line"> <span class="keyword">go</span> worker.wait()</div><div class="line"> worker.commitNewWork()</div><div class="line"></div><div class="line"> <span class="keyword">return</span> worker</div><div class="line">}</div></pre></td></tr></table></figure><p>worker的创建函数里也如法炮制，分别用单独线程去启动worker.updater()和wait()；最后worker.CommitNewWork()会开始准备一个新区块所需的基本数据，如Header，Txs, Uncles等。注意此时Agent尚未启动。</p><p>####Miner.Update()</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// update方法可以保持对下载事件的监听，请了解这是一段短型的update循环。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(self *Miner)</span> <span class="title">update</span><span class="params">()</span></span> {</div><div class="line"> <span class="comment">// 注册下载开始事件，下载结束事件，下载失败事件。</span></div><div class="line"> events := self.mux.Subscribe(downloader.StartEvent{}, downloader.DoneEvent{}, downloader.FailedEvent{})</div><div class="line">out:</div><div class="line"> <span class="keyword">for</span> ev := <span class="keyword">range</span> events.Chan() {</div><div class="line"> <span class="keyword">switch</span> ev.Data.(<span class="keyword">type</span>) {</div><div class="line"> <span class="keyword">case</span> downloader.StartEvent:</div><div class="line"> atomic.StoreInt32(&self.canStart, <span class="number">0</span>)</div><div class="line"> <span class="keyword">if</span> self.Mining() {</div><div class="line"> self.Stop()</div><div class="line"> atomic.StoreInt32(&self.shouldStart, <span class="number">1</span>)</div><div class="line"> log.Info(<span class="string">"Mining aborted due to sync"</span>)</div><div class="line"> }</div><div class="line"> <span class="keyword">case</span> downloader.DoneEvent, downloader.FailedEvent: <span class="comment">// 下载完成和失败都走相同的分支。</span></div><div class="line"> shouldStart := atomic.LoadInt32(&self.shouldStart) == <span class="number">1</span></div><div class="line"></div><div class="line"> atomic.StoreInt32(&self.canStart, <span class="number">1</span>)</div><div class="line"> atomic.StoreInt32(&self.shouldStart, <span class="number">0</span>)</div><div class="line"> <span class="keyword">if</span> shouldStart {</div><div class="line"> self.Start(self.coinbase) <span class="comment">// 执行Miner的start方法。</span></div><div class="line"> }</div><div class="line"> <span class="comment">// 处理完以后要取消订阅</span></div><div class="line"> events.Unsubscribe()</div><div class="line"> <span class="comment">// 跳出循环，不再监听</span></div><div class="line"> <span class="keyword">break</span> out</div><div class="line"> }</div><div class="line"> }</div><div class="line">}</div></pre></td></tr></table></figure><p>####Miner.Start()</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(self *Miner)</span> <span class="title">Start</span><span class="params">(coinbase common.Address)</span></span> {</div><div class="line">atomic.StoreInt32(&self.shouldStart, <span class="number">1</span>)</div><div class="line">self.SetEtherbase(coinbase)</div><div class="line"></div><div class="line"><span class="keyword">if</span> atomic.LoadInt32(&self.canStart) == <span class="number">0</span> {</div><div class="line">log.Info(<span class="string">"Network syncing, will start miner afterwards"</span>)</div><div class="line"><span class="keyword">return</span></div><div class="line">}</div><div class="line">atomic.StoreInt32(&self.mining, <span class="number">1</span>)</div><div class="line"></div><div class="line">log.Info(<span class="string">"Starting mining operation"</span>)</div><div class="line">self.worker.start()</div><div class="line">self.worker.commitNewWork()</div><div class="line">}</div></pre></td></tr></table></figure><p>####worker.start()</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(self *worker)</span> <span class="title">start</span><span class="params">()</span></span> {</div><div class="line">self.mu.Lock()</div><div class="line"><span class="keyword">defer</span> self.mu.Unlock()</div><div class="line"></div><div class="line">atomic.StoreInt32(&self.mining, <span class="number">1</span>)</div><div class="line"></div><div class="line"><span class="comment">// spin up agents</span></div><div class="line"><span class="keyword">for</span> agent := <span class="keyword">range</span> self.agents {</div><div class="line">agent.Start()</div><div class="line">}</div><div class="line">}</div></pre></td></tr></table></figure><p>这个update()会订阅(监听)几种事件，均跟Downloader相关。当收到Downloader的StartEvent时，意味者此时本节点正在从其他节点下载新区块，这时miner会立即停止进行中的挖掘工作，并继续监听；如果收到DoneEvent或FailEvent时，意味本节点的下载任务已结束-无论下载成功或失败-此时都可以开始挖掘新区块，并且此时会退出Downloader事件的监听。</p><p>从miner.Update()的逻辑可以看出，对于任何一个Ethereum网络中的节点来说，挖掘一个新区块和从其他节点下载、同步一个新区块，根本是相互冲突的。这样的规定，保证了在某个节点上，一个新区块只可能有一种来源，这可以大大降低可能出现的区块冲突，并避免全网中计算资源的浪费。</p><h3 id="worker的函数"><a href="#worker的函数" class="headerlink" title="worker的函数"></a>worker的函数</h3><p>worker的属性非常多而具体了，都是挖矿具体操作相关的，其中包括链本身的属性以及区块数据结构的属性。首先来看ChainConfig：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> ChainConfig <span class="keyword">struct</span> {</div><div class="line"> ChainId *big.Int <span class="string">`json:"chainId"`</span> <span class="comment">// 链id标识了当前链，主键唯一id，也用于replay protection重发保护（用来防止replay attack重发攻击：恶意重复或拖延正确数据传输的一种网络攻击手段）</span></div><div class="line"></div><div class="line"> HomesteadBlock *big.Int <span class="string">`json:"homesteadBlock,omitempty"`</span> <span class="comment">// 当前链Homestead，置为0</span></div><div class="line"></div><div class="line"> DAOForkBlock *big.Int <span class="string">`json:"daoForkBlock,omitempty"`</span> <span class="comment">// TheDAO硬分叉切换。</span></div><div class="line"> DAOForkSupport <span class="keyword">bool</span> <span class="string">`json:"daoForkSupport,omitempty"`</span> <span class="comment">// 结点是否支持或者反对DAO硬分叉。</span></div><div class="line"></div><div class="line"> <span class="comment">// EIP150 implements the Gas price changes (https://github.com/ethereum/EIPs/issues/150)</span></div><div class="line"> EIP150Block *big.Int <span class="string">`json:"eip150Block,omitempty"`</span> <span class="comment">// EIP150 HF block (nil = no fork)</span></div><div class="line"> EIP150Hash common.Hash <span class="string">`json:"eip150Hash,omitempty"`</span> <span class="comment">// EIP150 HF hash (needed for header only clients as only gas pricing changed)</span></div><div class="line"></div><div class="line"> EIP155Block *big.Int <span class="string">`json:"eip155Block,omitempty"`</span> <span class="comment">// EIP155 HF block，没有硬分叉置为0</span></div><div class="line"> EIP158Block *big.Int <span class="string">`json:"eip158Block,omitempty"`</span> <span class="comment">// EIP158 HF block，没有硬分叉置为0</span></div><div class="line"></div><div class="line"> ByzantiumBlock *big.Int <span class="string">`json:"byzantiumBlock,omitempty"`</span> <span class="comment">// Byzantium switch block (nil = no fork, 0 = already on byzantium)</span></div><div class="line"></div><div class="line"> <span class="comment">// Various consensus engines</span></div><div class="line"> Ethash *EthashConfig <span class="string">`json:"ethash,omitempty"`</span></div><div class="line"> Clique *CliqueConfig <span class="string">`json:"clique,omitempty"`</span></div><div class="line">}</div></pre></td></tr></table></figure><p>ChainConfig顾名思义就是链的配置属性。ChainConfig中包含了ChainID等属性，其中有很多都是针对以太坊历史发生的问题进行的专门配置。</p><ul><li>ChainId可以预防replay攻击。</li><li>Homestead是以太坊发展蓝图中的一个阶段。第一阶段是以太坊区块链面世，代号为frontier，第二个阶段即为当前阶段，代号为Homestead（家园），第三阶段为Metropolis（大都会），大都会又细分为两个小阶段，第一个是Byzantium（拜占庭）硬分叉(引入新型零知识证明算法以及pos权益证明共识算法)，第二个是Constantinople（君士坦丁堡）硬分叉（以太坊正式应用pow和pos混合链，解决拜占庭引发的问题）。最后一个阶段代号Serenity（宁静），最终版本的以太坊稳定运行。</li><li>2017年6月18日，以太坊上DAO（去中心自治组织）的一次大危机做出的相应调整。感兴趣的可以自行谷百。</li><li>2017年10月16日，以太坊的一次Byzantium拜占庭硬分叉。</li><li>EIPs(Ethereum Improvement Proposals)，是以太坊更新改善的一些方案，对应后面的数字就是以太坊github源码issue的编号。</li></ul><p>再来看下 Backend对象，Backend是一个自定义接口封装了所有挖矿所需方法。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Backend <span class="keyword">interface</span> {</div><div class="line">AccountManager() *accounts.Manager</div><div class="line">BlockChain() *core.BlockChain</div><div class="line">TxPool() *core.TxPool</div><div class="line">ChainDb() ethdb.Database</div><div class="line">}</div></pre></td></tr></table></figure><p>这里我们主要关注worker.updater()和wait()</p><p><img src="/2018/03/26/mine/worker.png" alt="img"></p><p>####worker.update()</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(self *worker)</span> <span class="title">update</span><span class="params">()</span></span> {</div><div class="line"><span class="keyword">defer</span> self.txSub.Unsubscribe()</div><div class="line"><span class="keyword">defer</span> self.chainHeadSub.Unsubscribe()</div><div class="line"><span class="keyword">defer</span> self.chainSideSub.Unsubscribe()</div><div class="line"></div><div class="line"><span class="keyword">for</span> {</div><div class="line"><span class="comment">// A real event arrived, process interesting content</span></div><div class="line"><span class="keyword">select</span> {</div><div class="line"><span class="comment">// Handle ChainHeadEvent</span></div><div class="line"><span class="keyword">case</span> <-self.chainHeadCh:</div><div class="line">self.commitNewWork()</div><div class="line"></div><div class="line"><span class="comment">// Handle ChainSideEvent</span></div><div class="line"><span class="keyword">case</span> ev := <-self.chainSideCh:</div><div class="line">self.uncleMu.Lock()</div><div class="line">self.possibleUncles[ev.Block.Hash()] = ev.Block</div><div class="line">self.uncleMu.Unlock()</div><div class="line"></div><div class="line"><span class="comment">// Handle TxPreEvent</span></div><div class="line"><span class="keyword">case</span> ev := <-self.txCh:</div><div class="line"><span class="comment">// Apply transaction to the pending state if we're not mining</span></div><div class="line"><span class="keyword">if</span> atomic.LoadInt32(&self.mining) == <span class="number">0</span> {</div><div class="line">self.currentMu.Lock()</div><div class="line">acc, _ := types.Sender(self.current.signer, ev.Tx)</div><div class="line">txs := <span class="keyword">map</span>[common.Address]types.Transactions{acc: {ev.Tx}}</div><div class="line">txset := types.NewTransactionsByPriceAndNonce(self.current.signer, txs)</div><div class="line"></div><div class="line">self.current.commitTransactions(self.mux, txset, self.chain, self.coinbase)</div><div class="line">self.updateSnapshot()</div><div class="line">self.currentMu.Unlock()</div><div class="line">} <span class="keyword">else</span> {</div><div class="line"><span class="comment">// If we're mining, but nothing is being processed, wake on new transactions</span></div><div class="line"><span class="keyword">if</span> self.config.Clique != <span class="literal">nil</span> && self.config.Clique.Period == <span class="number">0</span> {</div><div class="line">self.commitNewWork()</div><div class="line">}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// System stopped</span></div><div class="line"><span class="keyword">case</span> <-self.txSub.Err():</div><div class="line"><span class="keyword">return</span></div><div class="line"><span class="keyword">case</span> <-self.chainHeadSub.Err():</div><div class="line"><span class="keyword">return</span></div><div class="line"><span class="keyword">case</span> <-self.chainSideSub.Err():</div><div class="line"><span class="keyword">return</span></div><div class="line">}</div><div class="line">}</div><div class="line">}</div></pre></td></tr></table></figure><p>worker.update()分别监听ChainHeadEvent，ChainSideEvent，TxPreEvent几个事件，每个事件会触发worker不同的反应。ChainHeadEvent是指区块链中已经加入了一个新的区块作为整个链的链头，这时worker的回应是立即开始准备挖掘下一个新区块(也是够忙的)；ChainSideEvent指区块链中加入了一个新区块作为当前链头的旁支，worker会把这个区块收纳进possibleUncles[]数组，作为下一个挖掘新区块可能的Uncle之一；TxPreEvent是TxPool对象发出的，指的是一个新的交易tx被加入了TxPool，这时如果worker没有处于挖掘中，那么就去执行这个tx，并把它收纳进Work.txs数组，为下次挖掘新区块备用。</p><p>需要稍稍注意的是，ChainHeadEvent 并不一定是外部源发出。由于 worker 对象有个成员变量chain(eth.BlockChain)，所以当worker自己完成挖掘一个新区块，并把它写入数据库，加进区块链里成为新的链头时，worker自己也可以调用chain发出一个ChainHeadEvent，从而被worker.update()函数监听到，进入下一次区块挖掘。</p><p>####worker.wait()</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(self *worker)</span> <span class="title">wait</span><span class="params">()</span></span> {</div><div class="line"> <span class="keyword">for</span> {</div><div class="line"> mustCommitNewWork := <span class="literal">true</span></div><div class="line"> <span class="keyword">for</span> result := <span class="keyword">range</span> self.recv {</div><div class="line"> atomic.AddInt32(&self.atWork, <span class="number">-1</span>)</div><div class="line"></div><div class="line"> <span class="keyword">if</span> result == <span class="literal">nil</span> {</div><div class="line"> <span class="keyword">continue</span></div><div class="line"> }</div><div class="line"> block := result.Block</div><div class="line"> work := result.Work</div><div class="line"></div><div class="line"> <span class="comment">// Update the block hash in all logs since it is now available and not when the receipt/log of individual transactions were created.</span></div><div class="line"> <span class="keyword">for</span> _, r := <span class="keyword">range</span> work.receipts {</div><div class="line"> <span class="keyword">for</span> _, l := <span class="keyword">range</span> r.Logs {</div><div class="line"> l.BlockHash = block.Hash()</div><div class="line"> }</div><div class="line"> }</div><div class="line"> <span class="keyword">for</span> _, log := <span class="keyword">range</span> work.state.Logs() {</div><div class="line"> log.BlockHash = block.Hash()</div><div class="line"> }</div><div class="line"> stat, err := self.chain.WriteBlockAndState(block, work.receipts, work.state)</div><div class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"> log.Error(<span class="string">"Failed writing block to chain"</span>, <span class="string">"err"</span>, err)</div><div class="line"> <span class="keyword">continue</span></div><div class="line"> }</div><div class="line"> <span class="comment">// 检查是否是标准块，写入交易数据。</span></div><div class="line"> <span class="keyword">if</span> stat == core.CanonStatTy {</div><div class="line"> <span class="comment">// 受ChainHeadEvent事件的影响。</span></div><div class="line"> mustCommitNewWork = <span class="literal">false</span></div><div class="line"> }</div><div class="line"> <span class="comment">// 广播一个块声明插入链事件NewMinedBlockEvent</span></div><div class="line"> self.mux.Post(core.NewMinedBlockEvent{Block: block})</div><div class="line"> <span class="keyword">var</span> (</div><div class="line"> events []<span class="keyword">interface</span>{}</div><div class="line"> logs = work.state.Logs()</div><div class="line"> )</div><div class="line"> events = <span class="built_in">append</span>(events, core.ChainEvent{Block: block, Hash: block.Hash(), Logs: logs})</div><div class="line"> <span class="keyword">if</span> stat == core.CanonStatTy {</div><div class="line"> events = <span class="built_in">append</span>(events, core.ChainHeadEvent{Block: block})</div><div class="line"> }</div><div class="line"> self.chain.PostChainEvents(events, logs)</div><div class="line"></div><div class="line"> <span class="comment">// 将处理中的数据插入到区块中，等待确认</span></div><div class="line"> self.unconfirmed.Insert(block.NumberU64(), block.Hash())</div><div class="line"></div><div class="line"> <span class="keyword">if</span> mustCommitNewWork {</div><div class="line"> self.commitNewWork() <span class="comment">// 多次见到，顾名思义，就是提交新的work</span></div><div class="line"> }</div><div class="line"> }</div><div class="line"> }</div><div class="line">}</div></pre></td></tr></table></figure><p>worker.wait()会在一个channel处一直等待Agent完成挖掘发送回来的新Block和Work对象。这个Block会被写入数据库，加入本地的区块链试图成为最新的链头。注意，此时区块中的所有交易，假设都已经被执行过了，所以这里的操作，不会再去执行这些交易对象。</p><p>当这一切都完成，worker就会发送一条事件(NewMinedBlockEvent{})，等于通告天下：我挖出了一个新区块！这样监听到该事件的其他节点，就会根据自身的状况，来决定是否接受这个新区块成为全网中公认的区块链新的链头。至于这个公认过程如何实现，就属于共识算法的范畴了。</p><p>####worker.commitNewWork()</p><p>commitNewWork方法源码比较长，这里就不粘贴出来了，这个方法主要的工作是为新块准备基本数据，包括header，txs，uncles等。commitNewWork()会在worker内部多处被调用，注意它每次都是被直接调用，并没有以goroutine的方式启动。commitNewWork()内部使用sync.Mutex对全部操作做了隔离。这个函数的基本逻辑如下：</p><ol><li>准备新区块的时间属性Header.Time，一般均等于系统当前时间，不过要确保父区块的时间(parentBlock.Time())要早于新区块的时间，父区块当然来自当前区块链的链头了。</li><li>创建新区块的Header对象，其各属性中：Num可确定(父区块Num +1)；Time可确定；ParentHash可确定;其余诸如Difficulty，GasLimit等，均留待之后共识算法中确定。</li><li>调用Engine.Prepare()函数，完成Header对象的准备。</li><li>根据新区块的位置(Number)，查看它是否处于DAO硬分叉的影响范围内，如果是，则赋值予header.Extra。</li><li>根据已有的Header对象，创建一个新的Work对象，并用其更新worker.current成员变量。</li><li>如果配置信息中支持硬分叉，在Work对象的StateDB里应用硬分叉。</li><li>准备新区块的交易列表，来源是TxPool中那些最近加入的tx，并执行这些交易。</li><li>准备新区块的叔区块uncles[]，来源是worker.possibleUncles[]，而possibleUncles[]中的每个区块都从事件ChainSideEvent中搜集得到。注意叔区块最多有两个。</li><li>调用Engine.Finalize()函数，对新区块“定型”，填充上Header.Root, TxHash, ReceiptHash, UncleHash等几个属性。</li><li>如果上一个区块(即旧的链头区块)处于unconfirmedBlocks中，意味着它也是由本节点挖掘出来的，尝试去验证它已经被吸纳进主干链中。</li><li>把创建的Work对象，通过channel发送给每一个登记过的Agent，进行后续的挖掘。</li></ol><p>以上步骤中，4和6都是仅仅在该区块配置中支持DAO硬分叉，并且该区块的位置正好处于DAO硬分叉影响范围内时才会发生；其他步骤是普遍性的。commitNewWork()完成了待挖掘区块的组装，block.Header创建完毕，交易数组txs，叔区块Uncles[]都已取得，并且由于所有交易被执行完毕，相应的Receipt[]也已获得。万事俱备，可以交给Agent进行‘挖掘’了。</p><h3 id="CpuAgent的函数"><a href="#CpuAgent的函数" class="headerlink" title="CpuAgent的函数"></a>CpuAgent的函数</h3><p>CpuAgent中与mine相关的函数，主要是update()和mine():</p><p><img src="/2018/03/26/mine/./images/cpuagent.png" alt="img"></p><p>####CpuAgent.update()</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(self *CpuAgent)</span> <span class="title">update</span><span class="params">()</span></span> {</div><div class="line">out:</div><div class="line"> <span class="keyword">for</span> {</div><div class="line"> <span class="keyword">select</span> {</div><div class="line"> <span class="keyword">case</span> work := <-self.workCh:</div><div class="line"> self.mu.Lock()</div><div class="line"> <span class="keyword">if</span> self.quitCurrentOp != <span class="literal">nil</span> {</div><div class="line"> <span class="built_in">close</span>(self.quitCurrentOp)</div><div class="line"> }</div><div class="line"> self.quitCurrentOp = <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">struct</span>{})</div><div class="line"> <span class="keyword">go</span> self.mine(work, self.quitCurrentOp)</div><div class="line"> self.mu.Unlock()</div><div class="line"> <span class="keyword">case</span> <-self.stop:</div><div class="line"> self.mu.Lock()</div><div class="line"> <span class="keyword">if</span> self.quitCurrentOp != <span class="literal">nil</span> {</div><div class="line"> <span class="built_in">close</span>(self.quitCurrentOp)</div><div class="line"> self.quitCurrentOp = <span class="literal">nil</span></div><div class="line"> }</div><div class="line"> self.mu.Unlock()</div><div class="line"> <span class="keyword">break</span> out</div><div class="line"> }</div><div class="line"> }</div><div class="line">}</div></pre></td></tr></table></figure><p>CpuAgent.update()就是worker.commitNewWork()结束后发出Work对象的会一直监听相关channel，如果收到Work对象(显然由worker.commitNewWork()结束后发出)，就启动mine()函数；如果收到停止(mine)的消息，就退出一切相关操作。</p><p>####CpuAgent.mine()</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(self *CpuAgent)</span> <span class="title">mine</span><span class="params">(work *Work, stop <-<span class="keyword">chan</span> <span class="keyword">struct</span>{})</span></span> {</div><div class="line"> <span class="keyword">if</span> result, err := self.engine.Seal(self.chain, work.Block, stop); result != <span class="literal">nil</span> {</div><div class="line"> log.Info(<span class="string">"Successfully sealed new block"</span>, <span class="string">"number"</span>, result.Number(), <span class="string">"hash"</span>, result.Hash())</div><div class="line"> self.returnCh <- &Result{work, result}</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"> log.Warn(<span class="string">"Block sealing failed"</span>, <span class="string">"err"</span>, err)</div><div class="line"> }</div><div class="line"> self.returnCh <- <span class="literal">nil</span></div><div class="line"> }</div><div class="line">}</div></pre></td></tr></table></figure><p>CpuAgent.mine()会直接调用Engine.Seal()函数，利用Engine实现体的共识算法对传入的Block进行最终的授权，如果成功，就将Block同Work一起通过channel发还给worker，那边worker.wait()会接收并处理。</p><p>显然，这两个函数都没做什么实质性工作，它们只是负责调用<engine>接口实现体，待授权完成后将区块数据发送回worker。挖掘出一个区块的真正奥妙全在Engine实现体所代表的共识算法里。</engine></p><p><img src="/2018/03/26/mine/mine_block.png" alt="img"></p>]]></content>
<summary type="html">
<p>前几篇分别介绍了以太坊的基本概念，基本环节-交易，区块、区块链的存储方式等，这篇打算介绍一下“挖矿“得到新区块的整个过程，然后下一篇讲下不同共识算法的实现细节。<br></p>
</summary>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/categories/BlockChain/"/>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/tags/BlockChain/"/>
</entry>
<entry>
<title>区块存储</title>
<link href="https://yangchenglong11.github.io/2018/03/18/the-store-of-block/"/>
<id>https://yangchenglong11.github.io/2018/03/18/the-store-of-block/</id>
<published>2018-03-18T04:42:40.000Z</published>
<updated>2018-03-31T08:47:04.798Z</updated>
<content type="html"><![CDATA[<p>区块(Block)是以太坊的核心数据结构之一，Block包含Header和Body两部分。区块的存储是由leveldb完成的，leveldb的数据是以键值对存储的。<br><a id="more"></a><br><img src="/2018/03/18/the-store-of-block/core_data_structure.png" alt="data_structure"></p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// BlockChain 表示了一个规范的链,这个链通过一个包含了创世区块的数据库指定. BlockChain管理了链的插入,还原,重建等操作.</span></div><div class="line"><span class="comment">// 插入一个区块需要通过一系列指定的规则指定的两阶段的验证器.</span></div><div class="line"><span class="comment">// 使用Processor来对区块的交易进行处理. 状态的验证是第二阶段的验证器. 错误将导致插入终止.</span></div><div class="line"><span class="comment">// 需要注意的是GetBlock可能返回任意不在当前规范区块链中的区块,</span></div><div class="line"><span class="comment">// 但是GetBlockByNumber总是返回当前规范区块链中的区块.</span></div><div class="line"><span class="keyword">type</span> BlockChain <span class="keyword">struct</span> {</div><div class="line"> chainConfig *params.ChainConfig <span class="comment">// Chain & network configuration</span></div><div class="line"> cacheConfig *CacheConfig <span class="comment">// Cache configuration for pruning</span></div><div class="line"></div><div class="line"> db ethdb.Database <span class="comment">// Low level persistent database to store final content in</span></div><div class="line"> triegc *prque.Prque <span class="comment">// Priority queue mapping block numbers to tries to gc</span></div><div class="line"> gcproc time.Duration <span class="comment">// Accumulates canonical block processing for trie dumping</span></div><div class="line"></div><div class="line"> hc *HeaderChain <span class="comment">// 只包含了区块头的区块链</span></div><div class="line"> rmLogsFeed event.Feed <span class="comment">// 底层数据库</span></div><div class="line"> chainFeed event.Feed <span class="comment">// 下面是很多消息通知的组件</span></div><div class="line"> chainSideFeed event.Feed</div><div class="line"> chainHeadFeed event.Feed</div><div class="line"> logsFeed event.Feed</div><div class="line"> scope event.SubscriptionScope</div><div class="line"> genesisBlock *types.Block <span class="comment">// 创世区块</span></div><div class="line"></div><div class="line"> mu sync.RWMutex <span class="comment">// global mutex for locking chain operations</span></div><div class="line"> chainmu sync.RWMutex <span class="comment">// blockchain insertion lock</span></div><div class="line"> procmu sync.RWMutex <span class="comment">// block processor lock</span></div><div class="line"></div><div class="line"> checkpoint <span class="keyword">int</span> <span class="comment">// checkpoint counts towards the new checkpoint</span></div><div class="line"> currentBlock *types.Block <span class="comment">// Current head of the block chain 当前的区块头</span></div><div class="line"> currentFastBlock *types.Block <span class="comment">// Current head of the fast-sync chain (may be above the block chain!) 当前的快速同步的区块头</span></div><div class="line"></div><div class="line"> stateCache state.Database <span class="comment">// State database to reuse between imports (contains state cache)</span></div><div class="line"> bodyCache *lru.Cache <span class="comment">// Cache for the most recent block bodies</span></div><div class="line"> bodyRLPCache *lru.Cache <span class="comment">// Cache for the most recent block bodies in RLP encoded format</span></div><div class="line"> blockCache *lru.Cache <span class="comment">// Cache for the most recent entire blocks</span></div><div class="line"> futureBlocks *lru.Cache <span class="comment">// future blocks are blocks added for later processing 暂时还不能插入的区块存放位置</span></div><div class="line"></div><div class="line"> quit <span class="keyword">chan</span> <span class="keyword">struct</span>{} <span class="comment">// blockchain quit channel</span></div><div class="line"> running <span class="keyword">int32</span> <span class="comment">// running must be called atomically</span></div><div class="line"> <span class="comment">// procInterrupt must be atomically called</span></div><div class="line"> procInterrupt <span class="keyword">int32</span> <span class="comment">// interrupt signaler for block processing</span></div><div class="line"> wg sync.WaitGroup <span class="comment">// chain processing wait group for shutting down</span></div><div class="line"></div><div class="line"> engine consensus.Engine <span class="comment">// 一致性引擎</span></div><div class="line"> processor Processor <span class="comment">// block processor interface // 区块处理器接口</span></div><div class="line"> validator Validator <span class="comment">// block and state validator interface // 区块和状态验证器接口</span></div><div class="line"> vmConfig vm.Config <span class="comment">// 虚拟机的配置</span></div><div class="line"></div><div class="line"> badBlocks *lru.Cache <span class="comment">// Bad block cache 错误区块的缓存</span></div><div class="line">}</div></pre></td></tr></table></figure><p> Blockchain管理所有的Block, 让其组成一个单向链表。Headerchain管理所有的Header,也形成一个单向链表， Headerchain是Blockchain里面的一部分 。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> HeaderChain <span class="keyword">struct</span> {</div><div class="line">config *params.ChainConfig</div><div class="line"></div><div class="line">chainDb ethdb.Database</div><div class="line">genesisHeader *types.Header</div><div class="line"></div><div class="line">currentHeader atomic.Value <span class="comment">// Current head of the header chain (may be above the block chain!)</span></div><div class="line">currentHeaderHash common.Hash <span class="comment">// Hash of the current head of the header chain (prevent recomputing all the time)</span></div><div class="line"></div><div class="line">headerCache *lru.Cache <span class="comment">// Cache for the most recent block headers</span></div><div class="line">tdCache *lru.Cache <span class="comment">// Cache for the most recent block total difficulties</span></div><div class="line">numberCache *lru.Cache <span class="comment">// Cache for the most recent block numbers</span></div><div class="line"></div><div class="line">procInterrupt <span class="function"><span class="keyword">func</span><span class="params">()</span> <span class="title">bool</span></span></div><div class="line"><span class="function"></span></div><div class="line"><span class="function"><span class="title">rand</span> *<span class="title">mrand</span>.<span class="title">Rand</span></span></div><div class="line"><span class="function"><span class="title">engine</span> <span class="title">consensus</span>.<span class="title">Engine</span></span></div><div class="line"><span class="function">}</span></div></pre></td></tr></table></figure><p>以太坊的数据库体系-Merkle-Patricia Trie(MPT)， 它是由一系列节点组成的二叉树，在树底包含了源数据的大量叶子节点， 父节点是两个子节点的Hash值，一直到根节点。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// Header represents a block header in the Ethereum blockchain.</span></div><div class="line"><span class="keyword">type</span> Header <span class="keyword">struct</span> {</div><div class="line"> ParentHash common.Hash <span class="string">`json:"parentHash" gencodec:"required"`</span></div><div class="line"> UncleHash common.Hash <span class="string">`json:"sha3Uncles" gencodec:"required"`</span></div><div class="line"> Coinbase common.Address <span class="string">`json:"miner" gencodec:"required"`</span></div><div class="line"> Root common.Hash <span class="string">`json:"stateRoot" gencodec:"required"`</span></div><div class="line"> TxHash common.Hash <span class="string">`json:"transactionsRoot" gencodec:"required"`</span></div><div class="line"> ReceiptHash common.Hash <span class="string">`json:"receiptsRoot" gencodec:"required"`</span></div><div class="line"> Bloom Bloom <span class="string">`json:"logsBloom" gencodec:"required"`</span></div><div class="line"> Difficulty *big.Int <span class="string">`json:"difficulty" gencodec:"required"`</span></div><div class="line"> Number *big.Int <span class="string">`json:"number" gencodec:"required"`</span></div><div class="line"> GasLimit <span class="keyword">uint64</span> <span class="string">`json:"gasLimit" gencodec:"required"`</span></div><div class="line"> GasUsed <span class="keyword">uint64</span> <span class="string">`json:"gasUsed" gencodec:"required"`</span></div><div class="line"> Time *big.Int <span class="string">`json:"timestamp" gencodec:"required"`</span></div><div class="line"> Extra []<span class="keyword">byte</span> <span class="string">`json:"extraData" gencodec:"required"`</span></div><div class="line"> MixDigest common.Hash <span class="string">`json:"mixHash" gencodec:"required"`</span></div><div class="line"> Nonce BlockNonce <span class="string">`json:"nonce" gencodec:"required"`</span></div><div class="line">}</div></pre></td></tr></table></figure><p><img src="/2018/03/18/the-store-of-block/ethereum.png" alt="img"></p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// Block represents an entire block in the Ethereum blockchain.</span></div><div class="line"><span class="keyword">type</span> Block <span class="keyword">struct</span> {</div><div class="line"> header *Header</div><div class="line"> uncles []*Header</div><div class="line"> transactions Transactions</div><div class="line"></div><div class="line"> <span class="comment">// caches</span></div><div class="line"> hash atomic.Value</div><div class="line"> size atomic.Value</div><div class="line"></div><div class="line"> <span class="comment">// Td is used by package core to store the total difficulty</span></div><div class="line"> <span class="comment">// of the chain up to and including the block.</span></div><div class="line"> td *big.Int</div><div class="line"></div><div class="line"> <span class="comment">// These fields are used by package eth to track</span></div><div class="line"> <span class="comment">// inter-peer block relay.</span></div><div class="line"> ReceivedAt time.Time</div><div class="line"> ReceivedFrom <span class="keyword">interface</span>{}</div><div class="line">}</div></pre></td></tr></table></figure><p>Transaction是Body的重要数据结构，一个交易就是被外部拥有账户生成的加密签名的一段指令，序列化，然后提交给区块链。</p><p>在这里保存区块信息时，key一般是与hash相关的，value所保存的数据结构是经过RLP编码的。<br>在代码中，core/database_util.go中封装了区块存储和读取相关的代码。<br>在存储区块信息时，会将区块头和区块体分开进行存储。因此在区块的结构体中，能够看到Header和Body两个结构体。<br>区块头（Header）的存储格式为：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">headerPrefix + num (<span class="keyword">uint64</span> big endian) + hash -> rlpEncode(header)</div></pre></td></tr></table></figure><p>key是由区块头的前缀，区块号和区块hash构成。value是区块头的RLP编码。<br>区块体（Body）的存储格式为：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">bodyPrefix + num (<span class="keyword">uint64</span> big endian) + hash -> rlpEncode(block body)</div></pre></td></tr></table></figure><p>key是由区块体前缀，区块号和区块hash构成。value是区块体的RLP编码。<br>在database_util.go中，key的前缀可以区分leveldb中存储的是什么类型的数据。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">var</span> (</div><div class="line"> headHeaderKey = []<span class="keyword">byte</span>(<span class="string">"LastHeader"</span>)</div><div class="line"> headBlockKey = []<span class="keyword">byte</span>(<span class="string">"LastBlock"</span>)</div><div class="line"> headFastKey = []<span class="keyword">byte</span>(<span class="string">"LastFast"</span>)</div><div class="line"></div><div class="line"> <span class="comment">// Data item prefixes (use single byte to avoid mixing data types, avoid `i`).</span></div><div class="line"> headerPrefix = []<span class="keyword">byte</span>(<span class="string">"h"</span>) <span class="comment">// headerPrefix + num (uint64 big endian) + hash -> header</span></div><div class="line"> tdSuffix = []<span class="keyword">byte</span>(<span class="string">"t"</span>) <span class="comment">// headerPrefix + num (uint64 big endian) + hash + tdSuffix -> td</span></div><div class="line"> numSuffix = []<span class="keyword">byte</span>(<span class="string">"n"</span>) <span class="comment">// headerPrefix + num (uint64 big endian) + numSuffix -> hash</span></div><div class="line"> blockHashPrefix = []<span class="keyword">byte</span>(<span class="string">"H"</span>) <span class="comment">// blockHashPrefix + hash -> num (uint64 big endian)</span></div><div class="line"> bodyPrefix = []<span class="keyword">byte</span>(<span class="string">"b"</span>) <span class="comment">// bodyPrefix + num (uint64 big endian) + hash -> block body</span></div><div class="line"> blockReceiptsPrefix = []<span class="keyword">byte</span>(<span class="string">"r"</span>) <span class="comment">// blockReceiptsPrefix + num (uint64 big endian) + hash -> block receipts</span></div><div class="line"> lookupPrefix = []<span class="keyword">byte</span>(<span class="string">"l"</span>) <span class="comment">// lookupPrefix + hash -> transaction/receipt lookup metadata</span></div><div class="line"> bloomBitsPrefix = []<span class="keyword">byte</span>(<span class="string">"B"</span>) <span class="comment">// bloomBitsPrefix + bit (uint16 big endian) + section (uint64 big endian) + hash -> bloom bits</span></div><div class="line"></div><div class="line"> preimagePrefix = <span class="string">"secure-key-"</span> <span class="comment">// preimagePrefix + hash -> preimage</span></div><div class="line"> configPrefix = []<span class="keyword">byte</span>(<span class="string">"ethereum-config-"</span>) <span class="comment">// config prefix for the db</span></div><div class="line"></div><div class="line"> <span class="comment">// Chain index prefixes (use `i` + single byte to avoid mixing data types).</span></div><div class="line"> BloomBitsIndexPrefix = []<span class="keyword">byte</span>(<span class="string">"iB"</span>) <span class="comment">// BloomBitsIndexPrefix is the data table of a chain indexer to track its progress</span></div><div class="line"></div><div class="line"> <span class="comment">// used by old db, now only used for conversion</span></div><div class="line"> oldReceiptsPrefix = []<span class="keyword">byte</span>(<span class="string">"receipts-"</span>)</div><div class="line"> oldTxMetaSuffix = []<span class="keyword">byte</span>{<span class="number">0x01</span>}</div><div class="line"></div><div class="line"> ErrChainConfigNotFound = errors.New(<span class="string">"ChainConfig not found"</span>) <span class="comment">// general config not found error</span></div><div class="line"></div><div class="line"> preimageCounter = metrics.NewCounter(<span class="string">"db/preimage/total"</span>)</div><div class="line"> preimageHitCounter = metrics.NewCounter(<span class="string">"db/preimage/hits"</span>)</div><div class="line">)</div></pre></td></tr></table></figure><p>database_util.go最开始就定义了所有的前缀。这里的注释详细说明了每一个前缀存储了什么数据类型。<br>database_util.go中的其他方法则是对leveldb的操作。其中get方法是读取数据库中的内容，write则是向leveldb中写入数据。<br>要讲一个区块的信息写入数据库，则需要调用其中的WriteBlock方法。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// WriteBlock serializes a block into the database, header and body separately.</span></div><div class="line"> <span class="function"><span class="keyword">func</span> <span class="title">WriteBlock</span><span class="params">(db ethdb.Putter, block *types.Block)</span> <span class="title">error</span></span> {</div><div class="line"> <span class="comment">// Store the body first to retain database consistency</span></div><div class="line"> <span class="keyword">if</span> err := WriteBody(db, block.Hash(), block.NumberU64(), block.Body()); err != <span class="literal">nil</span> {</div><div class="line"> <span class="keyword">return</span> err</div><div class="line"> }</div><div class="line"> <span class="comment">// Store the header too, signaling full block ownership</span></div><div class="line"> <span class="keyword">if</span> err := WriteHeader(db, block.Header()); err != <span class="literal">nil</span> {</div><div class="line"> <span class="keyword">return</span> err</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> <span class="literal">nil</span></div><div class="line"> }</div></pre></td></tr></table></figure><p>这里我们看到，将一个区块信息写入数据库其实是分别将区块头和区块体写入数据库。<br>首先来看区块头的存储。区块头的存储是由WriteHeader方法完成的。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// WriteHeader serializes a block header into the database.</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">WriteHeader</span><span class="params">(db ethdb.Putter, header *types.Header)</span> <span class="title">error</span></span> {</div><div class="line"> data, err := rlp.EncodeToBytes(header)</div><div class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"> <span class="keyword">return</span> err</div><div class="line"> }</div><div class="line"> hash := header.Hash().Bytes()</div><div class="line"> num := header.Number.Uint64()</div><div class="line"> encNum := encodeBlockNumber(num)</div><div class="line"> key := <span class="built_in">append</span>(blockHashPrefix, hash...)</div><div class="line"> <span class="keyword">if</span> err := db.Put(key, encNum); err != <span class="literal">nil</span> {</div><div class="line"> log.Crit(<span class="string">"Failed to store hash to number mapping"</span>, <span class="string">"err"</span>, err)</div><div class="line"> }</div><div class="line"> key = <span class="built_in">append</span>(<span class="built_in">append</span>(headerPrefix, encNum...), hash...)</div><div class="line"> <span class="keyword">if</span> err := db.Put(key, data); err != <span class="literal">nil</span> {</div><div class="line"> log.Crit(<span class="string">"Failed to store header"</span>, <span class="string">"err"</span>, err)</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> <span class="literal">nil</span></div><div class="line">}</div></pre></td></tr></table></figure><p>这里首先对区块头进行了RLP编码，然后将区块号转换成为byte格式，开始组装key。<br>这里首先向数据库中存储了一条区块hash->区块号的键值对，然后才将区块头的信息写入数据库。<br>接下来是区块体的存储。区块体存储是由WriteBody方法实现。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// WriteBody serializes the body of a block into the database.</span></div><div class="line"> <span class="function"><span class="keyword">func</span> <span class="title">WriteBody</span><span class="params">(db ethdb.Putter, hash common.Hash, number <span class="keyword">uint64</span>, body *types.Body)</span> <span class="title">error</span></span> {</div><div class="line"> data, err := rlp.EncodeToBytes(body)</div><div class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"> <span class="keyword">return</span> err</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> WriteBodyRLP(db, hash, number, data)</div><div class="line"> }</div><div class="line"></div><div class="line"><span class="comment">// WriteBodyRLP writes a serialized body of a block into the database.</span></div><div class="line"> <span class="function"><span class="keyword">func</span> <span class="title">WriteBodyRLP</span><span class="params">(db ethdb.Putter, hash common.Hash, number <span class="keyword">uint64</span>, rlp rlp.RawValue)</span> <span class="title">error</span></span> {</div><div class="line"> key := <span class="built_in">append</span>(<span class="built_in">append</span>(bodyPrefix, encodeBlockNumber(number)...), hash.Bytes()...)</div><div class="line"> <span class="keyword">if</span> err := db.Put(key, rlp); err != <span class="literal">nil</span> {</div><div class="line"> log.Crit(<span class="string">"Failed to store block body"</span>, <span class="string">"err"</span>, err)</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> <span class="literal">nil</span></div><div class="line"> }</div></pre></td></tr></table></figure><p>WriteBody首先将区块体的信息进行RLP编码，然后调用WriteBodyRLP方法将区块体的信息写入数据库。key的组装方法如之前所述。</p><h2 id="交易存储"><a href="#交易存储" class="headerlink" title="交易存储"></a>交易存储</h2><p>交易主要在数据库中仅存储交易的Meta信息。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">txHash + txMetaSuffix -> rlpEncode(txMeta)</div></pre></td></tr></table></figure><p>交易的Meta信息结构体如下：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// TxLookupEntry is a positional metadata to help looking up the data content of</span></div><div class="line"><span class="comment">// a transaction or receipt given only its hash.</span></div><div class="line"> <span class="keyword">type</span> TxLookupEntry <span class="keyword">struct</span> {</div><div class="line"> BlockHash common.Hash</div><div class="line"> BlockIndex <span class="keyword">uint64</span></div><div class="line"> Index <span class="keyword">uint64</span></div><div class="line"> }</div></pre></td></tr></table></figure><p>这里，meta信息会存储块的hash，块号和块上第几笔交易这些信息。<br>交易Meta存储是以交易hash加交易的Meta前缀为key，Meta的RLP编码为value。<br>交易写入数据库是通过WriteTxLookupEntries方法实现的。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// WriteTxLookupEntries stores a positional metadata for every transaction from</span></div><div class="line"><span class="comment">// a block, enabling hash based transaction and receipt lookups.</span></div><div class="line"> <span class="function"><span class="keyword">func</span> <span class="title">WriteTxLookupEntries</span><span class="params">(db ethdb.Putter, block *types.Block)</span> <span class="title">error</span></span> {</div><div class="line"> <span class="comment">// Iterate over each transaction and encode its metadata</span></div><div class="line"> <span class="keyword">for</span> i, tx := <span class="keyword">range</span> block.Transactions() {</div><div class="line"> entry := TxLookupEntry{</div><div class="line"> BlockHash: block.Hash(),</div><div class="line"> BlockIndex: block.NumberU64(),</div><div class="line"> Index: <span class="keyword">uint64</span>(i),</div><div class="line"> }</div><div class="line"> data, err := rlp.EncodeToBytes(entry)</div><div class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"> <span class="keyword">return</span> err</div><div class="line"> }</div><div class="line"> <span class="keyword">if</span> err := db.Put(<span class="built_in">append</span>(lookupPrefix, tx.Hash().Bytes()...), data); err != <span class="literal">nil</span> {</div><div class="line"> <span class="keyword">return</span> err</div><div class="line"> }</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> <span class="literal">nil</span></div><div class="line"> }</div></pre></td></tr></table></figure><p>这里，在将交易meta入库时，会遍历块上的所有交易，并构造交易的meta信息，进行RLP编码。然后以交易hash为key，meta为value进行存储。<br>这样就将一笔交易写入数据库中。<br>从数据库中读取交易信息时通过GetTransaction方法获得的。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// GetTransaction retrieves a specific transaction from the database, along with</span></div><div class="line"><span class="comment">// its added positional metadata.</span></div><div class="line"> <span class="function"><span class="keyword">func</span> <span class="title">GetTransaction</span><span class="params">(db DatabaseReader, hash common.Hash)</span> <span class="params">(*types.Transaction, common.Hash, <span class="keyword">uint64</span>, <span class="keyword">uint64</span>)</span></span> {</div><div class="line"> <span class="comment">// Retrieve the lookup metadata and resolve the transaction from the body</span></div><div class="line"> blockHash, blockNumber, txIndex := GetTxLookupEntry(db, hash)</div><div class="line"> </div><div class="line"> <span class="keyword">if</span> blockHash != (common.Hash{}) {</div><div class="line"> body := GetBody(db, blockHash, blockNumber)</div><div class="line"> <span class="keyword">if</span> body == <span class="literal">nil</span> || <span class="built_in">len</span>(body.Transactions) <= <span class="keyword">int</span>(txIndex) {</div><div class="line"> log.Error(<span class="string">"Transaction referenced missing"</span>, <span class="string">"number"</span>, blockNumber, <span class="string">"hash"</span>, blockHash, <span class="string">"index"</span>, txIndex)</div><div class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, common.Hash{}, <span class="number">0</span>, <span class="number">0</span></div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> body.Transactions[txIndex], blockHash, blockNumber, txIndex</div><div class="line"> }</div><div class="line"> <span class="comment">// Old transaction representation, load the transaction and it's metadata separately</span></div><div class="line"> data, _ := db.Get(hash.Bytes())</div><div class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(data) == <span class="number">0</span> {</div><div class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, common.Hash{}, <span class="number">0</span>, <span class="number">0</span></div><div class="line"> }</div><div class="line"> <span class="keyword">var</span> tx types.Transaction</div><div class="line"> <span class="keyword">if</span> err := rlp.DecodeBytes(data, &tx); err != <span class="literal">nil</span> {</div><div class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, common.Hash{}, <span class="number">0</span>, <span class="number">0</span></div><div class="line"> }</div><div class="line"> <span class="comment">// Retrieve the blockchain positional metadata</span></div><div class="line"> data, _ = db.Get(<span class="built_in">append</span>(hash.Bytes(), oldTxMetaSuffix...))</div><div class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(data) == <span class="number">0</span> {</div><div class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, common.Hash{}, <span class="number">0</span>, <span class="number">0</span></div><div class="line"> }</div><div class="line"> <span class="keyword">var</span> entry TxLookupEntry</div><div class="line"> <span class="keyword">if</span> err := rlp.DecodeBytes(data, &entry); err != <span class="literal">nil</span> {</div><div class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, common.Hash{}, <span class="number">0</span>, <span class="number">0</span></div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> &tx, entry.BlockHash, entry.BlockIndex, entry.Index</div><div class="line"> }</div></pre></td></tr></table></figure><p>这个方法会首先通过交易hash从数据库中获取交易的meta信息，包括交易所在块的hash，块号和第几笔交易。<br>接下来使用块号和块hash获取从数据库中读取块的信息。<br>然后根据第几笔交易从块上获取交易的具体信息。<br>这里以太坊将交易的存储换成了新的存储方式，即交易的具体信息存储在块上，交易hash只对应交易的meta信息，并不包含交易的具体信息。<br>而以前的交易存储则是需要存储交易的具体信息和meta信息。<br>因此GetTransaction方法会支持原有的数据存储方式。</p>]]></content>
<summary type="html">
<p>区块(Block)是以太坊的核心数据结构之一，Block包含Header和Body两部分。区块的存储是由leveldb完成的，leveldb的数据是以键值对存储的。<br></p>
</summary>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/categories/BlockChain/"/>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/tags/BlockChain/"/>
</entry>
<entry>
<title>Merkle Patricia Tree (梅克尔帕特里夏树) 详解</title>
<link href="https://yangchenglong11.github.io/2018/03/16/merkle-patricia-tree/"/>
<id>https://yangchenglong11.github.io/2018/03/16/merkle-patricia-tree/</id>
<published>2018-03-16T04:42:40.000Z</published>
<updated>2018-03-31T06:22:41.609Z</updated>
<content type="html"><![CDATA[<h3 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a><strong>1. 前言</strong></h3><h4 id="1-1-概述"><a href="#1-1-概述" class="headerlink" title="1.1 概述"></a><strong>1.1 概述</strong></h4><p>Merkle Patricia Tree（又称为Merkle Patricia Trie）是一种经过改良的、融合了默克尔树和前缀树两种树结构优点的数据结构，是以太坊中用来组织管理账户数据、生成交易集合哈希的重要数据结构。<br><a id="more"></a><br>MPT树有以下几个作用：</p><ul><li>存储任意长度的key-value键值对数据；</li><li>提供了一种快速计算所维护数据集哈希标识的机制；</li><li>提供了快速状态回滚的机制；</li><li>提供了一种称为<strong>默克尔证明</strong>的证明方法，进行轻节点的扩展，实现简单支付验证；</li></ul><p>由于MPT结合了（1）前缀树（2）默克尔树两种树结构的特点与优势 ，因此在介绍MPT之前，我们首先简要地介绍下这两种树结构的特点。</p><h4 id="1-2-前缀树"><a href="#1-2-前缀树" class="headerlink" title="1.2 前缀树"></a><strong>1.2 前缀树</strong></h4><p>前缀树（又称字典树），用于保存关联数组，其键（key）的内容通常为字符串。前缀树节点在树中的位置是由其键的内容所决定的，即前缀树的key值被编码在根节点到该节点的路径中。</p><p>如下图所示，图中共有6个叶子节点，其key的值分别为（1）tc（2）tea（3）ted（4）ten（5）Ab（6）il。</p><p><img src="/2018/03/16/merkle-patricia-tree/trie.png" alt="trie"></p><p><strong>优势</strong>：</p><p>相比于哈希表，使用前缀树来进行查询拥有共同前缀key的数据时十分高效，例如在字典中查找前缀为pre的单词，对于哈希表来说，需要遍历整个表，时间效率为O(n)；然而对于前缀树来说，只需要在树中找到前缀为pre的节点，且遍历以这个节点为根节点的子树即可。</p><p>但是对于最差的情况（前缀为空串），时间效率为O(n)，仍然需要遍历整棵树，此时效率与哈希表相同。</p><p>相比于哈希表，在前缀树不会存在哈希冲突的问题。</p><p><strong>劣势</strong>：</p><ul><li>直接查找效率低下</li></ul><p>前缀树的查找效率是O(m)，m为所查找节点的key长度，而哈希表的查找效率为O(1)。且一次查找会有m次IO开销，相比于直接查找，无论是速率、还是对磁盘的压力都比较大。</p><ul><li>可能会造成空间浪费</li></ul><p>当存在一个节点，其key值内容很长（如一串很长的字符串），当树中没有与他相同前缀的分支时，为了存储该节点，需要创建许多非叶子节点来构建根节点到该节点间的路径，造成了存储空间的浪费。</p><h4 id="1-3-默克尔树"><a href="#1-3-默克尔树" class="headerlink" title="1.3 默克尔树"></a><strong>1.3 默克尔树</strong></h4><p>Merkle树是由计算机科学家 Ralph Merkle 在很多年前提出的，并以他本人的名字来命名,由于在比特币网络中用到了这种数据结构来进行数据正确性的验证，在这里简要地介绍一下merkle树的特点及原理。</p><p>在比特币网络中，merkle树被用来归纳一个区块中的所有交易，同时生成整个交易集合的数字指纹。此外，由于merkle树的存在，使得在比特币这种公链的场景下，扩展一种“轻节点”实现简单支付验证变成可能，关于轻节点的内容，将会下文详述。</p><p><strong>特点</strong></p><ul><li>默克尔树是一种树，大多数是二叉树，也可以多叉树，无论是几叉树，它都具有树结构的所有特点；</li><li>默克尔树叶子节点的value是数据项的内容，或者是数据项的哈希值；</li><li>非叶子节点的value根据其孩子节点的信息，然后按照Hash算法计算而得出的；</li></ul><p><strong>原理</strong></p><p>在比特币网络中，merkle树是自底向上构建的。在下图的例子中，首先将L1-L4四个单元数据哈希化，然后将哈希值存储至相应的叶子节点。这些节点是Hash0-0, Hash0-1, Hash1-0, Hash1-1</p><p><img src="/2018/03/16/merkle-patricia-tree/merkle.png" alt="merkle"></p><p>将相邻两个节点的哈希值合并成一个字符串，然后计算这个字符串的哈希，得到的就是这两个节点的父节点的哈希值。</p><p>如果该层的树节点个数是单数，那么对于最后剩下的树节点，这种情况就直接对它进行哈希运算，其父节点的哈希就是其哈希值的哈希值（对于单数个叶子节点，有着不同的处理方法，也可以采用复制最后一个叶子节点凑齐偶数个叶子节点的方式）。循环重复上述计算过程，最后计算得到最后一个节点的哈希值，将该节点的哈希值作为整棵树的哈希。</p><p>若两棵树的根哈希一致，则这两棵树的结构、节点的内容必然相同。</p><p>如上图所示，一棵有着4个叶子节点的树，计算代表整棵树的哈希需要经过7次计算，若采用将这四个叶子节点拼接成一个字符串进行计算，仅仅只需要一次哈希就可以实现，那么为什么要采用这种看似奇怪的方式呢？</p><p><strong>优势</strong>：</p><ul><li>快速重哈希</li></ul><p>默克尔树的特点之一就是当树节点内容发生变化时，能够在前一次哈希计算的基础上，仅仅将被修改的树节点进行哈希重计算，便能得到一个新的根哈希用来代表整棵树的状态。</p><ul><li>轻节点扩展</li></ul><p>采用默克尔树，可以在公链环境下扩展一种“轻节点”。轻节点的特点是对于每个区块，仅仅需要存储约80个字节大小的区块头数据，而不存储交易列表，回执列表等数据。然而通过轻节点，可以实现在<strong>非信任的公链环境中</strong>验证某一笔交易是否被收录在区块链账本的功能。这使得像比特币，以太坊这样的区块链能够运行在个人PC，智能手机等拥有小存储容量的终端上。</p><p><strong>劣势</strong>：</p><ul><li>存储空间开销大</li></ul><h3 id="2-结构设计"><a href="#2-结构设计" class="headerlink" title="2. 结构设计"></a><strong>2. 结构设计</strong></h3><p>在这一小节，将详细地介绍MPT (梅克尔帕特里夏树) 树的结构设计，以及采用这种结构设计的用意、优化点。</p><h4 id="2-1-节点分类"><a href="#2-1-节点分类" class="headerlink" title="2.1 节点分类"></a><strong>2.1 节点分类</strong></h4><p>如上文所述，尽管前缀树可以起到维护key-value数据的目的，但是其具有十分明显的局限性。无论是查询操作，还是对数据的增删改，不仅效率低下，且存储空间浪费严重。故，在以太坊中，为MPT树新增了几种不同类型的树节点，以尽量压缩整体的树高、降低操作的复杂度。</p><p>MPT树中，树节点可以分为以下四类：</p><ul><li>空节点</li><li>分支节点</li><li>叶子节点</li><li>扩展节点</li></ul><p><strong>空节点</strong></p><p>空节点用来表示空串。</p><p><strong>分支节点</strong></p><p>分支节点用来表示MPT树中所有拥有超过1个孩子节点以上的非叶子节点， 其定义如下所示：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> fullNode <span class="keyword">struct</span> {</div><div class="line"> Children [<span class="number">17</span>]node <span class="comment">// Actual trie node data to encode/decode (needs custom encoder)</span></div><div class="line"> flags nodeFlag</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// nodeFlag contains caching-related metadata about a node.</span></div><div class="line"><span class="keyword">type</span> nodeFlag <span class="keyword">struct</span> {</div><div class="line"> hash hashNode <span class="comment">// cached hash of the node (may be nil)</span></div><div class="line"> gen <span class="keyword">uint16</span> <span class="comment">// cache generation counter</span></div><div class="line"> dirty <span class="keyword">bool</span> <span class="comment">// whether the node has changes that must be written to the database</span></div><div class="line">}</div></pre></td></tr></table></figure><p>与前缀树相同，MPT同样是把key-value数据项的key编码在树的路径中，但是key的每一个字节值的范围太大（[0-127]），因此在以太坊中，在进行树操作之前，首先会进行一个key编码的转换（下节会详述），将一个字节的高低四位内容分拆成两个字节存储。通过编码转换，<code>key'</code>的每一位的值范围都在[0, 15]内。因此，一个分支节点的孩子至多只有16个。以太坊通过这种方式，减小了每个分支节点的容量，但是在一定程度上增加了树高。</p><p>分支节点的孩子列表中，最后一个元素是用来存储自身的内容。</p><p>此外，每个分支节点会有一个附带的字段<code>nodeFlag</code>，记录了一些辅助数据：</p><ul><li>节点哈希：若该字段不为空，则当需要进行哈希计算时，可以跳过计算过程而直接使用上次计算的结果（当节点变脏时，该字段被置空）；</li><li>脏标志：当一个节点被修改时，该标志位被置为1；</li><li>诞生标志：当该节点第一次被载入内存中（或被修改时），会被赋予一个计数值作为诞生标志，该标志会被作为节点驱除的依据，清除内存中“太老”的未被修改的节点，防止占用的内存空间过多；</li></ul><p><strong>叶子节点&&扩展节点</strong></p><p>叶子节点与扩展节点的定义相似，如下所示：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> shortNode <span class="keyword">struct</span> {</div><div class="line"> Key []<span class="keyword">byte</span></div><div class="line"> Val node</div><div class="line"> flags nodeFlag</div><div class="line">}</div></pre></td></tr></table></figure><p>其中关键的字段为：</p><ul><li>Key：用来存储<strong>属于该节点范围的key</strong>；</li><li>Val：用来存储该节点的内容；</li></ul><p>其中<code>Key</code>是MPT树实现树高压缩的关键！</p><p>如之前所提及的，前缀树中会出现严重的存储空间浪费的情况，如下图：</p><p><img src="/2018/03/16/merkle-patricia-tree/bad_trie.jpg" alt="bad_trie"></p><p>图中右侧有一长串节点，这些节点大部分只是充当非叶子节点，用来构建一条路径，目的只是为了存储该路径上的叶子节点。</p><p>针对这种情况，MPT树对此进行了优化：当MPT试图插入一个节点，插入过程中发现目前没有与该节点Key拥有相同前缀的路径。此时MPT把<strong>剩余的Key</strong>存储在叶子／扩展节点的Key字段中，充当一个”Shortcut“。</p><p>例如图中我们将红线所圈的节点称为node1, 将蓝线所圈的节点称为node2。node1与node2共享路径前缀t，但是node1在插入时，树中没有与oast有共同前缀的路径，<em>因此node1的key为oast</em>，实现了编码路径的压缩。</p><p><img src="/2018/03/16/merkle-patricia-tree/shortcut.jpg" alt="shortcut"></p><p>这种做法有以下几点优势：</p><ul><li>提高节点的查找效率，避免过多的磁盘访问；</li><li>减少存储空间浪费，避免存储无用的节点；</li></ul><p>此外Val字段用来存储叶子／扩展节点的内容，对于叶子节点来说，该字段存储的是一个数据项的内容；而对于扩展节点来说，该字段可以是以下两种内容：</p><ol><li>Val字段存储的是其孩子节点在数据库中存储的索引值（其实该索引值也是孩子节点的哈希值）；</li><li>Val字段存储的是其孩子节点的引用；</li></ol><blockquote><p>为什么设计在扩展节点的Val字段有可能存储一串哈希值作为孩子节点的索引呢？</p><p>在以太坊中，该哈希代表着另外一个节点在数据库中索引，即根据这个哈希值作为数据库中的索引，可以从数据库中读取出另外一个节点的内容。</p><p>这种设计的目的是：</p><p>（1）当整棵树被持久化到数据库中时，保持节点间的关联关系；</p><p>（2）从数据库中读取节点时，尽量避免不必要的IO开销；</p><p>在内存中，父节点与子节点之间关联关系可以通过引用、指针等编程手段实现，但是当树节点持久化到数据库是，父节点中会存储一个子节点在数据库中的索引值，以此保持关联关系。</p><p>同样，从数据库中读取节点时，本着最小IO开销的原则，仅需要读取那些需要用到的节点数据即可，因此若目前该节点已经包含所需要查找的信息时，便无须将其子节点再读取出来；反之，则根据子节点的哈希索引递归读取子节点，直至读取到所需要的信息。</p></blockquote><p>由于叶子／扩展节点共享一套定义，那么怎么来区分Val字段存储的到底是一个数据项的内容，还是一串哈希索引呢？在以太坊中，通过在Key中加入特殊的标志来区分两种类型的节点。</p><h4 id="2-2-key值编码"><a href="#2-2-key值编码" class="headerlink" title="2.2 key值编码"></a><strong>2.2 key值编码</strong></h4><p>在以太坊中，MPT树的key值共有三种不同的编码方式，以满足不同场景的不同需求，在这里对每一种进行介绍。</p><p>三种编码方式分别为：</p><ol><li>Raw编码（原生的字符）；</li><li>Hex编码（扩展的16进制编码）；</li><li>Hex-Prefix编码（16进制前缀编码）；</li></ol><p><strong>Raw编码</strong></p><p>Raw编码就是原生的key值，不做任何改变。这种编码方式的key，是MPT对外提供接口的默认编码方式。</p><blockquote><p>例如一条key为“cat”，value为“dog”的数据项，其Raw编码就是[‘c’, ‘a’, ‘t’]，换成ASCII表示方式就是[63, 61, 74]</p></blockquote><p><strong>Hex编码</strong></p><p>在介绍分支节点的时候，我们介绍了，为了减少分支节点孩子的个数，需要将key的编码进行转换，将原key的高低四位分拆成两个字节进行存储。这种转换后的key的编码方式，就是Hex编码。</p><p>从Raw编码向Hex编码的转换规则是：</p><ul><li>将Raw编码的每个字符，根据高4位低4位拆成两个字节；</li><li>若该Key对应的节点存储的是真实的数据项内容（即该节点是叶子节点），则在末位添加一个ASCII值为16的字符作为终止标志符；</li><li>若该key对应的节点存储的是另外一个节点的哈希索引（即该节点是扩展节点），则不加任何字符；</li></ul><blockquote><p>key为“cat”, value为“dog”的数据项，其Hex编码为[3, 15, 3, 13, 4, 10, 16]</p></blockquote><p><em>Hex编码用于对内存中MPT树节点key进行编码</em></p><p><strong>HP (Hex-Prefix)编码</strong></p><p>在介绍叶子／扩展节点时，我们介绍了这两种节点定义是共享的，即便持久化到数据库中，存储的方式也是一致的。那么当节点加载到内存是，同样需要通过一种额外的机制来区分节点的类型。于是以太坊就提出了一种<strong>HP编码</strong>对存储在数据库中的叶子／扩展节点的key进行编码区分。在将这两类节点持久化到数据库之前，首先会对该节点的key做编码方式的转换，即<em>从Hex编码转换成HP编码</em>。</p><p>HP编码的规则如下：</p><ul><li>若原key的末尾字节的值为16（即该节点是叶子节点），去掉该字节；</li><li>在key之前增加一个半字节，其中最低位用来编码原本key长度的奇偶信息，key长度为奇数，则该位为1；低2位中编码一个特殊的终止标记符，若该节点为叶子节点，则该位为1；</li><li>若原本key的长度为奇数，则在key之前再增加一个值为0x0的<strong>半字节</strong>；</li><li>将原本key的内容作压缩，即将两个字符以高4位低4位进行划分，存储在一个字节中（Hex扩展的逆过程）；</li></ul><blockquote><p>若Hex编码为[3, 15, 3, 13, 4, 10, 16]，则HP编码的值为[32, 63, 61, 74]</p></blockquote><p><em>HP编码用于对数据库中的树节点key进行编码</em></p><p><strong>转换关系</strong></p><p><img src="/2018/03/16/merkle-patricia-tree/conversion.png" alt="conversion"></p><p>以上三种编码方式的转换关系为：</p><ul><li>Raw编码：原生的key编码，是MPT对外提供接口中使用的编码方式，当数据项被插入到树中时，Raw编码被转换成Hex编码；</li><li>Hex编码：16进制扩展编码，用于对内存中树节点key进行编码，当树节点被持久化到数据库时，Hex编码被转换成HP编码；</li><li>HP编码：16进制前缀编码，用于对数据库中树节点key进行编码，当树节点被加载到内存时，HP编码被转换成Hex编码；</li></ul><h4 id="2-3-安全的MPT"><a href="#2-3-安全的MPT" class="headerlink" title="2.3 安全的MPT"></a><strong>2.3 安全的MPT</strong></h4><p>以上介绍的MPT树，可以用来存储内容为任何长度的key-value数据项。倘若数据项的key长度没有限制时，当树中维护的数据量较大时，仍然会造成整棵树的深度变得越来越深，会造成以下影响：</p><ul><li>查询一个节点可能会需要许多次IO读取，效率低下；</li><li>系统易遭受Dos攻击，攻击者可以通过在合约中存储特定的数据，“构造”一棵拥有一条很长路径的树，然后不断地调用<code>SLOAD</code>指令读取该树节点的内容，造成系统执行效率极度下降；</li><li>所有的key其实是一种明文的形式进行存储；</li></ul><p>为了解决以上问题，在以太坊中对MPT再进行了一次封装，对数据项的key进行了一次哈希计算，因此最终作为参数传入到MPT接口的数据项其实是<code>(sha3(key), value)</code></p><p><strong>优势</strong>：</p><ul><li>传入MPT接口的key是固定长度的（32字节），可以避免出现树中出现长度很长的路径；</li></ul><p><strong>劣势</strong>：</p><ul><li>每次树操作需要增加一次哈希计算；</li><li>需要在数据库中存储额外的<code>sha3(key)</code>与<code>key</code>之间的对应关系；</li></ul><h3 id="3-基本操作"><a href="#3-基本操作" class="headerlink" title="3. 基本操作"></a><strong>3. 基本操作</strong></h3><p>介绍完MPT树的组成结构，在这一章将介绍MPT几种核心的基本操作。</p><h4 id="3-1-Get"><a href="#3-1-Get" class="headerlink" title="3.1 Get"></a><strong>3.1 Get</strong></h4><p>一次Get操作的过程为：</p><ol><li><p>将需要查找Key的Raw编码转换成Hex编码，得到的内容称之为<em>搜索路径</em>；</p></li><li><p>从根节点开始搜寻与</p><p>搜索路径</p><p>内容一致的路径；</p><ol><li>若当前节点为叶子节点，存储的内容是数据项的内容，且<em>搜索路径的内容</em>与叶子节点的key一致，则表示找到该节点；反之则表示该节点在树中不存在。</li><li>若当前节点为扩展节点，且存储的内容是哈希索引，则利用哈希索引从数据库中加载该节点，再将<em>搜索路径</em>作为参数，对新解析出来的节点递归地调用查找函数。</li><li>若当前节点为扩展节点，存储的内容是另外一个节点的引用，且当前节点的key是<em>搜索路径</em>的前缀，则将<em>搜索路径</em>减去当前节点的key，将剩余的<em>搜索路径</em>作为参数，对其子节点递归地调用查找函数；若当前节点的key不是<em>搜索路径</em>的前缀，表示该节点在树中不存在。</li><li>若当前节点为分支节点，若<em>搜索路径</em>为空，则返回分支节点的存储内容；反之利用<em>搜索路径</em>的第一个字节选择分支节点的孩子节点，将剩余的<em>搜索路径</em>作为参数递归地调用查找函数。<br><a href="https://github.com/fengyfei/wizard/blob/master/blockchain/images/get.jpeg?raw=true" target="_blank" rel="external">https://github.com/fengyfei/wizard/blob/master/blockchain/images/get.jpeg?raw=true</a><br><img src="/2018/03/16/merkle-patricia-tree/get.jpeg" alt="get"></li></ol></li></ol><p>上图是一次查找key为”cat“节点的过程。</p><ol><li>将key”cat”转换成hex编码[3,15,3,13,4,10,T] （在末尾添加终止符是因为需要查找一个真实的数据项内容）；</li><li>当前节点是根节点，且是扩展节点，其key为3,15，则递归地对其子节点进行查找调用，剩余的搜索路径为[3,13,4,10,T];</li><li>当前节点是分支节点，以搜索路径的第一个字节内容3选择第4个孩子节点递归进行查找，剩余的搜索路径为[13,4,10,T]；</li><li>当前节点是叶子节点，且key与剩余的搜索路径一致，表示找到了该节点，返回Val为“dog”；</li></ol><h4 id="3-2-Insert"><a href="#3-2-Insert" class="headerlink" title="3.2 Insert"></a><strong>3.2 Insert</strong></h4><p>插入操作也是基于查找过程完成的，一个插入过程为：</p><ol><li>根据3.1中描述的查找步骤，首先找到与新插入节点拥有<strong>最长相同路径前缀</strong>的节点，记为Node；</li><li>若该Node为分支节点：<ol><li>剩余的搜索路径不为空，则将新节点作为一个叶子节点插入到对应的孩子列表中；</li><li>剩余的搜索路径为空（完全匹配），则将新节点的内容存储在分支节点的第17个孩子节点项中（Value）；</li></ol></li><li>若该节点为叶子／扩展节点：<ol><li>剩余的搜索路径与当前节点的key一致，则把当前节点Val更新即可；</li><li>剩余的搜索路径与当前节点的key不完全一致，则将<strong>叶子／扩展节点的孩子节点替换成分支节点</strong>，将新节点与当前节点key的共同前缀作为当前节点的key，将新节点与当前节点的孩子节点作为两个孩子插入到分支节点的孩子列表中，同时当前节点转换成了一个扩展节点（若新节点与当前节点没有共同前缀，则直接用生成的分支节点替换当前节点）；</li></ol></li><li>若插入成功，则将被修改节点的dirty标志置为true，hash标志置空（之前的结果已经不可能用），且将节点的诞生标记更新为<em>现在</em>；</li></ol><p><img src="/2018/03/16/merkle-patricia-tree/insert.jpeg" alt="insert"></p><p>上图是一次将key为“cau”, value为“dog1”节点插入的过程。</p><ol><li>将key”cau”转换成hex编码[3,15,3,13,4,11,T] ；</li><li>通过查找算法，找到左图蓝线圈出的节点node1，且拥有与新插入节点最长的共同前缀[3,15,3,13,4]；</li><li>新增一个分支节点node2，将node1的val与新节点作为孩子插入到node2的孩子列表中，将node1的val替换成node2；</li><li>node1变成了一个扩展节点；</li></ol><h4 id="3-3-Delete"><a href="#3-3-Delete" class="headerlink" title="3.3 Delete"></a><strong>3.3 Delete</strong></h4><p>删除操作与插入操作类似，都需要借助查找过程完成，一次删除过程为：</p><ol><li>根据3.1中描述的查找步骤，找到与需要插入的节点拥有<strong>最长相同路径前缀</strong>的节点，记为Node；</li><li>若Node为叶子／扩展节点：<ol><li>若剩余的搜索路径与node的Key完全一致，则将整个node删除；</li><li>若剩余的搜索路径与node的key不匹配，则表示需要删除的节点不存于树中，删除失败；</li><li>若node的key是剩余搜索路径的前缀，则对该节点的Val做递归的删除调用；</li></ol></li><li>若Node为分支节点：<ol><li>删除孩子列表中相应下标标志的节点；</li><li>删除结束，若Node的孩子个数只剩下一个，那么将分支节点替换成一个叶子／扩展节点；</li></ol></li><li>若删除成功，则将被修改节点的dirty标志置为true，hash标志置空（之前的结果已经不可能用），且将节点的诞生标记更新为<em>现在</em>；</li></ol><p><img src="/2018/03/16/merkle-patricia-tree/delete_1.jpeg" alt="delete_1"></p><p><img src="/2018/03/16/merkle-patricia-tree/delete_2.jpeg" alt="delete_2"></p><p>上面两幅图是一次将key为“cat”, value为“dog1”节点删除的过程。</p><ol><li>将key”cau”转换成hex编码[3,15,3,13,4,11,T] ；</li><li>通过查找算法，找到用叉表示的节点node1，从根节点到node1的路径与搜索路径完全一致；</li><li>从node1的父节点中删除该节点，父节点仅剩一个孩子节点，故将父节点转换成一个叶子节点；</li><li>新生成的叶子节点又与其父节点（扩展节点）发生了合并，最终生成了一个叶子节点包含了所有的信息（图2）；</li></ol><h4 id="3-4-Update"><a href="#3-4-Update" class="headerlink" title="3.4 Update"></a><strong>3.4 Update</strong></h4><p>更新操作就是3.2Insert与3.3Delete的结合。当用户调用Update函数时，若value不为空，则隐式地转为调用Insert；若value为空，则隐式地转为调用Delete，故在此不再赘述。</p><p>####3.5 Commit</p><p>Commit函数提供将<em>内存</em>中的MPT数据持久化到<em>数据库</em>的功能。在第一章中我们提到的MPT具有<em>快速计算所维护数据集哈希标识</em>以<em>快速状态回滚</em>的能力，也都是在该函数中实现的。</p><p>在commit完成后，所有<strong>变脏</strong>的树节点会重新进行哈希计算，并且将新内容写入数据库；最终新的根节点哈希将被作为MPT的最新状态被返回。</p><p>一次MPT树提交是一个递归调用的过程，在介绍MPT提交过程之前，我们首先介绍单个节点是如何进行哈希计算和存储的。</p><p><strong>单节点</strong></p><ol><li><p>首先是对该节点进行脏位的判断，若当前节点未被修改，则直接返回该节点的哈希值，调用结束（此外，若当前节点既未被修改，同时存在于内存的时间又”过长“，则将以该节点为根节点的子树从内存中驱除）；</p></li><li><p>该节点为脏节点，对该节点进行哈希重计算。首先是对当前节点的孩子节点进行哈希计算，对孩子节点的哈希计算是利用递归地对节点进行处理完成。这一步骤的目的是<strong>将孩子节点的信息各自转换成一个哈希值进行表示</strong>；。</p></li><li><p>对当前节点进行哈希计算。哈希计算利用sha256哈希算法对当前节点的</p><p><em>RLP编码</em></p><p>进行哈希计算；</p><ol><li>对于分支节点来说，该节点的RLP编码就是对其孩子列表的内容进行编码，且在第二步中，所有的孩子节点所有已经被转换成了一个哈希值；</li><li>对于叶子／扩展节点来说，该节点的RLP编码就是对其Key，Value字段进行编码。同样在第二步中，若Value指代的是另外一个节点的引用，则已经被转换成了一个哈希值（在第二步中，Key已经被转换成了HP编码）；</li></ol></li><li><p>将当前节点的数据存入数据库，存储的格式为[节点哈希值，节点的RLP编码]。</p></li><li><p>将自身的dirty标志置为false，并将计算所得的哈希值进行缓存；</p></li></ol><p><strong>MPT树的提交过程</strong></p><p>在理解单节点的提交过程后，MPT树的提交过程就是以根节点为入口，对根节点进行提交调用即可。</p><p><img src="http://upyun-assets.ethfans.org/uploads/photo/image/6be30274b9944d13854f54daa2351746.jpeg" alt="img"></p><p>上图展示一棵MPT被持久化的过程：</p><p>左下角的叶子节点计算得到哈希为0xaa，将其存入数据库中，并在其父节点中用哈希值进行替换；粉色的扩展节点计算得到哈希为0xcc，在父节点用中0xcc进行替换；递归至根节点，计算得到根节点的哈希为0xee，即整棵树的哈希为0xee。</p><p><strong>节点过老的判断依据</strong></p><blockquote><p>判断一个节点在内存中存在时间是否过长的依据是：</p><ol><li>该节点未被修改；</li><li>当前MPT的计数器减去节点的诞生标志超过了固定的上限；</li><li>每当MPT调用一次Commit函数，MPT的计数器发生自增；</li></ol></blockquote><p><strong>实现功能</strong></p><p>1.快速计算所维护数据集哈希标识</p><p>这个特点体现在单节点计算的第一步，即在节点哈希计算之前会对该节点的状态进行判断，只有当该节点的内容变脏，才会进行哈希重计算、数据库持久化等操作。如此一来，在某一次事务操作中，对整棵MPT树的部分节点的内容产生了修改，那么一次哈希重计算，仅需对这些被修改的节点、以及从这些节点到根节点路径上的节点进行重计算，便能重新获得整棵树的新哈希。</p><p>2.快速状态回滚</p><p>在公链的环境下，采用POW算法是可能会造成分叉而导致区块链状态进行回滚的。在以太坊中，由于出块时间短，这种分叉的几率很大，区块链状态回滚的现象很频繁。</p><p>所谓的状态回滚指的是：（1）区块链内容发生了重组织，链头发生切换（2）区块链的世界状态（账户信息）需要进行回滚，即对之前的操作进行撤销。</p><p>MPT树就提供了一种机制，可以当区块碰撞发生了，零延迟地完成世界状态的回滚。这种优势的代价就是需要浪费存储空间去冗余地存储每个节点的历史状态。</p><p><strong>每个节点在数据库中的存储都是值驱动的</strong>。当一个节点的内容发生了变化，其哈希相应改变，而MPT将哈希作为数据库中的索引，也就实现了对于<strong>每一个值</strong>，在数据库中都有一条<strong>确定的记录</strong>。而MPT是根据节点哈希来关联父子节点的，因此每当一个节点的内容发生变化，最终对于父节点来说，改变的只是一个哈希索引值；父节点的内容也由此改变，产生了一个新的父节点，递归地将这种影响传递到根节点。<strong>最终，一次改变对应创建了一条从被改节点到根节点的新路径，而旧节点依然可以根据旧根节点通过旧路径访问得到</strong>。</p><p>示例：</p><p><img src="http://upyun-assets.ethfans.org/uploads/photo/image/4e252f20afd44b5c8ffd6890417ce690.jpeg" alt="img"></p><p>在上图中，一个节点的内容由27变为45，就对应成创建了一条由蓝线圈出的新路径，通过复用绿线圈出的未修改节点信息，构造一棵新树，而旧路径依旧保留。故通过旧stateRoot，我们依旧能够查询到该节点的值为27。</p><p>所以，在以太坊中，发生分叉而进行世界状态回滚时，仅需要用旧的MPT根节点作为入口，即可完成“状态回滚”。</p><h3 id="4-轻节点扩展"><a href="#4-轻节点扩展" class="headerlink" title="4. 轻节点扩展"></a><strong>4. 轻节点扩展</strong></h3><p>接下来来介绍一个默克尔树，MPT能够提供的一个重要功能 - 默克尔证明，使用默克尔证明能够实现轻节点的扩展。</p><h4 id="4-1-什么是轻节点"><a href="#4-1-什么是轻节点" class="headerlink" title="4.1 什么是轻节点"></a><strong>4.1 什么是轻节点</strong></h4><p>在以太坊或比特币中，一个参与共识的全节点通常会维护整个区块链的数据，每个区块中的区块头信息，所有的交易，回执信息等。由于区块链的不可篡改性，这将导致随着时间的增加，整个区块链的数据体量会非常庞大。运行在个人PC或者移动终端的可能性显得微乎其微。为了解决这个问题，一种轻量级的，<strong>只存储区块头部信息</strong>的节点被提出。这种节点只需要维护链中所有的区块头信息（一个区块头的大小通常为几十个字节，普通的移动终端设备完全能够承受出）。</p><p>在公链的环境下，<strong>仅仅通过本地所维护的区块头信息</strong>，轻节点就能够证明某一笔交易是否存在与区块链中；某一个账户是否存在与区块链中，其余额是多少等功能。</p><h4 id="4-2-什么是默克尔证明"><a href="#4-2-什么是默克尔证明" class="headerlink" title="4.2 什么是默克尔证明"></a><strong>4.2 什么是默克尔证明</strong></h4><p>默克尔证明指一个轻节点向一个全节点发起一次证明请求，询问全节点完整的默克尔树中，是否存在一个指定的节点；全节点向轻节点返回一个默克尔证明路径，由轻节点进行计算，验证存在性。</p><h4 id="4-3-默克尔证明过程"><a href="#4-3-默克尔证明过程" class="headerlink" title="4.3 默克尔证明过程"></a><strong>4.3 默克尔证明过程</strong></h4><p>如有棵如下图所示的merkle树，如果某个轻节点想要验证<code>9Dog:64</code>这个树节点是否存在与默克尔树中，只需要向全节点发送该请求，全节点会返回一个<code>1FXq:18</code>, <code>ec20</code>,<br><code>8f74</code>的一个路径（默克尔路径，如图2黄色框所表示的）。得到路径之后，轻节点利用<code>9Dog:64</code>与<code>1FXq:18</code>求哈希，在与<code>ec20</code>求哈希，最后与<code>8f74</code>求哈希，得到的结果与本地维护的根哈希相比，是否相等。</p><p><img src="http://upyun-assets.ethfans.org/uploads/photo/image/362f6e0cac7d4a50b06bd95f7e6d1cfc.jpeg?raw=true" alt="img"></p><p><img src="http://upyun-assets.ethfans.org/uploads/photo/image/5409681f672c465db8b49ee4ac6f4dd0.jpeg?raw=true" alt="img"></p><h4 id="4-4默克尔证明安全性"><a href="#4-4默克尔证明安全性" class="headerlink" title="4.4默克尔证明安全性"></a><strong>4.4默克尔证明安全性</strong></h4><p>（1）若全节点返回的是一条恶意的路径？试图为一个不存在于区块链中的节点伪造一条合法的merkle路径，使得最终的计算结果与区块头中的默克尔根哈希相同。</p><p>由于哈希的计算具有不可预测性，使得一个恶意的“全”节点想要为一条不存在的节点伪造一条“伪路径”使得最终计算的根哈希与轻节点所维护的根哈希相同是不可能的。</p><p>（2）为什么不直接向全节点请求该节点是否存在于区块链中？</p><p>由于在公链的环境中，无法判断请求的全节点是否为恶意节点，因此直接向某一个或者多个全节点请求得到的结果是无法得到保证的。但是轻节点本地维护的区块头信息，是经过工作量证明验证的，也就是经过共识一定正确的，若利用全节点提供的默克尔路径，与代验证的节点进行哈希计算，若最终结果与本地维护的区块头中根哈希一致，则能够证明该节点一定存在于默克尔树中。</p><h4 id="4-5-简单支付验证"><a href="#4-5-简单支付验证" class="headerlink" title="4.5 简单支付验证"></a><strong>4.5 简单支付验证</strong></h4><p>在以太坊中，利用默克尔证明在轻节点中实现简单支付验证，即在无需维护具体交易信息的前提下，证明某一笔交易是否存在于区块链中。</p>]]></content>
<summary type="html">
<h3 id="1-前言"><a href="#1-前言" class="headerlink" title="1. 前言"></a><strong>1. 前言</strong></h3><h4 id="1-1-概述"><a href="#1-1-概述" class="headerlink" title="1.1 概述"></a><strong>1.1 概述</strong></h4><p>Merkle Patricia Tree（又称为Merkle Patricia Trie）是一种经过改良的、融合了默克尔树和前缀树两种树结构优点的数据结构，是以太坊中用来组织管理账户数据、生成交易集合哈希的重要数据结构。<br></p>
</summary>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/categories/BlockChain/"/>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/tags/BlockChain/"/>
</entry>
<entry>
<title>初识以太坊</title>
<link href="https://yangchenglong11.github.io/2018/03/13/the-basic-of-ethereum/"/>
<id>https://yangchenglong11.github.io/2018/03/13/the-basic-of-ethereum/</id>
<published>2018-03-13T04:42:40.000Z</published>
<updated>2018-03-31T06:22:31.474Z</updated>
<content type="html"><![CDATA[<p>前面几节我们介绍了比特币的一些情况，比特币是一个很伟大的发明，但它仍存在一些问题，比如我们上节讲过，比特币系统支持简单的脚本，但它的脚本语言存在一些严重的限制：<br><a id="more"></a></p><ul><li><strong>缺少图灵完备性</strong> – 这就是说，尽管比特币脚本语言可以支持多种计算，但是它不能支持所有的计算。最主要的缺失是循环语句。不支持循环语句的目的是避免交易确认时出现无限循环。理论上，对于脚本程序员来说，这是可以克服的障碍，因为任何循环都可以用多次重复if 语句的方式来模拟，但是这样做会导致脚本空间利用上的低效率，例如，实施一个替代的椭圆曲线签名算法可能将需要256次重复的乘法，而每次都需要单独编码。</li><li><strong>价值盲（Value-blindness）</strong>。UTXO脚本不能为账户的取款额度提供精细的的控制。例如，预言机合约（oracle contract）的一个强大应用是对冲合约，A和B各自向对冲合约中发送价值1000美元的比特币，30天以后，脚本向A发送价值1000美元的比特币，向B发送剩余的比特币。虽然实现对冲合约需要一个预言机（oracle）决定一比特币值多少美元，但是与现在完全中心化的解决方案相比，这一机制已经在减少信任和基础设施方面有了巨大的进步。然而，因为UTXO是不可分割的，为实现此合约，唯一的方法是非常低效地采用许多有不同面值的UTXO（例如对应于最大为30的每个k，有一个2^k的UTXO)并使预言机挑出正确的UTXO发送给A和B。</li><li><strong>缺少状态</strong> – UTXO只能是已花费或者未花费状态，这就没有给需要任何其它内部状态的多阶段合约或者脚本留出生存空间。这使得实现多阶段期权合约、去中心化的交换要约或者两阶段加密承诺协议（对确保计算奖励非常必要）非常困难。这也意味着UTXO只能用于建立简单的、一次性的合约，而不是例如去中心化组织这样的有着更加复杂的状态的合约，使得元协议难以实现。二元状态与价值盲结合在一起意味着另一个重要的应用-取款限额-是不可能实现的。</li><li><strong>区块链盲（Blockchain-blindness）</strong>- UTXO看不到区块链的数据，例如随机数和上一个区块的哈希。这一缺陷剥夺了脚本语言所拥有的基于随机性的潜在价值，严重地限制了博彩等其它领域应用。</li></ul><p>除了脚本语言外，比特币的共识机制也会导致很多的算力浪费，虽然这样是为了保持全局状态唯一。而且区块被挖掘的速度太慢，对于正常的中心化的金融系统来说，每秒千万条交易是很常见的，但在比特币区块链中很难达到这样的交易量。比特币的扩容问题严重限制了其在金融领域的落地，虽然有闪电网络的出现来解决这些问题，但总是不尽如人意。</p><p>以太坊（Ethereum）是一个建立在区块链技术之上的去中心化应用平台。它允许任何人在平台中建立和使用通过区块链技术运行的去中心化应用。如果说比特币是一个分布式账本的话，那么以太坊则是一个全球的计算机。以太坊通过建立终极的抽象的基础层-内置有图灵完备编程语言的区块链-使得任何人都能够创建合约和去中心化应用并在其中设立他们自由定义的所有权规则、交易方式和状态转换函数。</p><h3 id="以太坊账户"><a href="#以太坊账户" class="headerlink" title="以太坊账户"></a>以太坊账户</h3><p>在以太坊系统中，状态是由被称为“账户”（每个账户由一个20字节的地址）的对象和在两个账户之间转移价值和信息的状态转换构成的。以太坊的账户包含四个部分：</p><ul><li>随机数，用于确定每笔交易只能被处理一次的计数器</li><li>账户目前的以太币余额</li><li>账户的合约代码，如果有的话</li><li>账户的存储（默认为空）</li></ul><p>以太币（Ether）是以太坊内部的主要加密燃料，用于支付交易费用。一般而言，以太坊有两种类型的账户：外部所有的账户（由私钥控制的）和合约账户（由合约代码控制）。外部所有的账户没有代码，人们可以通过创建和签名一笔交易从一个外部账户发送消息。每当合约账户收到一条消息，合约内部的代码就会被激活，允许它对内部存储进行读取和写入，和发送其它消息或者创建合约。</p><h3 id="消息和交易"><a href="#消息和交易" class="headerlink" title="消息和交易"></a>消息和交易</h3><p>以太坊的消息在某种程度上类似于比特币的交易，但是两者之间存在三点重要的不同。第一，以太坊的消息可以由外部实体或者合约创建，然而比特币的交易只能从外部创建。第二，以太坊消息可以选择包含数据。第三，如果以太坊消息的接受者是合约账户，可以选择进行回应，这意味着以太坊消息也包含函数概念。</p><p>以太坊中“交易”是指存储从外部账户发出的消息的签名数据包。有两种类型的交易：<strong>消息通信</strong>和<strong>合约创建(也就是交易产生一个新的以太坊合约)</strong>。交易包含消息的接收者、用于确认发送者的签名、以太币账户余额、要发送的数据和两个被称为STARTGAS和GASPRICE的数值。为了防止代码的指数型爆炸和无限循环，每笔交易需要对执行代码所引发的计算步骤-包括初始消息和所有执行中引发的消息-做出限制。STARTGAS就是限制，有的地方也叫 gas limit ，GASPRICE是每一计算步骤需要支付矿工的费用。如果执行交易的过程中，“用完了瓦斯”，所有的状态改变恢复原状态，但是已经支付的交易费用不可收回了。如果执行交易中止时还剩余瓦斯，那么这些瓦斯将退还给发送者。创建合约有单独的交易类型和相应的消息类型；合约的地址是基于账号随机数和交易数据的哈希计算出来的。</p><p>消息机制的一个重要后果是以太坊的“头等公民”财产-合约与外部账户拥有同样权利，包括发送消息和创建其它合约的权利。这使得合约可以同时充当多个不同的角色，例如，用户可以使去中心化组织（一个合约）的一个成员成为一个中介账户（另一个合约），为一个偏执的使用定制的基于量子证明的兰波特签名（第三个合约）的个人和一个自身使用由五个私钥保证安全的账户（第四个合约）的共同签名实体提供居间服务。以太坊平台的强大之处在于去中心化的组织和代理合约不需要关心合约的每一参与方是什么类型的账户。</p><h3 id="代码执行"><a href="#代码执行" class="headerlink" title="代码执行"></a>代码执行</h3><p>以太坊合约的代码使用低级的基于堆栈的字节码的语言写成的，被称为“以太坊虚拟机代码”或者“EVM代码”。代码由一系列字节构成，每一个字节代表一种操作。一般而言，代码执行是无限循环，程序计数器每增加一（初始值为零）就执行一次操作，直到代码执行完毕或者遇到错误，<code>STOP</code>或者<code>RETURN</code>指令。操作可以访问三种存储数据的空间：</p><ul><li><strong>堆栈</strong>，一种后进先出的数据存储，32字节的数值可以入栈，出栈。</li><li><strong>内存</strong>，可无限扩展的字节队列。</li><li><strong>合约的长期存储</strong>，一个秘钥/数值的存储，其中秘钥和数值都是32字节大小，与计算结束即重置的堆栈和内存不同，存储内容将长期保持。</li></ul><p>代码可以象访问区块头数据一样访问数值，发送者和接受到的消息中的数据，代码还可以返回数据的字节队列作为输出。</p><p>EVM代码的正式执行模型令人惊讶地简单。当以太坊虚拟机运行时，它的完整的计算状态可以由元组<code>(block_state, transaction, message, code, memory, stack, pc, gas)</code>来定义，这里<code>block_state</code>是包含所有账户余额和存储的全局状态。每轮执行时，通过调出代码的第<code>pc</code>（程序计数器）个字节，当前指令被找到，每个指令都有定义自己如何影响元组。例如，<code>ADD</code>将两个元素出栈并将它们的和入栈，将<code>gas</code>（瓦斯）减一并将<code>pc</code>加一，<code>SSTORE</code>将顶部的两个元素出栈并将第二个元素插入到由第一个元素定义的合约存储位置，同样减少最多200的gas值并将<code>pc</code>加一，虽然有许多方法通过即时编译去优化以太坊，但以太坊的基础性的实施可以用几百行代码实现。</p><h3 id="区块链和挖矿"><a href="#区块链和挖矿" class="headerlink" title="区块链和挖矿"></a>区块链和挖矿</h3><p>虽然有一些不同，但以太坊的区块链在很多方面类似于比特币区块链。它们的区块链架构的不同在于，以太坊区块不仅包含交易记录和最近的状态，还包含区块序号和难度值。以太坊中的区块确认算法如下：</p><ol><li>检查区块引用的上一个区块是否存在和有效。</li><li>检查区块的时间戳是否比引用的上一个区块大，而且小于15分钟。</li><li>检查区块序号、难度值、 交易根，叔根和瓦斯限额（许多以太坊特有的底层概念）是否有效。</li><li>检查区块的工作量证明是否有效。</li><li>将<code>S[0]</code>赋值为上一个区块的<code>STATE_ROOT</code>。</li><li>将<code>TX</code>赋值为区块的交易列表，一共有<code>n</code>笔交易。对于属于<code>0……n-1</code>的<code>i</code>，进行状态转换<code>S[i+1] = APPLY(S[i],TX[i])</code>。如果任何一个转换发生错误，或者程序执行到此处所花费的瓦斯（gas）超过了<code>GASLIMIT</code>，返回错误。</li><li>用<code>S[n]</code>给<code>S_FINAL</code>赋值, 向矿工支付区块奖励。</li><li>检查<code>S-FINAL</code>是否与<code>STATE_ROOT</code>相同。如果相同，区块是有效的。否则，区块是无效的。</li></ol><p>这一确认方法乍看起来似乎效率很低，因为它需要存储每个区块的所有状态，但是事实上以太坊的确认效率可以与比特币相提并论。原因是状态存储在树结构中（tree structure），每增加一个区块只需要改变树结构的一小部分。因此，一般而言，两个相邻的区块的树结构的大部分应该是相同的，因此存储一次数据，可以利用指针（即子树哈希）引用两次。一种被称为“帕特里夏树”（“Patricia Tree”）的树结构可以实现这一点，其中包括了对默克尔树概念的修改，不仅允许改变节点，而且还可以插入和删除节点。另外，因为所有的状态信息是最后一个区块的一部分，所以没有必要存储全部的区块历史-这一方法如果能够可以应用到比特币系统中，经计算可以对存储空间有10-20倍的节省。</p><p>###交易流程</p><p>现在我们看下具体的一笔交易是如何执行的。假设你发送了一笔交易给以太坊网络处理，将以太坊状态转换成包含你的交易这个过程到底发生了什么？</p><p><img src="/2018/03/13/the-basic-of-ethereum/transaction.png" alt="transaction"></p><p>首先，为了可以被执行所有的交易必须都要符合最基础的一系列要求，包括：</p><ul><li>交易必须是正确格式化的RLP。”RLP”代表Recursive Length Prefix，它是一种数据格式，用来编码二进制数据嵌套数组。以太坊就是使用RLP格式序列化对象。</li><li>有效的交易签名。</li><li>有效的交易序号。账户中的nonce就是从此账户发送出去交易的计数。如果有效，那么交易序号一定等于发送账户中的nonce。</li><li>交易的STARTGAS 一定要等于或者大于交易使用的<strong>intrinsic gas</strong>，<strong>intrinsic gas</strong>包括：——-1.执行交易预订费用为21,000gas——-2.随交易发送的数据的gas费用（每字节数据或代码为0的费用为4gas，每个非零字节的数据或代码费用为68gas）——-3.如果交易是合约创建交易，还需要额外的32,000gas</li></ul><p><img src="/2018/03/13/the-basic-of-ethereum/intrinsic.png" alt="intrinsic"></p><ul><li>发送账户余额必须有足够的Ether来支付”前期”gas费用。前期gas费用的计算比较简单：首先，交易的STARTGAS乘以交易的GASPRICE得到最大的gas费用。然后，这个最大gas费用被加到从发送方传送给接收方的总值。如果账户余额不足，返回错误。</li></ul><p><img src="/2018/03/13/the-basic-of-ethereum/upfront.png" alt="upfront"></p><p>如何交易符合上面所说的所有要求，那么我们进行下面步骤。</p><p>第一步，我们从发送者的余额中扣除执行的前期费用，并为当前交易将发送者账户中的nonce增加1。此时，我们可以计算剩余的gas，将交易的总gas减去使用的<strong>intrinsic gas</strong>。</p><p><img src="/2018/03/13/the-basic-of-ethereum/remaining.png" alt="remaining"></p><p>第二步，开始执行交易。首先从发送者的账户转移价值到接收者账户。如果接收账户还不存在，创建此账户。如果接收账户是一个合约，运行合约的代码，直到代码运行结束或者瓦斯用完。如果没有合约接收交易，那么所有的交易费用就等于GASPRICE乘以交易的字节长度，交易的数据就与交易费用无关了。另外，需要注意的是，合约发起的消息可以对它们产生的计算分配瓦斯限额，如果子计算的瓦斯用完了，它只恢复到消息发出时的状态。因此，就像交易一样，合约也可以通过对它产生的子计算设置严格的限制，保护它们的计算资源。在交易执行的整个过程中，以太坊保持跟踪“子状态”。子状态是记录在交易中生成的信息的一种方式，当交易完成时会立即需要这些信息。具体来说，它包含：</p><ul><li>自毁集：在交易完成之后会被丢弃的账户集（如果存在的话）</li><li>日志系列：虚拟机的代码执行的归档和可检索的检查点</li><li>退款余额：交易完成之后需要退还给发送账户的总额。我们之前提到的以太坊中的存储需要付费，发送者要是清理了内存就会有退款。以太坊使用退款计数进行跟踪退款余额。退款计数从0开始并且每当合约删除了一些存储中的东西都会进行增加。</li></ul><p>第三步，交易所需的各种计算开始被处理。</p><p>当交易所需的步骤全部处理完成，并假设没有无效状态，通过确定退还给发送者的未使用的gas量，最终的状态也被确定。除了未使用的gas，发送者还会得到上面所说的“退款余额”中退还的一些津贴。</p><p>一旦发送者得到退款之后：</p><ul><li>gas的Ether就会矿工</li><li>交易使用的gas会被添加到区块的gas计数中（计数一直记录当前区块中所有交易使用的gas总量，这对于验证区块时是非常有用的）</li><li>所有在<strong>自毁集</strong>中的账户（如果存在的话）都会被删除</li></ul><p>最后，我们就有了一个新的状态以及交易创建的一系列日志。</p><h2 id="智能合约示例："><a href="#智能合约示例：" class="headerlink" title="智能合约示例："></a>智能合约示例：</h2><p><strong>储蓄钱包</strong>。 假设Alice想确保她的资金安全，但她担心丢失或者被黑客盗走私钥。她把以太币放到和Bob签订的一个合约里，如下所示，这合同是一个银行：</p><ul><li>Alice单独每天最多可提取1%的资金。</li><li>Bob单独每天最多可提取1%的资金，但Alice可以用她的私钥创建一个交易取消Bob的提现权限。</li><li>Alice 和 Bob 一起可以任意提取资金。 </li></ul><p>一般来讲，每天1%对Alice足够了，如果Alice想提现更多她可以联系Bob寻求帮助。如果Alice的私钥被盗，她可以立即找到Bob把她的资金转移到一个新合同里。如果她弄丢了她的私钥，Bob可以慢慢地把钱提出。如果Bob表现出了恶意，她可以关掉他的提现权限。</p>]]></content>
<summary type="html">
<p>前面几节我们介绍了比特币的一些情况，比特币是一个很伟大的发明，但它仍存在一些问题，比如我们上节讲过，比特币系统支持简单的脚本，但它的脚本语言存在一些严重的限制：<br></p>
</summary>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/categories/BlockChain/"/>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/tags/BlockChain/"/>
</entry>
<entry>
<title>比特币分析之交易、脚本</title>
<link href="https://yangchenglong11.github.io/2018/03/11/transaction-and-scripts-of-bitcoin/"/>
<id>https://yangchenglong11.github.io/2018/03/11/transaction-and-scripts-of-bitcoin/</id>
<published>2018-03-11T03:42:40.000Z</published>
<updated>2018-03-31T06:21:54.658Z</updated>
<content type="html"><![CDATA[<p>在此之前，我们已经对比特币有了一个大体上的了解，总的来说它就是一个分布式账本。上个文章中我们只是按照通俗的说法讲解了比特币，这篇文章我们更加细致的看下比特币中交易是如何完成的。我们日常的每笔交易是这样的：张三账上减￥200，李四账上加￥200。在比特币区块链中，交易不是这么简单，交易实际是通过脚本来完成，以承载更多的功能。</p><a id="more"></a><h2 id="未花费的交易输出-UTXO"><a href="#未花费的交易输出-UTXO" class="headerlink" title="未花费的交易输出(UTXO)"></a>未花费的交易输出(UTXO)</h2><p>先引入一个概念：未花费的交易输出——UTXO（Unspent Transaction Output）</p><p>比特币及其许多衍生品，都将用户的余额信息存储在UTXO结构中，系统的整个状态由一系列的“有效的输出”组成（可以将这些“有效的输出”想象成钱币）。每个UTXO都有拥有者和自身的价值属性。一笔交易在消费若干个UTXO同时也会生成若干个新的UTXO。比特币的交易都是基于UTXO上的，即交易的输入是之前交易未花费的输出，这笔交易的输出可以被当做下一笔新交易的输入。“有效的输出”中“有效”需满足下面几点约束：</p><p>1.每个被引用的输入必须有效，且未被使用过；<br>2.交易的签名必须与每笔输入的所有者签名匹配；<br>3.输入的总值必须等于或大于输出的总值。</p><p>因此，比特币系统中，用户的“余额”是该用户的私钥能够有效签名的所有UTXO的总和。</p><blockquote><p>挖矿奖励属于一个特殊的交易（称为coinbase交易），可以没有输入。<br>UTXO是交易的基本单元，不能再分割。<br>在比特币没有余额概念，只有分散到区块链里的UTXO</p></blockquote><p>随着钱从一个地址被移动到另一个地址的同时形成了一条所有权链，像这样：<br><img src="/2018/03/11/transaction-and-scripts-of-bitcoin/own_chain.jpg" alt="own_chain"></p><h2 id="比特币脚本"><a href="#比特币脚本" class="headerlink" title="比特币脚本"></a>比特币脚本</h2><p>比特币交易是首先要提供一个用于解锁UTXO（用私钥去匹配锁定脚本）的脚本（常称为解锁脚本：Signature script)，这也叫交易输入，<br>交易的输出则是指向一个脚本（称为锁定脚本：PubKey script），这个脚本表达了：谁的签名（签名是常见形式，并不一定必须是签名）能匹配这个输出地址，钱就支付给谁。</p><p>每一个比特币节点会通过同时执行这解锁和锁定脚本（不是当前的锁定脚本，是指上一个交易的锁定脚本）来验证一笔交易，脚本组合结果为真，则为有效交易。</p><blockquote><p>当解锁版脚本与锁定版脚本的设定条件相匹配时，执行组合有效脚本时才会显示结果为真</p></blockquote><p>如最为常见类型的比特币交易脚本（支付到公钥哈希：P2PKH（Pay-to-Public-Key-Hash））组合是这样：<br><img src="/2018/03/11/transaction-and-scripts-of-bitcoin/script.jpg" alt="script"></p><h3 id="常见交易脚本验证过程"><a href="#常见交易脚本验证过程" class="headerlink" title="常见交易脚本验证过程"></a>常见交易脚本验证过程</h3><p>比特币交易脚本语言是一种基于逆波兰表示法的基于栈的执行语言。</p><blockquote><p>比特币脚本语言包含基本算数计算、基本逻辑(比如if…then)、报错以及返回结果和一些加密指令，不支持循环。</p></blockquote><p>脚本语言通过从左至右地处理每个项目的方式执行脚本。</p><p>下面用两个图说明下常见类型的比特币交易脚本验证执行过程：<br><img src="/2018/03/11/transaction-and-scripts-of-bitcoin/script_run1.jpg" alt="script_run1"><br>上图为解锁脚本运行过程（主要是入栈）<br><img src="/2018/03/11/transaction-and-scripts-of-bitcoin/script_run2.jpg" alt="script_run2"><br>上图为锁定脚本运行过程（主要是出栈），最后的结果为真，说明交易有效。</p><h2 id="交易分析"><a href="#交易分析" class="headerlink" title="交易分析"></a>交易分析</h2><p>实际上比特币的交易被设计为可以纳入多个输入和输出。</p><h3 id="交易结构"><a href="#交易结构" class="headerlink" title="交易结构"></a>交易结构</h3><p>我们来看看完整的交易结构，<br><img src="/2018/03/11/transaction-and-scripts-of-bitcoin/tx_struce.jpg" alt="img"></p><blockquote><p>交易的锁定时间定义了能被加到区块链里的最早的交易时间。在大多数交易里，它被设置成0，用来表示立即执行。<br>如果锁定时间不是0并且小于5亿，就被视为区块高度，意指在这个指定的区块高度之前，该交易不会被包含在区块链里。<br>如果锁定时间大于5亿，则它被当作是一个Unix纪元时间戳（从1970年1月1日以来的秒数），并且在这个指定时间之前，该交易不会被包含在区块链里。</p></blockquote><p>交易的数据结构没有交易费的字段，交易费通过所有输入的总和，以及所有输出的总和之间的差来表示，即：</p><blockquote><p>交易费 = 求和（所有输入） - 求和（所有输出）</p></blockquote><h3 id="交易输入结构"><a href="#交易输入结构" class="headerlink" title="交易输入结构"></a>交易输入结构</h3><p>刚刚我们提过输入需要提供一个解锁脚本，现在来看看一个交易的输入结构：<br><img src="/2018/03/11/transaction-and-scripts-of-bitcoin/tx_input_struce.jpg" alt="img"></p><p>我们结合整个交易的结构里看输入结构就是这样子：<br><img src="/2018/03/11/transaction-and-scripts-of-bitcoin/tx_input_overview.jpg" alt="img"></p><h3 id="交易输出结构"><a href="#交易输出结构" class="headerlink" title="交易输出结构"></a>交易输出结构</h3><p>刚刚我们提过输出是指向一个解锁脚本，具体交易的输出结构为：<br><img src="/2018/03/11/transaction-and-scripts-of-bitcoin/tx_output_struce.jpg" alt="img"><br>我们结合整个交易的结构里看输出结构就是这样子：<br><img src="/2018/03/11/transaction-and-scripts-of-bitcoin/tx_output_overview.jpg" alt="img"></p><p>我们看一下通过比特币核心的 API 获得的真正的交易数据：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div></pre></td><td class="code"><pre><div class="line">{</div><div class="line"><span class="attr">"txid"</span>: <span class="string">"0627052b6f28912f2703066a912ea577f2ce4da4caa5a5fbd8a57286c345c2f2"</span>,</div><div class="line"><span class="attr">"size"</span>: <span class="number">258</span>,</div><div class="line"> <span class="attr">"version"</span>: <span class="number">1</span>,</div><div class="line"> <span class="attr">"locktime"</span>: <span class="number">0</span>,</div><div class="line"> <span class="attr">"vin"</span>: </div><div class="line"> [</div><div class="line"> {</div><div class="line"> <span class="attr">"txid"</span>:<span class="string">"7957a35fe64f80d234d76d83a2...8149a41d81de548f0a65a8a999f6f18"</span>,</div><div class="line"> <span class="attr">"vout"</span>:<span class="number">0</span>,</div><div class="line"> <span class="attr">"scriptSig"</span>: </div><div class="line"> {</div><div class="line"> <span class="attr">"asm"</span>:<span class="string">"3045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1decc..."</span>,</div><div class="line"><span class="attr">"hex"</span>:<span class="string">"483045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1de..."</span> </div><div class="line">},</div><div class="line"> <span class="attr">"sequence"</span>: <span class="number">4294967295</span></div><div class="line"> }</div><div class="line"> ], </div><div class="line"> <span class="attr">"vout"</span>: </div><div class="line"> [</div><div class="line"> {</div><div class="line"> <span class="attr">"value"</span>: <span class="number">0.01500000</span>,</div><div class="line"> <span class="attr">"n"</span>: <span class="number">0</span>,</div><div class="line"> <span class="attr">"scriptPubKey"</span>: </div><div class="line"> {</div><div class="line"><span class="attr">"asm"</span>: <span class="string">"OP_DUP OP_HASH160 ab68...5f654e7 OP_EQUALVERIFY OP_CHECKSIG"</span>,</div><div class="line"><span class="attr">"hex"</span>: <span class="string">"76a914ab68025513c3dbd2f7b92a94e0581f5d50f654e788ac"</span>, </div><div class="line"><span class="attr">"reqSigs"</span>: <span class="number">1</span>,</div><div class="line"><span class="attr">"type"</span>: <span class="string">"pubkeyhash"</span>,</div><div class="line"><span class="attr">"addresses"</span>: [<span class="string">"1GdK9UzpHBzqzX2A9JFP3Di4weBwqgmoQA"</span> ]</div><div class="line">} </div><div class="line"> },</div><div class="line"> {</div><div class="line"> <span class="attr">"value"</span>: <span class="number">0.08450000</span>,</div><div class="line"> <span class="attr">"n"</span>: <span class="number">1</span>,</div><div class="line"> <span class="attr">"scriptPubKey"</span>: </div><div class="line"> {</div><div class="line"><span class="attr">"asm"</span>: <span class="string">"OP_DUP OP_HASH160 7f9b1a...025a8 OP_EQUALVERIFY OP_CHECKSIG"</span>,</div><div class="line"><span class="attr">"hex"</span>: <span class="string">"76a9147f9b1a7fb68d60c536c2fd8aeaa53a8f3cc025a888ac"</span>, <span class="attr">"reqSigs"</span>: <span class="number">1</span>,</div><div class="line"><span class="attr">"type"</span>: <span class="string">"pubkeyhash"</span>,</div><div class="line"><span class="attr">"addresses"</span>: [<span class="string">"1Cdid9KFAaatwczBwBttQcwXYCpvK8h7FK"</span> ]</div><div class="line">} </div><div class="line"> }</div><div class="line">] </div><div class="line">}</div></pre></td></tr></table></figure><h2 id="智能合约雏形-应用场景说明"><a href="#智能合约雏形-应用场景说明" class="headerlink" title="智能合约雏形 - 应用场景说明"></a>智能合约雏形 - 应用场景说明</h2><p>由于交易是通过脚本来实现，脚本语言可以表达出无数的条件变种。</p><p>比特币的脚本目前常用的主要分为两种，一种是常见的P2PKH（支付给公钥哈希），另一种是P2SH（Pay-to-Script-Hash支付脚本哈希）。<br>P2SH支付中，锁定脚本被密码学哈希所取代，当一笔交易试图支付UTXO时，要解锁支付脚本，它必须含有与哈希相匹配的脚本。</p><p>这里不展开技术细节，下面说明一些应用场景，以便大家有更直观的认识。</p><ul><li><p>多重签名应用<br>合伙经营中，如只有一半以上的的股东同意签名就可以进行支付，可为公司治理提供管控便利，同时也能有效防范盗窃、挪用和遗失。</p><p>用于担保和争端调解，一个买家想和他不认识或不信任的某人交易，在一般情况交易正常进行时，买家不想任何第三方参与。那交易双方可以发起支付，但如果交易出现问题时，那第三方就可以根据裁定，使用自己的签名和裁定认可的一方共同签名来兑现这笔交易。</p></li><li><p>保证合同<br>保证合同是建造公众商品时的集资办法，公众商品是指一旦建成，任何人都可以免费享受到好处。标准的例子是灯塔，所有人都认同应该建造一个，但是对于个人航海者来说灯塔太贵了，灯塔同时也会方便其他航海者。<br>一个解决方案是向所有人集资，只有当筹集的资金超过所需的建造成本时，每个人才真正付钱，如果集资款不足，则谁都不用付钱。</p></li><li><p>依靠预言<br>假如老人想让他孙子继承遗产，继承时间是在他死后或者在孙子年满18岁时（也是一个带锁定时间交易），无论哪个条件先满足，他的孙子都可以得到遗产。<br>因为比特币节点可依靠预言对死亡条件进行判断，预言是指具有密钥对的服务器，当用户自定义的表达式被证明是真的，它能按照要求对交易签名。</p></li></ul><p>相信随着区块链的普及，会对未来的交易模式和商业结构带来巨大的影响。不过由于比特币的脚本语言不是图灵完备的，交易模式依旧有限，以太坊就是为解决这一问题而出现，之后的文章我会介绍以太坊是什么。</p>]]></content>
<summary type="html">
<p>在此之前，我们已经对比特币有了一个大体上的了解，总的来说它就是一个分布式账本。上个文章中我们只是按照通俗的说法讲解了比特币，这篇文章我们更加细致的看下比特币中交易是如何完成的。我们日常的每笔交易是这样的：张三账上减￥200，李四账上加￥200。在比特币区块链中，交易不是这么简单，交易实际是通过脚本来完成，以承载更多的功能。</p>
</summary>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/categories/BlockChain/"/>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/tags/BlockChain/"/>
</entry>
<entry>
<title>区块链是什么</title>
<link href="https://yangchenglong11.github.io/2018/03/09/the-basic-of-bitcoin/"/>
<id>https://yangchenglong11.github.io/2018/03/09/the-basic-of-bitcoin/</id>
<published>2018-03-09T04:42:40.000Z</published>
<updated>2018-03-31T06:35:37.890Z</updated>
<content type="html"><![CDATA[<p>区块链是一个<strong>具有共享状态的密码性安全交易的单机</strong>。<br><a id="more"></a></p><ul><li><strong>“密码性安全(Cryptographically secure)”</strong>是指用一个很难被解开的复杂数学机制算法来保证数字货币生产的安全性。将它想象成类似于防火墙的这种。它们使得欺骗系统近乎是一个不可能的事情（比如：构造一笔假的交易，消除一笔交易等等）。</li><li><strong>“交易的单机(Transactional singleton machine)”</strong>是指只有一个权威的机器实例为系统中产生的交易负责任。换句话说，只有一个全球真相是大家所相信的。</li><li><strong>“具有共享状态(With shared-state)”</strong>是指在这台机器上存储的状态是共享的，对每个人都是开放的。</li></ul><p>上面所说的复杂数学机制算法主要是指（1）密码学哈希函数，（2）非对称加密。简单的说，密码学哈希函数就是指任意长度的字符串、甚至文件体本身经过Hash函数工厂的加工，都会输出一个固定长度的字符串；同时，输入的字符串或者文件稍微做一丢丢的改动，Hash() 函数给出的输出结果都将发生翻天覆地的改变。注意，Hash()函数是公开的，任何人都能使用。</p><p>这个算法主要用于验证信息完整性——在一个信息后面放上这个信息的哈希值，收到信息之后收信人再算一遍哈希值，对比两者就知道这条信息是否被篡改过了。如果被篡改过，哪怕只有一bit，整个哈希值也会截然不同。而根据哈希函数的性质，没有人能够伪造出另一个消息具有同样的哈希值，也就是说篡改过的数据完全不可能通过哈希校验。</p><p>非对称加密就是指任何人手里都有两把钥匙，其中一把只有自己知道，叫做“私钥”，以及一把可以公布于众，叫做“公钥”；通过私钥加密的信息，必须通过公钥才能解密，连自己的私钥也无解。公钥可以通过私钥生成多把。</p><p>非对称加密除了用于信息加密之外，还有另一个用途，就是身份验证。因为通常情况我们假设一对公私钥，公钥是公开的，而私钥只有本人有，于是一个人如果有对应的私钥，我们就可以认定他是本人。其中一个重要的应用就是数字签名——某个消息后面，发信人对这个消息做哈希运算，然后用私钥加密。接着收信人首先对消息进行哈希运算，接着用相应的公钥解密数字签名，再对比两个哈希值，如果相同，就代表这个消息是本人发出的而且没有被篡改过。</p><p>单机不是说它不联网，是指全球只有一个机器或者叫系统，它是由全球所有的节点共同组成的，不属于任何组织或个人，也就是通常所说的去中心化。对于比特币来说这个系统存储着全球所有的交易信息，所以也有人把区块链叫做分布式账本，但区块链不是只可以存储交易，它还可以存储合约，数据等其他内容。</p><p>刚刚说了，区块链是一个分布式的系统，它也要解决分布式系统中的共识问题。在区块链之前，就已经有了很多关于分布式共识的研究，发明了许多共识算法。共识算法的核心在于如何解决某个变更在网络中是一致的，是被大家都承认的，同时这个信息是被确定的，不可推翻的。在区块链中则是让所有节点对于新增区块达成共识，也就是说，所有人都要认可新增的区块。</p><p>比特币区块链考虑的是公开匿名场景下的最坏保证，引入了工作量证明（Proof of Work）策略来规避少数人恶意破坏数据，并通过概率模型保证最后大家看到的就是合法的最长链。比特币网络需要所有试图参与者（矿工）都首先要付出挖矿的代价，进行算力消耗，越想拿到新区块的决定权，意味着抵押的算力越多。一旦失败，这些算力都会被没收掉，成为沉没成本。当网络中存在众多参与者时，个体试图拿到新区块决定权要付出的算力成本是巨大的，意味着进行一次作恶付出的代价已经超过可能带来的好处。简单来说比特币共识模型就是：模型中有公认的“价值”，每个节点说话都需要一定代价，诚实节点会受到奖励，而恶意节点由于只付出代价而收不到奖励，变相受到了惩罚。</p><p>然后我们来看一下比特币区块链是如何运行的，首先我们先从最基本的概念入手—公共账本。假如某几个人之间有频繁的金钱往来，每次交易结束后用现金支付可能会有点麻烦，你们想到了使用账本系统。每次交易后将交易信息写入账本，比如它记录了某个时间 A 转给了 B 1000 元等类似的信息。那么这个账本必须是公开的，每个人都可以查阅信息，同样的每个人也可以添加记录。等到了月底大家对账本的交易记录没有异议就会合计一下，假如某个人的交易显示为负值，则需要向系统交钱补上负值，如果为正值就可以从系统中取钱。</p><p>但这样的公共开放的系统存在一个问题，如何保证上面所记录的交易信息都是真实的呢？比如说 A 瞒着 B 在上面偷偷添加了一条 B 转给了 A 1000 元记录，也就是说我们怎么相信账本中所记载的信息都是准确无误的呢？这就用到了我们刚刚说的数字签名技术，记录者在添加每条记录时需在后加上数字签名。其他人通过公钥进行验证此记录是否由他本人添加，同时也可以防止他人对记录进行篡改。但这样仍存在一个问题，某个人可以将某条记录复制多次同样会导致系统出错。所以我们需要将每一条记录编号，这样哈希后的结果就是不一样的，也就是最终的数字签名也是不一样的，这样我们就保证了账本中所记载的信息都是准确无误，也就是解决了账本系统中的记录信任问题。</p><p>但这样虽然记录是没有问题的，但月底结算时如果出现有人无法支付欠款的情况该怎么办？为了解决这个问题，我们在每次添加交易时系统需要进行简单的验证，它会计算转账人的余额是否足够用来支出，假如不够，会拒绝添加此条记录。这样就可以避免出现上面提到的情况了。我们可以发现，我们完全不需要现实中的货币，只通过这个账本就可以进行所有的交易了。</p><p>现在我们的系统好像是可以解决各种问题了，但这样的一个完美的系统仍需运行在一个中心上，我们要把它运行在哪个可以信任的中心呢？这个系统要由谁来维护呢？这样的一看，我们刚刚解决的信任问题好像又出现了。。。问题好像出在了只有一个账本的身上，那假如我们让每个人都有一份账本，当某个人添加新纪录时他就向其他人广播这条记录，其他人收到消息后就记录在自己的账本中。这样我们的系统就变成了分布式的，看起来还是挺不错的。但还是存在问题，比如说 A 收到了一条 B 支付给 A 1000 元的消息，他怎么保证别人也收到了同样的一条消息呢，否则他就无法在接下来的交易中使用这 1000 元。而且即使别人收到相同的消息，但如何保证其他人接受的顺序是一致的呢？这是个比较麻烦的事情。</p><p>当每个人都拥有一份账本时，这个系统已经成为了分布式系统。我们刚刚提出来的问题其实就是分布式系统中的共识问题。我们刚刚讲过了，比特币区块链使用的是工作量证明策略来解决这个问题的。当系统中状态发生变化也就是有新纪录产生时，系统中可能会出现多个不同版本的账本，工作量证明策略选择相信消耗最多资源的那份账本。它基本的思路如下：假如某个人给了你一份交易记录并说，我发现一个数字，将这个数字放在交易记录的后面然后经过哈希算法（这里我们假设是SHA256）之后得到的哈希值前30位为零，我们可以想下找到这样一个数字有多难，SHA256产生的结果为256位，要求前30位为零的概率大概是二的三十次方分之一，差不多是十亿分之一。。。而且这个算法是无法逆向推倒的，所以为了找到这个值，除了暴力枚举外没有什么更好的办法了。而且当我们经过简单的计算验证发现它的哈希值确实符合要求时，我们有理由相信他是付出了很多的工作量的，这就是工作量证明。更加重要的是，这份工作量证明和这份交易记录紧密相关，假如你对记录做了某个微小的改动，哈希值将会发生很大的变化，就需要重新计算才能达到要求。</p><p>再回过头来看分布式账本，我们如果想要在系统中只保留一份账本的话，假如每一条交易记录都需要付出这么多的工作量显然是有些不合适的，我们可以将账单整理成区块，区块包含了一系列的交易记录已及相对应的工作量证明，其中不同区块的工作量证明的要求可能是不一样的，这个我们待会再说怎么计算，这里我们先以 60 个零为例，同需要在每份转账记录后加入发送者的数字签名一样，这里只有拥有工作量证明的区块才是合法的区块。为了保证这些区块的顺序，每个区块都要将前一个区块的哈希值加入到当前区块的头部，当你想要改变之前某个区块的内容或者交换区块顺序时，你就要重新计算这个区块以后的所有区块的工作量证明，这样整个系统就变成了区块的一套链，所以叫区块链。</p><p><img src="/2018/03/09/the-basic-of-bitcoin/hash_chain.png" alt="hash_chain.png"></p><p>在这种体系下，世界上任何一个人都可以成为区块的建立者，他们都可以接受网络中的交易信息并把它们整理成区块，然后花大量的计算完成工作量证明，一旦找到符合要求的值后，他们就将区块广播出去，为了奖励这个建立者的付出，当他完成区块的挖掘后，系统会奖励他一定数额的资产。这样看来，对于试图建立区块也就是通常所说的矿工来说，挖矿就像是买彩票，每个人都想尽快猜出那个数字，进而挖出区块拿到奖励。对于只是想通过他来交易的人来说，他只要接受矿工的广播然后把它加到自己链的末尾就可以了。但这样仍存在一个问题，假如某一时刻，我们收到了两个不同的区块链，我们应该相信哪一个呢？这种情况是很有可能出现的，虽然全网会尽力控制在一个周期内只有一个节点能够成功挖出区块，但是不能够完全避免多个节点同时挖出区块的可能性，这样同时计算出的矿工会沿着自己计算出来的链继续计算下一个区块，所以就会出现两条链，也叫分叉。遇到这种情况时，我们倾向于选择两条链中的较长的那一条，也就是付出工作量较多的那一条。如果目前两条链相等，可以等待下一个区块的产生，这样就可以做出选择，所以即使没有中心机构维持，所有人也都自己维持自己的那份区块链，我们就达成了一个去中心化的共识。</p><p><img src="/2018/03/09/the-basic-of-bitcoin/bifurcate.png" alt="bifurcate"></p><p>下面我们可以看下这个系统到底有多可信，我们来尝试下在这个系统中伪造信息欺骗他人有多难。比如说 Alice 想要用一个伪造的区块来欺骗 Bob ，于是他将一个包含了 一条 Alice 转账给 Bob 1000 元的消息广播给 Bob，但他并没有将这个区块广播给其他人，这样其他人就会以为 Alice 还拥有那 1000 元，为了欺骗其他人，他需要在其他人之前找到工作量证明，这是很有可能发生的。但同时 Bob 也会收到其他人的广播， 为了让 Bob 相信 Alice ，Alice 必须伪造 Bob 接收到的那个假区块后面的所有区块，这样的话，他可能就此陷入泥潭不能自拔了。。。必须一直计算下去。但仍凭他怎么计算，他的算力如果不是达到了系统的 51%，总会在某个区块被其他人先算出区块，导致他恶意创造的链被抛弃，同时他也浪费了大量算力。所以为了防止伪造情况出现，要求在区块挖出后，当有 6 个区块在其区块后又被挖掘出来，这个区块才会被真正承认。同时为了调整区块被挖出来的频率，公作量证明的难度也是不断变化的，保证大约每 10 分钟挖出一个区块。</p><p><img src="/2018/03/09/the-basic-of-bitcoin/double-spending.png" alt="double-spending"></p>]]></content>
<summary type="html">
<p>区块链是一个<strong>具有共享状态的密码性安全交易的单机</strong>。<br></p>
</summary>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/categories/BlockChain/"/>
<category term="BlockChain" scheme="https://yangchenglong11.github.io/tags/BlockChain/"/>
</entry>
<entry>
<title>Network学习笔记</title>
<link href="https://yangchenglong11.github.io/2018/01/14/Network%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0/"/>
<id>https://yangchenglong11.github.io/2018/01/14/Network学习笔记/</id>
<published>2018-01-14T04:42:40.000Z</published>
<updated>2018-01-14T04:50:22.642Z</updated>
<content type="html"><![CDATA[<p>整理了一些网络学习中易混淆的名词。<br><a id="more"></a></p><p>说到网络，不得不提的就是分层模型。</p><table><thead><tr><th>七层模型</th><th>五层模型</th><th>四层模型</th></tr></thead><tbody><tr><td>应用层</td><td></td><td></td><td></td></tr><tr><td>表示层</td><td>应用层</td><td>应用层</td></tr><tr><td>会话层</td><td></td><td></td><td></td></tr><tr><td>传输层</td><td>传输层</td><td>传输层</td></tr><tr><td>网络层</td><td>网络层</td><td>网络层</td></tr><tr><td>数据链路层</td><td>数据链路层</td><td>链接层/实体层</td></tr><tr><td>物理层</td><td>物理层</td></tr></tbody></table><p>七层模型的上三层归为应用层即为TCP/IP五层模型，五层模型的下两层归为链接层或者说实体层即为四层模型。</p><p>也就是说，所谓的五层或者四层，其实可以认为是方便理解而形成的潜规则，而具体的实施肯定还是得根据七层的标准来。毕竟每一层都有每一层各自的功能，而为了完成每一层的功能，就需要大家遵守相关的规则，也就是协议。</p><p><strong>IP</strong></p><p> IP协议（网际协议）对应于网络层，在网络通信中，网络组件的寻址对信息的路由选择和传输来说是相当关键的。相同网络中的两台机器间的消息传输有各自的技术协定。LAN 是通过提供6字节的唯一标识符（“MAC”地址）在机器间发送消息的。SNA 网络中的每台机器都有一个逻辑单元及与其相应的网络地址。</p><p><strong>TCP</strong> </p><p>TCP（Transmission Control Protocol，传输控制协议）是基于连接的协议，也就是说，在正式收发数据前，必须和对方建立可靠的连接。一个TCP连接必须要经过三次“对话”才能建立起来，握手过程中传送的包里不包含数据，三次握手完毕后，客户端与服务器才正式开始传送数据。</p><p>理想状态下，TCP连接一旦建立，在通信双方中的任何一方主动关闭连接之前，TCP 连接都将被一直保持下去。</p><p>断开连接时服务器和客户端均可以主动发起断开TCP连接的请求，断开过程需要经过“四次握手”(过程就不细写了，就是服务器和客户端交互，最终确定断开)</p><p>通过序列化应答和必要时重发数据包，TCP 为应用程序提供了可靠的传输流和虚拟连接服务，使一台计算机发出的字节流无差错地发往网络上的其他计算机，对可靠性要求高的数据通信系统往往使用TCP协议传输数据。TCP 还提供有效流控制，全双工操作和多路传输技术。</p><p>TCP是传输层协议，每一条TCP连接只能是点到点的，数据包在网络传输过程中，HTTP被封装在TCP包内。</p><p><strong>UDP</strong></p><p>UDP（User Data Protocol，用户数据报协议）是与TCP相对应的协议。它是面向非连接的协议，它不与对方建立连接，而是直接就把数据包发送过去！</p><p>UDP 适用于一次只传送少量数据、对可靠性要求不高的应用环境。比如，我们经常使用“ping”命令来测试两台主机之间TCP/IP通信是否正常，其实 “ping”命令的原理就是向对方主机发送UDP数据包，然后对方主机确认收到数据包，如果数据包是否到达的消息及时反馈回来，那么网络就是通的。例如， 在默认状态下，一次“ping”操作发送4个数据包。大家可以看到，发送的数据包数量是4包，收到的也是4包（因为对方主机收到后会发回一 个确认收到的数据包）。</p><p>这充分说明了UDP协议是面向非连接的协议，没有建立连接的过程。正因为UDP协议没有连接的过程，使得UDP的开销更小数据传输速率更高，因为不必进行收发数据的确认，所以UDP的实时性更好通信效果高。但也正因为如此，它的可靠性不如TCP协议高，可能会出现先发未必先至现象。QQ就使用UDP发消息，因此有时会出现收不到消息的情况。</p><p>UDP支持一对一，一对多，多对一和多对多的交互通信，可以实现广播。</p><p>知道了TCP和UDP的区别，就不难理解为何采用TCP传输协议的MSN比采用UDP的QQ传输文件慢了，但并不能说QQ的通信是不安全的，因为程序员可以手动对UDP的数据收发进行验证，比如发送方对每个数据包进行编号然后由接收方进行验证啊什么的，即使是这样，UDP因为在底层协议的封装上没有采用类似TCP的“三次握手”而实现了TCP所无法达到的传输效率。</p><p><strong>HTTP</strong></p><p>HTTP协议是Hyper Text Transfer Protocol（超文本传输协议）的缩写,是用于从万维网（WWW:World Wide Web ）服务器传输超文本到本地浏览器的传送协议。</p><p>HTTP是一个基于TCP/IP通信协议来传递数据（HTML 文件, 图片文件, 查询结果等）。</p><p>HTTP是一个属于应用层的面向对象的协议，由于其简捷、快速的方式，适用于分布式超媒体信息系统。</p><p>HTTP协议工作于客户端-服务端架构为上。浏览器作为HTTP客户端通过URL向HTTP服务端即WEB服务器发送所有请求。Web服务器根据接收到的请求后，向客户端发送响应信息。</p><p>HTTP协议定义Web客户端如何从Web服务器请求Web页面，以及服务器如何把Web页面传送给客户端。HTTP协议采用了请求/响应模型。客户端向服务器发送一个请求报文，请求报文包含请求的方法、URL、协议版本、请求头部和请求数据。服务器以一个状态行作为响应，响应的内容包括协议的版本、成功或者错误代码、服务器信息、响应头部和响应数据。</p><p>HTTP连接最显著的特点是客户端发送的每次请求都需要服务器回送响应，在请求结束后，会主动释放连接。从建立连接到关闭连接的过程称为“一次连接”。HTTP是无状态的短连接，就是经过一次一来一回的即断开连接。无法记录状态，所以引进session还有cookie弥补他无状态的缺陷。</p><p><strong>WebSocket</strong></p><p>WebSocket也是一种协议，并且也是基于TCP协议的。具体流程是WebSocket通过HTTP先发送一个标记了 Upgrade 的请求，服务端解析后开始建立TCP连接，省去了HTTP长连接每次请求都要上传header的冗余，可以理解为WebSocket是HTTP的优化，但WebSocket不仅仅在Web应用程序上得到支持。</p><p>WebSocket是在H5之后发布的，可以实现双向通信，也是基于TCP的应用层协议。 在他的握手请求中会比HTTP协议多出两行代码:</p><pre><code>Upgrade: websocketConnection: Upgrade</code></pre><p>他保证了服务器识别该协议为WebSocket协议。</p><p>总结一下，TCP/IP协议是传输层协议，主要解决数据如何在网络中传输，而HTTP是应用层协议，主要解决如何包装数据。</p><p>关于TCP/IP和HTTP协议的关系，网络有一段比较容易理解的介绍：“我们在传输数据时，可以只使用(传输层)TCP/IP协议，但是那样的话，如果没有应用层，便无法识别数据内容。如果想要使传输的数据有意义，则必须使用到应用层协议。应用层协议有很多，除 HTTP ，WebSocket外还有 FTP、TELNET 等，也可以自己定义应用层协议。WEB使用HTTP协议作应用层协议，以封装HTTP文本信息，然后使用TCP/IP做传输层协议将它发到网络上。”</p><p>HTTP通信过程属于“你推一下，我走一下”的方式，客户端不发请求则服务器永远无法发送数据给客户端。在HTTP上实现长连接以前可用AJAX轮询(就是隔一段时间发送ajax来保证数据，但这样会耗费大量资源)。而WebSocket则在进行第一次HTTP请求之后，其他全部采用TCP通道进行双向通讯。所以，HTTP和WebSocket虽都是基于TCP协议，但是两者属于完全不同的两种通讯方式。</p><p><strong>Socket</strong></p><p>Socket是对TCP/IP协议的封装，Socket本身并不是协议，而是一个调用接口(API)。通过Socket，我们才能使用TCP/IP协议。实际上，Socket跟TCP/IP协议没有必然的联系。</p><p>Socket编程接口在设计的时候，就希望也能适应其他的网络协议。所以说，Socket的出现只是使得程序员更方便地使用TCP/IP协议栈而已，是对TCP/IP协议的抽象，从而形成了我们知道的一些最基本的函数接口，比如create、listen、connect、accept、send、read和write等等。<br>建立Socket连接至少需要一对套接字，其中一个运行于客户端，称为ClientSocket ，另一个运行于服务器端，称为ServerSocket 。</p><p>套接字之间的连接过程分为三个步骤：服务器监听，客户端请求，连接确认。</p><p>1、服务器监听：服务器端套接字并不定位具体的客户端套接字，而是处于等待连接的状态，实时监控网络状态，等待客户端的连接请求。</p><p>2、客户端请求：指客户端的套接字提出连接请求，要连接的目标是服务器端的套接字。</p><p>为此，客户端的套接字必须首先描述它要连接的服务器的套接字，指出服务器端套接字的地址和端口号，然后就向服务器端套接字提出连接请求。</p><p>3、连接确认：当服务器端套接字监听到或者说接收到客户端套接字的连接请求时，就响应客户端套接字的请求，建立一个新的线程，把服务器端套接字的描述发给客户端，一旦客户端确认了此描述，双方就正式建立连接。</p><p>而服务器端套接字继续处于监听状态，继续接收其他客户端套接字的连接请求。</p><p>传输层的TCP是基于网络层的IP协议的，而应用层的HTTP协议又是基于传输层的TCP协议的，而Socket本身不算是协议，就像上面所说，它只是提供了一个针对TCP或者UDP编程的接口。</p>]]></content>
<summary type="html">
<p>整理了一些网络学习中易混淆的名词。<br></p>
</summary>
<category term="Network" scheme="https://yangchenglong11.github.io/categories/Network/"/>
<category term="Network" scheme="https://yangchenglong11.github.io/tags/Network/"/>
</entry>
<entry>
<title>processor_scheduling</title>
<link href="https://yangchenglong11.github.io/2017/09/12/processor-scheduling/"/>
<id>https://yangchenglong11.github.io/2017/09/12/processor-scheduling/</id>
<published>2017-09-12T09:42:57.000Z</published>
<updated>2018-07-10T11:20:10.256Z</updated>
<content type="html"><![CDATA[<h1 id="单处理器调度"><a href="#单处理器调度" class="headerlink" title="单处理器调度"></a>单处理器调度</h1><p>在多道程序环境下，主存中有着多个进程，其数目往往多于处理机数目。这就要求系统能按某种算法，动态地把处理机分配给就绪队列中的一个进程，使之执行。<br><a id="more"></a><br>分配处理机的任务是由处理机调度程序完成的。由于处理机是最重要的计算机资源，提高处理机的利用率及改善系统性能(吞吐量、响应时间)，在很大程度上取决于处理机调度性能的好坏，因而，处理机调度便成为操作系统设计的中心问题之一。 </p><p>在多道程序系统中，一个作业被提交后必须经过处理机调度后，方能获得处理机执行。 对于批量型作业而言，通常需要经历作业调度(又称高级调度或长程调度)和进程调度(又称低级调度或短程调度)两个过程后方能获得处理机；对于终端型作业，则通常只需经过进程调度即可获得处理机。在较完善的操作系统中，为提高内存的利用率，往往还设置了中级调度(又称中程调度)。对于上述的每一级调度，又都可采用不同的调度方式和调度算法。对于一个批处理型作业，从进入系统并驻留在外存的后备队列开始，直至作业运行完毕，可能要经历上述的三级调度。今天主要是对处理机调度层次做较详细的介绍。 </p><p>典型的有4种类型的调度如下表：</p><table><thead><tr><th>项目</th><th>说明</th></tr></thead><tbody><tr><td>长程调度</td><td>决定加入到待执行的进程池中</td></tr><tr><td>中程调度</td><td>决定加入到部分或全部在内存中的进程集合中</td></tr><tr><td>短程调度</td><td>决定哪一个可运行的进程将被处理器执行</td></tr><tr><td>I/O调度</td><td>决定哪一个进程挂起的I/O请求将被可用的I/O设备处理</td></tr></tbody></table><p>其中I/O调度以后再讲，剩下的三种调度类型属于处理器调度，这是我们接下来要讲的重点。我们今天具体讲短程调度，并且只考虑单处理器中的调度情况。</p><h2 id="处理器调度的类型"><a href="#处理器调度的类型" class="headerlink" title="处理器调度的类型"></a>处理器调度的类型</h2><p>处理器调度的目标是以满足系统目标（如响应时间，吞吐率，处理器效率）的方式，把进程分配到一个或多个处理器中执行。在许多系统中，这个调度活动分成三个独立的功能：长程，中程和短程调度。他们的名字表明了在执行这些功能时的相对时间比例。</p><p>如下图，将调度功能结合到了进程状态转换图中。创建新进程时，执行长程调度，它决定是否把进程添加到当前活跃的进程集合中。中程调度是交换功能的一部分，它决定是否把进程添加到那些至少部分在内存中并且可以被执行的进程集合中。进程调度真正决定下一次决定执行哪一个就绪进程。</p><p><img src="/2017/09/12/processor-scheduling/schedule_conversion.png" alt="schedule_conversion"></p><p>下图重新组织了上图的进程转换图，用于表示调度功能的嵌套。</p><p><img src="/2017/09/12/processor-scheduling/scheduling_level.png" alt="scheduling_level"></p><p>由于调度决定了那个进程必须等待，哪个进程可以继续运行，因此它影响着系统的性能。从下图中体现的更明显，该图给出了在一个进程状态转换过程中所涉及的队列。从根本上说，调度是属于队列管理方面的问题，用来在排队环境中减少延迟和优化性能。</p><p><img src="/2017/09/12/processor-scheduling/schedule_queue.png" alt="schedule_queue"></p><p>###长程调度</p><p>长程调度(LongTerm Scheduling)又称为作业调度或高级调度(High Level Scheduling) 长程调度程序决定哪一个程序可以进入到系统中处理，因此，它控制系统并发度。在批处理系统中，是以作业为基本单位从外存调入内存的。 一旦允许进入，一个作业或用户程序就成为一个进程，并被添加到供短程调度程序使用的队列中等待调度。</p><p>在批处理系统或通用的操作系统中的批处理部分中，新提交的作业被发送到磁盘，并保存在一个批处理队列中，在长程调度程序运行的时候，从队列中创建相应的作业。这里有两个决策。首先调度程序必须决定什么时候操作系统能够接纳一个作业或多个作业；第二，调度程序必须决定接受哪个作业或哪些作业，并将其转变成进程。简单考虑下这两个决策。</p><p>关于何时创建一个新进程的决策通常由要求的系统并发度来驱动。创建的进程越多，每个进程可以执行的时间所占百分比就越小。为了给当前的进程集提供满意的服务，长程调度程序可能限制系统并发度，每当一个作业终止时，调度程序可决定增加一个或多个新作业。当处理器的空闲时间片超过了一定的阙值，也可能会启动长程调度程序。</p><p>关于应将哪些作业从外存调入内存，这将取决于所采用的调度算法。最简单的是先来先服务调度算法，这是指将最早进入外存的作业最先调入内存；较常用的一种算法是短作业优先调度算法，是将外存上最短的作业最先调入内存;另一种较常用的是基于作业优先级的调度算法，该算法是将外存上优先级最高的作业优先调入内存；比较好的一种算法是“响应比高者优先”的调度算法。</p><p>在批处理系统中，作业进入系统后，总是先驻留在外存的后备队列上，因此需要有作业调度的过程，以便将它们分批地装入内存。然而在分时系统中，为了做到及时响应，用户通过键盘输入的命令或数据等都是被直接送入内存的，因而无需再配置上述的作业调度机制，但也需要有某些限制性措施来限制进入系统的用户数。即，如果系统尚未饱和，将接纳所有授权用户，否则，将拒绝接纳。类似地，在实时系统中通常也不需要作业调度。 </p><h3 id="中程调度"><a href="#中程调度" class="headerlink" title="中程调度"></a>中程调度</h3><p>中程调度(Medium-Term Scheduling)又称中级调度(Intermediate Level Scheduling)。引入中级调度的主要目的是为了提高内存利用率和系统吞吐量。为此，应使那些暂时不能运行的进程不再占用宝贵的内存资源，而将它们调至外存上去等待，把此时的进程状态称为就绪驻外存状态或挂起状态。当这些进程重又具备运行条件且内存又稍有空闲时，由中级调度来决定把外存上的那些又具备运行条件的就绪进程重新调入内存，并修改其状态为就绪状态，挂在就绪队列上等待进程调度。中级调度实际上就是存储器管理中的对换功能， 关于存储管理，我们之后再讲。</p><h3 id="短程调度"><a href="#短程调度" class="headerlink" title="短程调度"></a>短程调度</h3><p>从执行的频繁程度考虑这些调度类型，长程调度程序执行的频率相对较低，并且仅仅是粗略的决定是否接受新进程以及接受哪一个。为进行交换决定，中程调度程序执行的略微频繁一些。短程调度程序，也称作分派程序，执行的最频繁，并且精确地决定下一次执行哪一个进程。当可能导致当前进程阻塞或可能抢占当前运行进程的事件发生时，调用短程调度程序。这类事件包括：时钟中断，I/O中断，操作系统调用，信号。</p><h2 id="调度算法"><a href="#调度算法" class="headerlink" title="调度算法"></a>调度算法</h2><h3 id="短程调度准则"><a href="#短程调度准则" class="headerlink" title="短程调度准则"></a>短程调度准则</h3><p>短程调度的主要目标是按照优化系统一个或多个方面行为的方式来分配处理器时间。通常需要对可能被评估的各种调度策略建立一系列规则。</p><p>通常使用的规则可以按两维来分类。首先可以区分为面向用户的准则和面向系统的准则。面向用户的准则与单个用户或进程感知到的系统行为有关。例如交互式系统中的响应时间。响应时间是指从提交一条请求到输出响应所经历的时间间隔，这个时间数量对用户是可见的。我们希望调度策略能给各种用户提供好的服务，则对于响应时间，可以定义一个阙值，如2秒。调度机制的目标是使平均响应时间为2秒或小于2秒的用户数目达到最大。</p><p>另一个准则则是面向系统的，其重点是处理器使用的效果和效率。关于这类准则的一个例子是吞吐量，也就是进程完成的速度。该准则的重点是系统的性能，而不是提供给用户的服务。因此吞吐量是系统管理员所关注的，而不是普通用户所关注的。</p><p>面向用户的准则在所有系统中都是非常重要的，而面向系统的原则在单用户系统中的重要性就低一些。在单用户系统中，只要系统对用户应用程序的响应时间是可以接受的，则实现处理器高利用率或高吞吐量可能不是很重要。</p><p>另一维的划分是根据这些准则是否与性能直接相关。与性能直接相关的准则是定量的，通常可以很容易的度量，比如响应时间和吞吐量。与性能无关的准则或者本质上是定性的，或者不容易测量和分析，比如可预测性。我们希望提供给用户的服务能够随时间的进行展现给用户一贯相同的特性，而与系统执行的其他工作无关。</p><p>下表是几种重要的调度准则：</p><hr><p> <strong>面向用户，与性能相关</strong></p><ul><li><p>周转时间：是指从作业被提交给系统开始，到作业完成为止的这段时间间隔 。它包括四部分时间:作业在外存后备队列上等待 (作业)调度的时间，进程在就绪队列上等待进程调度的时间，进程在 CPU 上执行的时间， 以及进程等待 I/O 操作完成的时间。 总的来说就是实际执行时间加上等待资源时间</p></li><li><p>响应时间：是从用户通过键盘提交一个请求开始，直至系统首次产生响应为止的时间。从用户的角度看，相对于周转时间，这是一种更好的度量。该调度原则应该试图达到较低的响应时间，并且在响应时间可接受的范围内，使得可以交互的用户数目达到最大</p></li><li>最后期限：当可以指定进程完成的最后期限时，调度原则将降低其他目标，使得满足最后期限的作业数目的百分比达到最大</li></ul><hr><p> <strong>面向用户，其他</strong></p><ul><li>可预测性：无论系统的负载如何，一个给定的工作运行的总时间量和总代价是相同的。用户不希望响应时间或周转时间的变化太大。</li></ul><hr><p> <strong>面向系统，与性能相关</strong></p><ul><li>吞吐量：是指在单位时间内系统所完成的作业数 。调度策略应该试图使得每个单位时间完成的进程数目达到最大。</li><li>处理器利用率：这是处理器忙的时间百分比。对于大、中型多用户系统，由于 CPU 价格十分昂贵，致使处理机的利用率成为衡量系统性能的十分重要的指标 ，但对于单用户微机或某些实时系统，此准则就不那么重要了。 </li></ul><hr><p> <strong>面向系统，其他</strong></p><ul><li>公平性：在没有来自用户的指导或其他系统提供的指导时，进程应该被平等的对待</li><li>强制优先级：调度策略应该优先选择高优先级的进程</li><li>平衡资源：调度策略将保持系统中所有资源处于繁忙状态，较少使用紧缺资源的进程应该受到照顾</li></ul><hr><h3 id="优先级的使用"><a href="#优先级的使用" class="headerlink" title="优先级的使用"></a>优先级的使用</h3><p>在许多系统中，每个进程都只被指定一个优先级，调度程序总是选择具有较高优先级的进程。如下图：</p><p><img src="/2017/09/12/processor-scheduling/priority_queue.png" alt="priority_queue"></p><p>纯粹的优先级调度方案的一个问题是优先级的进程可能会长时间处于饥饿状态，如果不希望这样，一个进程的优先级应该随着它的时间或执行历史而改变。</p><h3 id="选择调度策略"><a href="#选择调度策略" class="headerlink" title="选择调度策略"></a>选择调度策略</h3><p>如下表列出了关于本节所分析的各种调度策略的一些简要信息。</p><p><strong>选择函数</strong>确定在就绪进程中选择哪一个进程在下一次执行。这个函数可以基于优先级，资源需求或者该进程的执行特性。对于最后一种情况，下面的三个量是非常重要的：</p><ul><li>w：到现在为止，在系统中停留的时间</li><li>e：到现在为止，花费的执行时间</li><li>s：进程所需要的总服务时间，包括e</li></ul><p>例如，选择函数max[w]，表示先来先服务（First-Come-First-Served，FCFS）的原则</p><p><strong>决策模型</strong>说明选择函数在被执行的瞬间的处理方式，通常可以分为以下两类：</p><ul><li>非抢占：在这种情况下，一旦进程处于运行状态，它就不断执行直到终止，或者因为等待I/O或请求某些操作系统服务而阻塞自己</li><li><p>抢占：当前正在运行的进程可能被操作系统中断，并转移到就绪态。</p><p>与非抢占策略相比，抢占策略可能会导致较大的开销，但是可能对所有的进程会提供较好的服务，因为他们避免了任何一个进程独占处理器太长的时间。</p></li></ul><table><thead><tr><th>类别</th><th>选择函数</th><th>决策模式</th><th>吞吐量</th><th>响应时间</th><th>开销</th><th>对进程的影响</th><th>饥饿</th></tr></thead><tbody><tr><td>先来先服务（FCFS）</td><td>max[w]</td><td>非抢占</td><td>不强调</td><td>可能很高，特别是当进程的执行时间差别很大时</td><td>最小</td><td>对短进程不利，对I/O密集型的进程不利</td><td>无</td></tr><tr><td>轮转</td><td>常数</td><td>抢占（在时间片用完时）</td><td>如果时间片小，吞吐量会很低</td><td>为短进程提供很好的响应时间</td><td>最小</td><td>公平对待</td><td>无</td></tr><tr><td>最短进程优先（SPN）</td><td>min[s]</td><td>非抢占</td><td>高</td><td>为短进程提供很好的响应时间</td><td>可能比较高</td><td>对长进程不利</td><td>可能</td></tr><tr><td>最短剩余时间（SRT）</td><td>min[s-e]</td><td>抢占（在时间片用完时）</td><td>高</td><td>提供很好的响应时间</td><td>可能比较高</td><td>对长进程不利</td><td>可能</td></tr><tr><td>最短响应比优先（HRRN）</td><td>max[(w+s)/s]</td><td>非抢占</td><td>高</td><td>为短进程提供很好的响应时间</td><td>可能比较高</td><td>很好的平衡</td><td>无</td></tr><tr><td>反馈</td><td>见下文</td><td>抢占（在时间片用完时）</td><td>不强调</td><td>不强调</td><td>可能比较高</td><td>可能对I/O密集型的进程有利</td><td>可能</td></tr></tbody></table><p>在描述各种调度策略时，使用下表中的进程集合作为实例。可以把他们想象成批处理作业，服务时间是所需要的整个执行时间。</p><table><thead><tr><th>进程</th><th>到达时间</th><th>服务时间</th></tr></thead><tbody><tr><td>A</td><td>0</td><td>3</td></tr><tr><td>B</td><td>2</td><td>6</td></tr><tr><td>C</td><td>4</td><td>4</td></tr><tr><td>D</td><td>6</td><td>5</td></tr><tr><td>E</td><td>8</td><td>2</td></tr></tbody></table><p>对于上表中的例子，下图显示了一个周期内，每种策略的执行模式以及执行结果。每个进程的结束时间是确定的，根据这一点，可以确定周转时间，即其在队列中的驻留时间Tr，我们用一个更有用的数字，归一化周转时间它是周转时间与服务时间的比率，它表示一个进程的相对延迟。一般情况，进程的执行时间越长，可以容忍的延迟时间就越长，其最小值为1.0</p><h4 id="先来先服务"><a href="#先来先服务" class="headerlink" title="先来先服务"></a>先来先服务</h4><p>这是最简单的策略，也叫做先进先出。当每个进程就绪后，它就加入就绪队列。当前正在运行的进程停止执行时，选择在就绪队列中存在时间最长的进程运行。</p><p>它执行长进程比执行短进程更好。考虑下面的例子</p><table><thead><tr><th>进程</th><th>到达时间</th><th>服务时间（Ts）</th><th>开始时间</th><th>结束时间</th><th>周转时间（Tr）</th><th>Tr/Ts</th></tr></thead><tbody><tr><td>W</td><td>0</td><td>1</td><td>0</td><td>1</td><td>1</td><td>1</td></tr><tr><td>X</td><td>1</td><td>100</td><td>1</td><td>100</td><td>100</td><td>1</td></tr><tr><td>Y</td><td>2</td><td>1</td><td>101</td><td>102</td><td>100</td><td>100</td></tr><tr><td>Z</td><td>3</td><td>100</td><td>102</td><td>202</td><td>199</td><td>1.99</td></tr><tr><td>平均值</td><td></td><td></td><td></td><td></td><td>100</td><td>26</td></tr></tbody></table><p>进程Y的归一化周转时间与其他进程相比显得不协调：它在系统中的总时间是所需要的处理时间的100倍。当一个短进程紧随着一个长进程之后到达时就会发生这种情况。再和进程Z比较，在这个极端的例子中，长进程Z的周转时间几乎是Y的两倍，但是它的归一化等待时间低于2.0</p><p>FCFS的另一个难点是相对于I/O密集型的进程，它更有利于处理器密集型的进程。考虑一组进程，其中有一个进程大多时候都使用处理器（处理器密集型），还有许多进程大多数时候进行I/O操作（I/O密集型）。如果一个处理器密集型的进程正在运行，则所有I/O密集型的进程因为没有被分配I/O资源都必须等待。有一些进程可能在I/O队列中，进行I/O操作（阻塞态），但是当处理器密集型的进程正在执行时，他们可能I/O操作完成，已经回到就绪队列。这时，大多数或所有I/O设备都可能是空闲的，即使他们可能还有工作要做。在当前进程离开运行态时，就绪的I/O密集型的进程迅速通过运行态，又阻塞在I/O事件上，如果处理器密集型的进程也被阻塞了，则处理器空闲。因此这可能会导致处理器和I/O设备都没有得到充分利用。</p><h4 id="轮转"><a href="#轮转" class="headerlink" title="轮转"></a>轮转</h4><p>为了减少在FCFS策略下短作业的不利情况，一种简单的方法是采用基于时钟的抢占策略，这类方法中，最简单的是轮转算法。以一个周期性间隔产生时钟中断，当中断发生时，当前正在运行的进程被置于就绪队列中，然后基于FCFS策略选择下一个就绪作业运行，也叫做时间片轮转，每个进程在被抢占前都给定一片时间。</p><p>对于轮转法，最主要的问题是时间片的长度。一般情况时间片最好大于一次典型的交互所需要的时间。如果小于这个时间，大多数进程都需要至少两个时间片。注意，当一个时间片比运行时间最长的进程还要长时，轮转法退化成FCFS。</p><p>轮转法在通用的分时系统或事务处理系统中都特别有效。它的一个缺点是依赖于处理器密集型的进程和I/O密集型的进程的不同。通常I/O密集型的进程比处理器密集型的进程使用处理器的时间短，假如同时有这两种进程，可能会有如下情况：一个I/O密集型的进程只使用处理器很短的一段时间，然后因为I/O而被阻塞，等待I/O操作的完成，然后加入到就绪队列；另一方面，一个处理器密集型的进程在执行过程中通常使用一个完整的时间片并立即返回到就绪队列中。因此，处理器密集型的进程不公平的使用了大部分处理器时间，从而导致I/O密集型的进程性能降低。</p><p>在此基础上有人提出一种改进的轮转法，叫做虚拟轮转法，可以避免这种不公平性。如图所示，新进程到达并加入就绪队列，是基于FCFS管理的。当一个正在运行的进程的时间片用完了，它返回到就绪队列。当一个进程为I/O而被阻塞时，它加入到一个I/O队列，这些都和原来的一样，它的新特点是解除了I/O阻塞的进程都被转移到一个FCFS辅助队列中，当进行一次调度决策时，辅助队列中的进程优于就绪队列中进程，当一个进程从辅助队列中调度时，它的运行时间一般不会长于基本时间段减去它上一次从就绪队列中被选择运行的总时间。</p><p><img src="/2017/09/12/processor-scheduling/vrr.png" alt="vrr"></p><h4 id="最短进程优先"><a href="#最短进程优先" class="headerlink" title="最短进程优先"></a>最短进程优先</h4><p>减少FCFS固有的对长进程的偏向的另一种方法是最短进程优先(SPN)，这是一个非抢占的策略，其原则是下一次选择预计处理时间最短的进程。因此，短进程将会越过长作业，跳到队列头。</p><p>SPN的难点在于需要知道或至少需要估计每个进程所需要的处理时间。对于批处理作业操作系统，系统要求程序员估计该值，并提供给操作系统。如果程序猿的估计远低于实际运行时间，系统就可能终止改作业。</p><p>SPN的风险在于只要持续不断的提供更短的进程，长进程就有可能饥饿。另一方面，尽管SPN减少了对长作业的偏向，但是由于缺少抢占机制，它对分时系统或事务处理环境仍然不理想。</p><h4 id="最短剩余时间"><a href="#最短剩余时间" class="headerlink" title="最短剩余时间"></a>最短剩余时间</h4><p>最短剩余时间(SRT)是针对SPN增加了抢占机制的版本，在这种情况下，调度程序总是选择预期剩余时间最短的进程，当一个新进程加入到就绪队列时，他可能比当前运行的进程具有更短的剩余时间。只要新进程就绪，调度程序就可能抢占当前正在运行的进程。和SPN一样，调度程序正在执行选择函数时必须有关于处理时间的估计，并且存在长进程饥饿的危险。</p><p>SRT不像FCFS那样偏向长进程，也不像轮转那样会产生额外的中断，从而减少了开销。另一方面，他必须记录过去的服务时间，从而增加了开销。从周转时间来看，SRT比SPN有更好的性能，因为相对于一个正在运行的长作业，短作业可以立即被选择运行。</p><h4 id="最高响应比优先"><a href="#最高响应比优先" class="headerlink" title="最高响应比优先"></a>最高响应比优先</h4><p>在上面的讲解中，我们引入了归一化周转时间的概念，它是周转时间和实际服务时间的比率，可作为性能度量。对于每个单独的进程，我们都希望该值最小，并且希望所有进程的平均值也最小。</p><p>考虑该比率 R= (w+s)/s，其中R为响应比。w为等待处理器时间，s为预计的服务时间。如果该进程被立即调度，则R等于归一化周转时间。注意，R最小值即为1.0</p><p>调度规则为在当前进程完成或被阻塞时，选择R值最大的就绪进程。该方法非常具有吸引力，因为它说明进程的年龄。当偏向短作业时，长进程由于得不到服务的时间不短的增加，从而增大了比值，最终在竞争中胜了短进程。不过该策略同样需要估计服务时间。</p><h4 id="反馈"><a href="#反馈" class="headerlink" title="反馈"></a>反馈</h4><p>在上面所讲的3种算法中，如果没有关于各个进程相对长度的信息，他们都无法使用。不过我们可以换一种思路，如果不能获得剩余的执行时间，我们可以关注已经执行了的时间。</p><p>具体如下：调度基于抢占策略并且使用动态优先级机制，当一个进程第一次进入系统中时，它被放置在RQ0，当它第一次被抢占后并返回就绪状态时，它被放置在RQ1，在随后的时间里，每当它被抢占时，它被降级到下一个低优先级队列中。一个短进程很快会执行完，不会在就绪队列中降很多级。一个长进程会逐级下降。新到的进程和短进程优先于老进程和长进程。在每个队列中，除了在优先级最低的队列中之外，都是用最简单的FCFS机制。一旦一个进程处于优先级最低的队列中，它就不可能再降级，但是会重复的返回该队列，知道运行结束，因此，该队列可按照轮转方式调度。</p><p><img src="/2017/09/12/processor-scheduling/feedback.png" alt="feedback"> </p><p>该方案存在的一个问题是长进程的周转时间可能惊人的增加。事实上，如果频繁的有新作业进入系统，就有可能出现饥饿的情况。为补偿这一点，可以按照队列改变抢占次数：从RQ0中调度的进程允许执行一个时间单位，然后被抢占；从RQ1中调度的进程允许执行两个时间单位等等。一般而言，从RQi中调度的进程允许执行2^i的时间，然后才被抢占。</p><p>即使给较低的优先级分配较长的时间，长进程仍然有可能饥饿。一种可能的补救方法是当一个进程在它的当前队列中等待服务的时间超过一定的时间量后，把它提升到一个优先级较高的队列中。</p><p>####调度策略的比较</p><p><img src="/2017/09/12/processor-scheduling/compare.png" alt="compare"></p><table><thead><tr><th>进程</th><th>A</th><th>B</th><th>C</th><th>D</th><th>E</th><th>平均值</th></tr></thead><tbody><tr><td>到达时间</td><td>0</td><td>2</td><td>4</td><td>6</td><td>8</td><td></td></tr><tr><td>服务时间(Ts)</td><td>3</td><td>6</td><td>4</td><td>5</td><td>2</td><td></td></tr><tr><td></td><td></td><td></td><td>FCFS</td><td></td><td></td><td></td></tr><tr><td>完成时间</td><td>3</td><td>9</td><td>13</td><td>18</td><td>20</td><td></td></tr><tr><td>周转时间(Tr)</td><td>3</td><td>7</td><td>9</td><td>12</td><td>12</td><td>8.60</td></tr><tr><td>Tr/Ts</td><td>1.00</td><td>1.17</td><td>2.25</td><td>2.40</td><td>6.00</td><td>2.56</td></tr><tr><td></td><td></td><td></td><td>RR q=1</td><td></td><td></td><td></td></tr><tr><td>完成时间</td><td>4</td><td>18</td><td>17</td><td>20</td><td>15</td><td></td></tr><tr><td>周转时间(Tr)</td><td>4</td><td>16</td><td>13</td><td>14</td><td>7</td><td>10.80</td></tr><tr><td>Tr/Ts</td><td>1.33</td><td>2.67</td><td>3.25</td><td>2.80</td><td>3.50</td><td>2.71</td></tr><tr><td></td><td></td><td></td><td>RR q=4</td><td></td><td></td><td></td></tr><tr><td>完成时间</td><td>3</td><td>17</td><td>11</td><td>20</td><td>19</td><td></td></tr><tr><td>周转时间(Tr)</td><td>3</td><td>15</td><td>7</td><td>14</td><td>11</td><td>10.00</td></tr><tr><td>Tr/Ts</td><td>1.00</td><td>2.5</td><td>1.75</td><td>2.80</td><td>5.50</td><td>2.71</td></tr><tr><td></td><td></td><td></td><td>SPN</td><td></td><td></td><td></td></tr><tr><td>完成时间</td><td>3</td><td>9</td><td>15</td><td>20</td><td>11</td><td></td></tr><tr><td>周转时间(Tr)</td><td>3</td><td>7</td><td>11</td><td>14</td><td>3</td><td>7.60</td></tr><tr><td>Tr/Ts</td><td>1.00</td><td>1.71</td><td>2.75</td><td>2.80</td><td>1.50</td><td>1.84</td></tr><tr><td></td><td></td><td></td><td>SRT</td><td></td><td></td><td></td></tr><tr><td>完成时间</td><td>3</td><td>15</td><td>8</td><td>20</td><td>10</td><td></td></tr><tr><td>周转时间(Tr)</td><td>3</td><td>13</td><td>4</td><td>14</td><td>2</td><td>7.20</td></tr><tr><td>Tr/Ts</td><td>1.00</td><td>2.17</td><td>1.00</td><td>2.80</td><td>1.00</td><td>1.59</td></tr><tr><td></td><td></td><td></td><td>HRRN</td><td></td><td></td><td></td></tr><tr><td>完成时间</td><td>3</td><td>9</td><td>13</td><td>20</td><td>15</td><td></td></tr><tr><td>周转时间(Tr)</td><td>3</td><td>7</td><td>9</td><td>14</td><td>7</td><td>8.00</td></tr><tr><td>Tr/Ts</td><td>1.00</td><td>1.17</td><td>2.25</td><td>2.80</td><td>3.5</td><td>2.14</td></tr><tr><td></td><td></td><td></td><td>FB q=1</td><td></td><td></td><td></td></tr><tr><td>完成时间</td><td>4</td><td>20</td><td>16</td><td>19</td><td>11</td><td></td></tr><tr><td>周转时间(Tr)</td><td>4</td><td>18</td><td>12</td><td>13</td><td>3</td><td>10.00</td></tr><tr><td>Tr/Ts</td><td>1.33</td><td>3.00</td><td>3.00</td><td>2.60</td><td>1.5</td><td>2.29</td></tr><tr><td></td><td></td><td></td><td>FB q=2</td><td></td><td></td><td></td></tr><tr><td>完成时间</td><td>4</td><td>17</td><td>18</td><td>20</td><td>14</td><td></td></tr><tr><td>周转时间(Tr)</td><td>4</td><td>15</td><td>14</td><td>14</td><td>6</td><td>10.60</td></tr><tr><td>Tr/Ts</td><td>1.33</td><td>2.50</td><td>3.50</td><td>2.80</td><td>3.00</td><td>2.63</td></tr></tbody></table>]]></content>
<summary type="html">
<h1 id="单处理器调度"><a href="#单处理器调度" class="headerlink" title="单处理器调度"></a>单处理器调度</h1><p>在多道程序环境下，主存中有着多个进程，其数目往往多于处理机数目。这就要求系统能按某种算法，动态地把处理机分配给就绪队列中的一个进程，使之执行。<br></p>
</summary>
<category term="OS" scheme="https://yangchenglong11.github.io/categories/OS/"/>
<category term="OS" scheme="https://yangchenglong11.github.io/tags/OS/"/>
</entry>
<entry>
<title>process</title>
<link href="https://yangchenglong11.github.io/2017/09/06/process/"/>
<id>https://yangchenglong11.github.io/2017/09/06/process/</id>
<published>2017-09-06T02:36:26.000Z</published>
<updated>2018-07-10T13:52:20.478Z</updated>
<content type="html"><![CDATA[<h3 id="操作系统与进程"><a href="#操作系统与进程" class="headerlink" title="操作系统与进程"></a>操作系统与进程</h3><p>操作系统是计算机硬件和应用程序之间的一层软件，对应用程序和工具提供了支持。操作系统是硬件资源的统一抽象表示，为了使应用程序可以请求和访问，操作系统需要管理他们的使用，以达到以下目的：<br><a id="more"></a></p><ul><li>资源对多个应用程序是可用的，即可以共享；</li><li>物理处理器在多个应用程序间切换以保证所有程序都在执行中；</li><li>处理器和I/O设备能得到充分的利用。</li></ul><p>现代操作系统所采用的是进程模型来实现这些功能。</p><hr><h3 id="进程是什么"><a href="#进程是什么" class="headerlink" title="进程是什么"></a>进程是什么</h3><p>一般来说，一个应用程序对应于一个或多个进程，操作系统为了有序的去管理应用程序的执行，需要控制其对应的进程的执行。为了控制进程的执行，我们首先需要区分不同的进程，然后跟踪并记录其执行时的状态，只有这样，当我们将某个挂起的进程重新获得处理机时，其继续执行才可以不受挂起的影响。</p><p>进程是由一组元素组成的实体，它的两个基本元素是程序代码和与代码相关联的数据集。在进程执行时，任意给定一个时间，进程都可以唯一的被表征为以下元素：</p><ul><li>标识符：跟这个进程相关的唯一标识符，用来区别其他进程；</li><li>状态：根据进程是否在执行而划分的状态；</li><li>优先级：相对于其他进程的优先级；</li><li>程序计数器：程序中即将被执行的下一条指令的地址；</li><li>内存指针：包括进程代码和进程相关数据的指针，还有和其他进程共享内存块的指针；</li><li>上下文数据：进程执行时处理器的寄存器中的数据；</li><li>I/O 状态信息：包括显示的 I/O 请求，分配给进程的 I/O 设备和被进程使用的文件列表等；</li><li>记账信息：可能包括处理器时间总和，使用的时间数总和，时间限制等。</li></ul><p>这些信息存放在一个叫做进程控制块的数据结构中，该控制块由操作系统创建和管理，它是操作系统支持多进程和提供多处理的关键工具。</p><hr><h3 id="两状态进程模型"><a href="#两状态进程模型" class="headerlink" title="两状态进程模型"></a>两状态进程模型</h3><p>操作系统的基本职责是控制程序的执行，这包括确定交替执行的方式和给进程分配资源。在设计控制进程的程序时，第一步就是描述进程所表现出的行为。通过观察可以发现，在任何时刻，一个进程要么正在执行，要么没有执行，暂时我们先建立一个简单的模型：一个进程可以处于运行态或未运行态。</p><p>当操作系统创建一个新进程时，它将该进程以未运行态加入到系统中，操作系统知道这个进程是存在的，并正在等待执行机会。当前正在执行的进程不时的被中断，操作系统中的分派器将选择一个新进程运行，前一个进程从运行态转移到未运行态，另外一个进程转换到运行态。未运行的进程应保存在某种类型的队列中，以等待他们的执行时机。</p><p><img src="/2017/09/06/process/two_state_conversion.png" alt="two_state_conversion"></p><p><img src="/2017/09/06/process/two_state_queue.png" alt="two_state_queue"></p><p>进程的生存周期都围绕着进程的创建和终止。我们接下来就来看下进程的这两个过程。</p><hr><h3 id="进程的创建"><a href="#进程的创建" class="headerlink" title="进程的创建"></a>进程的创建</h3><p>当一个新进程添加到那些正在被管理的进程集合中去时，操作系统需要建立用于管理该进程的数据结构，并在内存中给它分配地址空间。通常会有4个事件会导致创建一个进程：</p><ul><li>新的批处理作业</li><li>交互登录</li><li>操作系统因为提供一项服务而创建</li><li>由现有的进程派生</li></ul><p>一般的，操作系统创建进程的方式对用户和应用程序都是透明的，大多数的操作系统允许一个进程引发另一个进程的创建。当操作系统为另一个进程的显式请求创建一个进程时，这个动作称为进程派生。当一个进程派生另一个进程时，前一个称作父进程，被派生的进程叫做子进程。在典型的情况下，相关进程需要相互之间的通信与合作。</p><hr><h3 id="进程的终止"><a href="#进程的终止" class="headerlink" title="进程的终止"></a>进程的终止</h3><p>任何一个计算机系统都必须为进程提供表示其完成的方法，批处理作业中应该包含一个Halt 指令或用于终止的操作系统显示服务调用来终止其执行。在前一种情况下，Halt 指令将产生一个中断，警告操作系统一个进程已经完成。对交互式应用程序，用户的行为将指出何时进程完成。所有这些行为最终会导致进程发送给操作系统一个服务请求，以终止自己的执行。下面列出了进程终止的典型原因：</p><ul><li>正常完成：进程自己执行一个系统调用，表示它已经结束运行</li><li>超过时限：进程运行时间超过规定的时限</li><li>无可用内存：系统无法满足进程需要的内存空间</li><li>越界：进程试图访问不允许访问的内存单元</li><li>保护错误：进程试图使用不允许使用的资源或文件，或者试图以一种不正确的方式使用，如往只读文件中写</li><li>算术错误：进程试图进行被禁止的计算，如除以零</li><li>无效指令：进程试图执行一个不存在的指令（通常是由于转移到了数据区并企图执行数据）</li><li>操作员或操作系统干涉：字面意思</li><li>父进程请求：父进程通常具有终止其任意后代进程的权力</li></ul><hr><h3 id="五状态模型"><a href="#五状态模型" class="headerlink" title="五状态模型"></a>五状态模型</h3><p>再来看下上面提到的排队队列，进程处理器一般以轮转方式操作这个队列。但是，这样存在一些问题：存在着一些处于非运行状态但已经就绪等待执行的进程，而同时存在另外的一些处于阻塞状态等待I/O操作结束然后继续执行的进程，这两种进程虽然都是非运行态，其对应的处理是不一样的。所以我们的模型需要进行一些更改，需要将非运行状态分成两个状态：就绪和阻塞。再加上原来提到的新建和退出两个状态，新的模型中就有了5个状态：</p><p><img src="/2017/09/06/process/five_state.png" alt="five"></p><ul><li>运行态：该进程正在执行，目前讨论只有一个处理器的情况，所以一次最多只有一个进程处于这个状态</li><li>就绪态：进程做好了准备，只要有机会就开始执行</li><li>阻塞态：进程在某些事情发生前不能执行，比如I/O操作完成</li><li>新建态：刚刚创建的进程，操作系统还没有把它加入到可执行进程组中，通常是进程控制块已经创建但还没有加载到内存中的新进程。</li><li>退出态：操作系统从可执行进程组中释放出的进程</li></ul><p>进程处于新建态时，操作系统所需要的关于该进程的信息保存在内存中的进程表中，但进程自身还未进入内存，就是将要执行的程序代码不在内存中，也没有为与这个程序相关的数据分配空间。当程序处于新建态时，程序保存在外村中。进程处于退出态时，与作业相关的的表和其他信息临时被操作系统保留起来，这给辅助程序或支持程序提供了提取所需信息的时间。</p><p>显然，现在我们的排队队列只有一个是不够的了，我们先使用两个队列：就绪队列和阻塞队列。进入系统的每个进程被放置在就绪队列中，对于没有优先级的方案，这只是一个先进先出的队列，。当一个正在运行的进程被移出处理器时，他根据情况或者被终止，或者被放置在就绪或阻塞队列中，当某个事件发生后，所有位于阻塞队列中等待这个事件的进程都被转换到就绪队列中。</p><p><img src="/2017/09/06/process/one_queue.png" alt="one_queue"></p><h4 id="优化"><a href="#优化" class="headerlink" title="优化"></a>优化</h4><p>在这种情况下，当某个事件发生时，操作系统必须扫描整个阻塞队列，搜索那些等待该事件的进程，在大型操作系统中，队列中可能会有成百上千的进程，所以改为多个队列将会很有效，我们可以根据其等待的事件类型进行划分，这样是比较方便的。同样的，我们也可以根据优先级去修改就绪队列，这样操作系统很容易确定哪个就绪进程具有较高的优先级且等待时间最长。</p><p><img src="/2017/09/06/process/more_queue.png" alt="more_queue"></p><hr><h3 id="被挂起的进程"><a href="#被挂起的进程" class="headerlink" title="被挂起的进程"></a>被挂起的进程</h3><p>前面描述的三个基本状态提供了一种为进程行为建立模型的系统方法，并指导操作系统的实现。但是，这样的设计仍然是存在问题的。上面我们所说的每个被执行的进程都是被完全载入内存的，所有队列中的所有进程必须驻留在内存中，但因为I/O处理比计算速度慢得多，即使是多道程序设计，大多时候处理器仍然可能处于空闲状态。</p><p>一种解决方法是扩充内存，但是内存的价格比较贵。。。另一种解决办法是交换，也就是把内存中某个部分的一部分或全部移到磁盘中。当内存中没有处于就绪状态的进程时，操作系统就把被阻塞的进程换出到磁盘中的挂起队列，这是暂时保存从内存中被驱逐出的进程队列，或者说被挂起的进程队列。操作系统在此之后取出挂起队列中的另一个进程，或者接受一个新进程的请求，将其纳入内存运行。</p><p>为了使用前面描述的交换，在我们的进程行为模型中需要增加另一个状态：挂起态。当内存中的所有进程都处于阻塞态时，操作系统可以把其中的一个进程置于挂起态，并将它转移到磁盘，内存中释放的空间可被调入的另一个进程使用。</p><p><img src="/2017/09/06/process/one_suspend.png" alt="one_suspend"></p><p>当操作系统已经执行了一个换出操作，它可以有两种将一个进程取到内存中的选择：可以接纳一个新进创建的进程，或调入一个以前被挂起的进程。显然，通常比较倾向于调入一个以前被挂起的进程，给它提供服务，而不是增加系统中的负载总数。但是这样做也还是有问题的，所有已经挂起的进程在挂起时都处于阻塞态。显然，这时候把被阻塞的进程取回内存没有什么意义，因为它仍然没有准备好执行。但是，因为挂起状态中的每个进程最初是阻塞在一个特定的时间上，当这个事件发生时，进程就不再阻塞，可以继续执行。</p><p>因此，我们需要重新考虑设计方式。这里有两个独立的概念：进程是否在等待一个事件（阻塞与否）以及进程是否已经被换出内存（挂起与否）。由这两个独立的概念可以引出四个状态：</p><ul><li>就绪态：进程在内存中并可以执行</li><li>阻塞态：进程在内存中并等待一个事件</li><li>阻塞/挂起态：进程在外存中并等待一个事件</li><li>就绪/挂起态：进程在外存中，但是只要被载入内存就可以执行</li></ul><p><img src="/2017/09/06/process/more_suspend.png" alt="more_suspend"></p>]]></content>
<summary type="html">
<h3 id="操作系统与进程"><a href="#操作系统与进程" class="headerlink" title="操作系统与进程"></a>操作系统与进程</h3><p>操作系统是计算机硬件和应用程序之间的一层软件，对应用程序和工具提供了支持。操作系统是硬件资源的统一抽象表示，为了使应用程序可以请求和访问，操作系统需要管理他们的使用，以达到以下目的：<br></p>
</summary>
<category term="OS" scheme="https://yangchenglong11.github.io/categories/OS/"/>
<category term="OS" scheme="https://yangchenglong11.github.io/tags/OS/"/>
</entry>
<entry>
<title>iptables</title>
<link href="https://yangchenglong11.github.io/2017/07/15/iptables/"/>
<id>https://yangchenglong11.github.io/2017/07/15/iptables/</id>
<published>2017-07-15T08:09:51.000Z</published>
<updated>2018-03-15T08:13:00.805Z</updated>
<content type="html"><![CDATA[<p>iptables利用的是数据包过滤机制</p><a id="more"></a><h2 id="Netfilter-与-iptables-的关系"><a href="#Netfilter-与-iptables-的关系" class="headerlink" title="Netfilter 与 iptables 的关系"></a>Netfilter 与 iptables 的关系</h2><p>Linux 系统在内核中提供了对报文数据包过滤和修改的官方项目名为 Netfilter，它指的是 Linux 内核中的一个框架，它可以用于在不同阶段将某些钩子函数（hook）作用域网络协议栈。Netfilter 本身并不对数据包进行过滤，它只是允许可以过滤数据包或修改数据包的函数挂接到内核网络协议栈中的适当位置。这些函数是可以自定义的。</p><p>iptables 是用户层的工具，它提供命令行接口，能够向 Netfilter 中添加规则策略，从而实现报文过滤，修改等功能。Linux 系统中并不止有 iptables 能够生成防火墙规则，其他的工具如 firewalld 等也能实现类似的功能。</p><h2 id="使用-iptables-进行包过滤"><a href="#使用-iptables-进行包过滤" class="headerlink" title="使用 iptables 进行包过滤"></a>使用 iptables 进行包过滤</h2><p>iptables利用的是数据包过滤机制，他会分析数据包的报头数据，根据报头数据与定义的规则来决定该数据包是否可以进入主机或者是被丢弃。也就是说，根据数据包的分析资料对比预先定义的规则内容，若数据包数据数据与规则内容相同则进行动作，否则就继续下一条规则的对比。重点是对比与分析的顺序。</p><p><img src="https://github.com/fengyfei/WikiImages/blob/master/iptables/iptables_01.png?raw=true" alt="tu"></p><p> 在上图中，当网络数据包开始Rule 1 的比对时，如果比对结果符合Rule 1，此时这个网络数据包就会进行Action 1 的动作，而不会理会后续的 Rule 2，Rule 3等规则了。如果一个一个规则对比下去，所有规则都不符合的话， 就会执行默认操作(Policy)，来决定这个数据包的去向。</p><h3 id="表"><a href="#表" class="headerlink" title="表"></a>表</h3><p>上图中列出的规则仅是iptables众多表格当中的一个链而已。iptables中有多个表格，每个表都定义自己的默认策略与规则。</p><p>iptables 根据功能分类，内建有多个表，如包过滤（filter）或者网络地址转换（NAT）。iptables 中共有 4 个表：filter，nat，mangle 和 raw。filter 表主要实现过滤功能，nat 表实现 NAT 功能，mangle 表用于修改分组数据，raw 用于配置数据包，raw中的数据包不会被系统跟踪。</p><h3 id="链"><a href="#链" class="headerlink" title="链"></a>链</h3><p>一个 iptables 链就是一个规则集，这些规则按序与包含某种特征的数据包进行比较匹配。</p><p>每个表都有一组内置链，用户还可以添加自定义的链。最重要的内置链是 filter 表中的 INPUT、OUTPUT 和 FORWARD 链。</p><ul><li>filter,用于路由网络数据包<ul><li>INPUT 发往本机的报文</li><li>OUTPUT 由本机发出的报文</li><li>FORWARD 经由本机转发的报文</li></ul></li><li>nat,用于NAT表。这个表主要用来进行来源与目的地的IP或port的转换，与linux本机无关，主要与linux主机后的局域网内计算机相关。<ul><li>PREROUTING 网络数据包到达服务器时可以被修改</li><li>POSTROUTING 网络数据包在即将从服务器发出时可以被修改</li><li>OUTPUT 网络数据包流出服务器</li></ul></li><li>mangle,用于修改网络数据包的表，如TOS(Type Of Service),TTL(Time To Live),等<ul><li>INPUT 发往本机的报文</li><li>OUTPUT 由本机发出的报文</li><li>FORWARD 经由本机转发的报文</li><li>PREROUTING 网络数据包到达服务器时可以被修改</li><li>POSTROUTING 网络数据包在即将从服务器发出时可以被修改</li></ul></li><li>raw, 用于决定数据包是否被跟踪机制处理<ul><li>OUTPUT 由本机发出的报文</li><li>PREROUTING 网络数据包到达服务器时可以被修改</li></ul></li></ul><p>3.数据包过滤匹配流程</p><p> 1>.规则表之间的优先顺序</p><p> 依次应用：raw、mangle、nat、filter表</p><p> 2>.规则链之间的优先顺序</p><p> 入站数据流向</p><p> 转发数据流向</p><p> 出站数据流向</p><p> 3>.规则链内部各条防火墙规则之间的优先顺序</p><p>所以如果liunx是作为服务器，就要让客户端可以访问你的服务，就得要处理 filter 的 INPUT 链； 而如果你的 Linux 是作为局域网络的路由器，那么就得要分析 nat 的各个链以及 filter 的 FORWARD 链才行。也就是说， 其实各个表格的链结之间是有关系的。</p><p><img src="https://github.com/fengyfei/WikiImages/blob/master/iptables/iptables_02.png?raw=true" alt="tu"></p><p> iptables 可以控制三种封包的流向：</p><ul><li>封包进入 Linux 主机使用资源 (路径 A)： 在路由判断后确定是向 Linux 主机要求数据的封包，主要就会透过 filter 的 INPUT 链来进行控管；</li><li>封包经由 Linux 主机的转递，没有使用主机资源，而是向后端主机流动 (路径 B)： 在路由判断之前进行封包表头的修订作业后，发现到封包主要是要透过防火墙而去后端，此时封包就会透过路径 B 来跑动。 也就是说，该封包的目标并非我们的 Linux 本机。主要经过的链是 filter 的 FORWARD 以及 nat 的 POSTROUTING, PREROUTING。</li><li>封包由 Linux 本机发送出去 (路径 C)： 例如响应客户端的要求，或者是 Linux 本机主动送出的封包，都是透过路径 C 来跑的。先是透过路由判断， 决定了输出的路径后，再透过 filter 的 OUTPUT 链来传送。当然，最终还是会经过 nat 的 POSTROUTING 链。</li></ul><p>iptables 至少有三个预设的 table (filter, nat, mangle)，较常用的是本机的 filter 表格， 也是默认表格。另一个则是后端主机的 nat 表格，至于 mangle 较少使用，所以这里我们并不会讨论 mangle。 由于不同的 table 他们的链不一样，导致使用的指令语法或多或少都有点差异。 这里，我们主要将针对 filter 这个默认表格的三条链来做介绍。</p><h4 id="显示当前规则"><a href="#显示当前规则" class="headerlink" title="显示当前规则"></a>显示当前规则</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div></pre></td><td class="code"><pre><div class="line">[root@www ~]# iptables [-t tables] [-L] [-nv]</div><div class="line">选项与参数：</div><div class="line">-t ：后面接 table ，例如 nat 或 filter ，若省略此项目，则使用默认的 filter</div><div class="line">-L ：列出目前的 table 的规则</div><div class="line">-n ：不进行 IP 与 HOSTNAME 的反查，显示讯息的速度会快很多</div><div class="line">-v ：列出更多的信息，包括通过该规则的封包总位数、相关的网络接口等</div><div class="line"></div><div class="line">范例：列出 filter table 三条链的规则</div><div class="line">[root@www ~]# iptables -L -n</div><div class="line">Chain INPUT (policy ACCEPT) <==INPUT 链，预设为可接受</div><div class="line">target prot opt source destination <==说明栏</div><div class="line">ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED <==第 1 条规则</div><div class="line">ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 <==第 2 条规则</div><div class="line">ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 <==第 3 条规则</div><div class="line">ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 <==以下类推</div><div class="line">REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited</div><div class="line"></div><div class="line">Chain FORWARD (policy ACCEPT) <==针对 FORWARD 链，且预设政策为可接受</div><div class="line">target prot opt source destination</div><div class="line">REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited</div><div class="line"></div><div class="line">Chain OUTPUT (policy ACCEPT) <==针对 OUTPUT 链，且预设政策为可接受</div><div class="line">target prot opt source destination</div><div class="line"></div><div class="line">范例：列出 nat table 三条链的规则</div><div class="line">[root@www ~]# iptables -t nat -L -n</div><div class="line">Chain PREROUTING (policy ACCEPT)</div><div class="line">target prot opt source destination</div><div class="line"></div><div class="line">Chain POSTROUTING (policy ACCEPT)</div><div class="line">target prot opt source destination</div><div class="line"></div><div class="line">Chain OUTPUT (policy ACCEPT)</div><div class="line">target prot opt source destination</div></pre></td></tr></table></figure><p>在上表中，每一个 Chain 就是前面提到的每个链， Chain 那一行里面括号的 policy 就是预设的政策， 那底下的 target, prot 释义如下：</p><ul><li>target：代表进行的动作， ACCEPT 是放行，而 REJECT 则是拒绝，此外，也有 DROP (丢弃) 的项目；</li><li>prot：代表使用的封包协议，主要有 tcp, udp 及 icmp 三种封包格式；</li><li>opt：额外的选项说明；</li><li>source ：代表此规则是针对哪个“来源 IP”进行限制；</li><li>destination ：代表此规则是针对哪个“目标 IP”进行限制。</li></ul><p>在输出结果中，第一个范例因为没有加上 -t 的选项，所以默认就是 filter 这个表格内的 INPUT, OUTPUT, FORWARD 三条链的规则。若针对单机来说，INPUT 与 FORWARD 算是比较重要的管制防火墙链， 所以你可以发现最后一条规则的政策是 REJECT (拒绝) 。虽然 INPUT 与 FORWARD 的政策是放行 (ACCEPT)， 不过在最后一条规则就已经将全部的封包都拒绝了。</p><p>不过这个指令的观察只是作个格式化的查阅，要详细解释每个规则会比较不容易解析。举例来说， 我们将 INPUT 的 5 条规则依据输出结果来说明一下，结果会变成：</p><ol><li>只要是封包状态为 RELATED,ESTABLISHED 就予以接受</li><li>只要封包协议是 icmp 类型的，就予以放行</li><li>无论任何来源 (0.0.0.0/0) 且要去任何目标的封包，不论任何封包格式 (prot 为 all)，通通都接受</li><li>只要是传给 port 22 的主动式联机 tcp 封包就接受</li><li>全部的封包信息通通拒绝</li></ol><p>可以看下第 3 条规则，怎么会所有的封包信息都予以接受，如果都接受的话，那么后续的规则根本就不会有用了。 其实那条规则是仅针对每部主机都有的内部循环测试网络 (lo) 接口。如果没有列出接口，那么我们就很容易搞错。 所以，建议使用 iptables-save 这个指令来观察防火墙规则。因为 iptables-save 会列出完整的防火墙规则，只是并没有规格化输出而已。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div></pre></td><td class="code"><pre><div class="line">[root@www ~]# iptables-save [-t table]</div><div class="line">选项与参数：</div><div class="line">-t ：可以仅针对某些表格来输出，例如仅针对 nat 或 filter 等等</div><div class="line"></div><div class="line">[root@www ~]# iptables-save</div><div class="line"><span class="meta">#</span><span class="bash"> Generated by iptables-save v1.4.7 on Fri Jul 22 15:51:52 2011</span></div><div class="line">*filter <==星号开头的指的是表格，这里为 filter</div><div class="line">:INPUT ACCEPT [0:0] <==冒号开头的指的是链，三条内建的链</div><div class="line">:FORWARD ACCEPT [0:0] <==三条内建链的政策都是 ACCEPT </div><div class="line">:OUTPUT ACCEPT [680:100461]</div><div class="line">-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT <==针对 INPUT 的规则</div><div class="line">-A INPUT -p icmp -j ACCEPT</div><div class="line">-A INPUT -i lo -j ACCEPT <==这条很重要，针对本机内部接口开放</div><div class="line">-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT</div><div class="line">-A INPUT -j REJECT --reject-with icmp-host-prohibited</div><div class="line">-A FORWARD -j REJECT --reject-with icmp-host-prohibited <==针对 FORWARD 的规则</div><div class="line">COMMIT</div><div class="line"><span class="meta">#</span><span class="bash"> Completed on Fri Jul 22 15:51:52 2011</span></div></pre></td></tr></table></figure><p>其中，每个表都包含链和规则，链的详细说明是:<chain-name> <chain-policy> [<packet-counter>:<byte-counter>]，即[<packet-counter>:<byte-counter>]表示该链已经匹配了多少个 [包，字节]</byte-counter></packet-counter></byte-counter></packet-counter></chain-policy></chain-name></p><p>由上面的输出来看，在内容含有 lo 的那条规则当中，” -i lo” 指的就是由 lo 适配卡进来的封包, 这样看就清楚多了。 不过，既然这个规则不是我们想要的，下面看下如何修改规则</p><h4 id="清除规则"><a href="#清除规则" class="headerlink" title="清除规则:"></a>清除规则:</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line">[root@www ~]# iptables [-t tables] [-FXZ]</div><div class="line">选项与参数：</div><div class="line">-F ：清除所有的已订定的规则；</div><div class="line">-X ：杀掉所有使用者 "自定义" 的 chain ；</div><div class="line">-Z ：将所有的 chain 的计数与流量统计都归零</div><div class="line"></div><div class="line">范例：清除本机防火墙 (filter) 的所有规则</div><div class="line">[root@www ~]# iptables -F</div><div class="line">[root@www ~]# iptables -X</div><div class="line">[root@www ~]# iptables -Z</div></pre></td></tr></table></figure><p>这三个指令会将本机防火墙的所有规则都清除，但却不会改变预设政策(policy)</p><h4 id="定义预设-policy"><a href="#定义预设-policy" class="headerlink" title="定义预设 (policy)"></a>定义预设 (policy)</h4><p>预设规则指当您的数据包不在您设定的规则之内时，则该包的通过与否，以 Policy 的设定为准</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div></pre></td><td class="code"><pre><div class="line">[root@www ~]# iptables [-t nat] -P [INPUT,OUTPUT,FORWARD] [ACCEPT,DROP]</div><div class="line">选项与参数：</div><div class="line">-P ：定义政策( Policy )。注意，这个 P 为大写</div><div class="line">ACCEPT ：该封包可接受</div><div class="line">DROP ：该封包直接丢弃，不会让 client 端知道为何被丢弃。</div><div class="line"></div><div class="line">范例：将本机的 INPUT 设定为 DROP ，其他设定为 ACCEPT</div><div class="line">[root@www ~]# iptables -P INPUT DROP</div><div class="line">[root@www ~]# iptables -P OUTPUT ACCEPT</div><div class="line">[root@www ~]# iptables -P FORWARD ACCEPT</div><div class="line">[root@www ~]# iptables-save</div><div class="line"><span class="meta">#</span><span class="bash"> Generated by iptables-save v1.4.7 on Fri Jul 22 15:56:34 2011</span></div><div class="line">*filter</div><div class="line">:INPUT DROP [0:0]</div><div class="line">:FORWARD ACCEPT [0:0]</div><div class="line">:OUTPUT ACCEPT [0:0]</div><div class="line">COMMIT</div><div class="line"><span class="meta">#</span><span class="bash"> Completed on Fri Jul 22 15:56:34 2011</span></div></pre></td></tr></table></figure><h4 id="数据包的基础比对：IP-网域及接口装置"><a href="#数据包的基础比对：IP-网域及接口装置" class="headerlink" title="数据包的基础比对：IP, 网域及接口装置"></a>数据包的基础比对：IP, 网域及接口装置</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div></pre></td><td class="code"><pre><div class="line">[root@www ~]# iptables [-AI 链名] [-io 网络接口] [-p 协议] \</div><div class="line"><span class="meta">></span><span class="bash"> [-s 来源IP/网域] [-d 目标IP/网域] -j [ACCEPT|DROP|REJECT|LOG]</span></div><div class="line">选项与参数：</div><div class="line">-AI 链名：针对某的链进行规则的 "插入" 或 "累加"</div><div class="line"> -A ：新增加一条规则，该规则增加在原本规则的最后面。例如原本已经有四条规则，</div><div class="line"> 使用 -A 就可以加上第五条规则！</div><div class="line"> -I ：插入一条规则。如果没有指定此规则的顺序，默认是插入变成第一条规则。</div><div class="line"> 例如原本有四条规则，使用 -I 则该规则变成第一条，而原本四条变成 2~5 号</div><div class="line"> 链 ：有 INPUT, OUTPUT, FORWARD 等，此链名称又与 -io 有关，请看底下。</div><div class="line"></div><div class="line">-io 网络接口：设定封包进出的接口规范</div><div class="line"> -i ：封包所进入的那个网络接口，例如 eth0, lo 等接口。需与 INPUT 链配合；</div><div class="line"> -o ：封包所传出的那个网络接口，需与 OUTPUT 链配合；</div><div class="line"></div><div class="line">-p 协定：设定此规则适用于哪种封包格式</div><div class="line"> 主要的封包格式有： tcp, udp, icmp 及 all 。</div><div class="line"></div><div class="line">-s 来源 IP/网域：设定此规则之封包的来源项目，可指定单纯的 IP 或包括网域，例如：</div><div class="line"> IP ：192.168.0.100</div><div class="line"> 网域：192.168.0.0/24, 192.168.0.0/255.255.255.0 均可。</div><div class="line"> 若规范为"不许"时，则加上 ! 即可，例如：</div><div class="line"> -s ! 192.168.100.0/24 表示不许 192.168.100.0/24 的封包来源；</div><div class="line"></div><div class="line">-d 目标 IP/网域：同 -s ，只不过这里指的是目标的 IP 或网域。</div><div class="line"></div><div class="line">-j ：后面接动作，主要的动作有接受(ACCEPT)、丢弃(DROP)、拒绝(REJECT)及记录(LOG)</div></pre></td></tr></table></figure><p>iptables 的基本参数就如同上面所示的，仅只谈到 IP 、网域与接口装置等等的信息， 至于 TCP, UDP 包特有的端口与状态则在下小节才会谈到。 好，先让我们来看看最基本的几个规则，例如开放 lo 这个本机的接口以及某个 IP 来源。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div></pre></td><td class="code"><pre><div class="line">范例一:所有的来自 lo 这个接口的数据包，都予以接受</div><div class="line">[root@linux ~]# iptables -A INPUT -i lo -j ACCEPT</div><div class="line"><span class="meta">#</span><span class="bash"> 仔细看上面并没有列出 -s, -d 等等的规则，这表示:不论本包来自何处或去到哪里， </span></div><div class="line"><span class="meta">#</span><span class="bash"> 只要是来自 lo 这个接口，就予以接受。</span></div><div class="line"></div><div class="line">范例：只要是来自内网的 (192.168.100.0/24) 的封包通通接受</div><div class="line">[root@www ~]# iptables -A INPUT -i eth1 -s 192.168.100.0/24 -j ACCEPT</div><div class="line"></div><div class="line">范例：只要是来自 192.168.100.10 就接受，但 192.168.100.230 这个恶意来源就丢弃</div><div class="line">[root@www ~]# iptables -A INPUT -i eth1 -s 192.168.100.10 -j ACCEPT</div><div class="line">[root@www ~]# iptables -A INPUT -i eth1 -s 192.168.100.230 -j DROP</div><div class="line"></div><div class="line">[root@www ~]# iptables-save</div><div class="line"><span class="meta">#</span><span class="bash"> Generated by iptables-save v1.4.7 on Fri Jul 22 16:00:43 2011</span></div><div class="line">*filter</div><div class="line">:INPUT DROP [0:0]</div><div class="line">:FORWARD ACCEPT [0:0]</div><div class="line">:OUTPUT ACCEPT [17:1724]</div><div class="line">-A INPUT -i lo -j ACCEPT</div><div class="line">-A INPUT -s 192.168.100.0/24 -i eth1 -j ACCEPT</div><div class="line">-A INPUT -s 192.168.100.10/32 -i eth1 -j ACCEPT</div><div class="line">-A INPUT -s 192.168.100.230/32 -i eth1 -j DROP</div><div class="line">COMMIT</div><div class="line"><span class="meta">#</span><span class="bash"> Completed on Fri Jul 22 16:00:43 2011</span></div></pre></td></tr></table></figure><h4 id="记录某个规则的日志"><a href="#记录某个规则的日志" class="headerlink" title="记录某个规则的日志"></a>记录某个规则的日志</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">[root@linux ~]# iptables -A INPUT -s 192.168.2.200 -j LOG</div><div class="line"> </div><div class="line">[root@linux ~]# iptables -L -n</div><div class="line">target prot opt source destination</div><div class="line">LOG all -- 192.168.2.200 0.0.0.0/0 LOG flags 0 level 4</div></pre></td></tr></table></figure><p>看到输出结果的最左边，会出现的是 LOG 。只要有 包来自 192.168.2.200 这个 IP 时， 那么该 包<br>的相关信息就会被写入到日志中，亦即是 /var/log/messages 这个文件当中。LOG 这个动作仅在进行记录而已，并不会影响到这个数据包的其它规则比对的。</p><h4 id="TCP-UDP-的规则比对"><a href="#TCP-UDP-的规则比对" class="headerlink" title="TCP, UDP 的规则比对"></a>TCP, UDP 的规则比对</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">[root@linux ~]# iptables [-AI 链] [-io 网络接口] [-p tcp,udp] </div><div class="line"><span class="meta">></span><span class="bash"> [-s 来源IP/网域] [--sport 端口范围] </span></div><div class="line"><span class="meta">></span><span class="bash"> [-d 目标IP/网域] [--dport 端口范围] -j [ACCEPT|DROP|REJECT]</span></div><div class="line"></div><div class="line">参数:</div><div class="line">选项与参数：</div><div class="line">--sport 端口范围：限制来源的端口号码，端口号码可以是连续的，例如 1024:65535</div><div class="line">--dport 端口范围：限制目标的端口号码。</div></pre></td></tr></table></figure><p>事实上就是多了那个 –sport 及 –dport 这两个选项，重点在那个 port number， 仅有 tcp 与 udp 封包具有端口，因此你想要使用 –dport, –sport 时，得要加上 -p tcp 或 -p udp 的参数才会成功。底下让我们来进行几个小测试:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">范例一:想要联机进入本机 port 21 的数据包都丢掉:</div><div class="line">[root@linux ~]# iptables -A INPUT -i eth0 -p tcp --dport 21 -j DROP</div><div class="line">范例二:想连到我这部主机的(upd port 137,138 tcp port 139,445) 就放行 </div><div class="line">[root@linux ~]# iptables -A INPUT -i eth0 -p udp --dport 137:138 -j ACCEPT </div><div class="line">[root@linux ~]# iptables -A INPUT -i eth0 -p tcp --dport 139 -j ACCEPT </div><div class="line">[root@linux ~]# iptables -A INPUT -i eth0 -p tcp --dport 445 -j ACCEPT</div></pre></td></tr></table></figure><p>你可以利用 UDP 与 TCP 协议所拥有的端口号码来进行某些服务的开放或关闭。</p><p>例如:只要来自 192.168.1.0/24 的 1024:65535 端口的数据包，只要想要联机到本机的 ssh port 就予以抵挡，可以这样做:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">[root@linux ~]# iptables -A INPUT -i eth0 -p tcp -s 192.168.1.0/24 \</div><div class="line"> --sport 1024:65534 --dport ssh -j DROP</div></pre></td></tr></table></figure><p>如果你有使用到 –sport 及 –dport 的参数时，忘了指定 -p tcp 或 -p udp，就会出现如下的错误:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">[root@k8s ~]# iptables -A INPUT -i eth0 --dport 21 -j DROP</div><div class="line">iptables v1.4.21: unknown option "--dport"</div><div class="line">Try `iptables -h' or 'iptables --help' for more information.</div></pre></td></tr></table></figure><p>在 iptables里面还支持 –syn 的处理方式，我们以底下的例子来说明:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">范例:将来自任何地方来源 port 1:1023 的主动联机到本机端的 1:1023 联机丢弃 </div><div class="line">[root@linux ~]# iptables -A INPUT -i eth0 -p tcp --sport 1:1023 \</div><div class="line"> --dport 1:1023 --syn -j DROP</div></pre></td></tr></table></figure><h4 id="状态模块-MAC-与-STATE"><a href="#状态模块-MAC-与-STATE" class="headerlink" title="状态模块:MAC 与 STATE"></a>状态模块:MAC 与 STATE</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line">[root@linux ~]# iptables -A INPUT [-m state] [--state 状态]</div><div class="line">选项与参数：</div><div class="line">-m ：一些 iptables 的外挂模块，主要常见的有：</div><div class="line"> state ：状态模块</div><div class="line"> mac ：网络卡硬件地址 (hardware address)</div><div class="line">--state ：一些数据包的状态，主要有：</div><div class="line"> INVALID ：无效的数据包，例如数据破损的数据包状态</div><div class="line"> ESTABLISHED：已经联机成功的联机状态；</div><div class="line"> NEW ：想要新建立联机的数据包状态；</div><div class="line"> RELATED ：这个最常用，表示这个数据包是与我们主机发送出去的数据包有关</div><div class="line"></div><div class="line">范例:只要已建立或相关数据包就予以通过，只要是不合法数据包就丢弃</div><div class="line">[root@linux ~]# iptables -A INPUT -m state \</div><div class="line">--state RELATED,ESTABLISHED -j ACCEPT</div><div class="line">[root@linux ~]# iptables -A INPUT -m state --state INVALID -j DROP</div></pre></td></tr></table></figure><p>所以说，如果你的 Linux 主机只想要作为 client 的用途，不许所有主动对你联机的来源， 那么你可以这样做即可:</p><ol><li>清除所有已经存在的规则 (iptables -F…)</li><li>设定预设 ，除了 INPUT 预设为 DROP 其它为预设 ACCEPT;</li><li>开放本机的 lo 可以自由放行;</li><li>设定有相关的数据包状态可以联机进入本机。</li></ol><p>你可以在某个 script 上面这样做即可:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#</span><span class="bash">!/bin/bash</span></div><div class="line">PATH=/sbin:/bin:/usr/sbin:/usr/bin export PATH</div><div class="line">iptables -F</div><div class="line">iptables -X</div><div class="line">iptables -Z </div><div class="line">iptables -P INPUT DROP</div><div class="line">iptables -P OUTPUT ACCEPT</div><div class="line">iptables -P FORWARD ACCEPT</div><div class="line">iptables -A INPUT -i lo -j ACCEPT</div><div class="line">iptables -A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT</div><div class="line"><span class="meta">#</span><span class="bash">iptables -A INPUT -i eth0 -s 192.168.1.0/24 -j ACCEPT</span></div></pre></td></tr></table></figure><p>那如果局域网络内有其它的主机时，再将上表最后一行的 # 取消，就可以接受来自本地 LAN 的其它主机的联机了。 如果你担心某些 LAN 内的恶意来源主机会主动的对你联机时，那你还可以针对信任的本地端主机的 MAC 进行过滤。这次的状态则是 MAC 的比对。举例来说:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">范例一:针对 域网络内的 aa:bb:cc:dd:ee:ff 主机开放其联机</div><div class="line">[root@linux ~]# iptables -A INPUT -m mac --mac-source aa:bb:cc:dd:ee:ff \</div><div class="line"> -j ACCEPT</div><div class="line">参数:</div><div class="line">--mac-source :是来源主机的 MAC .</div></pre></td></tr></table></figure><h4 id="ICMP-包规则的比对"><a href="#ICMP-包规则的比对" class="headerlink" title="ICMP 包规则的比对"></a>ICMP 包规则的比对</h4><p>在网络基 的 ICMP 协议当中我们知道 ICMP 的格式相当的多，而且很多 ICMP 包的类型格式都是为了<br>要用来进行网络 测用的。所以最好不要将所有的 ICMP 数据包都丢弃。通常我们会把 ICMP type 8 (echo<br>request) 拿掉而已，让远程主机不知道我们是否存在，也不会接受 ping 的响应就是了。ICMP 包格式<br>的处理是这样的:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line">[root@linux ~]# iptables -A INPUT [-p icmp] [--icmp-type 类型] -j ACCEPT</div><div class="line"></div><div class="line">选项与参数：</div><div class="line">--icmp-type ：后面必须要接 ICMP 的数据包类型，也可以使用代号，</div><div class="line"> 例如 8 代表 echo request 的意思。</div><div class="line"><span class="meta">#</span><span class="bash">范例:让 0,3,4,11,12,14,16,18 的 ICMP <span class="built_in">type</span> 可以进入 机: </span></div><div class="line">[root@linux ~]# vi somefile</div><div class="line"><span class="meta">#</span><span class="bash">!/bin/bash</span></div><div class="line">icmp_type= 0 3 4 11 12 14 16 18 </div><div class="line">for typeicmp in icmp_type</div><div class="line">do</div><div class="line"> iptables -A INPUT -i eth0 -p icmp --icmp-type typeicmp -j ACCEPT</div><div class="line">done</div><div class="line">[root@linux ~]# sh somefile</div></pre></td></tr></table></figure><p>这样就能够开放部分的 ICMP 数据包格式进入本机进行网络检测的工作了。</p>]]></content>
<summary type="html">
<p>iptables利用的是数据包过滤机制</p>
</summary>
<category term="Linux" scheme="https://yangchenglong11.github.io/categories/Linux/"/>
<category term="Linux" scheme="https://yangchenglong11.github.io/tags/Linux/"/>
</entry>
<entry>
<title>flag</title>
<link href="https://yangchenglong11.github.io/2017/07/12/flag/"/>
<id>https://yangchenglong11.github.io/2017/07/12/flag/</id>
<published>2017-07-12T08:14:32.000Z</published>
<updated>2018-02-12T12:04:12.448Z</updated>
<content type="html"><![CDATA[<p><strong>flag</strong> 是Go 标准库提供的解析命令行参数的包。flag跟以往的传统命令行解析不同,它将参数定义, 参数默认值, 命令帮助, 参数读取集成到一起,摆脱以前繁琐的命令判断流程.</p><a id="more"></a><p><strong>使用方式：</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">flag.Type(name, defValue, usage)</div></pre></td></tr></table></figure><p>其中Type为String, Int, Bool等；并返回一个相应类型的指针。</p><figure class="highlight sqf"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="built_in">flag</span>.TypeVar(&flagvar, <span class="built_in">name</span>, defValue, usage)</div></pre></td></tr></table></figure><p>将flag绑定到一个变量上。</p><p>##返回指针<br>通过flag.String, flag.Int, flag.Bool等可以直接返回一个参数指针,因此后续使用的时候需要加*进行取值:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">str = flag.String(<span class="string">"mystring"</span>, <span class="string">"default-value"</span>, <span class="string">"mystring is my test flag of String"</span>)</div><div class="line">i = flag.Int(<span class="string">"myint"</span>, <span class="number">123</span>, <span class="string">"test int flag"</span>)</div></pre></td></tr></table></figure></p><p>##序列化到变量<br>通过flag.StringVar, flag.IntVar, flag.BoolVar等,可以直接把值序列化到变量:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"> flag.StringVar(&vStr, <span class="string">"mystring2"</span>, <span class="string">"default-value2"</span>, <span class="string">"mystring2 is my test flag of String"</span>)</div><div class="line">flag.IntVar(&vI, <span class="string">"myint2"</span>, <span class="number">123</span>, <span class="string">"test int flag"</span>)</div></pre></td></tr></table></figure></p><p><strong>自定义flag</strong></p><p>只要实现flag.Value接口即可：</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">type Value<span class="built_in"> interface </span>{</div><div class="line"> String() string</div><div class="line"> <span class="builtin-name">Set</span>(string) error</div><div class="line">}</div></pre></td></tr></table></figure><p>通过如下方式定义该flag：</p><figure class="highlight delphi"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">flag.<span class="keyword">Var</span>(&flagvar, <span class="keyword">name</span>, usage)</div></pre></td></tr></table></figure><p>example：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"><span class="keyword">import</span>(</div><div class="line"><span class="string">"flag"</span></div><div class="line"><span class="string">"fmt"</span></div><div class="line"><span class="string">"strconv"</span></div><div class="line">) </div><div class="line"></div><div class="line"><span class="keyword">type</span> percentage <span class="keyword">float32</span></div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(p *percentage)</span> <span class="title">Set</span><span class="params">(s <span class="keyword">string</span>)</span> <span class="title">error</span></span> {</div><div class="line">v, err := strconv.ParseFloat(s, <span class="number">32</span>)</div><div class="line">*p = percentage(v)</div><div class="line"><span class="keyword">return</span> err</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(p *percentage)</span> <span class="title">String</span><span class="params">()</span> <span class="title">string</span></span> {</div><div class="line"><span class="keyword">return</span> fmt.Sprintf(<span class="string">"%f"</span>, *p) </div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</div><div class="line">namePtr := flag.String(<span class="string">"name"</span>, <span class="string">"yang"</span>, <span class="string">"user's name"</span>)</div><div class="line">agePtr := flag.Int(<span class="string">"age"</span>, <span class="number">20</span>, <span class="string">"user's age"</span>)</div><div class="line">vipPtr := flag.Bool(<span class="string">"vip"</span>, <span class="literal">true</span>, <span class="string">"is a vip user"</span>)</div><div class="line"></div><div class="line"><span class="keyword">var</span> email <span class="keyword">string</span></div><div class="line"></div><div class="line">flag.StringVar(&email, <span class="string">"email"</span>, <span class="string">"[email protected]"</span>, <span class="string">"user's email"</span>)</div><div class="line"></div><div class="line"><span class="keyword">var</span> pop percentage</div><div class="line"></div><div class="line">flag.Var(&pop, <span class="string">"pop"</span>, <span class="string">"popularity"</span>)</div><div class="line"></div><div class="line">flag.Parse()</div><div class="line"></div><div class="line">others := flag.Args()</div><div class="line">fmt.Println(<span class="string">"name:"</span>, *namePtr)</div><div class="line">fmt.Println(<span class="string">"age:"</span>, *agePtr)</div><div class="line">fmt.Println(<span class="string">"vip:"</span>, *vipPtr)</div><div class="line">fmt.Println(<span class="string">"pop:"</span>, pop)</div><div class="line">fmt.Println(<span class="string">"email:"</span>, email)</div><div class="line">fmt.Println(<span class="string">"other:"</span>, others)</div><div class="line">}</div></pre></td></tr></table></figure><p>输出：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> go run test.go </span></div><div class="line">name: yang</div><div class="line">age: 20</div><div class="line">vip: true</div><div class="line">email: [email protected]</div><div class="line">other: []</div></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> go run test.go -name golang -age 4 -vip=<span class="literal">true</span> -pop 99 简洁 高并发 等等</span></div><div class="line">name: golang</div><div class="line">age: 4</div><div class="line">vip: true</div><div class="line">pop: 99</div><div class="line">email: [email protected]</div><div class="line">other: [简洁 高并发 等等]</div></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> go run test.go -h</span></div><div class="line">Usage of go /var/folders/6y/z4y473d12fj8ss2gkr276dx80000gn/T/go-build217780999/command-line-arguments/_obj/exe/test :</div><div class="line"> -age=20: user's age</div><div class="line"> -email="[email protected]": user's email</div><div class="line"> -name="yang": user's name</div><div class="line"> -pop=0.0: popularity</div><div class="line"> -vip=true: is a vip user</div></pre></td></tr></table></figure><h1 id="pflag"><a href="#pflag" class="headerlink" title="pflag"></a>pflag</h1><p>pflag是flag扩展,主要是加入了对短参数的支持.</p><h2 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h2><figure class="highlight sh"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">go get github.com/ogier/pflag</div></pre></td></tr></table></figure><h2 id="测试"><a href="#测试" class="headerlink" title="测试"></a>测试</h2><figure class="highlight sh"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">go <span class="built_in">test</span> github.com/ogier/pflag</div></pre></td></tr></table></figure><h2 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h2><p>如果你之前已经import了flag包,那么直接把那行改为:</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">import</span> flag <span class="string">"github.com/ogier/pflag"</span></div></pre></td></tr></table></figure><h2 id="短参数支持"><a href="#短参数支持" class="headerlink" title="短参数支持"></a>短参数支持</h2><p>在函数后面加上P即可:</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"> pflag.StringVarP(&p1, <span class="string">"stringflag"</span>, <span class="string">"s"</span>, <span class="string">"test-p-flag-string"</span>, <span class="string">"test pflag for string."</span>)</div><div class="line">pflag.IntVarP(&p2, <span class="string">"intflag"</span>, <span class="string">"i"</span>, <span class="number">12345</span>, <span class="string">"test pflag for int."</span>)</div></pre></td></tr></table></figure><h2 id="程序运行结果"><a href="#程序运行结果" class="headerlink" title="程序运行结果"></a>程序运行结果</h2><figure class="highlight sh"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">$ go run main.go</div><div class="line"></div><div class="line">Usage of /private/var/folders/98/pmb6b_8x5w5319m9ts6h6z6m0000gn/T/Build main.go and run1go:</div><div class="line"> -i, --intflag=12345: <span class="built_in">test</span> pflag <span class="keyword">for</span> int.</div><div class="line"> -s, --stringflag=<span class="string">"test-p-flag-string"</span>: <span class="built_in">test</span> pflag <span class="keyword">for</span> string.</div></pre></td></tr></table></figure>]]></content>
<summary type="html">
<p><strong>flag</strong> 是Go 标准库提供的解析命令行参数的包。flag跟以往的传统命令行解析不同,它将参数定义, 参数默认值, 命令帮助, 参数读取集成到一起,摆脱以前繁琐的命令判断流程.</p>
</summary>
<category term="Golang" scheme="https://yangchenglong11.github.io/categories/Golang/"/>
<category term="Golang" scheme="https://yangchenglong11.github.io/tags/Golang/"/>
</entry>
<entry>
<title>channel示例之GoTocket</title>
<link href="https://yangchenglong11.github.io/2017/05/26/channel%E7%A4%BA%E4%BE%8B%E4%B9%8BGoTocket/"/>
<id>https://yangchenglong11.github.io/2017/05/26/channel示例之GoTocket/</id>
<published>2017-05-26T03:54:34.000Z</published>
<updated>2017-10-28T14:02:48.429Z</updated>
<content type="html"><![CDATA[<p>该代码为批量处理人员信息，即将人员信息中的居住地进行统一处理。</p><a id="more"></a><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div><div class="line">114</div><div class="line">115</div><div class="line">116</div><div class="line">117</div><div class="line">118</div><div class="line">119</div><div class="line">120</div><div class="line">121</div><div class="line">122</div><div class="line">123</div><div class="line">124</div><div class="line">125</div><div class="line">126</div><div class="line">127</div><div class="line">128</div><div class="line">129</div><div class="line">130</div><div class="line">131</div><div class="line">132</div><div class="line">133</div><div class="line">134</div><div class="line">135</div><div class="line">136</div><div class="line">137</div><div class="line">138</div><div class="line">139</div><div class="line">140</div><div class="line">141</div><div class="line">142</div><div class="line">143</div><div class="line">144</div><div class="line">145</div><div class="line">146</div><div class="line">147</div><div class="line">148</div><div class="line">149</div><div class="line">150</div><div class="line">151</div><div class="line">152</div><div class="line">153</div><div class="line">154</div><div class="line">155</div><div class="line">156</div><div class="line">157</div><div class="line">158</div><div class="line">159</div><div class="line">160</div><div class="line">161</div><div class="line">162</div><div class="line">163</div><div class="line">164</div><div class="line">165</div><div class="line">166</div><div class="line">167</div><div class="line">168</div><div class="line">169</div><div class="line">170</div><div class="line">171</div><div class="line">172</div><div class="line">173</div><div class="line">174</div><div class="line">175</div><div class="line">176</div><div class="line">177</div><div class="line">178</div><div class="line">179</div><div class="line">180</div><div class="line">181</div><div class="line">182</div><div class="line">183</div><div class="line">184</div><div class="line">185</div><div class="line">186</div><div class="line">187</div><div class="line">188</div><div class="line">189</div><div class="line">190</div><div class="line">191</div><div class="line">192</div><div class="line">193</div><div class="line">194</div><div class="line">195</div><div class="line">196</div><div class="line">197</div><div class="line">198</div><div class="line">199</div><div class="line">200</div><div class="line">201</div><div class="line">202</div><div class="line">203</div><div class="line">204</div><div class="line">205</div><div class="line">206</div><div class="line">207</div><div class="line">208</div><div class="line">209</div><div class="line">210</div><div class="line">211</div><div class="line">212</div><div class="line">213</div><div class="line">214</div><div class="line">215</div><div class="line">216</div><div class="line">217</div><div class="line">218</div><div class="line">219</div><div class="line">220</div><div class="line">221</div><div class="line">222</div><div class="line">223</div><div class="line">224</div><div class="line">225</div><div class="line">226</div><div class="line">227</div><div class="line">228</div><div class="line">229</div><div class="line">230</div><div class="line">231</div><div class="line">232</div><div class="line">233</div><div class="line">234</div><div class="line">235</div><div class="line">236</div><div class="line">237</div><div class="line">238</div><div class="line">239</div><div class="line">240</div><div class="line">241</div><div class="line">242</div><div class="line">243</div><div class="line">244</div><div class="line">245</div><div class="line">246</div><div class="line">247</div><div class="line">248</div><div class="line">249</div><div class="line">250</div><div class="line">251</div><div class="line">252</div><div class="line">253</div><div class="line">254</div><div class="line">255</div><div class="line">256</div><div class="line">257</div><div class="line">258</div><div class="line">259</div><div class="line">260</div><div class="line">261</div><div class="line">262</div><div class="line">263</div><div class="line">264</div><div class="line">265</div><div class="line">266</div><div class="line">267</div><div class="line">268</div><div class="line">269</div><div class="line">270</div><div class="line">271</div><div class="line">272</div><div class="line">273</div><div class="line">274</div><div class="line">275</div><div class="line">276</div><div class="line">277</div><div class="line">278</div><div class="line">279</div><div class="line">280</div><div class="line">281</div><div class="line">282</div><div class="line">283</div><div class="line">284</div><div class="line">285</div><div class="line">286</div><div class="line">287</div><div class="line">288</div><div class="line">289</div><div class="line">290</div><div class="line">291</div><div class="line">292</div><div class="line">293</div><div class="line">294</div><div class="line">295</div><div class="line">296</div><div class="line">297</div><div class="line">298</div><div class="line">299</div><div class="line">300</div><div class="line">301</div><div class="line">302</div><div class="line">303</div><div class="line">304</div><div class="line">305</div><div class="line">306</div><div class="line">307</div><div class="line">308</div><div class="line">309</div><div class="line">310</div><div class="line">311</div><div class="line">312</div><div class="line">313</div><div class="line">314</div><div class="line">315</div><div class="line">316</div><div class="line">317</div><div class="line">318</div><div class="line">319</div><div class="line">320</div><div class="line">321</div><div class="line">322</div><div class="line">323</div><div class="line">324</div><div class="line">325</div><div class="line">326</div><div class="line">327</div><div class="line">328</div><div class="line">329</div><div class="line">330</div><div class="line">331</div><div class="line">332</div><div class="line">333</div><div class="line">334</div><div class="line">335</div><div class="line">336</div><div class="line">337</div><div class="line">338</div><div class="line">339</div><div class="line">340</div><div class="line">341</div><div class="line">342</div><div class="line">343</div><div class="line">344</div><div class="line">345</div><div class="line">346</div><div class="line">347</div><div class="line">348</div><div class="line">349</div><div class="line">350</div><div class="line">351</div><div class="line">352</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line"><span class="string">"fmt"</span></div><div class="line"><span class="string">"time"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="comment">//定义员工数据结构</span></div><div class="line"><span class="keyword">type</span> Person <span class="keyword">struct</span> {</div><div class="line">Name <span class="keyword">string</span></div><div class="line">Age <span class="keyword">uint8</span></div><div class="line">Address Addr</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//定义地址数据结构</span></div><div class="line"><span class="keyword">type</span> Addr <span class="keyword">struct</span> {</div><div class="line">city <span class="keyword">string</span></div><div class="line">district <span class="keyword">string</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//定义处理接口，方法Batch被声明为实现批量处理人员信息功能的方法，</span></div><div class="line"><span class="comment">//其方法声明中的两个通道分别对该方法和该方法的调用方使用它的方式进行了约束</span></div><div class="line"><span class="keyword">type</span> PersonHandler <span class="keyword">interface</span> {</div><div class="line">Batch(origs <-<span class="keyword">chan</span> Person) <-<span class="keyword">chan</span> Person</div><div class="line">Handle(orig *Person)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//定义空结构体，为其添加方法，实现PersonHandler接口</span></div><div class="line"><span class="keyword">type</span> PersonHandlerImpl <span class="keyword">struct</span>{}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(handler PersonHandlerImpl)</span> <span class="title">Batch</span><span class="params">(origs <-<span class="keyword">chan</span> Person)</span> <-<span class="title">chan</span> <span class="title">Person</span></span> {</div><div class="line">dests := <span class="built_in">make</span>(<span class="keyword">chan</span> Person, <span class="number">100</span>)</div><div class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</div><div class="line"><span class="keyword">for</span> p := <span class="keyword">range</span> origs {</div><div class="line">handler.Handle(&p)</div><div class="line">dests <- p</div><div class="line">}</div><div class="line">fmt.Println(<span class="string">"All the information has been handled."</span>)</div><div class="line"><span class="comment">//在发送方关闭通道</span></div><div class="line"><span class="built_in">close</span>(dests)</div><div class="line">}()</div><div class="line"><span class="keyword">return</span> dests</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(handler PersonHandlerImpl)</span> <span class="title">Handle</span><span class="params">(orig *Person)</span></span> {</div><div class="line"><span class="keyword">if</span> orig.Address.district == <span class="string">"Haidian"</span> {</div><div class="line">orig.Address.district = <span class="string">"Shijingshan"</span></div><div class="line">}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//定义要被处理的数据并初始化</span></div><div class="line"><span class="keyword">var</span> personTotal = <span class="number">200</span></div><div class="line"></div><div class="line"><span class="keyword">var</span> persons []Person = <span class="built_in">make</span>([]Person, personTotal)</div><div class="line"></div><div class="line"><span class="keyword">var</span> personCount <span class="keyword">int</span></div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">init</span><span class="params">()</span></span> {</div><div class="line"><span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">200</span>; i++ {</div><div class="line">name := fmt.Sprintf(<span class="string">"%s%d"</span>, <span class="string">"P"</span>, i)</div><div class="line">p := Person{name, <span class="number">32</span>, Addr{<span class="string">"Beijing"</span>, <span class="string">"Haidian"</span>}}</div><div class="line">persons[i] = p</div><div class="line">}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//main函数中首先获取handler，初始化origs通道，将人员信息通过origs通道传入</span></div><div class="line"><span class="comment">//Batch中处理，处理后的信息放入dests通道中，并将dests通道返回。</span></div><div class="line"><span class="comment">//通道初始化完成后，fecthPerson获取人员信息放入到origs中，savePerson从dests中接收处理过的信息进行保存</span></div><div class="line"><span class="comment">//其中sign通道作用为在批处理完全执行结束之前阻塞主Goroutine</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</div><div class="line">handler := getPersonHandler()</div><div class="line">origs := <span class="built_in">make</span>(<span class="keyword">chan</span> Person, <span class="number">100</span>)</div><div class="line">dests := handler.Batch(origs)</div><div class="line">fecthPerson(origs)</div><div class="line">sign := savePerson(dests)</div><div class="line"><-sign</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">getPersonHandler</span><span class="params">()</span> <span class="title">PersonHandler</span></span> {</div><div class="line"><span class="keyword">return</span> PersonHandlerImpl{}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">savePerson</span><span class="params">(dest <-<span class="keyword">chan</span> Person)</span> <-<span class="title">chan</span> <span class="title">byte</span></span> {</div><div class="line">sign := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">byte</span>, <span class="number">1</span>)</div><div class="line"></div><div class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</div><div class="line">ok := <span class="literal">true</span></div><div class="line"><span class="keyword">var</span> p Person</div><div class="line"><span class="keyword">for</span> {</div><div class="line"><span class="keyword">select</span> {</div><div class="line"><span class="keyword">case</span> p, ok = <-dest:</div><div class="line">{</div><div class="line"><span class="keyword">if</span> !ok {</div><div class="line">fmt.Println(<span class="string">"All the information has been saved."</span>)</div><div class="line">sign <- <span class="number">0</span></div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line">savePerson1(p)</div><div class="line">}</div><div class="line"><span class="keyword">case</span> ok = <-<span class="function"><span class="keyword">func</span><span class="params">()</span> <span class="title">chan</span> <span class="title">bool</span></span> {</div><div class="line">timeout := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">bool</span>, <span class="number">1</span>)</div><div class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</div><div class="line">time.Sleep(time.Millisecond)</div><div class="line">timeout <- <span class="literal">false</span></div><div class="line">}()</div><div class="line"><span class="keyword">return</span> timeout</div><div class="line">}() :</div><div class="line">fmt.Println(<span class="string">"TimeOut!"</span>)</div><div class="line">sign <- <span class="number">0</span></div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">if</span> !ok {</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line">}</div><div class="line">}()</div><div class="line"><span class="keyword">return</span> sign</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">fecthPerson</span><span class="params">(origs <span class="keyword">chan</span><- Person)</span></span> {</div><div class="line"><span class="comment">//调用cap函数确定origs是否为缓冲通道</span></div><div class="line">origsCap := <span class="built_in">cap</span>(origs)</div><div class="line">buffered := origsCap > <span class="number">0</span></div><div class="line"><span class="comment">//以origsCap的一半作为Goroutine票池的总数，创建票池</span></div><div class="line">goTicketTotal := origsCap / <span class="number">2</span></div><div class="line">goTicket := initGoTicket(goTicketTotal)</div><div class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</div><div class="line"><span class="keyword">for</span> {</div><div class="line">p, ok := fecthPerson1()</div><div class="line"><span class="keyword">if</span> !ok {</div><div class="line"><span class="keyword">for</span> {</div><div class="line"><span class="comment">//如果为非缓冲通道或者所有goroutine已完成工作，跳出循环</span></div><div class="line"><span class="keyword">if</span> !buffered || <span class="built_in">len</span>(goTicket) == goTicketTotal {</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line">time.Sleep(time.Nanosecond)</div><div class="line">}</div><div class="line">fmt.Println(<span class="string">"All the information has been fetched."</span>)</div><div class="line"><span class="comment">//在发送方关闭通道</span></div><div class="line"><span class="built_in">close</span>(origs)</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//如果为缓冲通道，从goTicket接受一个值，表示有一个goroutine被占用</span></div><div class="line"><span class="comment">//当操作完成后，向其中发送一个值，表示接解除占用</span></div><div class="line"><span class="keyword">if</span> buffered {</div><div class="line"><-goTicket</div><div class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</div><div class="line">origs <- p</div><div class="line">goTicket <- <span class="number">1</span></div><div class="line">}()</div><div class="line">} <span class="keyword">else</span> {</div><div class="line">origs <- p</div><div class="line">}</div><div class="line">}</div><div class="line">}()</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//goTicket是为了限制该程序启用的goroutine的数量而声明的一个缓冲通道</span></div><div class="line"><span class="comment">//根据传进来的total初始化通道，total即表示可以启用goroutine数量</span></div><div class="line"><span class="comment">//每当启用一个goroutine时从该通道中接受一个值表示可用goroutine少了一个</span></div><div class="line"><span class="comment">//即每个goroutine要想启动必须要有ticket。上述是在origs为缓冲条件下，即整个过程为异步完成情况下</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">initGoTicket</span><span class="params">(total <span class="keyword">int</span>)</span> <span class="title">chan</span> <span class="title">byte</span></span> {</div><div class="line"><span class="keyword">var</span> goTicket <span class="keyword">chan</span> <span class="keyword">byte</span></div><div class="line"><span class="keyword">if</span> total == <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> goTicket</div><div class="line">}</div><div class="line">goTicket = <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">byte</span>, total)</div><div class="line"><span class="keyword">for</span> i := <span class="number">0</span>; i < total; i++ {</div><div class="line">goTicket <- <span class="number">1</span></div><div class="line">}</div><div class="line"><span class="keyword">return</span> goTicket</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">fecthPerson1</span><span class="params">()</span> <span class="params">(Person, <span class="keyword">bool</span>)</span></span> {</div><div class="line"><span class="keyword">if</span> personCount < personTotal {</div><div class="line">p := persons[personCount]</div><div class="line">personCount++</div><div class="line"><span class="keyword">return</span> p, <span class="literal">true</span></div><div class="line">}</div><div class="line"><span class="keyword">return</span> Person{}, <span class="literal">false</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">savePerson1</span><span class="params">(p Person)</span> <span class="title">bool</span></span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">true</span></div><div class="line">}</div><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line"><span class="string">"fmt"</span></div><div class="line"><span class="string">"time"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="comment">//定义员工数据结构</span></div><div class="line"><span class="keyword">type</span> Person <span class="keyword">struct</span> {</div><div class="line">Name <span class="keyword">string</span></div><div class="line">Age <span class="keyword">uint8</span></div><div class="line">Address Addr</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//定义地址数据结构</span></div><div class="line"><span class="keyword">type</span> Addr <span class="keyword">struct</span> {</div><div class="line">city <span class="keyword">string</span></div><div class="line">district <span class="keyword">string</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//定义处理接口，方法Batch被声明为实现批量处理人员信息功能的方法，</span></div><div class="line"><span class="comment">//其方法声明中的两个通道分别对该方法和该方法的调用方使用它的方式进行了约束</span></div><div class="line"><span class="keyword">type</span> PersonHandler <span class="keyword">interface</span> {</div><div class="line">Batch(origs <-<span class="keyword">chan</span> Person) <-<span class="keyword">chan</span> Person</div><div class="line">Handle(orig *Person)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//定义空结构体，为其添加方法，实现PersonHandler接口</span></div><div class="line"><span class="keyword">type</span> PersonHandlerImpl <span class="keyword">struct</span>{}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(handler PersonHandlerImpl)</span> <span class="title">Batch</span><span class="params">(origs <-<span class="keyword">chan</span> Person)</span> <-<span class="title">chan</span> <span class="title">Person</span></span> {</div><div class="line">dests := <span class="built_in">make</span>(<span class="keyword">chan</span> Person, <span class="number">100</span>)</div><div class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</div><div class="line"><span class="keyword">for</span> p := <span class="keyword">range</span> origs {</div><div class="line">handler.Handle(&p)</div><div class="line">dests <- p</div><div class="line">}</div><div class="line">fmt.Println(<span class="string">"All the information has been handled."</span>)</div><div class="line"><span class="comment">//在发送方关闭通道</span></div><div class="line"><span class="built_in">close</span>(dests)</div><div class="line">}()</div><div class="line"><span class="keyword">return</span> dests</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(handler PersonHandlerImpl)</span> <span class="title">Handle</span><span class="params">(orig *Person)</span></span> {</div><div class="line"><span class="keyword">if</span> orig.Address.district == <span class="string">"Haidian"</span> {</div><div class="line">orig.Address.district = <span class="string">"Shijingshan"</span></div><div class="line">}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//定义要被处理的数据并初始化</span></div><div class="line"><span class="keyword">var</span> personTotal = <span class="number">200</span></div><div class="line"></div><div class="line"><span class="keyword">var</span> persons []Person = <span class="built_in">make</span>([]Person, personTotal)</div><div class="line"></div><div class="line"><span class="keyword">var</span> personCount <span class="keyword">int</span></div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">init</span><span class="params">()</span></span> {</div><div class="line"><span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">200</span>; i++ {</div><div class="line">name := fmt.Sprintf(<span class="string">"%s%d"</span>, <span class="string">"P"</span>, i)</div><div class="line">p := Person{name, <span class="number">32</span>, Addr{<span class="string">"Beijing"</span>, <span class="string">"Haidian"</span>}}</div><div class="line">persons[i] = p</div><div class="line">}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//main函数中首先获取handler，初始化origs通道，将人员信息通过origs通道传入</span></div><div class="line"><span class="comment">//Batch中处理，处理后的信息放入dests通道中，并将dests通道返回。</span></div><div class="line"><span class="comment">//通道初始化完成后，fecthPerson获取人员信息放入到origs中，savePerson从dests中接收处理过的信息进行保存</span></div><div class="line"><span class="comment">//其中sign通道作用为在批处理完全执行结束之前阻塞主Goroutine</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</div><div class="line">handler := getPersonHandler()</div><div class="line">origs := <span class="built_in">make</span>(<span class="keyword">chan</span> Person, <span class="number">100</span>)</div><div class="line">dests := handler.Batch(origs)</div><div class="line">fecthPerson(origs)</div><div class="line">sign := savePerson(dests)</div><div class="line"><-sign</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">getPersonHandler</span><span class="params">()</span> <span class="title">PersonHandler</span></span> {</div><div class="line"><span class="keyword">return</span> PersonHandlerImpl{}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">savePerson</span><span class="params">(dest <-<span class="keyword">chan</span> Person)</span> <-<span class="title">chan</span> <span class="title">byte</span></span> {</div><div class="line">sign := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">byte</span>, <span class="number">1</span>)</div><div class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</div><div class="line"><span class="keyword">for</span> {</div><div class="line">p, ok := <-dest</div><div class="line"><span class="keyword">if</span> !ok {</div><div class="line">fmt.Println(<span class="string">"All the information has been saved."</span>)</div><div class="line">sign <- <span class="number">0</span></div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line">savePerson1(p)</div><div class="line">}</div><div class="line">}()</div><div class="line"><span class="keyword">return</span> sign</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">fecthPerson</span><span class="params">(origs <span class="keyword">chan</span><- Person)</span></span> {</div><div class="line"><span class="comment">//调用cap函数确定origs是否为缓冲通道</span></div><div class="line">origsCap := <span class="built_in">cap</span>(origs)</div><div class="line">buffered := origsCap > <span class="number">0</span></div><div class="line"><span class="comment">//以origsCap的一半作为Goroutine票池的总数，创建票池</span></div><div class="line">goTicketTotal := origsCap / <span class="number">2</span></div><div class="line">goTicket := initGoTicket(goTicketTotal)</div><div class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</div><div class="line"><span class="keyword">for</span> {</div><div class="line">p, ok := fecthPerson1()</div><div class="line"><span class="keyword">if</span> !ok {</div><div class="line"><span class="keyword">for</span> {</div><div class="line"><span class="comment">//如果为非缓冲通道或者所有goroutine已完成工作，跳出循环</span></div><div class="line"><span class="keyword">if</span> !buffered || <span class="built_in">len</span>(goTicket) == goTicketTotal {</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line">time.Sleep(time.Nanosecond)</div><div class="line">}</div><div class="line">fmt.Println(<span class="string">"All the information has been fetched."</span>)</div><div class="line"><span class="comment">//在发送方关闭通道</span></div><div class="line"><span class="built_in">close</span>(origs)</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//如果为缓冲通道，从goTicket接受一个值，表示有一个goroutine被占用</span></div><div class="line"><span class="comment">//当操作完成后，向其中发送一个值，表示接解除占用</span></div><div class="line"><span class="keyword">if</span> buffered {</div><div class="line"><-goTicket</div><div class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</div><div class="line">origs <- p</div><div class="line">goTicket <- <span class="number">1</span></div><div class="line">}()</div><div class="line">} <span class="keyword">else</span> {</div><div class="line">origs <- p</div><div class="line">}</div><div class="line">}</div><div class="line">}()</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//goTicket是为了限制该程序启用的goroutine的数量而声明的一个缓冲通道</span></div><div class="line"><span class="comment">//根据传进来的total初始化通道，total即表示可以启用goroutine数量</span></div><div class="line"><span class="comment">//每当启用一个goroutine时从该通道中接受一个值表示可用goroutine少了一个</span></div><div class="line"><span class="comment">//即每个goroutine要想启动必须要有ticket。上述是在origs为缓冲条件下，即整个过程为异步完成情况下</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">initGoTicket</span><span class="params">(total <span class="keyword">int</span>)</span> <span class="title">chan</span> <span class="title">byte</span></span> {</div><div class="line"><span class="keyword">var</span> goTicket <span class="keyword">chan</span> <span class="keyword">byte</span></div><div class="line"><span class="keyword">if</span> total == <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> goTicket</div><div class="line">}</div><div class="line">goTicket = <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">byte</span>, total)</div><div class="line"><span class="keyword">for</span> i := <span class="number">0</span>; i < total; i++ {</div><div class="line">goTicket <- <span class="number">1</span></div><div class="line">}</div><div class="line"><span class="keyword">return</span> goTicket</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">fecthPerson1</span><span class="params">()</span> <span class="params">(Person, <span class="keyword">bool</span>)</span></span> {</div><div class="line"><span class="keyword">if</span> personCount < personTotal {</div><div class="line">p := persons[personCount]</div><div class="line">personCount++</div><div class="line"><span class="keyword">return</span> p, <span class="literal">true</span></div><div class="line">}</div><div class="line"><span class="keyword">return</span> Person{}, <span class="literal">false</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">savePerson1</span><span class="params">(p Person)</span> <span class="title">bool</span></span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">true</span></div><div class="line">}</div></pre></td></tr></table></figure>]]></content>
<summary type="html">
<p>该代码为批量处理人员信息，即将人员信息中的居住地进行统一处理。</p>
</summary>
<category term="Golang" scheme="https://yangchenglong11.github.io/categories/Golang/"/>
<category term="Golang" scheme="https://yangchenglong11.github.io/tags/Golang/"/>
</entry>
<entry>
<title>cobra</title>
<link href="https://yangchenglong11.github.io/2017/05/16/cobra/"/>
<id>https://yangchenglong11.github.io/2017/05/16/cobra/</id>
<published>2017-05-16T10:14:34.000Z</published>
<updated>2018-02-12T12:03:04.869Z</updated>
<content type="html"><![CDATA[<p>cobra是一个命令行框架, 它可以生成和解析命令行。 Cobra具有非常干净的界面和简单的设计，而不需要不必要的构造函数或初始化方法。</p><a id="more"></a><p>用Cobra命令构建的应用程序被设计成尽可能方便用户使用。可以在命令之前或之后放置flag。可以使用短的和长的flag。命令甚至不需要完全键入，help是自动生成的，它是应用程序使用help命令或–help flag时调用的特定命令.</p><h3 id="概念"><a href="#概念" class="headerlink" title="概念"></a>概念</h3><p>Cobra是建立在命令、参数结构之上的。</p><p>Commands代表行动，Args是事物，Flags表示对行为的修饰。</p><p>比较好的应用程序使用起来就像句子一样。用户将知道如何使用应用程序因为他们本来就知道如何使用它。</p><p>一般模式就像下面一样：</p><figure class="highlight brainfuck"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="comment">VERB</span> <span class="comment">NOUN</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">ADJECTIVE</span><span class="string">.</span> <span class="comment">or</span> <span class="comment">APPNAME</span> <span class="comment">COMMAND</span> <span class="comment">ARG</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">FLAG</span></div></pre></td></tr></table></figure><p>一些好的现实世界的例子可以更好地说明这一点。</p><p>在下面的例子中，server 是一个命令， port是一个参数</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">hugo server --port=1313</div></pre></td></tr></table></figure><h3 id="Commands"><a href="#Commands" class="headerlink" title="Commands"></a>Commands</h3><p>命令是应用程序的中心点。应用程序支持的每个交互都将包含在命令中。命令可以拥有子命令并且可选地运行操作。</p><p>命令具有以下结构：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Command <span class="keyword">struct</span> {</div><div class="line"> Use <span class="keyword">string</span> <span class="comment">// The one-line usage message.</span></div><div class="line"> Short <span class="keyword">string</span> <span class="comment">// The short description shown in the 'help' output.</span></div><div class="line"> Long <span class="keyword">string</span> <span class="comment">// The long message shown in the 'help <this-command>' output.</span></div><div class="line"> Run <span class="function"><span class="keyword">func</span><span class="params">(cmd *Command, args []<span class="keyword">string</span>)</span> // <span class="title">Run</span> <span class="title">runs</span> <span class="title">the</span> <span class="title">command</span>.</span></div><div class="line"><span class="function">}</span></div></pre></td></tr></table></figure><h3 id="flags"><a href="#flags" class="headerlink" title="flags"></a>flags</h3><p>flags是修改命令行为的一种方式。Cobra完全兼容POSIX命令行模式以及flags。</p><p>flag的功能由pflag库提供，它是flag标准库的一个fork，保持相同的接口的同时兼容POSIX标准。</p><h3 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h3><p>Cobra的工作原理是创建一组命令，然后将它们组织成一棵树。命令树树定义应用程序的结构。</p><p>一旦每个命令都定义了相应的flag，那么树将被分配给最终执行的command。</p><h3 id="Cobra提供的功能"><a href="#Cobra提供的功能" class="headerlink" title="Cobra提供的功能"></a>Cobra提供的功能</h3><ul><li>简易的子命令行模式，如 app server， app fetch等等</li><li>完全兼容posix命令行模式</li><li>嵌套子命令subcommand</li><li>支持全局，局部，串联flags</li><li>使用Cobra很容易的生成应用程序和命令，使用cobra create appname和cobra add cmdname</li><li>如果命令输入错误，将提供智能建议，如 app srver，将提示srver没有，是否是app server</li><li>自动生成commands和flags的帮助信息</li><li>自动生成详细的help信息，如app help</li><li>自动识别-h，–help帮助flag</li><li>自动生成应用程序在bash下命令自动完成功能</li><li>自动生成应用程序的man手册</li><li>命令行别名</li><li>自定义help和usage信息</li><li>可选的紧密集成的<a href="http://www.ctolib.com/http://github.com/spf13/viper" target="_blank" rel="external">viper</a> apps</li></ul><h3 id="如何使用"><a href="#如何使用" class="headerlink" title="如何使用"></a>如何使用</h3><p>下面简单介绍一下如何使用Cobra，基本能够满足一般命令行程序的需求，如果需要更多功能，可以研究一下源码<a href="http://www.ctolib.com/https://github.com/spf13/cobra" target="_blank" rel="external">github</a>。</p><h2 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h2><p>cobra的源工程地址在github.com/spf13/cobra/cobra, 同其他Go开源项目一样,我们通过go get来拉取:</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ <span class="keyword">go</span> get -u github.com/spf13/cobra/cobra</div></pre></td></tr></table></figure><p>安装完成后，打开GOPATH目录，bin目录下应该有已经编译好的cobra或cobra.exe程序，当然你也可以使用源代码自己生成一个最新的cobra程序。</p><p>通过下面语句import到工程:</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">import</span> <span class="string">"github.com/spf13/cobra"</span></div></pre></td></tr></table></figure><h2 id="通过cobra命令生成工程"><a href="#通过cobra命令生成工程" class="headerlink" title="通过cobra命令生成工程"></a>通过cobra命令生成工程</h2><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> cobra init cobra-demo</span></div><div class="line">Your Cobra application is ready at</div><div class="line">/Users/yang/lib/go/src/cobra-demo</div><div class="line">Give it a try by going there and running `go run main.go`</div><div class="line">Add commands to it by running `cobra add [cmdname]`</div></pre></td></tr></table></figure><p>生成后的工程目录结构如下:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> tree cobra-demo</span></div><div class="line">cobra-demo</div><div class="line">├── LICENSE</div><div class="line">├── cmd</div><div class="line">│ └── root.go</div><div class="line">└── main.go</div></pre></td></tr></table></figure><p>cobra生成的工程, 默认文件都放在cmd目录下, 以root.go作为根命令. cobra可以嵌套<a href="https://github.com/ShevYan/GoTutor/blob/master/flag-demo" target="_blank" rel="external">flag</a>, <a href="https://github.com/ShevYan/GoTutor/blob/master/pflag-demo" target="_blank" rel="external">pflag</a>, <a href="https://github.com/ShevYan/GoTutor/blob/master/viper-demo" target="_blank" rel="external">viper</a>一起使用. 其中init()里面调用了initConfig(), 并可以调用pflag:</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">init</span><span class="params">()</span></span> {</div><div class="line">cobra.OnInitialize(initConfig)</div><div class="line"></div><div class="line"><span class="comment">// Here you will define your flags and configuration settings.</span></div><div class="line"><span class="comment">// Cobra supports Persistent Flags, which, if defined here,</span></div><div class="line"><span class="comment">// will be global for your application.</span></div><div class="line"></div><div class="line">RootCmd.PersistentFlags().StringVar(&cfgFile, <span class="string">"config"</span>, <span class="string">""</span>, <span class="string">"config file (default is $HOME/.cobra-demo.yaml)"</span>)</div><div class="line"><span class="comment">// Cobra also supports local flags, which will only run</span></div><div class="line"><span class="comment">// when this action is called directly.</span></div><div class="line">RootCmd.Flags().BoolP(<span class="string">"toggle"</span>, <span class="string">"t"</span>, <span class="literal">false</span>, <span class="string">"Help message for toggle"</span>)</div><div class="line">}</div></pre></td></tr></table></figure><p>flag的操作中, 分为全局的flag和当前子命令的flag. 什么是子命令呢? cobra支持多级子命令, 比如: mycmd sub1 sub2, 其中mycmd是根命令, sub1是mycmd的子命令, sub2是sub1的子命令. 因此, RootCmd.PersistentFlags()是一个全局flag, RootCmd.Flags()是当前命令的flag. 由于是根命令,其实二者是相同的, 但是如果在子命令中, 二者是不同的.</p><p>下面是使用viper的地方:</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// initConfig reads in config file and ENV variables if set.</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">initConfig</span><span class="params">()</span></span> {</div><div class="line"><span class="keyword">if</span> cfgFile != <span class="string">""</span> { <span class="comment">// enable ability to specify config file via flag</span></div><div class="line">viper.SetConfigFile(cfgFile)</div><div class="line">}</div><div class="line"></div><div class="line">viper.SetConfigName(<span class="string">".cobra-demo"</span>) <span class="comment">// name of config file (without extension)</span></div><div class="line">viper.AddConfigPath(<span class="string">"$HOME"</span>) <span class="comment">// adding home directory as first search path</span></div><div class="line">viper.AutomaticEnv() <span class="comment">// read in environment variables that match</span></div><div class="line"></div><div class="line"><span class="comment">// If a config file is found, read it in.</span></div><div class="line"><span class="keyword">if</span> err := viper.ReadInConfig(); err == <span class="literal">nil</span> {</div><div class="line">fmt.Println(<span class="string">"Using config file:"</span>, viper.ConfigFileUsed())</div><div class="line">}</div><div class="line">}</div></pre></td></tr></table></figure><p>cobra对象初始化:</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// This represents the base command when called without any subcommands</span></div><div class="line"><span class="keyword">var</span> RootCmd = &cobra.Command{</div><div class="line">Use: <span class="string">"cobra-demo"</span>,</div><div class="line">Short: <span class="string">"A brief description of your application"</span>,</div><div class="line">Long: <span class="string">`A longer description that spans multiple lines and likely contains</span></div><div class="line"><span class="string">demos and usage of using your application. For demo:</span></div><div class="line"><span class="string"></span></div><div class="line"><span class="string">Cobra is a CLI library for Go that empowers applications.</span></div><div class="line"><span class="string">This application is a tool to generate the needed files</span></div><div class="line"><span class="string">to quickly create a Cobra application.`</span>,</div><div class="line"><span class="comment">// Uncomment the following line if your bare application</span></div><div class="line"><span class="comment">// has an action associated with it:</span></div><div class="line">Run: <span class="function"><span class="keyword">func</span><span class="params">(cmd *cobra.Command, args []<span class="keyword">string</span>)</span></span> { },</div><div class="line">}</div></pre></td></tr></table></figure><p>上面这个对象就是cobra的根命令对象, 默认生成的时候Run字段是被注释的, 打开以后就可以在函数里面写入自己的处理函数. 下面是运行结果:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> go run main.go -h</span></div><div class="line">A longer description that spans multiple lines and likely contains</div><div class="line">demos and usage of using your application. For demo:</div><div class="line"></div><div class="line">Cobra is a CLI library for Go that empowers applications.</div><div class="line">This application is a tool to generate the needed files</div><div class="line">to quickly create a Cobra application.</div><div class="line"></div><div class="line">Usage:</div><div class="line"> cobra-demo [flags]</div><div class="line"></div><div class="line">Flags:</div><div class="line"> --config="": config file (default is $HOME/.cobra-demo.yaml)</div><div class="line"> -t, --toggle[=false]: Help message for toggle</div></pre></td></tr></table></figure><p>执行下列命令, 生成子命令:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">cobra add serve</div><div class="line">cobra add config</div><div class="line">cobra add create -p 'configCmd'</div></pre></td></tr></table></figure><p>之后,工程结构变为:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> tree cobra-demo</span></div><div class="line">cobra-demo</div><div class="line">├── LICENSE</div><div class="line">├── cmd</div><div class="line">│ ├── config.go</div><div class="line">│ ├── create.go</div><div class="line">│ ├── root.go</div><div class="line">│ └── serve.go</div><div class="line">└── main.go</div><div class="line"></div><div class="line">1 directory, 8 files</div></pre></td></tr></table></figure><p>通过上述命令,我们生成了3个子命令, config, serve, create:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> go run main.go -h</span></div><div class="line">A longer description that spans multiple lines and likely contains</div><div class="line">demos and usage of using your application. For demo:</div><div class="line"></div><div class="line">Cobra is a CLI library for Go that empowers applications.</div><div class="line">This application is a tool to generate the needed files</div><div class="line">to quickly create a Cobra application.</div><div class="line"></div><div class="line">Usage:</div><div class="line"> cobra-demo [flags]</div><div class="line"> cobra-demo [command]</div><div class="line"></div><div class="line">Available Commands:</div><div class="line"> config A brief description of your command</div><div class="line"> serve A brief description of your command</div><div class="line"></div><div class="line">Flags:</div><div class="line"> --config="": config file (default is $HOME/.cobra-demo.yaml)</div><div class="line"> -t, --toggle[=false]: Help message for toggle</div></pre></td></tr></table></figure><p>执行config子命令</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">$ <span class="keyword">go</span> run main.<span class="keyword">go</span> config</div><div class="line">config called</div></pre></td></tr></table></figure><p>执行config的create子命令</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">$ <span class="keyword">go</span> run main.<span class="keyword">go</span> config create</div><div class="line">create called</div></pre></td></tr></table></figure><p>我们可以看到代码里面config是作为根命令的子命令:</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">init</span><span class="params">()</span></span> {</div><div class="line">RootCmd.AddCommand(configCmd)</div><div class="line"></div><div class="line"><span class="comment">// Here you will define your flags and configuration settings.</span></div><div class="line"></div><div class="line"><span class="comment">// Cobra supports Persistent Flags which will work for this command</span></div><div class="line"><span class="comment">// and all subcommands, e.g.:</span></div><div class="line">configCmd.PersistentFlags().String(<span class="string">"foo"</span>, <span class="string">""</span>, <span class="string">"A help for foo"</span>)</div><div class="line"></div><div class="line"><span class="comment">// Cobra supports local flags which will only run when this command</span></div><div class="line"><span class="comment">// is called directly, e.g.:</span></div><div class="line">configCmd.Flags().BoolP(<span class="string">"toggle"</span>, <span class="string">"t"</span>, <span class="literal">false</span>, <span class="string">"Help message for toggle"</span>)</div><div class="line"></div><div class="line">}</div></pre></td></tr></table></figure><p>当我们输入错误时，会有相应提示</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line">$ <span class="keyword">go</span> run main.<span class="keyword">go</span> serv </div><div class="line">Error: unknown command <span class="string">"serv"</span> <span class="keyword">for</span> <span class="string">"cobra-demo"</span></div><div class="line"></div><div class="line">Did you mean this?</div><div class="line"> serve</div><div class="line"></div><div class="line">Run <span class="string">'cobra-demo --help'</span> <span class="keyword">for</span> usage.</div><div class="line">unknown command <span class="string">"serv"</span> <span class="keyword">for</span> <span class="string">"cobra-demo"</span></div><div class="line"></div><div class="line">Did you mean this?</div><div class="line"> serve</div><div class="line"></div><div class="line">exit status <span class="number">1</span></div></pre></td></tr></table></figure>]]></content>
<summary type="html">
<p>cobra是一个命令行框架, 它可以生成和解析命令行。 Cobra具有非常干净的界面和简单的设计，而不需要不必要的构造函数或初始化方法。</p>
</summary>
<category term="Golang" scheme="https://yangchenglong11.github.io/categories/Golang/"/>
<category term="Golang" scheme="https://yangchenglong11.github.io/tags/Golang/"/>
</entry>
<entry>
<title>bufio 源码解读</title>
<link href="https://yangchenglong11.github.io/2017/05/04/bufio/"/>
<id>https://yangchenglong11.github.io/2017/05/04/bufio/</id>
<published>2017-05-04T03:24:34.000Z</published>
<updated>2017-10-28T13:59:55.712Z</updated>
<content type="html"><![CDATA[<p>bufio包实现了有缓冲的I/O。它包装一个io.Reader或io.Writer接口对象，创建另一个也实现了该接口，且同时还提供了缓冲和一些文本I/O的帮助函数的对象。<br><a id="more"></a></p><p>Reader 部分：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div><div class="line">114</div><div class="line">115</div><div class="line">116</div><div class="line">117</div><div class="line">118</div><div class="line">119</div><div class="line">120</div><div class="line">121</div><div class="line">122</div><div class="line">123</div><div class="line">124</div><div class="line">125</div><div class="line">126</div><div class="line">127</div><div class="line">128</div><div class="line">129</div><div class="line">130</div><div class="line">131</div><div class="line">132</div><div class="line">133</div><div class="line">134</div><div class="line">135</div><div class="line">136</div><div class="line">137</div><div class="line">138</div><div class="line">139</div><div class="line">140</div><div class="line">141</div><div class="line">142</div><div class="line">143</div><div class="line">144</div><div class="line">145</div><div class="line">146</div><div class="line">147</div><div class="line">148</div><div class="line">149</div><div class="line">150</div><div class="line">151</div><div class="line">152</div><div class="line">153</div><div class="line">154</div><div class="line">155</div><div class="line">156</div><div class="line">157</div><div class="line">158</div><div class="line">159</div><div class="line">160</div><div class="line">161</div><div class="line">162</div><div class="line">163</div><div class="line">164</div><div class="line">165</div><div class="line">166</div><div class="line">167</div><div class="line">168</div><div class="line">169</div><div class="line">170</div><div class="line">171</div><div class="line">172</div><div class="line">173</div><div class="line">174</div><div class="line">175</div><div class="line">176</div><div class="line">177</div><div class="line">178</div><div class="line">179</div><div class="line">180</div><div class="line">181</div><div class="line">182</div><div class="line">183</div><div class="line">184</div><div class="line">185</div><div class="line">186</div><div class="line">187</div><div class="line">188</div><div class="line">189</div><div class="line">190</div><div class="line">191</div><div class="line">192</div><div class="line">193</div><div class="line">194</div><div class="line">195</div><div class="line">196</div><div class="line">197</div><div class="line">198</div><div class="line">199</div><div class="line">200</div><div class="line">201</div><div class="line">202</div><div class="line">203</div><div class="line">204</div><div class="line">205</div><div class="line">206</div><div class="line">207</div><div class="line">208</div><div class="line">209</div><div class="line">210</div><div class="line">211</div><div class="line">212</div><div class="line">213</div><div class="line">214</div><div class="line">215</div><div class="line">216</div><div class="line">217</div><div class="line">218</div><div class="line">219</div><div class="line">220</div><div class="line">221</div><div class="line">222</div><div class="line">223</div><div class="line">224</div><div class="line">225</div><div class="line">226</div><div class="line">227</div><div class="line">228</div><div class="line">229</div><div class="line">230</div><div class="line">231</div><div class="line">232</div><div class="line">233</div><div class="line">234</div><div class="line">235</div><div class="line">236</div><div class="line">237</div><div class="line">238</div><div class="line">239</div><div class="line">240</div><div class="line">241</div><div class="line">242</div><div class="line">243</div><div class="line">244</div><div class="line">245</div><div class="line">246</div><div class="line">247</div><div class="line">248</div><div class="line">249</div><div class="line">250</div><div class="line">251</div><div class="line">252</div><div class="line">253</div><div class="line">254</div><div class="line">255</div><div class="line">256</div><div class="line">257</div><div class="line">258</div><div class="line">259</div><div class="line">260</div><div class="line">261</div><div class="line">262</div><div class="line">263</div><div class="line">264</div><div class="line">265</div><div class="line">266</div><div class="line">267</div><div class="line">268</div><div class="line">269</div><div class="line">270</div><div class="line">271</div><div class="line">272</div><div class="line">273</div><div class="line">274</div><div class="line">275</div><div class="line">276</div><div class="line">277</div><div class="line">278</div><div class="line">279</div><div class="line">280</div><div class="line">281</div><div class="line">282</div><div class="line">283</div><div class="line">284</div><div class="line">285</div><div class="line">286</div><div class="line">287</div><div class="line">288</div><div class="line">289</div><div class="line">290</div><div class="line">291</div><div class="line">292</div><div class="line">293</div><div class="line">294</div><div class="line">295</div><div class="line">296</div><div class="line">297</div><div class="line">298</div><div class="line">299</div><div class="line">300</div><div class="line">301</div><div class="line">302</div><div class="line">303</div><div class="line">304</div><div class="line">305</div><div class="line">306</div><div class="line">307</div><div class="line">308</div><div class="line">309</div><div class="line">310</div><div class="line">311</div><div class="line">312</div><div class="line">313</div><div class="line">314</div><div class="line">315</div><div class="line">316</div><div class="line">317</div><div class="line">318</div><div class="line">319</div><div class="line">320</div><div class="line">321</div><div class="line">322</div><div class="line">323</div><div class="line">324</div><div class="line">325</div><div class="line">326</div><div class="line">327</div><div class="line">328</div><div class="line">329</div><div class="line">330</div><div class="line">331</div><div class="line">332</div><div class="line">333</div><div class="line">334</div><div class="line">335</div><div class="line">336</div><div class="line">337</div><div class="line">338</div><div class="line">339</div><div class="line">340</div><div class="line">341</div><div class="line">342</div><div class="line">343</div><div class="line">344</div><div class="line">345</div><div class="line">346</div><div class="line">347</div><div class="line">348</div><div class="line">349</div><div class="line">350</div><div class="line">351</div><div class="line">352</div><div class="line">353</div><div class="line">354</div><div class="line">355</div><div class="line">356</div><div class="line">357</div><div class="line">358</div><div class="line">359</div><div class="line">360</div><div class="line">361</div><div class="line">362</div><div class="line">363</div><div class="line">364</div><div class="line">365</div><div class="line">366</div><div class="line">367</div><div class="line">368</div><div class="line">369</div><div class="line">370</div><div class="line">371</div><div class="line">372</div><div class="line">373</div><div class="line">374</div><div class="line">375</div><div class="line">376</div><div class="line">377</div><div class="line">378</div><div class="line">379</div><div class="line">380</div><div class="line">381</div><div class="line">382</div><div class="line">383</div><div class="line">384</div><div class="line">385</div><div class="line">386</div><div class="line">387</div><div class="line">388</div><div class="line">389</div><div class="line">390</div><div class="line">391</div><div class="line">392</div><div class="line">393</div><div class="line">394</div><div class="line">395</div><div class="line">396</div><div class="line">397</div><div class="line">398</div><div class="line">399</div><div class="line">400</div><div class="line">401</div><div class="line">402</div><div class="line">403</div><div class="line">404</div><div class="line">405</div><div class="line">406</div><div class="line">407</div><div class="line">408</div><div class="line">409</div><div class="line">410</div><div class="line">411</div><div class="line">412</div><div class="line">413</div><div class="line">414</div><div class="line">415</div><div class="line">416</div><div class="line">417</div><div class="line">418</div><div class="line">419</div><div class="line">420</div><div class="line">421</div><div class="line">422</div><div class="line">423</div><div class="line">424</div><div class="line">425</div><div class="line">426</div><div class="line">427</div><div class="line">428</div><div class="line">429</div><div class="line">430</div><div class="line">431</div><div class="line">432</div><div class="line">433</div><div class="line">434</div><div class="line">435</div><div class="line">436</div><div class="line">437</div><div class="line">438</div><div class="line">439</div><div class="line">440</div><div class="line">441</div><div class="line">442</div><div class="line">443</div><div class="line">444</div><div class="line">445</div><div class="line">446</div><div class="line">447</div><div class="line">448</div><div class="line">449</div><div class="line">450</div><div class="line">451</div><div class="line">452</div><div class="line">453</div><div class="line">454</div><div class="line">455</div><div class="line">456</div><div class="line">457</div><div class="line">458</div><div class="line">459</div><div class="line">460</div><div class="line">461</div><div class="line">462</div><div class="line">463</div><div class="line">464</div><div class="line">465</div><div class="line">466</div><div class="line">467</div><div class="line">468</div><div class="line">469</div><div class="line">470</div><div class="line">471</div><div class="line">472</div><div class="line">473</div><div class="line">474</div><div class="line">475</div><div class="line">476</div><div class="line">477</div><div class="line">478</div><div class="line">479</div><div class="line">480</div><div class="line">481</div><div class="line">482</div><div class="line">483</div><div class="line">484</div><div class="line">485</div><div class="line">486</div><div class="line">487</div><div class="line">488</div><div class="line">489</div><div class="line">490</div><div class="line">491</div><div class="line">492</div><div class="line">493</div><div class="line">494</div><div class="line">495</div><div class="line">496</div><div class="line">497</div><div class="line">498</div><div class="line">499</div><div class="line">500</div><div class="line">501</div><div class="line">502</div><div class="line">503</div><div class="line">504</div><div class="line">505</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// 设置默认缓存大小</span></div><div class="line"><span class="keyword">const</span> (</div><div class="line">defaultBufSize = <span class="number">4096</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="comment">// 定义错误类型</span></div><div class="line"><span class="keyword">var</span> (</div><div class="line">ErrInvalidUnreadByte = errors.New(<span class="string">"bufio: invalid use of UnreadByte"</span>)</div><div class="line">ErrInvalidUnreadRune = errors.New(<span class="string">"bufio: invalid use of UnreadRune"</span>)</div><div class="line">ErrBufferFull = errors.New(<span class="string">"bufio: buffer full"</span>)</div><div class="line">ErrNegativeCount = errors.New(<span class="string">"bufio: negative count"</span>)</div><div class="line">)</div><div class="line"></div><div class="line"><span class="comment">// bufio.Reader 结构包装了一个 io.Reader 对象，提供缓存功能，同时实现了 io.Reader 接口。</span></div><div class="line"><span class="comment">// 其结构没有任何导出的字段。</span></div><div class="line"><span class="keyword">type</span> Reader <span class="keyword">struct</span> {</div><div class="line">buf []<span class="keyword">byte</span> <span class="comment">// 缓存</span></div><div class="line">rd io.Reader <span class="comment">// 底层的 io.Reader</span></div><div class="line">r, w <span class="keyword">int</span> <span class="comment">// r:从buf中读走的字节（偏移）；w:buf 中填充内容的偏移；</span></div><div class="line">err error <span class="comment">// 读过程中遇到的错误</span></div><div class="line">lastByte <span class="keyword">int</span> <span class="comment">// 最后一次读到的字节</span></div><div class="line">lastRuneSize <span class="keyword">int</span> <span class="comment">// 最后一次读到的 Rune 的大小</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// 设置连续读取最大尝试次数</span></div><div class="line"><span class="keyword">const</span> maxConsecutiveEmptyReads = <span class="number">100</span></div><div class="line"></div><div class="line"><span class="comment">// NewReaderSize 将 rd 封装成一个带缓存的 bufio.Reader 对象，</span></div><div class="line"><span class="comment">// 缓存大小由 size 指定（如果小于 16 则会被设置为 16）。</span></div><div class="line"><span class="comment">// 如果 rd 的基类型就是有足够缓存的 bufio.Reader 类型，则直接将</span></div><div class="line"><span class="comment">// rd 转换为基类型返回。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">NewReaderSize</span><span class="params">(rd io.Reader, size <span class="keyword">int</span>)</span> *<span class="title">Reader</span></span> {</div><div class="line"><span class="comment">// 已经是bufio.Reader类型，且缓存大小不小于 size，则直接返回</span></div><div class="line">b, ok := rd.(*Reader)</div><div class="line"> <span class="comment">// 判断缓存是否足够</span></div><div class="line"><span class="keyword">if</span> ok && <span class="built_in">len</span>(b.buf) >= size {</div><div class="line"><span class="keyword">return</span> b</div><div class="line">}</div><div class="line"> <span class="comment">// 缓存大小不会小于 minReadBufferSize （16字节）</span></div><div class="line"><span class="keyword">if</span> size < minReadBufferSize {</div><div class="line">size = minReadBufferSize</div><div class="line">}</div><div class="line"> <span class="comment">// 构造一个bufio.Reader实例</span></div><div class="line">r := <span class="built_in">new</span>(Reader)</div><div class="line">r.reset(<span class="built_in">make</span>([]<span class="keyword">byte</span>, size), rd)</div><div class="line"><span class="keyword">return</span> r</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// 创建默认大小缓存的 bufio.Reader</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">NewReader</span><span class="params">(rd io.Reader)</span> *<span class="title">Reader</span></span> {</div><div class="line"><span class="keyword">return</span> NewReaderSize(rd, defaultBufSize)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// 提供reset的接口</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">Reset</span><span class="params">(r io.Reader)</span></span> {</div><div class="line">b.reset(b.buf, r)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Reset丢弃缓冲中的数据，清除任何错误，将b重设为其下层从r读取数据。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">reset</span><span class="params">(buf []<span class="keyword">byte</span>, r io.Reader)</span></span> {</div><div class="line">*b = Reader{</div><div class="line">buf: buf,</div><div class="line">rd: r,</div><div class="line">lastByte: <span class="number">-1</span>,</div><div class="line">lastRuneSize: <span class="number">-1</span>,</div><div class="line">}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// 定义错误：读出字节数为负</span></div><div class="line"><span class="keyword">var</span> errNegativeRead = errors.New(<span class="string">"bufio: reader returned negative count from Read"</span>)</div><div class="line"></div><div class="line"><span class="comment">// 将新块读入缓冲区</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">fill</span><span class="params">()</span></span> {</div><div class="line"><span class="comment">// 将 r 之前的数据覆盖，即，将现有数据滑动到开始。</span></div><div class="line"><span class="keyword">if</span> b.r > <span class="number">0</span> {</div><div class="line"><span class="built_in">copy</span>(b.buf, b.buf[b.r:b.w])</div><div class="line">b.w -= b.r</div><div class="line">b.r = <span class="number">0</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">if</span> b.w >= <span class="built_in">len</span>(b.buf) {</div><div class="line"><span class="built_in">panic</span>(<span class="string">"bufio: tried to fill full buffer"</span>)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// 将新数据读到 buf 中，如果说读到数据则返回；如果读到数据为空，继续尝试有限次数。</span></div><div class="line"><span class="keyword">for</span> i := maxConsecutiveEmptyReads; i > <span class="number">0</span>; i-- {</div><div class="line">n, err := b.rd.Read(b.buf[b.w:])</div><div class="line"><span class="keyword">if</span> n < <span class="number">0</span> {</div><div class="line"><span class="built_in">panic</span>(errNegativeRead)</div><div class="line">}</div><div class="line">b.w += n</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line">b.err = err</div><div class="line"><span class="keyword">return</span></div><div class="line">}</div><div class="line"><span class="keyword">if</span> n > <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span></div><div class="line">}</div><div class="line">}</div><div class="line">b.err = io.ErrNoProgress</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// 返回读取过程中的错误，调用后清空</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">readErr</span><span class="params">()</span> <span class="title">error</span></span> {</div><div class="line">err := b.err</div><div class="line">b.err = <span class="literal">nil</span></div><div class="line"><span class="keyword">return</span> err</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Peek返回输入流的下n个字节，而不会移动读取位置。</span></div><div class="line"><span class="comment">// 返回的[]byte只在下一次调用读取操作前合法。</span></div><div class="line"><span class="comment">// 如果Peek返回的切片长度比n小，它也会返会一个错误说明原因。</span></div><div class="line"><span class="comment">// 如果n比缓冲尺寸还大，返回的错误将是ErrBufferFull。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">Peek</span><span class="params">(n <span class="keyword">int</span>)</span> <span class="params">([]<span class="keyword">byte</span>, error)</span></span> {</div><div class="line"> <span class="comment">// 检查参数</span></div><div class="line"><span class="keyword">if</span> n < <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span>, ErrNegativeCount</div><div class="line">}</div><div class="line"></div><div class="line"> <span class="comment">// 当 buf 中数据量小于 n，且 buf 未满时，向其中填充数据，直至填满；如果有错误则停止。</span></div><div class="line"><span class="keyword">for</span> b.w-b.r < n && b.w-b.r < <span class="built_in">len</span>(b.buf) && b.err == <span class="literal">nil</span> {</div><div class="line">b.fill() </div><div class="line">}</div><div class="line"></div><div class="line"> <span class="comment">// 最多读 len(buf) 长度</span></div><div class="line"><span class="keyword">if</span> n > <span class="built_in">len</span>(b.buf) {</div><div class="line"><span class="keyword">return</span> b.buf[b.r:b.w], ErrBufferFull</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// 0 <= n <= len(b.buf)</span></div><div class="line"><span class="keyword">var</span> err error</div><div class="line"><span class="keyword">if</span> avail := b.w - b.r; avail < n {</div><div class="line"><span class="comment">// 缓存中没有足够数据，设置错误</span></div><div class="line">n = avail</div><div class="line">err = b.readErr()</div><div class="line"><span class="keyword">if</span> err == <span class="literal">nil</span> {</div><div class="line">err = ErrBufferFull</div><div class="line">}</div><div class="line">}</div><div class="line"><span class="keyword">return</span> b.buf[b.r : b.r+n], err</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Discard 跳过后续的 n 个字节的数据，返回跳过的字节数。</span></div><div class="line"><span class="comment">// 如果结果小于 n，将返回错误信息。</span></div><div class="line"><span class="comment">// 如果 n 小于缓存中的数据长度，则不会从底层提取数据。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">Discard</span><span class="params">(n <span class="keyword">int</span>)</span> <span class="params">(discarded <span class="keyword">int</span>, err error)</span></span> {</div><div class="line"> <span class="comment">// 检查参数</span></div><div class="line"><span class="keyword">if</span> n < <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> <span class="number">0</span>, ErrNegativeCount</div><div class="line">}</div><div class="line"><span class="keyword">if</span> n == <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span></div><div class="line">}</div><div class="line">remain := n</div><div class="line"><span class="keyword">for</span> {</div><div class="line"> <span class="comment">// 获取缓存中的数据长度，如果没有数据，添加数据后再次获取长度。</span></div><div class="line">skip := b.Buffered()</div><div class="line"><span class="keyword">if</span> skip == <span class="number">0</span> {</div><div class="line">b.fill()</div><div class="line">skip = b.Buffered()</div><div class="line">}</div><div class="line"> <span class="comment">// 如果缓冲长度大于n，设置跳过长度为n。</span></div><div class="line"><span class="keyword">if</span> skip > remain {</div><div class="line">skip = remain</div><div class="line">}</div><div class="line">b.r += skip</div><div class="line">remain -= skip</div><div class="line"><span class="keyword">if</span> remain == <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> n, <span class="literal">nil</span></div><div class="line">}</div><div class="line"> <span class="comment">// 如果缓冲长度小于n，返回实际跳过长度及错误。</span></div><div class="line"><span class="keyword">if</span> b.err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> n - remain, b.readErr()</div><div class="line">}</div><div class="line">}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Read 从 b 中读出数据到 p 中，返回读出的字节数和遇到的错误。</span></div><div class="line"><span class="comment">// 如果缓存不为空，则只能读出缓存中的数据，不会从底层 io.Reader</span></div><div class="line"><span class="comment">// 中提取数据，如果缓存为空，则：</span></div><div class="line"><span class="comment">// 1、len(p) >= 缓存大小，则跳过缓存，直接从底层 io.Reader 中读</span></div><div class="line"><span class="comment">// 出到 p 中。</span></div><div class="line"><span class="comment">// 2、len(p) < 缓存大小，则先将数据从底层 io.Reader 中读取到缓存</span></div><div class="line"><span class="comment">// 中，再从缓存读取到 p 中。</span></div><div class="line"></div><div class="line"><span class="comment">// 读取到达结尾时，返回值n将为0而 err 将为 io.EOF。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">Read</span><span class="params">(p []<span class="keyword">byte</span>)</span> <span class="params">(n <span class="keyword">int</span>, err error)</span></span> {</div><div class="line">n = <span class="built_in">len</span>(p)</div><div class="line"><span class="keyword">if</span> n == <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> <span class="number">0</span>, b.readErr()</div><div class="line">}</div><div class="line"><span class="keyword">if</span> b.r == b.w {</div><div class="line"> <span class="comment">// 如果缓冲为 0，且错误不为空，直接返回 0 及错误。</span></div><div class="line"><span class="keyword">if</span> b.err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="number">0</span>, b.readErr()</div><div class="line">}</div><div class="line"><span class="keyword">if</span> <span class="built_in">len</span>(p) >= <span class="built_in">len</span>(b.buf) {</div><div class="line"><span class="comment">//跳过缓存，直接从底层 io.Reader 中读取, 避免 copy。</span></div><div class="line">n, b.err = b.rd.Read(p)</div><div class="line"><span class="keyword">if</span> n < <span class="number">0</span> {</div><div class="line"><span class="built_in">panic</span>(errNegativeRead)</div><div class="line">}</div><div class="line"><span class="keyword">if</span> n > <span class="number">0</span> {</div><div class="line">b.lastByte = <span class="keyword">int</span>(p[n<span class="number">-1</span>])</div><div class="line">b.lastRuneSize = <span class="number">-1</span></div><div class="line">}</div><div class="line"><span class="keyword">return</span> n, b.readErr()</div><div class="line">}</div><div class="line"><span class="comment">// 先将数据读到 buf 中，再读到 p 中。</span></div><div class="line"><span class="comment">// 不要使用 fill ，那将会循环读取。</span></div><div class="line">b.r = <span class="number">0</span></div><div class="line">b.w = <span class="number">0</span></div><div class="line">n, b.err = b.rd.Read(b.buf)</div><div class="line"><span class="keyword">if</span> n < <span class="number">0</span> {</div><div class="line"><span class="built_in">panic</span>(errNegativeRead)</div><div class="line">}</div><div class="line"><span class="keyword">if</span> n == <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> <span class="number">0</span>, b.readErr()</div><div class="line">}</div><div class="line">b.w += n</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// 如果缓存不为空，则将缓存中数据复制到 p 中。</span></div><div class="line">n = <span class="built_in">copy</span>(p, b.buf[b.r:b.w])</div><div class="line">b.r += n</div><div class="line">b.lastByte = <span class="keyword">int</span>(b.buf[b.r<span class="number">-1</span>])</div><div class="line">b.lastRuneSize = <span class="number">-1</span></div><div class="line"><span class="keyword">return</span> n, <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// ReadByte读取一个字节并返回该字节。如果没有可用的数据，会返回错误。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">ReadByte</span><span class="params">()</span> <span class="params">(<span class="keyword">byte</span>, error)</span></span> {</div><div class="line">b.lastRuneSize = <span class="number">-1</span></div><div class="line"><span class="keyword">for</span> b.r == b.w {</div><div class="line"><span class="keyword">if</span> b.err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="number">0</span>, b.readErr()</div><div class="line">}</div><div class="line">b.fill() <span class="comment">// buffer is empty</span></div><div class="line">}</div><div class="line">c := b.buf[b.r]</div><div class="line">b.r++</div><div class="line">b.lastByte = <span class="keyword">int</span>(c)</div><div class="line"><span class="keyword">return</span> c, <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// UnreadByte吐出最近一次读取操作读取的最后一个字节。（只能吐出最后一个，多次调用会出问题）</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">UnreadByte</span><span class="params">()</span> <span class="title">error</span></span> {</div><div class="line"><span class="keyword">if</span> b.lastByte < <span class="number">0</span> || b.r == <span class="number">0</span> && b.w > <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> ErrInvalidUnreadByte</div><div class="line">}</div><div class="line"><span class="comment">// b.r > 0 || b.w == 0</span></div><div class="line"><span class="keyword">if</span> b.r > <span class="number">0</span> {</div><div class="line">b.r--</div><div class="line">} <span class="keyword">else</span> {</div><div class="line"><span class="comment">// b.r == 0 && b.w == 0</span></div><div class="line">b.w = <span class="number">1</span></div><div class="line">}</div><div class="line">b.buf[b.r] = <span class="keyword">byte</span>(b.lastByte)</div><div class="line">b.lastByte = <span class="number">-1</span></div><div class="line">b.lastRuneSize = <span class="number">-1</span></div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// ReadRune 读取一个 utf-8 编码的 unicode 码值，返回该码值、其编码长度以及可能的错误。</span></div><div class="line"><span class="comment">// 如果 utf-8 编码非法，读取位置只移动1字节，返回 U+FFFD，返回值 size 为1而 err 为 nil。</span></div><div class="line"><span class="comment">// 如果没有可用的数据，会返回错误。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">ReadRune</span><span class="params">()</span> <span class="params">(r <span class="keyword">rune</span>, size <span class="keyword">int</span>, err error)</span></span> {</div><div class="line"><span class="keyword">for</span> b.r+utf8.UTFMax > b.w && !utf8.FullRune(b.buf[b.r:b.w]) && b.err == <span class="literal">nil</span> && b.w-b.r < <span class="built_in">len</span>(b.buf) {</div><div class="line">b.fill() <span class="comment">// b.w-b.r < len(buf) => buffer is not full</span></div><div class="line">}</div><div class="line">b.lastRuneSize = <span class="number">-1</span></div><div class="line"><span class="keyword">if</span> b.r == b.w {</div><div class="line"><span class="keyword">return</span> <span class="number">0</span>, <span class="number">0</span>, b.readErr()</div><div class="line">}</div><div class="line">r, size = <span class="keyword">rune</span>(b.buf[b.r]), <span class="number">1</span></div><div class="line"><span class="keyword">if</span> r >= utf8.RuneSelf {</div><div class="line">r, size = utf8.DecodeRune(b.buf[b.r:b.w])</div><div class="line">}</div><div class="line">b.r += size</div><div class="line">b.lastByte = <span class="keyword">int</span>(b.buf[b.r<span class="number">-1</span>])</div><div class="line">b.lastRuneSize = size</div><div class="line"><span class="keyword">return</span> r, size, <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// UnreadRune 吐出最近一次 ReadRune 调用读取的 unicode 码值。</span></div><div class="line"><span class="comment">// 如果最近一次读取不是调用的ReadRune，会返回错误。（从这点看，UnreadRune比UnreadByte严格很多）</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">UnreadRune</span><span class="params">()</span> <span class="title">error</span></span> {</div><div class="line"><span class="keyword">if</span> b.lastRuneSize < <span class="number">0</span> || b.r < b.lastRuneSize {</div><div class="line"><span class="keyword">return</span> ErrInvalidUnreadRune</div><div class="line">}</div><div class="line">b.r -= b.lastRuneSize</div><div class="line">b.lastByte = <span class="number">-1</span></div><div class="line">b.lastRuneSize = <span class="number">-1</span></div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Buffered 返回缓存中未读取的数据的长度。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">Buffered</span><span class="params">()</span> <span class="title">int</span></span> { <span class="keyword">return</span> b.w - b.r }</div><div class="line"></div><div class="line"><span class="comment">// ReadSlice 在 b 中查找 delim 并返回 delim 及其之前的所有数据。</span></div><div class="line"><span class="comment">// 该操作会读出数据，返回的切片是已读出的数据的引用，切片中的数据</span></div><div class="line"><span class="comment">// 在下一次读取操作之前是有效的。</span></div><div class="line"><span class="comment">//</span></div><div class="line"><span class="comment">// 如果找到 delim，则返回查找结果，err 返回 nil。</span></div><div class="line"><span class="comment">// 如果未找到 delim，则：</span></div><div class="line"><span class="comment">// 1、缓存不满，则将缓存填满后再次查找。</span></div><div class="line"><span class="comment">// 2、缓存是满的，则返回整个缓存，err 返回 ErrBufferFull。</span></div><div class="line"><span class="comment">//</span></div><div class="line"><span class="comment">// 如果未找到 delim 且遇到错误（通常是 io.EOF），则返回缓存中的所</span></div><div class="line"><span class="comment">// 有数据和遇到的错误。</span></div><div class="line"><span class="comment">//</span></div><div class="line"><span class="comment">// 因为返回的数据有可能被下一次的读写操作修改，所以大多数操作应该</span></div><div class="line"><span class="comment">// 使用 ReadBytes 或 ReadString，它们返回的是数据的拷贝。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">ReadSlice</span><span class="params">(delim <span class="keyword">byte</span>)</span> <span class="params">(line []<span class="keyword">byte</span>, err error)</span></span> {</div><div class="line"><span class="keyword">for</span> {</div><div class="line"><span class="comment">// 在 buf 中查找 delim</span></div><div class="line"><span class="keyword">if</span> i := bytes.IndexByte(b.buf[b.r:b.w], delim); i >= <span class="number">0</span> {</div><div class="line">line = b.buf[b.r : b.r+i+<span class="number">1</span>]</div><div class="line">b.r += i + <span class="number">1</span></div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Pending error?</span></div><div class="line"><span class="keyword">if</span> b.err != <span class="literal">nil</span> {</div><div class="line">line = b.buf[b.r:b.w]</div><div class="line">b.r = b.w</div><div class="line">err = b.readErr()</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Buffer full?</span></div><div class="line"><span class="keyword">if</span> b.Buffered() >= <span class="built_in">len</span>(b.buf) {</div><div class="line">b.r = b.w</div><div class="line">line = b.buf</div><div class="line">err = ErrBufferFull</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line"></div><div class="line">b.fill() <span class="comment">// buffer is not full</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Handle last byte, if any.</span></div><div class="line"><span class="keyword">if</span> i := <span class="built_in">len</span>(line) - <span class="number">1</span>; i >= <span class="number">0</span> {</div><div class="line">b.lastByte = <span class="keyword">int</span>(line[i])</div><div class="line">b.lastRuneSize = <span class="number">-1</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">return</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// ReadLine 是一个低水平的行读取原语，大多数情况下，应该使用</span></div><div class="line"><span class="comment">// ReadBytes('\n') 或 ReadString('\n')，或者使用一个 Scanner。</span></div><div class="line"><span class="comment">//</span></div><div class="line"><span class="comment">// ReadLine 通过调用 ReadSlice 方法实现，返回的也是缓存的切片。用于</span></div><div class="line"><span class="comment">// 读取一行数据，不包括行尾标记（\n 或 \r\n）。</span></div><div class="line"><span class="comment">//</span></div><div class="line"><span class="comment">// 只要能读出数据，err 就为 nil。如果没有数据可读，则 isPrefix 返回</span></div><div class="line"><span class="comment">// false，err 返回 io.EOF。</span></div><div class="line"><span class="comment">//</span></div><div class="line"><span class="comment">// 如果找到行尾标记，则返回查找结果，isPrefix 返回 false。</span></div><div class="line"><span class="comment">// 如果未找到行尾标记，则：</span></div><div class="line"><span class="comment">// 1、缓存不满，则将缓存填满后再次查找。</span></div><div class="line"><span class="comment">// 2、缓存是满的，则返回整个缓存，isPrefix 返回 true。</span></div><div class="line"><span class="comment">//</span></div><div class="line"><span class="comment">// 整个数据尾部“有一个换行标记”和“没有换行标记”的读取结果是一样。</span></div><div class="line"><span class="comment">//</span></div><div class="line"><span class="comment">// 如果 ReadLine 读取到换行标记，则调用 UnreadByte 撤销的是换行标记，</span></div><div class="line"><span class="comment">// 而不是返回的数据。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">ReadLine</span><span class="params">()</span> <span class="params">(line []<span class="keyword">byte</span>, isPrefix <span class="keyword">bool</span>, err error)</span></span> {</div><div class="line">line, err = b.ReadSlice(<span class="string">'\n'</span>)</div><div class="line"><span class="keyword">if</span> err == ErrBufferFull {</div><div class="line"><span class="comment">// Handle the case where "\r\n" straddles the buffer.</span></div><div class="line"><span class="keyword">if</span> <span class="built_in">len</span>(line) > <span class="number">0</span> && line[<span class="built_in">len</span>(line)<span class="number">-1</span>] == <span class="string">'\r'</span> {</div><div class="line"><span class="comment">// Put the '\r' back on buf and drop it from line.</span></div><div class="line"><span class="comment">// Let the next call to ReadLine check for "\r\n".</span></div><div class="line"><span class="keyword">if</span> b.r == <span class="number">0</span> {</div><div class="line"><span class="comment">// should be unreachable</span></div><div class="line"><span class="built_in">panic</span>(<span class="string">"bufio: tried to rewind past start of buffer"</span>)</div><div class="line">}</div><div class="line">b.r--</div><div class="line">line = line[:<span class="built_in">len</span>(line)<span class="number">-1</span>]</div><div class="line">}</div><div class="line"><span class="keyword">return</span> line, <span class="literal">true</span>, <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">if</span> <span class="built_in">len</span>(line) == <span class="number">0</span> {</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line">line = <span class="literal">nil</span></div><div class="line">}</div><div class="line"><span class="keyword">return</span></div><div class="line">}</div><div class="line">err = <span class="literal">nil</span></div><div class="line"></div><div class="line"><span class="keyword">if</span> line[<span class="built_in">len</span>(line)<span class="number">-1</span>] == <span class="string">'\n'</span> {</div><div class="line">drop := <span class="number">1</span></div><div class="line"><span class="keyword">if</span> <span class="built_in">len</span>(line) > <span class="number">1</span> && line[<span class="built_in">len</span>(line)<span class="number">-2</span>] == <span class="string">'\r'</span> {</div><div class="line">drop = <span class="number">2</span></div><div class="line">}</div><div class="line">line = line[:<span class="built_in">len</span>(line)-drop]</div><div class="line">}</div><div class="line"><span class="keyword">return</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// ReadBytes 功能同 ReadSlice，只不过返回的是缓存的拷贝。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">ReadBytes</span><span class="params">(delim <span class="keyword">byte</span>)</span> <span class="params">([]<span class="keyword">byte</span>, error)</span></span> {</div><div class="line"><span class="comment">// Use ReadSlice to look for array,</span></div><div class="line"><span class="comment">// accumulating full buffers.</span></div><div class="line"><span class="keyword">var</span> frag []<span class="keyword">byte</span></div><div class="line"><span class="keyword">var</span> full [][]<span class="keyword">byte</span></div><div class="line"><span class="keyword">var</span> err error</div><div class="line"><span class="keyword">for</span> {</div><div class="line"><span class="keyword">var</span> e error</div><div class="line">frag, e = b.ReadSlice(delim)</div><div class="line"><span class="keyword">if</span> e == <span class="literal">nil</span> { <span class="comment">// got final fragment</span></div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line"><span class="keyword">if</span> e != ErrBufferFull { <span class="comment">// unexpected error</span></div><div class="line">err = e</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Make a copy of the buffer.</span></div><div class="line">buf := <span class="built_in">make</span>([]<span class="keyword">byte</span>, <span class="built_in">len</span>(frag))</div><div class="line"><span class="built_in">copy</span>(buf, frag)</div><div class="line">full = <span class="built_in">append</span>(full, buf)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Allocate new buffer to hold the full pieces and the fragment.</span></div><div class="line">n := <span class="number">0</span></div><div class="line"><span class="keyword">for</span> i := <span class="keyword">range</span> full {</div><div class="line">n += <span class="built_in">len</span>(full[i])</div><div class="line">}</div><div class="line">n += <span class="built_in">len</span>(frag)</div><div class="line"></div><div class="line"><span class="comment">// Copy full pieces and fragment in.</span></div><div class="line">buf := <span class="built_in">make</span>([]<span class="keyword">byte</span>, n)</div><div class="line">n = <span class="number">0</span></div><div class="line"><span class="keyword">for</span> i := <span class="keyword">range</span> full {</div><div class="line">n += <span class="built_in">copy</span>(buf[n:], full[i])</div><div class="line">}</div><div class="line"><span class="built_in">copy</span>(buf[n:], frag)</div><div class="line"><span class="keyword">return</span> buf, err</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// ReadString 功能同 ReadBytes，只不过返回的是字符串。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">ReadString</span><span class="params">(delim <span class="keyword">byte</span>)</span> <span class="params">(<span class="keyword">string</span>, error)</span></span> {</div><div class="line">bytes, err := b.ReadBytes(delim)</div><div class="line"><span class="keyword">return</span> <span class="keyword">string</span>(bytes), err</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// WriteTo方法实现了 io.WriterTo 接口。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">WriteTo</span><span class="params">(w io.Writer)</span> <span class="params">(n <span class="keyword">int64</span>, err error)</span></span> {</div><div class="line">n, err = b.writeBuf(w)</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span></div><div class="line">}</div><div class="line"> </div><div class="line"> <span class="comment">// 如果 b.rd 已经实现了 io.WriterTo 接口，直接调用并返回。</span></div><div class="line"><span class="keyword">if</span> r, ok := b.rd.(io.WriterTo); ok {</div><div class="line">m, err := r.WriteTo(w)</div><div class="line">n += m</div><div class="line"><span class="keyword">return</span> n, err</div><div class="line">}</div><div class="line"></div><div class="line"> <span class="comment">// 如果 w 已经实现了 io.ReaderFrom 接口，调用并返回。</span></div><div class="line"><span class="keyword">if</span> w, ok := w.(io.ReaderFrom); ok {</div><div class="line">m, err := w.ReadFrom(b.rd)</div><div class="line">n += m</div><div class="line"><span class="keyword">return</span> n, err</div><div class="line">}</div><div class="line"></div><div class="line"> <span class="comment">// buffer not full</span></div><div class="line"><span class="keyword">if</span> b.w-b.r < <span class="built_in">len</span>(b.buf) {</div><div class="line">b.fill() </div><div class="line">}</div><div class="line"></div><div class="line"> <span class="comment">// b.r < b.w => buffer is not empty</span></div><div class="line"><span class="keyword">for</span> b.r < b.w {</div><div class="line">m, err := b.writeBuf(w)</div><div class="line">n += m</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> n, err</div><div class="line">}</div><div class="line">b.fill() <span class="comment">// buffer is empty</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">if</span> b.err == io.EOF {</div><div class="line">b.err = <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">return</span> n, b.readErr()</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// 定义错误：写入字节数为负</span></div><div class="line"><span class="keyword">var</span> errNegativeWrite = errors.New(<span class="string">"bufio: writer returned negative count from Write"</span>)</div><div class="line"></div><div class="line"><span class="comment">// 将 Reader.buf 中的数据写到 io.Writer 中。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Reader)</span> <span class="title">writeBuf</span><span class="params">(w io.Writer)</span> <span class="params">(<span class="keyword">int64</span>, error)</span></span> {</div><div class="line">n, err := w.Write(b.buf[b.r:b.w])</div><div class="line"><span class="keyword">if</span> n < <span class="number">0</span> {</div><div class="line"><span class="built_in">panic</span>(errNegativeWrite)</div><div class="line">}</div><div class="line">b.r += n</div><div class="line"><span class="keyword">return</span> <span class="keyword">int64</span>(n), err</div><div class="line">}</div></pre></td></tr></table></figure><p>Writer 部分：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div><div class="line">114</div><div class="line">115</div><div class="line">116</div><div class="line">117</div><div class="line">118</div><div class="line">119</div><div class="line">120</div><div class="line">121</div><div class="line">122</div><div class="line">123</div><div class="line">124</div><div class="line">125</div><div class="line">126</div><div class="line">127</div><div class="line">128</div><div class="line">129</div><div class="line">130</div><div class="line">131</div><div class="line">132</div><div class="line">133</div><div class="line">134</div><div class="line">135</div><div class="line">136</div><div class="line">137</div><div class="line">138</div><div class="line">139</div><div class="line">140</div><div class="line">141</div><div class="line">142</div><div class="line">143</div><div class="line">144</div><div class="line">145</div><div class="line">146</div><div class="line">147</div><div class="line">148</div><div class="line">149</div><div class="line">150</div><div class="line">151</div><div class="line">152</div><div class="line">153</div><div class="line">154</div><div class="line">155</div><div class="line">156</div><div class="line">157</div><div class="line">158</div><div class="line">159</div><div class="line">160</div><div class="line">161</div><div class="line">162</div><div class="line">163</div><div class="line">164</div><div class="line">165</div><div class="line">166</div><div class="line">167</div><div class="line">168</div><div class="line">169</div><div class="line">170</div><div class="line">171</div><div class="line">172</div><div class="line">173</div><div class="line">174</div><div class="line">175</div><div class="line">176</div><div class="line">177</div><div class="line">178</div><div class="line">179</div><div class="line">180</div><div class="line">181</div><div class="line">182</div><div class="line">183</div><div class="line">184</div><div class="line">185</div><div class="line">186</div><div class="line">187</div><div class="line">188</div><div class="line">189</div><div class="line">190</div><div class="line">191</div><div class="line">192</div><div class="line">193</div><div class="line">194</div><div class="line">195</div><div class="line">196</div><div class="line">197</div><div class="line">198</div><div class="line">199</div><div class="line">200</div><div class="line">201</div><div class="line">202</div><div class="line">203</div><div class="line">204</div><div class="line">205</div><div class="line">206</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// Writer实现了 io.Writer 接口并为其添加缓冲。</span></div><div class="line"><span class="comment">// 如果在向一个 Writer 类型写入时遇到了错误，该对象将不再接受任何数据，且所有写操作都会返回该错误。</span></div><div class="line"><span class="comment">// 在所有数据都写入后，调用者有义务调用 Flush 方法以保证所有的数据都交给了下层的 io.Writer。</span></div><div class="line"><span class="keyword">type</span> Writer <span class="keyword">struct</span> {</div><div class="line">err error</div><div class="line">buf []<span class="keyword">byte</span></div><div class="line">n <span class="keyword">int</span></div><div class="line">wr io.Writer</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// NewWriterSize 将 w 封装成一个带缓存的 bufio.Writer 对象，</span></div><div class="line"><span class="comment">// 缓存大小由 size 指定（如果小于 4096 则会被设置为 4096）。</span></div><div class="line"><span class="comment">// 如果 wr 的基类型就是有足够缓存的 bufio.Writer 类型，则直接将</span></div><div class="line"><span class="comment">// wr 转换为基类型返回。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">NewWriterSize</span><span class="params">(w io.Writer, size <span class="keyword">int</span>)</span> *<span class="title">Writer</span></span> {</div><div class="line"><span class="comment">// Is it already a Writer?</span></div><div class="line">b, ok := w.(*Writer)</div><div class="line"><span class="keyword">if</span> ok && <span class="built_in">len</span>(b.buf) >= size {</div><div class="line"><span class="keyword">return</span> b</div><div class="line">}</div><div class="line"><span class="keyword">if</span> size <= <span class="number">0</span> {</div><div class="line">size = defaultBufSize</div><div class="line">}</div><div class="line"><span class="keyword">return</span> &Writer{</div><div class="line">buf: <span class="built_in">make</span>([]<span class="keyword">byte</span>, size),</div><div class="line">wr: w,</div><div class="line">}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// NewWriter 相当于 NewWriterSize(wr, 4096)。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">NewWriter</span><span class="params">(w io.Writer)</span> *<span class="title">Writer</span></span> {</div><div class="line"><span class="keyword">return</span> NewWriterSize(w, defaultBufSize)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Reset 将 b 的底层 Writer 重新指定为 w，同时丢弃缓存中的所有数据，复位</span></div><div class="line"><span class="comment">// 所有标记和错误信息。相当于创建了一个新的 bufio.Writer。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Writer)</span> <span class="title">Reset</span><span class="params">(w io.Writer)</span></span> {</div><div class="line">b.err = <span class="literal">nil</span></div><div class="line">b.n = <span class="number">0</span></div><div class="line">b.wr = w</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Flush 将缓存中的数据提交到底层的 io.Writer 中。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Writer)</span> <span class="title">Flush</span><span class="params">()</span> <span class="title">error</span></span> {</div><div class="line"><span class="keyword">if</span> b.err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> b.err</div><div class="line">}</div><div class="line"><span class="keyword">if</span> b.n == <span class="number">0</span> {</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span></div><div class="line">}</div><div class="line"> </div><div class="line"> <span class="comment">// 将缓冲数据写入底层数据流</span></div><div class="line">n, err := b.wr.Write(b.buf[<span class="number">0</span>:b.n])</div><div class="line"> </div><div class="line"> <span class="comment">// 当未全部写完时，设置错误</span></div><div class="line"><span class="keyword">if</span> n < b.n && err == <span class="literal">nil</span> {</div><div class="line">err = io.ErrShortWrite</div><div class="line">}</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">if</span> n > <span class="number">0</span> && n < b.n {</div><div class="line"> <span class="comment">// 将已写过内容覆盖</span></div><div class="line"><span class="built_in">copy</span>(b.buf[<span class="number">0</span>:b.n-n], b.buf[n:b.n])</div><div class="line">}</div><div class="line">b.n -= n</div><div class="line">b.err = err</div><div class="line"><span class="keyword">return</span> err</div><div class="line">}</div><div class="line">b.n = <span class="number">0</span></div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Available 返回缓存中未使用的空间的长度</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Writer)</span> <span class="title">Available</span><span class="params">()</span> <span class="title">int</span></span> { <span class="keyword">return</span> <span class="built_in">len</span>(b.buf) - b.n }</div><div class="line"></div><div class="line"><span class="comment">// Buffered 返回缓存中未提交的数据的长度</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Writer)</span> <span class="title">Buffered</span><span class="params">()</span> <span class="title">int</span></span> { <span class="keyword">return</span> b.n }</div><div class="line"></div><div class="line"><span class="comment">// Write 将 p 的内容写入缓冲。</span></div><div class="line"><span class="comment">// 返回写入的字节数。</span></div><div class="line"><span class="comment">// 如果返回值 nn < len(p)，还会返回一个错误说明原因。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Writer)</span> <span class="title">Write</span><span class="params">(p []<span class="keyword">byte</span>)</span> <span class="params">(nn <span class="keyword">int</span>, err error)</span></span> {</div><div class="line"><span class="keyword">for</span> <span class="built_in">len</span>(p) > b.Available() && b.err == <span class="literal">nil</span> {</div><div class="line"><span class="keyword">var</span> n <span class="keyword">int</span></div><div class="line"><span class="keyword">if</span> b.Buffered() == <span class="number">0</span> {</div><div class="line"><span class="comment">// Large write, empty buffer.</span></div><div class="line"><span class="comment">// Write directly from p to avoid copy.</span></div><div class="line">n, b.err = b.wr.Write(p)</div><div class="line">} <span class="keyword">else</span> {</div><div class="line">n = <span class="built_in">copy</span>(b.buf[b.n:], p)</div><div class="line">b.n += n</div><div class="line">b.Flush()</div><div class="line">}</div><div class="line">nn += n</div><div class="line">p = p[n:]</div><div class="line">}</div><div class="line"><span class="keyword">if</span> b.err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> nn, b.err</div><div class="line">}</div><div class="line">n := <span class="built_in">copy</span>(b.buf[b.n:], p)</div><div class="line">b.n += n</div><div class="line">nn += n</div><div class="line"><span class="keyword">return</span> nn, <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// WriteByte 写入单个字节。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Writer)</span> <span class="title">WriteByte</span><span class="params">(c <span class="keyword">byte</span>)</span> <span class="title">error</span></span> {</div><div class="line"><span class="keyword">if</span> b.err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> b.err</div><div class="line">}</div><div class="line"><span class="keyword">if</span> b.Available() <= <span class="number">0</span> && b.Flush() != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> b.err</div><div class="line">}</div><div class="line">b.buf[b.n] = c</div><div class="line">b.n++</div><div class="line"><span class="keyword">return</span> <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// WriteRune写入一个unicode码值（的utf-8编码），返回写入的字节数和可能的错误。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Writer)</span> <span class="title">WriteRune</span><span class="params">(r <span class="keyword">rune</span>)</span> <span class="params">(size <span class="keyword">int</span>, err error)</span></span> {</div><div class="line"><span class="keyword">if</span> r < utf8.RuneSelf {</div><div class="line">err = b.WriteByte(<span class="keyword">byte</span>(r))</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="number">0</span>, err</div><div class="line">}</div><div class="line"><span class="keyword">return</span> <span class="number">1</span>, <span class="literal">nil</span></div><div class="line">}</div><div class="line"><span class="keyword">if</span> b.err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="number">0</span>, b.err</div><div class="line">}</div><div class="line">n := b.Available()</div><div class="line"><span class="keyword">if</span> n < utf8.UTFMax {</div><div class="line"><span class="keyword">if</span> b.Flush(); b.err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> <span class="number">0</span>, b.err</div><div class="line">}</div><div class="line">n = b.Available()</div><div class="line"><span class="keyword">if</span> n < utf8.UTFMax {</div><div class="line"><span class="comment">// Can only happen if buffer is silly small.</span></div><div class="line"><span class="keyword">return</span> b.WriteString(<span class="keyword">string</span>(r))</div><div class="line">}</div><div class="line">}</div><div class="line">size = utf8.EncodeRune(b.buf[b.n:], r)</div><div class="line">b.n += size</div><div class="line"><span class="keyword">return</span> size, <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// WriteString 功能同 Write，只不过写入的是字符串。</span></div><div class="line"><span class="comment">// 返回写入的字节数。如果返回值nn < len(s)，还会返回一个错误说明原因。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Writer)</span> <span class="title">WriteString</span><span class="params">(s <span class="keyword">string</span>)</span> <span class="params">(<span class="keyword">int</span>, error)</span></span> {</div><div class="line">nn := <span class="number">0</span></div><div class="line"><span class="keyword">for</span> <span class="built_in">len</span>(s) > b.Available() && b.err == <span class="literal">nil</span> {</div><div class="line">n := <span class="built_in">copy</span>(b.buf[b.n:], s)</div><div class="line">b.n += n</div><div class="line">nn += n</div><div class="line">s = s[n:]</div><div class="line">b.Flush()</div><div class="line">}</div><div class="line"><span class="keyword">if</span> b.err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> nn, b.err</div><div class="line">}</div><div class="line">n := <span class="built_in">copy</span>(b.buf[b.n:], s)</div><div class="line">b.n += n</div><div class="line">nn += n</div><div class="line"><span class="keyword">return</span> nn, <span class="literal">nil</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// ReadFrom实现了io.ReaderFrom接口。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="params">(b *Writer)</span> <span class="title">ReadFrom</span><span class="params">(r io.Reader)</span> <span class="params">(n <span class="keyword">int64</span>, err error)</span></span> {</div><div class="line"><span class="keyword">if</span> b.Buffered() == <span class="number">0</span> {</div><div class="line"><span class="keyword">if</span> w, ok := b.wr.(io.ReaderFrom); ok {</div><div class="line"><span class="keyword">return</span> w.ReadFrom(r)</div><div class="line">}</div><div class="line">}</div><div class="line"><span class="keyword">var</span> m <span class="keyword">int</span></div><div class="line"><span class="keyword">for</span> {</div><div class="line"><span class="keyword">if</span> b.Available() == <span class="number">0</span> {</div><div class="line"><span class="keyword">if</span> err1 := b.Flush(); err1 != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">return</span> n, err1</div><div class="line">}</div><div class="line">}</div><div class="line">nr := <span class="number">0</span></div><div class="line"><span class="keyword">for</span> nr < maxConsecutiveEmptyReads {</div><div class="line">m, err = r.Read(b.buf[b.n:])</div><div class="line"><span class="keyword">if</span> m != <span class="number">0</span> || err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line">nr++</div><div class="line">}</div><div class="line"><span class="keyword">if</span> nr == maxConsecutiveEmptyReads {</div><div class="line"><span class="keyword">return</span> n, io.ErrNoProgress</div><div class="line">}</div><div class="line">b.n += m</div><div class="line">n += <span class="keyword">int64</span>(m)</div><div class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</div><div class="line"><span class="keyword">break</span></div><div class="line">}</div><div class="line">}</div><div class="line"><span class="keyword">if</span> err == io.EOF {</div><div class="line"><span class="comment">// If we filled the buffer exactly, flush preemptively.</span></div><div class="line"><span class="keyword">if</span> b.Available() == <span class="number">0</span> {</div><div class="line">err = b.Flush()</div><div class="line">} <span class="keyword">else</span> {</div><div class="line">err = <span class="literal">nil</span></div><div class="line">}</div><div class="line">}</div><div class="line"><span class="keyword">return</span> n, err</div><div class="line">}</div></pre></td></tr></table></figure><p>ReadWriter 部分：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// ReadWriter类型保管了指向Reader和Writer类型的指针，因此实现了io.ReadWriter接口。</span></div><div class="line"><span class="keyword">type</span> ReadWriter <span class="keyword">struct</span> {</div><div class="line">*Reader</div><div class="line">*Writer</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// NewReadWriter申请创建一个新的、将读写操作分派给r和w 的ReadWriter。</span></div><div class="line"><span class="function"><span class="keyword">func</span> <span class="title">NewReadWriter</span><span class="params">(r *Reader, w *Writer)</span> *<span class="title">ReadWriter</span></span> {</div><div class="line"><span class="keyword">return</span> &ReadWriter{r, w}</div><div class="line">}</div></pre></td></tr></table></figure>]]></content>
<summary type="html">
<p>bufio包实现了有缓冲的I/O。它包装一个io.Reader或io.Writer接口对象，创建另一个也实现了该接口，且同时还提供了缓冲和一些文本I/O的帮助函数的对象。<br></p>
</summary>
<category term="Golang" scheme="https://yangchenglong11.github.io/categories/Golang/"/>
<category term="Golang" scheme="https://yangchenglong11.github.io/tags/Golang/"/>
</entry>
<entry>
<title>Docker 搭建 MongoDB 分片集群</title>
<link href="https://yangchenglong11.github.io/2017/04/23/docker%E6%90%AD%E5%BB%BAmongoDB%E5%88%86%E7%89%87%E9%9B%86%E7%BE%A4/"/>
<id>https://yangchenglong11.github.io/2017/04/23/docker搭建mongoDB分片集群/</id>
<published>2017-04-23T06:24:34.000Z</published>
<updated>2017-10-28T13:44:32.998Z</updated>
<content type="html"><![CDATA[<p>如何使用 Docker 搭建 MongoDB 分片集群。</p><a id="more"></a><p><strong>一、编写 dockerfile。</strong></p><p><strong>1、在适当目录下创建 mongod 的 dockerfile。</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> vi dockerfile</span></div></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#</span><span class="bash">version 1.0</span></div><div class="line">from ubuntu</div><div class="line"><span class="meta">#</span><span class="bash">maintainer </span></div><div class="line">maintainer hdx</div><div class="line"></div><div class="line"><span class="meta">#</span><span class="bash">install </span></div><div class="line">run apt-get clean</div><div class="line">run apt-get update</div><div class="line">run apt-get install -y vim</div><div class="line">run apt-get install -y openssh-server</div><div class="line">run mkdir -p /var/run/sshd</div><div class="line"></div><div class="line"><span class="meta">#</span><span class="bash">open port 22 20001</span></div><div class="line">expose 22</div><div class="line">expose 20001</div><div class="line"><span class="meta">#</span><span class="bash">cmd [<span class="string">"/usr/sbin/sshd"</span>,<span class="string">"-D"</span>]</span></div><div class="line"></div><div class="line">run apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927</div><div class="line"></div><div class="line">run echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.2 main" | sudo tee /etc/apt/sources.list.d/mongodb-org.list</div><div class="line"><span class="meta">#</span><span class="bash">install mongodb</span></div><div class="line">run apt-get update</div><div class="line">run apt-get install -y mongodb-org</div><div class="line"><span class="meta">#</span><span class="bash">create the mongodb data directory</span></div><div class="line">run mkdir -p /data/db</div><div class="line">entrypoint ["usr/bin/mongod"]</div></pre></td></tr></table></figure><p><strong>2、在适当目录下创建 mongod 的 dockerfile_mongos。</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> vi dockerfile_mongos</span></div></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">from ubuntu/mongo:latest</div><div class="line">entrypoint ["usr/bin/mongos"]</div></pre></td></tr></table></figure><p><strong>二、通过 dockerfile 生成 image 镜像。</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> sudo docker build -t ubuntu/mongo:latest -<./dockerfile</span></div><div class="line"><span class="meta">$</span><span class="bash"> sudo docker build -t ubuntu/mongos:latest -<./dockerfile_mongos</span></div></pre></td></tr></table></figure><p>查看 image 的生成情况。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> sudo docker images</span></div></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"> REPOSITORY TAG IMAGE ID CREATED SIZE</div><div class="line">ubuntu/mongos latest 6de9188a6c9c 37 hours ago 459MB</div><div class="line">ubuntu/mongo latest e2e287510648 37 hours ago 459MB</div></pre></td></tr></table></figure><p>发现 image 已经生成，然后通过 image 来创建容器了。</p><p><strong>三、通过 image 镜像构建 mongo 集群。</strong></p><p><strong>1、创建2个分片服务(shardsvr)，每个 shardsvr 包含4个副本，其中1个主节点，2个从节点，1个仲裁节点。</strong></p><p>-d 表示后台运行</p><p>-p 绑定 host 主机与 docker 的端口，第一个20001代表 host 主机端口，第二个代表对应的 docker 端口，绑定后可以通过调用 host 主机 ip:port 来访问 docker 启动的 mongoDB，我的主机端口为 10.0.0.116，所以下面的命令中IP都为这个。</p><p>注意：一定不能退在最后添加 <code>—fork</code>，使 mongo 服务后台运行，这样 docker 会任务无事可做而自杀！</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 20001:20001 --name rs1_container1 ubuntu/mongo:latest --shardsvr --port 20001 --replSet rs1</span></div><div class="line"></div><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 20002:20001 --name rs1_container2 ubuntu/mongo:latest --shardsvr --port 20001 --replSet rs1</span></div><div class="line"></div><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 20003:20001 --name rs1_container3 ubuntu/mongo:latest --shardsvr --port 20001 --replSet rs1</span></div><div class="line"></div><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 20004:20001 --name rs1_container4 ubuntu/mongo:latest --shardsvr --port 20001 --replSet rs1</span></div></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 20011:20001 --name rs2_container1 ubuntu/mongo:latest --shardsvr --port 20001 --replSet rs2</span></div><div class="line"></div><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 20012:20001 --name rs2_container2 ubuntu/mongo:latest --shardsvr --port 20001 --replSet rs2</span></div><div class="line"></div><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 20013:20001 --name rs2_container3 ubuntu/mongo:latest --shardsvr --port 20001 --replSet rs2</span></div><div class="line"></div><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 20014:20001 --name rs2_container4 ubuntu/mongo:latest --shardsvr --port 20001 --replSet rs2</span></div></pre></td></tr></table></figure><p><strong>2、创建2个配置服务(configsvr)。</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 21001:20001 --name config_container1 ubuntu/mongo:latest --configsvr --dbpath /data/db --replSet crs --port 20001</span></div><div class="line"></div><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 21002:20001 --name config_container2 ubuntu/mongo:latest --configsvr --dbpath /data/db --replSet crs --port 20001</span></div></pre></td></tr></table></figure><p>注意： <code>--dbpath /data/db</code> 一定要指定，因为<code>—configsvr</code> 默认路径是 <code>/data/configdb</code>，如果找不到会报错，然后 docker 直接自杀！</p><p> <code>--replSet crs</code> 从 mongoDB 版本3.2以后支持 configsvr 的副本集。</p><p><strong>3、启动2个路由服务（mongos）。</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 22001:20001 --name mongos_container1 ubuntu/mongos:latest --configdb crs/10.0.0.116:21001,10.0.0.116:21002 --port 20001 </span></div><div class="line"></div><div class="line"><span class="meta">$</span><span class="bash"> sudo docker run -d -p 22002:20001 --name mongos_container2 ubuntu/mongos:latest --configdb crs/10.0.0.116:21001,10.0.0.116:21002 --port 20001</span></div></pre></td></tr></table></figure><p>注意： <code>crs/10.0.0.116:21001,10.0.0.116:21002</code> mongodb 版本3.2以后通过这种形式指定 configdb，否则报错，ip 要写宿主机的实际 ip。</p><p><strong>4、查看当前 docker 服务启动情况。</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="meta">$</span><span class="bash"> sudo docker ps </span></div><div class="line">CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES</div><div class="line">d3bcc8c6ae02 ubuntu/mongos:latest "usr/bin/mongos --..." 17 hours ago Up 5 seconds 22/tcp, 0.0.0.0:22002->20001/tcp mongos_container2</div><div class="line">532a28409c5d ubuntu/mongos:latest "usr/bin/mongos --..." 17 hours ago Up 8 seconds 22/tcp, 0.0.0.0:22001->20001/tcp mongos_container1</div><div class="line">a071366a458d ubuntu/mongo:latest "usr/bin/mongod --..." 17 hours ago Up 14 seconds 22/tcp, 0.0.0.0:21002->20001/tcp config_container2</div><div class="line">88a34cbe67a6 ubuntu/mongo:latest "usr/bin/mongod --..." 17 hours ago Up 18 seconds 22/tcp, 0.0.0.0:21001->20001/tcp config_container1</div><div class="line">8cec58a3fdc4 ubuntu/mongo:latest "usr/bin/mongod --..." 17 hours ago Up 26 seconds 22/tcp, 0.0.0.0:20014->20001/tcp rs2_container4</div><div class="line">910881c88d92 ubuntu/mongo:latest "usr/bin/mongod --..." 17 hours ago Up 29 seconds 22/tcp, 0.0.0.0:20013->20001/tcp rs2_container3</div><div class="line">f69972ae2b0a ubuntu/mongo:latest "usr/bin/mongod --..." 17 hours ago Up 31 seconds 22/tcp, 0.0.0.0:20012->20001/tcp rs2_container2</div><div class="line">c6e8cece4ef1 ubuntu/mongo:latest "usr/bin/mongod --..." 17 hours ago Up 35 seconds 22/tcp, 0.0.0.0:20011->20001/tcp rs2_container1</div><div class="line">d93dba9c36a1 ubuntu/mongo:latest "usr/bin/mongod --..." 17 hours ago Up 48 seconds 22/tcp, 0.0.0.0:20004->20001/tcp rs1_container4</div><div class="line">f49ebcbfec7d ubuntu/mongo:latest "usr/bin/mongod --..." 17 hours ago Up 51 seconds 22/tcp, 0.0.0.0:20003->20001/tcp rs1_container3</div><div class="line">3f683b8848b3 ubuntu/mongo:latest "usr/bin/mongod --..." 17 hours ago Up 54 seconds 22/tcp, 0.0.0.0:20002->20001/tcp rs1_container2</div><div class="line">0f0792844d9d ubuntu/mongo:latest "usr/bin/mongod --..." 17 hours ago Up 56 seconds 22/tcp, 0.0.0.0:20001->20001/tcp rs1_container1</div></pre></td></tr></table></figure><p>发现分片、配置服务、和路由服务都启动起来了。</p><p><strong>四、mongo基本操作。</strong></p><p>服务启动起来了，但是服务都是互相独立的，所以，接下来我们将这些服务器串联起来。</p><p><strong>1、初始化分片 rs1 副本集。</strong></p><p>任意选择 rs1 分片的一个副本，连接并进行配置</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#</span><span class="bash"> 主机上如果安装了 mongoDB，使用如下命令：</span></div><div class="line">yangs-MacBook-Air:~ yang$ mongo 10.0.0.116:20001</div><div class="line"><span class="meta">#</span><span class="bash"> 这个是利用 docker 连接 mongoDB 命令，连接时二者选其一即可</span></div><div class="line">yangs-MacBook-Air:~ yang$ docker exec -it mongos_container2 mongo 10.0.0.116:20001</div><div class="line"></div><div class="line"></div><div class="line"><span class="meta">#</span><span class="bash"> 切换数据库</span></div><div class="line">use admin</div><div class="line"><span class="meta">#</span><span class="bash"> 写配置文件</span></div><div class="line">config = {_id:"rs1",members:[ {_id:0,host:"10.0.0.116:20001"}, {_id:1,host:"10.0.0.116:20002"}, {_id:2,host:"10.0.0.116:20003"},{_id:3,host:"10.0.0.116:20004",arbiterOnly:true}] }</div><div class="line"><span class="meta">#</span><span class="bash"> 初始化副本集</span></div><div class="line">rs.initiate(config)</div><div class="line"><span class="meta">#</span><span class="bash"> 查看副本集状态</span></div><div class="line">rs.status()</div></pre></td></tr></table></figure><p>结果：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div></pre></td><td class="code"><pre><div class="line">0f0792844d9d:20001(mongod-3.2.15)[PRIMARY:rs1] test> rs.status()</div><div class="line">{</div><div class="line"> "set": "rs1",</div><div class="line"> "date": ISODate("2017-07-27T02:00:03.335Z"),</div><div class="line"> "myState": 1,</div><div class="line"> "term": NumberLong("4"),</div><div class="line"> "heartbeatIntervalMillis": NumberLong("2000"),</div><div class="line"> "members": [</div><div class="line"> {</div><div class="line"> "_id": 0,</div><div class="line"> "name": "10.0.0.116:20001",</div><div class="line"> "health": 1,</div><div class="line"> "state": 1,</div><div class="line"> "stateStr": "PRIMARY",</div><div class="line"> "uptime": 442,</div><div class="line"> "optime": {</div><div class="line"> "ts": Timestamp(1501120374, 1),</div><div class="line"> "t": NumberLong("4")</div><div class="line"> },</div><div class="line"> "optimeDate": ISODate("2017-07-27T01:52:54Z"),</div><div class="line"> "electionTime": Timestamp(1501120373, 1),</div><div class="line"> "electionDate": ISODate("2017-07-27T01:52:53Z"),</div><div class="line"> "configVersion": 1,</div><div class="line"> "self": true</div><div class="line"> },</div><div class="line"> {</div><div class="line"> "_id": 1,</div><div class="line"> "name": "10.0.0.116:20002",</div><div class="line"> "health": 1,</div><div class="line"> "state": 2,</div><div class="line"> "stateStr": "SECONDARY",</div><div class="line"> "uptime": 436,</div><div class="line"> "optime": {</div><div class="line"> "ts": Timestamp(1501120374, 1),</div><div class="line"> "t": NumberLong("4")</div><div class="line"> },</div><div class="line"> "optimeDate": ISODate("2017-07-27T01:52:54Z"),</div><div class="line"> "lastHeartbeat": ISODate("2017-07-27T02:00:03.139Z"),</div><div class="line"> "lastHeartbeatRecv": ISODate("2017-07-27T02:00:03.139Z"),</div><div class="line"> "pingMs": NumberLong("3"),</div><div class="line"> "syncingTo": "10.0.0.116:20003",</div><div class="line"> "configVersion": 1</div><div class="line"> },</div><div class="line"> {</div><div class="line"> "_id": 2,</div><div class="line"> "name": "10.0.0.116:20003",</div><div class="line"> "health": 1,</div><div class="line"> "state": 2,</div><div class="line"> "stateStr": "SECONDARY",</div><div class="line"> "uptime": 436,</div><div class="line"> "optime": {</div><div class="line"> "ts": Timestamp(1501120374, 1),</div><div class="line"> "t": NumberLong("4")</div><div class="line"> },</div><div class="line"> "optimeDate": ISODate("2017-07-27T01:52:54Z"),</div><div class="line"> "lastHeartbeat": ISODate("2017-07-27T02:00:03.136Z"),</div><div class="line"> "lastHeartbeatRecv": ISODate("2017-07-27T02:00:01.665Z"),</div><div class="line"> "pingMs": NumberLong("1"),</div><div class="line"> "syncingTo": "10.0.0.116:20001",</div><div class="line"> "configVersion": 1</div><div class="line"> },</div><div class="line"> {</div><div class="line"> "_id": 3,</div><div class="line"> "name": "10.0.0.116:20004",</div><div class="line"> "health": 1,</div><div class="line"> "state": 7,</div><div class="line"> "stateStr": "ARBITER",</div><div class="line"> "uptime": 431,</div><div class="line"> "lastHeartbeat": ISODate("2017-07-27T02:00:03.136Z"),</div><div class="line"> "lastHeartbeatRecv": ISODate("2017-07-27T01:59:59.780Z"),</div><div class="line"> "pingMs": NumberLong("1"),</div><div class="line"> "configVersion": 1</div><div class="line"> }</div><div class="line"> ],</div><div class="line"> "ok": 1</div><div class="line">}</div></pre></td></tr></table></figure><p><strong>2、初始化分片 rs2 副本集。</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#</span><span class="bash"> 任意选择 rs2 分片的一个副本</span></div><div class="line">yangs-MacBook-Air:~ yang$ mongo 10.0.0.116:20011</div><div class="line"><span class="meta">#</span><span class="bash"> 切换数据库</span></div><div class="line">use admin</div><div class="line"><span class="meta">#</span><span class="bash"> 写配置文件</span></div><div class="line">config = {_id:"rs2",members:[ {_id:0,host:"10.0.0.116:20011"}, {_id:1,host:"10.0.0.116:20012"}, {_id:2,host:"10.0.0.116:20013"},{_id:3,host:"10.0.0.116:20014",arbiterOnly:true}] }</div><div class="line"><span class="meta">#</span><span class="bash"> 初始化副本集</span></div><div class="line">rs.initiate(config)</div><div class="line"><span class="meta">#</span><span class="bash"> 查看副本集状态</span></div><div class="line">rs.status()</div></pre></td></tr></table></figure><p><strong>3、初始化配置服务副本集。</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#</span><span class="bash"> 任意选择 crs 分片的一个副本</span></div><div class="line">yangs-MacBook-Air:~ yang$ mongo 10.0.0.116:21001</div><div class="line"><span class="meta">#</span><span class="bash"> 切换数据库</span></div><div class="line">use admin</div><div class="line"><span class="meta">#</span><span class="bash"> 写配置文件</span></div><div class="line">config = {_id:"crs", configsvr:true, members:[ {_id:0,host:"10.0.0.116:21001"}, {_id:1,host:"10.0.0.116:21002"} ] }</div><div class="line"><span class="meta">#</span><span class="bash"> 初始化副本集</span></div><div class="line">rs.initiate(config)</div><div class="line"><span class="meta">#</span><span class="bash"> 查看副本集状态</span></div><div class="line">rs.status()</div></pre></td></tr></table></figure><p><strong>4、通过 mongos 添加分片关系到 configsvr。</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">yangs-MacBook-Air:~ yang$ mongo 10.0.0.116:22001</div><div class="line">use admin</div><div class="line">db.runCommand({addshard:"rs1/10.0.0.116:20001,10.0.0.116:20002,10.0.0.116:20003,10.0.0.116:20004"})</div><div class="line">db.runCommand({addshard:"rs2/10.0.0.116:20011,10.0.0.116:20012,10.0.0.116:20013,10.0.0.116:20014"})</div></pre></td></tr></table></figure><p>查询结果如下：仲裁节点没有显示。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line">532a28409c5d:20001(mongos-3.2.15)[mongos] admin> db.runCommand({listshards:1})</div><div class="line">{</div><div class="line"> "shards": [</div><div class="line"> {</div><div class="line"> "_id": "rs1",</div><div class="line"> "host": "rs1/10.0.0.116:20001,10.0.0.116:20002,10.0.0.116:20003"</div><div class="line"> },</div><div class="line"> {</div><div class="line"> "_id": "rs2",</div><div class="line"> "host": "rs2/10.0.0.116:20011,10.0.0.116:20012,10.0.0.116:20013"</div><div class="line"> }</div><div class="line"> ],</div><div class="line"> "ok": 1</div><div class="line">}</div></pre></td></tr></table></figure><p><strong>5、设置数据库、集合分片。</strong></p><p>设置索引， 为了数据被均衡分发，我们创建散列索引</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#</span><span class="bash"> 在 mongos 中</span></div><div class="line">use mydb</div><div class="line">db.person.ensureIndex({id: "hashed"})</div></pre></td></tr></table></figure><p> 设置集合分片</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#</span><span class="bash"> 在 mongos 中</span></div><div class="line">db.runCommand({enablesharding:"mydb"})</div><div class="line">db.runCommand({shardcollection:"mydb.person", key:{id:"hashed"}})</div></pre></td></tr></table></figure><p>设置数据库 mydb、mydb 中 person 集合应用分片，片键为 person 集合的 id 字段。</p><p>说明：并不是数据库中所有集合都分片，只有设置了shardcollection 才分片，因为不是所有的集合都需要分片。</p><p><strong>6、测试分片结果。</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">use mydb</div><div class="line">for (i =0; i<5000;i++){</div><div class="line">db.person.save({id:i, company:"smartestee"})}</div></pre></td></tr></table></figure><p>测试结果如下：发现已经成功分片，rs1 和 rs2 较为均匀，这和片键有关系，具体情况请查询如何选择片键。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div><div class="line">114</div><div class="line">115</div><div class="line">116</div><div class="line">117</div><div class="line">118</div><div class="line">119</div><div class="line">120</div><div class="line">121</div><div class="line">122</div><div class="line">123</div><div class="line">124</div><div class="line">125</div><div class="line">126</div><div class="line">127</div><div class="line">128</div><div class="line">129</div><div class="line">130</div><div class="line">131</div><div class="line">132</div><div class="line">133</div><div class="line">134</div><div class="line">135</div><div class="line">136</div><div class="line">137</div><div class="line">138</div><div class="line">139</div><div class="line">140</div><div class="line">141</div><div class="line">142</div><div class="line">143</div><div class="line">144</div><div class="line">145</div><div class="line">146</div><div class="line">147</div><div class="line">148</div><div class="line">149</div><div class="line">150</div><div class="line">151</div><div class="line">152</div><div class="line">153</div><div class="line">154</div><div class="line">155</div><div class="line">156</div><div class="line">157</div><div class="line">158</div><div class="line">159</div><div class="line">160</div><div class="line">161</div><div class="line">162</div><div class="line">163</div><div class="line">164</div><div class="line">165</div><div class="line">166</div><div class="line">167</div><div class="line">168</div><div class="line">169</div><div class="line">170</div><div class="line">171</div><div class="line">172</div><div class="line">173</div><div class="line">174</div><div class="line">175</div><div class="line">176</div><div class="line">177</div><div class="line">178</div><div class="line">179</div><div class="line">180</div><div class="line">181</div><div class="line">182</div><div class="line">183</div><div class="line">184</div><div class="line">185</div><div class="line">186</div><div class="line">187</div><div class="line">188</div><div class="line">189</div><div class="line">190</div><div class="line">191</div><div class="line">192</div><div class="line">193</div><div class="line">194</div><div class="line">195</div><div class="line">196</div><div class="line">197</div><div class="line">198</div><div class="line">199</div><div class="line">200</div><div class="line">201</div><div class="line">202</div><div class="line">203</div><div class="line">204</div><div class="line">205</div><div class="line">206</div><div class="line">207</div><div class="line">208</div><div class="line">209</div><div class="line">210</div><div class="line">211</div><div class="line">212</div><div class="line">213</div><div class="line">214</div><div class="line">215</div><div class="line">216</div><div class="line">217</div><div class="line">218</div><div class="line">219</div><div class="line">220</div><div class="line">221</div><div class="line">222</div><div class="line">223</div><div class="line">224</div><div class="line">225</div><div class="line">226</div><div class="line">227</div><div class="line">228</div><div class="line">229</div><div class="line">230</div><div class="line">231</div><div class="line">232</div><div class="line">233</div><div class="line">234</div><div class="line">235</div><div class="line">236</div><div class="line">237</div><div class="line">238</div><div class="line">239</div><div class="line">240</div><div class="line">241</div><div class="line">242</div><div class="line">243</div><div class="line">244</div><div class="line">245</div><div class="line">246</div><div class="line">247</div><div class="line">248</div><div class="line">249</div><div class="line">250</div><div class="line">251</div><div class="line">252</div><div class="line">253</div><div class="line">254</div><div class="line">255</div><div class="line">256</div><div class="line">257</div><div class="line">258</div><div class="line">259</div><div class="line">260</div><div class="line">261</div><div class="line">262</div><div class="line">263</div><div class="line">264</div><div class="line">265</div><div class="line">266</div><div class="line">267</div><div class="line">268</div><div class="line">269</div><div class="line">270</div><div class="line">271</div><div class="line">272</div><div class="line">273</div><div class="line">274</div><div class="line">275</div><div class="line">276</div><div class="line">277</div><div class="line">278</div><div class="line">279</div><div class="line">280</div><div class="line">281</div><div class="line">282</div><div class="line">283</div><div class="line">284</div><div class="line">285</div><div class="line">286</div><div class="line">287</div><div class="line">288</div><div class="line">289</div><div class="line">290</div><div class="line">291</div><div class="line">292</div><div class="line">293</div><div class="line">294</div><div class="line">295</div><div class="line">296</div><div class="line">297</div><div class="line">298</div><div class="line">299</div><div class="line">300</div><div class="line">301</div><div class="line">302</div><div class="line">303</div><div class="line">304</div><div class="line">305</div><div class="line">306</div><div class="line">307</div><div class="line">308</div><div class="line">309</div><div class="line">310</div><div class="line">311</div><div class="line">312</div><div class="line">313</div><div class="line">314</div><div class="line">315</div><div class="line">316</div><div class="line">317</div><div class="line">318</div><div class="line">319</div><div class="line">320</div><div class="line">321</div><div class="line">322</div><div class="line">323</div><div class="line">324</div><div class="line">325</div><div class="line">326</div><div class="line">327</div><div class="line">328</div><div class="line">329</div><div class="line">330</div><div class="line">331</div><div class="line">332</div><div class="line">333</div><div class="line">334</div><div class="line">335</div><div class="line">336</div><div class="line">337</div><div class="line">338</div><div class="line">339</div><div class="line">340</div><div class="line">341</div><div class="line">342</div><div class="line">343</div><div class="line">344</div><div class="line">345</div><div class="line">346</div></pre></td><td class="code"><pre><div class="line">532a28409c5d:20001(mongos-3.2.15)[mongos] mydb> db.person.stats()</div><div class="line">{</div><div class="line"> "sharded": true,</div><div class="line"> "capped": false,</div><div class="line"> "ns": "mydb.person",</div><div class="line"> "count": 5000,</div><div class="line"> "size": 277766,</div><div class="line"> "storageSize": 196608,</div><div class="line"> "totalIndexSize": 323584,</div><div class="line"> "indexSizes": {</div><div class="line"> "_id_": 122880,</div><div class="line"> "id_hashed": 200704</div><div class="line"> },</div><div class="line"> "avgObjSize": 55.5532,</div><div class="line"> "nindexes": 2,</div><div class="line"> "nchunks": 4,</div><div class="line"> "shards": {</div><div class="line"> "rs1": {</div><div class="line"> "ns": "mydb.person",</div><div class="line"> "count": 2480,</div><div class="line"> "size": 137486,</div><div class="line"> "avgObjSize": 55,</div><div class="line"> "storageSize": 94208,</div><div class="line"> "capped": false,</div><div class="line"> "wiredTiger": {</div><div class="line"> "metadata": {</div><div class="line"> "formatVersion": 1</div><div class="line"> },</div><div class="line"> "creationString": "access_pattern_hint=none,allocation_size=4KB,app_metadata=(formatVersion=1),block_allocation=best,block_compressor=snappy,cache_resident=false,checksum=on,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=,name=),exclusive=false,extractor=,format=btree,huffman_key=,huffman_value=,ignore_in_memory_cache_size=false,immutable=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=4KB,key_format=q,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,log=(enabled=true),lsm=(auto_throttle=true,bloom=true,bloom_bit_count=16,bloom_config=,bloom_hash_count=8,bloom_oldest=false,chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB,merge_max=15,merge_min=0),memory_page_max=10m,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,source=,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,type=file,value_format=u",</div><div class="line"> "type": "file",</div><div class="line"> "uri": "statistics:table:collection-16-4938398221887072838",</div><div class="line"> "LSM": {</div><div class="line"> "bloom filter false positives": 0,</div><div class="line"> "bloom filter hits": 0,</div><div class="line"> "bloom filter misses": 0,</div><div class="line"> "bloom filter pages evicted from cache": 0,</div><div class="line"> "bloom filter pages read into cache": 0,</div><div class="line"> "bloom filters in the LSM tree": 0,</div><div class="line"> "chunks in the LSM tree": 0,</div><div class="line"> "highest merge generation in the LSM tree": 0,</div><div class="line"> "queries that could have benefited from a Bloom filter that did not exist": 0,</div><div class="line"> "sleep for LSM checkpoint throttle": 0,</div><div class="line"> "sleep for LSM merge throttle": 0,</div><div class="line"> "total size of bloom filters": 0</div><div class="line"> },</div><div class="line"> "block-manager": {</div><div class="line"> "allocations requiring file extension": 0,</div><div class="line"> "blocks allocated": 0,</div><div class="line"> "blocks freed": 0,</div><div class="line"> "checkpoint size": 40960,</div><div class="line"> "file allocation unit size": 4096,</div><div class="line"> "file bytes available for reuse": 36864,</div><div class="line"> "file magic number": 120897,</div><div class="line"> "file major version number": 1,</div><div class="line"> "file size in bytes": 94208,</div><div class="line"> "minor version number": 0</div><div class="line"> },</div><div class="line"> "btree": {</div><div class="line"> "btree checkpoint generation": 4,</div><div class="line"> "column-store fixed-size leaf pages": 0,</div><div class="line"> "column-store internal pages": 0,</div><div class="line"> "column-store variable-size RLE encoded values": 0,</div><div class="line"> "column-store variable-size deleted values": 0,</div><div class="line"> "column-store variable-size leaf pages": 0,</div><div class="line"> "fixed-record size": 0,</div><div class="line"> "maximum internal page key size": 368,</div><div class="line"> "maximum internal page size": 4096,</div><div class="line"> "maximum leaf page key size": 2867,</div><div class="line"> "maximum leaf page size": 32768,</div><div class="line"> "maximum leaf page value size": 67108864,</div><div class="line"> "maximum tree depth": 0,</div><div class="line"> "number of key/value pairs": 0,</div><div class="line"> "overflow pages": 0,</div><div class="line"> "pages rewritten by compaction": 0,</div><div class="line"> "row-store internal pages": 0,</div><div class="line"> "row-store leaf pages": 0</div><div class="line"> },</div><div class="line"> "cache": {</div><div class="line"> "bytes currently in the cache": 21314,</div><div class="line"> "bytes read into cache": 16816,</div><div class="line"> "bytes written from cache": 0,</div><div class="line"> "checkpoint blocked page eviction": 0,</div><div class="line"> "data source pages selected for eviction unable to be evicted": 0,</div><div class="line"> "hazard pointer blocked page eviction": 0,</div><div class="line"> "in-memory page passed criteria to be split": 0,</div><div class="line"> "in-memory page splits": 0,</div><div class="line"> "internal pages evicted": 0,</div><div class="line"> "internal pages split during eviction": 0,</div><div class="line"> "leaf pages split during eviction": 0,</div><div class="line"> "modified pages evicted": 0,</div><div class="line"> "overflow pages read into cache": 0,</div><div class="line"> "overflow values cached in memory": 0,</div><div class="line"> "page split during eviction deepened the tree": 0,</div><div class="line"> "page written requiring lookaside records": 0,</div><div class="line"> "pages read into cache": 2,</div><div class="line"> "pages read into cache requiring lookaside entries": 0,</div><div class="line"> "pages requested from the cache": 1,</div><div class="line"> "pages written from cache": 0,</div><div class="line"> "pages written requiring in-memory restoration": 0,</div><div class="line"> "tracked dirty bytes in the cache": 0,</div><div class="line"> "unmodified pages evicted": 0</div><div class="line"> },</div><div class="line"> "cache_walk": {</div><div class="line"> "Average difference between current eviction generation when the page was last considered": 0,</div><div class="line"> "Average on-disk page image size seen": 0,</div><div class="line"> "Clean pages currently in cache": 0,</div><div class="line"> "Current eviction generation": 0,</div><div class="line"> "Dirty pages currently in cache": 0,</div><div class="line"> "Entries in the root page": 0,</div><div class="line"> "Internal pages currently in cache": 0,</div><div class="line"> "Leaf pages currently in cache": 0,</div><div class="line"> "Maximum difference between current eviction generation when the page was last considered": 0,</div><div class="line"> "Maximum page size seen": 0,</div><div class="line"> "Minimum on-disk page image size seen": 0,</div><div class="line"> "On-disk page image sizes smaller than a single allocation unit": 0,</div><div class="line"> "Pages created in memory and never written": 0,</div><div class="line"> "Pages currently queued for eviction": 0,</div><div class="line"> "Pages that could not be queued for eviction": 0,</div><div class="line"> "Refs skipped during cache traversal": 0,</div><div class="line"> "Size of the root page": 0,</div><div class="line"> "Total number of pages currently in cache": 0</div><div class="line"> },</div><div class="line"> "compression": {</div><div class="line"> "compressed pages read": 1,</div><div class="line"> "compressed pages written": 0,</div><div class="line"> "page written failed to compress": 0,</div><div class="line"> "page written was too small to compress": 0,</div><div class="line"> "raw compression call failed, additional data available": 0,</div><div class="line"> "raw compression call failed, no additional data available": 0,</div><div class="line"> "raw compression call succeeded": 0</div><div class="line"> },</div><div class="line"> "cursor": {</div><div class="line"> "bulk-loaded cursor-insert calls": 0,</div><div class="line"> "create calls": 1,</div><div class="line"> "cursor-insert key and value bytes inserted": 0,</div><div class="line"> "cursor-remove key bytes removed": 0,</div><div class="line"> "cursor-update value bytes updated": 0,</div><div class="line"> "insert calls": 0,</div><div class="line"> "next calls": 0,</div><div class="line"> "prev calls": 1,</div><div class="line"> "remove calls": 0,</div><div class="line"> "reset calls": 1,</div><div class="line"> "restarted searches": 0,</div><div class="line"> "search calls": 0,</div><div class="line"> "search near calls": 0,</div><div class="line"> "truncate calls": 0,</div><div class="line"> "update calls": 0</div><div class="line"> },</div><div class="line"> "reconciliation": {</div><div class="line"> "dictionary matches": 0,</div><div class="line"> "fast-path pages deleted": 0,</div><div class="line"> "internal page key bytes discarded using suffix compression": 0,</div><div class="line"> "internal page multi-block writes": 0,</div><div class="line"> "internal-page overflow keys": 0,</div><div class="line"> "leaf page key bytes discarded using prefix compression": 0,</div><div class="line"> "leaf page multi-block writes": 0,</div><div class="line"> "leaf-page overflow keys": 0,</div><div class="line"> "maximum blocks required for a page": 0,</div><div class="line"> "overflow values written": 0,</div><div class="line"> "page checksum matches": 0,</div><div class="line"> "page reconciliation calls": 0,</div><div class="line"> "page reconciliation calls for eviction": 0,</div><div class="line"> "pages deleted": 0</div><div class="line"> },</div><div class="line"> "session": {</div><div class="line"> "object compaction": 0,</div><div class="line"> "open cursor count": 1</div><div class="line"> },</div><div class="line"> "transaction": {</div><div class="line"> "update conflicts": 0</div><div class="line"> }</div><div class="line"> },</div><div class="line"> "nindexes": 2,</div><div class="line"> "totalIndexSize": 159744,</div><div class="line"> "indexSizes": {</div><div class="line"> "_id_": 61440,</div><div class="line"> "id_hashed": 98304</div><div class="line"> },</div><div class="line"> "ok": 1</div><div class="line"> },</div><div class="line"> "rs2": {</div><div class="line"> "ns": "mydb.person",</div><div class="line"> "count": 2520,</div><div class="line"> "size": 140280,</div><div class="line"> "avgObjSize": 55,</div><div class="line"> "storageSize": 102400,</div><div class="line"> "capped": false,</div><div class="line"> "wiredTiger": {</div><div class="line"> "metadata": {</div><div class="line"> "formatVersion": 1</div><div class="line"> },</div><div class="line"> "creationString": "access_pattern_hint=none,allocation_size=4KB,app_metadata=(formatVersion=1),block_allocation=best,block_compressor=snappy,cache_resident=false,checksum=on,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=,name=),exclusive=false,extractor=,format=btree,huffman_key=,huffman_value=,ignore_in_memory_cache_size=false,immutable=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=4KB,key_format=q,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,log=(enabled=true),lsm=(auto_throttle=true,bloom=true,bloom_bit_count=16,bloom_config=,bloom_hash_count=8,bloom_oldest=false,chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB,merge_max=15,merge_min=0),memory_page_max=10m,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,source=,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,type=file,value_format=u",</div><div class="line"> "type": "file",</div><div class="line"> "uri": "statistics:table:collection-16--4365114664988837133",</div><div class="line"> "LSM": {</div><div class="line"> "bloom filter false positives": 0,</div><div class="line"> "bloom filter hits": 0,</div><div class="line"> "bloom filter misses": 0,</div><div class="line"> "bloom filter pages evicted from cache": 0,</div><div class="line"> "bloom filter pages read into cache": 0,</div><div class="line"> "bloom filters in the LSM tree": 0,</div><div class="line"> "chunks in the LSM tree": 0,</div><div class="line"> "highest merge generation in the LSM tree": 0,</div><div class="line"> "queries that could have benefited from a Bloom filter that did not exist": 0,</div><div class="line"> "sleep for LSM checkpoint throttle": 0,</div><div class="line"> "sleep for LSM merge throttle": 0,</div><div class="line"> "total size of bloom filters": 0</div><div class="line"> },</div><div class="line"> "block-manager": {</div><div class="line"> "allocations requiring file extension": 0,</div><div class="line"> "blocks allocated": 0,</div><div class="line"> "blocks freed": 0,</div><div class="line"> "checkpoint size": 45056,</div><div class="line"> "file allocation unit size": 4096,</div><div class="line"> "file bytes available for reuse": 40960,</div><div class="line"> "file magic number": 120897,</div><div class="line"> "file major version number": 1,</div><div class="line"> "file size in bytes": 102400,</div><div class="line"> "minor version number": 0</div><div class="line"> },</div><div class="line"> "btree": {</div><div class="line"> "btree checkpoint generation": 4,</div><div class="line"> "column-store fixed-size leaf pages": 0,</div><div class="line"> "column-store internal pages": 0,</div><div class="line"> "column-store variable-size RLE encoded values": 0,</div><div class="line"> "column-store variable-size deleted values": 0,</div><div class="line"> "column-store variable-size leaf pages": 0,</div><div class="line"> "fixed-record size": 0,</div><div class="line"> "maximum internal page key size": 368,</div><div class="line"> "maximum internal page size": 4096,</div><div class="line"> "maximum leaf page key size": 2867,</div><div class="line"> "maximum leaf page size": 32768,</div><div class="line"> "maximum leaf page value size": 67108864,</div><div class="line"> "maximum tree depth": 0,</div><div class="line"> "number of key/value pairs": 0,</div><div class="line"> "overflow pages": 0,</div><div class="line"> "pages rewritten by compaction": 0,</div><div class="line"> "row-store internal pages": 0,</div><div class="line"> "row-store leaf pages": 0</div><div class="line"> },</div><div class="line"> "cache": {</div><div class="line"> "bytes currently in the cache": 24852,</div><div class="line"> "bytes read into cache": 19676,</div><div class="line"> "bytes written from cache": 0,</div><div class="line"> "checkpoint blocked page eviction": 0,</div><div class="line"> "data source pages selected for eviction unable to be evicted": 0,</div><div class="line"> "hazard pointer blocked page eviction": 0,</div><div class="line"> "in-memory page passed criteria to be split": 0,</div><div class="line"> "in-memory page splits": 0,</div><div class="line"> "internal pages evicted": 0,</div><div class="line"> "internal pages split during eviction": 0,</div><div class="line"> "leaf pages split during eviction": 0,</div><div class="line"> "modified pages evicted": 0,</div><div class="line"> "overflow pages read into cache": 0,</div><div class="line"> "overflow values cached in memory": 0,</div><div class="line"> "page split during eviction deepened the tree": 0,</div><div class="line"> "page written requiring lookaside records": 0,</div><div class="line"> "pages read into cache": 2,</div><div class="line"> "pages read into cache requiring lookaside entries": 0,</div><div class="line"> "pages requested from the cache": 1,</div><div class="line"> "pages written from cache": 0,</div><div class="line"> "pages written requiring in-memory restoration": 0,</div><div class="line"> "tracked dirty bytes in the cache": 0,</div><div class="line"> "unmodified pages evicted": 0</div><div class="line"> },</div><div class="line"> "cache_walk": {</div><div class="line"> "Average difference between current eviction generation when the page was last considered": 0,</div><div class="line"> "Average on-disk page image size seen": 0,</div><div class="line"> "Clean pages currently in cache": 0,</div><div class="line"> "Current eviction generation": 0,</div><div class="line"> "Dirty pages currently in cache": 0,</div><div class="line"> "Entries in the root page": 0,</div><div class="line"> "Internal pages currently in cache": 0,</div><div class="line"> "Leaf pages currently in cache": 0,</div><div class="line"> "Maximum difference between current eviction generation when the page was last considered": 0,</div><div class="line"> "Maximum page size seen": 0,</div><div class="line"> "Minimum on-disk page image size seen": 0,</div><div class="line"> "On-disk page image sizes smaller than a single allocation unit": 0,</div><div class="line"> "Pages created in memory and never written": 0,</div><div class="line"> "Pages currently queued for eviction": 0,</div><div class="line"> "Pages that could not be queued for eviction": 0,</div><div class="line"> "Refs skipped during cache traversal": 0,</div><div class="line"> "Size of the root page": 0,</div><div class="line"> "Total number of pages currently in cache": 0</div><div class="line"> },</div><div class="line"> "compression": {</div><div class="line"> "compressed pages read": 1,</div><div class="line"> "compressed pages written": 0,</div><div class="line"> "page written failed to compress": 0,</div><div class="line"> "page written was too small to compress": 0,</div><div class="line"> "raw compression call failed, additional data available": 0,</div><div class="line"> "raw compression call failed, no additional data available": 0,</div><div class="line"> "raw compression call succeeded": 0</div><div class="line"> },</div><div class="line"> "cursor": {</div><div class="line"> "bulk-loaded cursor-insert calls": 0,</div><div class="line"> "create calls": 1,</div><div class="line"> "cursor-insert key and value bytes inserted": 0,</div><div class="line"> "cursor-remove key bytes removed": 0,</div><div class="line"> "cursor-update value bytes updated": 0,</div><div class="line"> "insert calls": 0,</div><div class="line"> "next calls": 0,</div><div class="line"> "prev calls": 1,</div><div class="line"> "remove calls": 0,</div><div class="line"> "reset calls": 1,</div><div class="line"> "restarted searches": 0,</div><div class="line"> "search calls": 0,</div><div class="line"> "search near calls": 0,</div><div class="line"> "truncate calls": 0,</div><div class="line"> "update calls": 0</div><div class="line"> },</div><div class="line"> "reconciliation": {</div><div class="line"> "dictionary matches": 0,</div><div class="line"> "fast-path pages deleted": 0,</div><div class="line"> "internal page key bytes discarded using suffix compression": 0,</div><div class="line"> "internal page multi-block writes": 0,</div><div class="line"> "internal-page overflow keys": 0,</div><div class="line"> "leaf page key bytes discarded using prefix compression": 0,</div><div class="line"> "leaf page multi-block writes": 0,</div><div class="line"> "leaf-page overflow keys": 0,</div><div class="line"> "maximum blocks required for a page": 0,</div><div class="line"> "overflow values written": 0,</div><div class="line"> "page checksum matches": 0,</div><div class="line"> "page reconciliation calls": 0,</div><div class="line"> "page reconciliation calls for eviction": 0,</div><div class="line"> "pages deleted": 0</div><div class="line"> },</div><div class="line"> "session": {</div><div class="line"> "object compaction": 0,</div><div class="line"> "open cursor count": 1</div><div class="line"> },</div><div class="line"> "transaction": {</div><div class="line"> "update conflicts": 0</div><div class="line"> }</div><div class="line"> },</div><div class="line"> "nindexes": 2,</div><div class="line"> "totalIndexSize": 163840,</div><div class="line"> "indexSizes": {</div><div class="line"> "_id_": 61440,</div><div class="line"> "id_hashed": 102400</div><div class="line"> },</div><div class="line"> "ok": 1</div><div class="line"> }</div><div class="line"> },</div><div class="line"> "ok": 1</div><div class="line">}</div></pre></td></tr></table></figure>]]></content>
<summary type="html">
<p>如何使用 Docker 搭建 MongoDB 分片集群。</p>
</summary>
<category term="MongoDB" scheme="https://yangchenglong11.github.io/categories/MongoDB/"/>
<category term="MongoDB" scheme="https://yangchenglong11.github.io/tags/MongoDB/"/>
</entry>
<entry>
<title>Linux 的I/O 模型</title>
<link href="https://yangchenglong11.github.io/2017/04/11/Linux-%E7%9A%84I-O-%E6%A8%A1%E5%9E%8B/"/>
<id>https://yangchenglong11.github.io/2017/04/11/Linux-的I-O-模型/</id>
<published>2017-04-11T13:05:37.000Z</published>
<updated>2017-10-28T13:17:40.143Z</updated>
<content type="html"><![CDATA[<p>进行后台开发时，经常需要进行数据传输，其数据传输归根结底还是Linux 的I/O 操作，今天就来讲下 Linux 的I/O 模型。</p><a id="more"></a><h2 id="1、介绍"><a href="#1、介绍" class="headerlink" title="1、介绍"></a>1、介绍</h2><p>Linux 的内核将所有外部设备都看做一个文件来操作（一切皆文件），对一个文件的读写操作会调用内核提供的系统命令，返回一个file descriptor（fd，文件描述符）。而对一个socket的读写也会有相应的描述符，称为socket fd（socket文件描述符），描述符就是一个数字，指向内核中的一个结构体（文件路径，数据区等一些属性）。</p><p>服务器端编程经常需要构造高性能的IO模型，Linux的IO模型有五种：</p><ul><li>同步阻塞IO（BlockingIO）：即传统的IO模型。</li><li>同步非阻塞IO（Non-blockingIO）：默认创建的socket都是阻塞的，非阻塞IO要求socket被设置为NONBLOCK。注意这里所说的NIO并非Java的NIO（NewIO）库。</li><li>IO多路复用（IOMultiplexing）：即经典的Reactor设计模式，有时也称为异步阻塞IO，Linux中的epoll就是这种模型。</li><li>信号驱动式I/O（signal-driven I/O）：即基于信号驱动的IO（SignalDrivenIO）模型。</li><li>异步IO（AsynchronousIO）：即经典的Proactor设计模式，也称为异步非阻塞IO。</li></ul><p><strong>同步和异步的概念描述的是用户线程与内核的交互方式：同步是指用户线程发起IO请求后需要等待或者轮询内核IO操作完成后才能继续执行；而异步是指用户线程发起IO请求后仍继续执行，当内核IO操作完成后会通知用户线程，或者调用用户线程注册的回调函数</strong>。<br><strong>阻塞和非阻塞的概念描述的是用户线程调用内核IO操作的方式：阻塞是指IO操作需要彻底完成后才返回到用户空间；而非阻塞是指IO操作被调用后立即返回给用户一个状态值，无需等到IO操作彻底完成。</strong></p><p>接下来，我们详细分析五种常见的IO模型的实现原理。为了方便描述，我们统一使用IO的读操作作为示例。</p><h4 id="1、阻塞I-O模型"><a href="#1、阻塞I-O模型" class="headerlink" title="1、阻塞I/O模型"></a>1、阻塞I/O模型</h4><p>阻塞I/O是最流行的I/O模型。它符合人们最常见的思考逻辑。<strong>阻塞就是进程 “被” 休息, CPU处理其它进程去了</strong>。在网络I/O的时候，进程发起<code>recvform</code>系统调用，然后进程就被阻塞了，什么也不干，直到数据准备好，并且将数据从内核复制到用户进程，最后进程再处理数据，在等待数据到处理数据的两个阶段，整个进程都被阻塞。不能处理别的网络I/O。</p><p>图示：</p><p> <img src="http://blog.anxpp.com/usr/uploads/2016/05/1140040694.png" alt="01"></p><p>阻塞IO的特点就是在IO执行的两个阶段都被block了。</p><h4 id="2、非阻塞I-O模型"><a href="#2、非阻塞I-O模型" class="headerlink" title="2、非阻塞I/O模型"></a>2、非阻塞I/O模型</h4><p>在网络I/O时候，非阻塞I/O也会进行recvform系统调用，检查数据是否准备好，与阻塞I/O不一样，”非阻塞将大的整片时间的阻塞分成N多的小的阻塞, 所以进程不断地有机会 ‘被’ CPU光顾”。</p><p>也就是说非阻塞的recvform系统调用调用之后，进程并没有被阻塞，内核马上返回给进程，如果数据还没准备好，此时会返回一个error。进程在返回之后，可以干点别的事情，然后再发起recvform系统调用。重复上面的过程，循环往复的进行recvform系统调用。这个过程通常被称之为<code>轮询</code>。轮询检查内核数据，直到数据准备好，再拷贝数据到进程，进行数据处理。需要注意，拷贝数据整个过程，进程仍然是属于阻塞的状态。</p><p>图示：</p><p><img src="http://blog.anxpp.com/usr/uploads/2016/05/2665563581.png" alt="02"></p><p>非阻塞 IO的特点是用户进程需要<strong>不断的主动询问</strong>kernel数据是否准备好</p><h4 id="3、I-O复用模型"><a href="#3、I-O复用模型" class="headerlink" title="3、I/O复用模型"></a>3、I/O复用模型</h4><p>可以看出，由于非阻塞的调用，轮询占据了很大一部分过程，轮询会消耗大量的CPU时间。结合前面两种模式。如果轮询不是进程的用户态，而是有人帮忙就好了。多路复用正好处理这样的问题。</p><p>多路复用有两个特别的系统调用<code>select</code>或<code>poll</code>。select调用是内核级别的，select轮询相对非阻塞的轮询的区别在于—前者可以等待多个socket，当其中任何一个socket的数据准好了，就能返回进行可读，然后进程再进行recvform系统调用，将数据由内核拷贝到用户进程，当然这个过程是阻塞的。多路复用有两种阻塞，select或poll调用之后，会阻塞进程，与第一种阻塞不同在于，此时的select不是等到socket数据全部到达再处理, 而是有了一部分数据就会调用用户进程来处理。如何知道有一部分数据到达了呢？监视的事情交给了内核，内核负责数据到达的处理。</p><p>图示：</p><p><img src="http://blog.anxpp.com/usr/uploads/2016/05/860854051.png" alt="03"></p><p>多路复用的特点是通过一种机制一个进程能同时等待IO文件描述符，内核监视这些文件描述符（套接字描述符），其中的任意一个进入读就绪状态，select， poll，epoll函数就可以返回。对于监视的方式，又可以分为 select， poll， epoll三种方式。</p><p>I/O多路复用技术通过把多个I/O的阻塞复用到同一个select的阻塞上，从而使得系统在单线程的情况下可以同时处理多个客户端请求。</p><p>与传统的多线程模型相比，I/O多路复用的最大优势就是系统开销小，系统不需要创建新的额外线程，也不需要维护这些线程的运行，降低了系统的维护工作量，节省了系统资源。</p><p>主要的应用场景：</p><ul><li> 服务器需要同时处理多个处于监听状态或多个连接状态的套接字。</li><li> 服务器需要同时处理多种网络协议的套接字。</li></ul><p>支持I/O多路复用的系统调用主要有select、pselect、poll、epoll。</p><p>而当前推荐使用的是epoll，优势如下：</p><ul><li> 支持一个进程打开的socket fd不受限制。</li><li> I/O效率不会随着fd数目的增加而线性下将。</li><li> 使用mmap加速内核与用户空间的消息传递。</li><li> epoll拥有更加简单的API。</li></ul><h4 id="4、信号驱动I-O模型"><a href="#4、信号驱动I-O模型" class="headerlink" title="4、信号驱动I/O模型"></a>4、信号驱动I/O模型</h4><p> 在这种模型下，我们首先开启套接字的信号驱动式I/O功能，并通过sigaction系统调用安装一个信号处理函数。改系统调用将立即返回，我们的进程继续工作，也就是说他没有被阻塞。当数据报准备好读取时，内核就为该进程产生一个SIGIO信号。我们随后就可以在信号处理函数中调用read读取数据报，并通知主循环数据已经准备好待处理，也可以立即通知主循环，让它读取数据报。</p><p>图示：</p><p> <img src="http://blog.anxpp.com/usr/uploads/2016/05/3322063871.png" alt="04"></p><p>无论如何处理SIGIO信号，这种模型的优势在于等待数据报到达期间进程不被阻塞。主循环可以继续执行，只要等到来自信号处理函数的通知：既可以是数据已准备好被处理，也可以是数据报已准备好被读取。</p><h4 id="5、异步I-O"><a href="#5、异步I-O" class="headerlink" title="5、异步I/O"></a>5、异步I/O</h4><p>相对于同步I/O，异步I/O不是顺序执行。用户进程进行<code>aio_read</code>系统调用之后，无论内核数据是否准备好，都会直接返回给用户进程，然后用户态进程可以去做别的事情。等到socket数据准备好了，内核直接复制数据给进程，然后从内核向进程发送通知。I/O两个阶段，进程都是非阻塞的。</p><p>图示：</p><p> <img src="http://blog.anxpp.com/usr/uploads/2016/05/4059852491.png" alt="05"></p>]]></content>
<summary type="html">
<p>进行后台开发时，经常需要进行数据传输，其数据传输归根结底还是Linux 的I/O 操作，今天就来讲下 Linux 的I/O 模型。</p>
</summary>
<category term="Linux" scheme="https://yangchenglong11.github.io/categories/Linux/"/>
<category term="Linux" scheme="https://yangchenglong11.github.io/tags/Linux/"/>
</entry>
<entry>
<title>mgo 断线重连</title>
<link href="https://yangchenglong11.github.io/2017/04/03/mgo-%E6%96%AD%E7%BA%BF%E9%87%8D%E8%BF%9E/"/>
<id>https://yangchenglong11.github.io/2017/04/03/mgo-断线重连/</id>
<published>2017-04-03T11:21:25.000Z</published>
<updated>2017-10-28T13:41:51.987Z</updated>
<content type="html"><![CDATA[<p>Mongo 使用起来很方便，但有时因为网络抖动导致连接断开，而后台服务却不知道，这样就会导致服务出现问题，今天来看下怎样实现 mgo 断线自动重连。<br><a id="more"></a></p><p>使用 mgo 时，当远端的 session 挂掉, 其实底层已经做了重连的机制，但是没有通知上层的 session 来更新当前的 session，而致使当前的 session 不可以用。所以为了实现断线重连，可以使用下面两种方法：</p><ul><li>不只是使用一个 master session ，而是通过调用 session.Copy() 来创建多个session。每当需要处理独立请求时，通过调用 Copy() 来为每个请求创建一个独立的 session。每当需要一个新 session 时，都会进行 Refresh , 然后在主 session 上进行复制，接着使用该 session 完成操作，使用后将其关闭。当多个任务同时进行时，会创建多个连接，mgo 对连接数作了限制，默认配置的连接数上限是4096，当然一般情况下是足够使用的，当然为了实现更高要求可以将其设置的更大些。此外，调用 Copy() 是非常 cheap 的，不用担心因调用而产生的资源消耗；</li><li>仍是用一个 session ，但在每次调用 session 时，调用 Session.Refresh() 方法，该方法会更新 session 状态，这样就可以和远端 session 状态保持一致。(如果觉得每次都要 refresh 有些麻烦，也可以在调用函数出错时，对错误进行判断，因mgo断线而抛出的错误是 io.EOF，如果错误类型为 io.EOF，则调用 Session.Refresh() 方法重新进行操作，但mgo断线也可能抛出其他错误，采用上面的方法更为保险)。</li></ul>]]></content>
<summary type="html">
<p>Mongo 使用起来很方便，但有时因为网络抖动导致连接断开，而后台服务却不知道，这样就会导致服务出现问题，今天来看下怎样实现 mgo 断线自动重连。<br></p>
</summary>
<category term="MongoDB" scheme="https://yangchenglong11.github.io/categories/MongoDB/"/>
<category term="Golang" scheme="https://yangchenglong11.github.io/tags/Golang/"/>
<category term="MongoDB" scheme="https://yangchenglong11.github.io/tags/MongoDB/"/>
</entry>
<entry>
<title>Mongodb 数据导出/导入</title>
<link href="https://yangchenglong11.github.io/2017/03/17/Mongodb%E6%95%B0%E6%8D%AE%E5%AF%BC%E5%87%BA-%E5%AF%BC%E5%85%A5/"/>
<id>https://yangchenglong11.github.io/2017/03/17/Mongodb数据导出-导入/</id>
<published>2017-03-17T06:43:34.000Z</published>
<updated>2017-10-28T13:41:38.647Z</updated>
<content type="html"><![CDATA[<p>因为工作环境的切换或是进行备份，我们可能需要将 Mongo 中的数据进行转移，今天我们来看下 Mongo 怎样进行数据导出/导入。<br><a id="more"></a><br>一、导出工具mongoexport**</p><p>Mongodb中的mongoexport工具可以把一个collection导出成JSON格式或CSV格式的文件。可以通过参数指定导出的数据项，也可以根据指定的条件导出数据。mongoexport具体用法如下所示：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div></pre></td><td class="code"><pre><div class="line">[root@localhost mongodb]# ./bin/mongoexport --help</div><div class="line">Export MongoDB data to CSV, TSV or JSON files.</div><div class="line"></div><div class="line">options:</div><div class="line"> --help produce help message</div><div class="line"> -v [ --verbose ] be more verbose (include multiple times for more </div><div class="line"> verbosity e.g. -vvvvv)</div><div class="line"> --version print the program's version and exit</div><div class="line"> -h [ --host ] arg mongo host to connect to ( <set name>/s1,s2 for </div><div class="line"> sets)</div><div class="line"> --port arg server port. Can also use --host hostname:port</div><div class="line"> --ipv6 enable IPv6 support (disabled by default)</div><div class="line"> -u [ --username ] arg username</div><div class="line"> -p [ --password ] arg password</div><div class="line"> --dbpath arg directly access mongod database files in the given </div><div class="line"> path, instead of connecting to a mongod server - </div><div class="line"> needs to lock the data directory, so cannot be used</div><div class="line"> if a mongod is currently accessing the same path</div><div class="line"> --directoryperdb if dbpath specified, each db is in a separate </div><div class="line"> directory</div><div class="line"> --journal enable journaling</div><div class="line"> -d [ --db ] arg database to use</div><div class="line"> -c [ --collection ] arg collection to use (some commands)</div><div class="line"> -f [ --fields ] arg comma separated list of field names e.g. -f </div><div class="line"> name,age</div><div class="line"> --fieldFile arg file with fields names - 1 per line</div><div class="line"> -q [ --query ] arg query filter, as a JSON string</div><div class="line"> --csv export to csv instead of json</div><div class="line"> -o [ --out ] arg output file; if not specified, stdout is used</div><div class="line"> --jsonArray output to a json array rather than one object per </div><div class="line"> line</div><div class="line"> -k [ --slaveOk ] arg (=1) use secondaries for export if available, default </div><div class="line"> true</div></pre></td></tr></table></figure><p>参数说明：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line">-h:指明数据库宿主机的IP</div><div class="line"></div><div class="line">-u:指明数据库的用户名</div><div class="line"></div><div class="line">-p:指明数据库的密码</div><div class="line"></div><div class="line">-d:指明数据库的名字</div><div class="line"></div><div class="line">-c:指明collection的名字</div><div class="line"></div><div class="line">-f:指明要导出那些列</div><div class="line"></div><div class="line">-o:指明到要导出的文件名</div><div class="line"></div><div class="line">-q:指明导出数据的过滤条件</div></pre></td></tr></table></figure><p>实例：test库中存在着一个students集合，集合中数据如下：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="meta">></span><span class="bash"> db.students.find()</span></div><div class="line">{ "_id" : ObjectId("5031143350f2481577ea81e5"), "classid" : 1, "age" : 20, "name" : "kobe" }</div><div class="line">{ "_id" : ObjectId("5031144a50f2481577ea81e6"), "classid" : 1, "age" : 23, "name" : "nash" }</div><div class="line">{ "_id" : ObjectId("5031145a50f2481577ea81e7"), "classid" : 2, "age" : 18, "name" : "james" }</div><div class="line">{ "_id" : ObjectId("5031146a50f2481577ea81e8"), "classid" : 2, "age" : 19, "name" : "wade" }</div><div class="line">{ "_id" : ObjectId("5031147450f2481577ea81e9"), "classid" : 2, "age" : 19, "name" : "bosh" }</div><div class="line">{ "_id" : ObjectId("5031148650f2481577ea81ea"), "classid" : 2, "age" : 25, "name" : "allen" }</div><div class="line">{ "_id" : ObjectId("5031149b50f2481577ea81eb"), "classid" : 1, "age" : 19, "name" : "howard" }</div><div class="line">{ "_id" : ObjectId("503114a750f2481577ea81ec"), "classid" : 1, "age" : 22, "name" : "paul" }</div><div class="line">{ "_id" : ObjectId("503114cd50f2481577ea81ed"), "classid" : 2, "age" : 24, "name" : "shane" }</div></pre></td></tr></table></figure><p>由上可以看出文档中存在着3个字段：classid、age、name</p><p>1.直接导出数据到文件中</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">[root@localhost mongodb]# ./bin/mongoexport -d test -c students -o students.dat</div><div class="line">connected to: 127.0.0.1</div><div class="line">exported 9 records</div></pre></td></tr></table></figure><p>命令执行完后使用ll命令查看，发现目录下生成了一个students.dat的文件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">-rw-r--r-- 1 root root 869 Aug 21 00:05 students.dat</div></pre></td></tr></table></figure><p>查看该文件信息，具体信息如下：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line">[root@localhost mongodb]# cat students.dat </div><div class="line">{ "_id" : { "$oid" : "5031143350f2481577ea81e5" }, "classid" : 1, "age" : 20, "name" : "kobe" }</div><div class="line">{ "_id" : { "$oid" : "5031144a50f2481577ea81e6" }, "classid" : 1, "age" : 23, "name" : "nash" }</div><div class="line">{ "_id" : { "$oid" : "5031145a50f2481577ea81e7" }, "classid" : 2, "age" : 18, "name" : "james" }</div><div class="line">{ "_id" : { "$oid" : "5031146a50f2481577ea81e8" }, "classid" : 2, "age" : 19, "name" : "wade" }</div><div class="line">{ "_id" : { "$oid" : "5031147450f2481577ea81e9" }, "classid" : 2, "age" : 19, "name" : "bosh" }</div><div class="line">{ "_id" : { "$oid" : "5031148650f2481577ea81ea" }, "classid" : 2, "age" : 25, "name" : "allen" }</div><div class="line">{ "_id" : { "$oid" : "5031149b50f2481577ea81eb" }, "classid" : 1, "age" : 19, "name" : "howard" }</div><div class="line">{ "_id" : { "$oid" : "503114a750f2481577ea81ec" }, "classid" : 1, "age" : 22, "name" : "paul" }</div><div class="line">{ "_id" : { "$oid" : "503114cd50f2481577ea81ed" }, "classid" : 2, "age" : 24, "name" : "shane" }</div></pre></td></tr></table></figure><p>参数说明：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">-d:指明使用的库，本例中为test</div><div class="line"></div><div class="line">-c:指明要导出的集合，本例中为students</div><div class="line"></div><div class="line">-o:指明要导出的文件名，本例中为students.dat</div></pre></td></tr></table></figure><p>从上面的结果可以看出，我们在导出数据时没有显示指定导出样式 ，默认导出了JSON格式的数据。如果我们需要导出CSV格式的数据，则需要使用–csv参数，具体如下所示：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line">[root@localhost mongodb]# ./bin/mongoexport -d test -c students --csv -f classid,name,age -o students_csv.dat</div><div class="line">connected to: 127.0.0.1</div><div class="line">exported 9 records</div><div class="line">[root@localhost mongodb]# cat students_csv.dat </div><div class="line">classid,name,age</div><div class="line">1.0,"kobe",20.0</div><div class="line">1.0,"nash",23.0</div><div class="line">2.0,"james",18.0</div><div class="line">2.0,"wade",19.0</div><div class="line">2.0,"bosh",19.0</div><div class="line">2.0,"allen",25.0</div><div class="line">1.0,"howard",19.0</div><div class="line">1.0,"paul",22.0</div><div class="line">2.0,"shane",24.0</div><div class="line">[root@localhost mongodb]#</div></pre></td></tr></table></figure><p>参数说明：</p><p>-csv：指明要导出为csv格式</p><p>-f：指明需要导出classid、name、age这3列的数据</p><p>由上面结果可以看出，mongoexport成功地将数据根据csv格式导出到了students_csv.dat文件中。</p><p><strong>二、导入工具mongoimport</strong></p><p>Mongodb中的mongoimport工具可以把一个特定格式文件中的内容导入到指定的collection中。该工具可以导入JSON格式数据，也可以导入CSV格式数据。具体使用如下所示：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div></pre></td><td class="code"><pre><div class="line">[root@localhost mongodb]# ./bin/mongoimport --help</div><div class="line">options:</div><div class="line"> --help produce help message</div><div class="line"> -v [ --verbose ] be more verbose (include multiple times for more </div><div class="line"> verbosity e.g. -vvvvv)</div><div class="line"> --version print the program's version and exit</div><div class="line"> -h [ --host ] arg mongo host to connect to ( <set name>/s1,s2 for sets)</div><div class="line"> --port arg server port. Can also use --host hostname:port</div><div class="line"> --ipv6 enable IPv6 support (disabled by default)</div><div class="line"> -u [ --username ] arg username</div><div class="line"> -p [ --password ] arg password</div><div class="line"> --dbpath arg directly access mongod database files in the given </div><div class="line"> path, instead of connecting to a mongod server - </div><div class="line"> needs to lock the data directory, so cannot be used </div><div class="line"> if a mongod is currently accessing the same path</div><div class="line"> --directoryperdb if dbpath specified, each db is in a separate </div><div class="line"> directory</div><div class="line"> --journal enable journaling</div><div class="line"> -d [ --db ] arg database to use</div><div class="line"> -c [ --collection ] arg collection to use (some commands)</div><div class="line"> -f [ --fields ] arg comma separated list of field names e.g. -f name,age</div><div class="line"> --fieldFile arg file with fields names - 1 per line</div><div class="line"> --ignoreBlanks if given, empty fields in csv and tsv will be ignored</div><div class="line"> --type arg type of file to import. default: json (json,csv,tsv)</div><div class="line"> --file arg file to import from; if not specified stdin is used</div><div class="line"> --drop drop collection first </div><div class="line"> --headerline CSV,TSV only - use first line as headers</div><div class="line"> --upsert insert or update objects that already exist</div><div class="line"> --upsertFields arg comma-separated fields for the query part of the </div><div class="line"> upsert. You should make sure this is indexed</div><div class="line"> --stopOnError stop importing at first error rather than continuing</div><div class="line"> --jsonArray load a json array, not one item per line. Currently </div><div class="line"> limited to 4MB.</div></pre></td></tr></table></figure><p>参数说明：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line">-h:指明数据库宿主机的IP</div><div class="line"></div><div class="line">-u:指明数据库的用户名</div><div class="line"></div><div class="line">-p:指明数据库的密码</div><div class="line"></div><div class="line">-d:指明数据库的名字</div><div class="line"></div><div class="line">-c:指明collection的名字</div><div class="line"></div><div class="line">-f:指明要导入那些列</div></pre></td></tr></table></figure><p>示例：先删除students中的数据，并验证</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="meta">></span><span class="bash"> db.students.remove()</span></div><div class="line"><span class="meta">></span><span class="bash"> db.students.find()</span></div><div class="line"><span class="meta">></span><span class="bash"></span></div></pre></td></tr></table></figure><p>然后再导入上面导出的students.dat文件中的内容</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">[root@localhost mongodb]# ./bin/mongoimport -d test -c students students.dat </div><div class="line">connected to: 127.0.0.1</div><div class="line">imported 9 objects</div><div class="line">[root@localhost mongodb]#</div></pre></td></tr></table></figure><p>参数说明：</p><p>-d:指明数据库名，本例中为test</p><p>-c:指明collection名，本例中为students</p><p>students.dat：导入的文件名</p><p>查询students集合中的数据</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="meta">></span><span class="bash"> db.students.find()</span></div><div class="line">{ "_id" : ObjectId("5031143350f2481577ea81e5"), "classid" : 1, "age" : 20, "name" : "kobe" }</div><div class="line">{ "_id" : ObjectId("5031144a50f2481577ea81e6"), "classid" : 1, "age" : 23, "name" : "nash" }</div><div class="line">{ "_id" : ObjectId("5031145a50f2481577ea81e7"), "classid" : 2, "age" : 18, "name" : "james" }</div><div class="line">{ "_id" : ObjectId("5031146a50f2481577ea81e8"), "classid" : 2, "age" : 19, "name" : "wade" }</div><div class="line">{ "_id" : ObjectId("5031147450f2481577ea81e9"), "classid" : 2, "age" : 19, "name" : "bosh" }</div><div class="line">{ "_id" : ObjectId("5031148650f2481577ea81ea"), "classid" : 2, "age" : 25, "name" : "allen" }</div><div class="line">{ "_id" : ObjectId("5031149b50f2481577ea81eb"), "classid" : 1, "age" : 19, "name" : "howard" }</div><div class="line">{ "_id" : ObjectId("503114a750f2481577ea81ec"), "classid" : 1, "age" : 22, "name" : "paul" }</div><div class="line">{ "_id" : ObjectId("503114cd50f2481577ea81ed"), "classid" : 2, "age" : 24, "name" : "shane" }</div><div class="line"><span class="meta">></span><span class="bash"></span></div></pre></td></tr></table></figure><p>证明数据导入成功</p><p>上面演示的是导入JSON格式的文件中的内容，如果要导入CSV格式文件中的内容，则需要通过–type参数指定导入格式，具体如下所示：</p><p>先删除数据</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="meta">></span><span class="bash"> db.students.remove()</span></div><div class="line"><span class="meta">></span><span class="bash"> db.students.find()</span></div><div class="line"><span class="meta">></span><span class="bash"></span></div></pre></td></tr></table></figure><p>再导入之前导出的students_csv.dat文件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">[root@localhost mongodb]# ./bin/mongoimport -d test -c students --type csv --headerline --file students_csv.dat </div><div class="line">connected to: 127.0.0.1</div><div class="line">imported 10 objects</div><div class="line">[root@localhost mongodb]#</div></pre></td></tr></table></figure><p>参数说明：</p><p>-type:指明要导入的文件格式</p><p>-headerline:指明第一行是列名，不需要导入</p><p>-file：指明要导入的文件</p><p>查询students集合，验证导入是否成功：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="meta">></span><span class="bash"> db.students.find()</span></div><div class="line">{ "_id" : ObjectId("503266029355c632cd118ad8"), "classid" : 1, "name" : "kobe", "age" : 20 }</div><div class="line">{ "_id" : ObjectId("503266029355c632cd118ad9"), "classid" : 1, "name" : "nash", "age" : 23 }</div><div class="line">{ "_id" : ObjectId("503266029355c632cd118ada"), "classid" : 2, "name" : "james", "age" : 18 }</div><div class="line">{ "_id" : ObjectId("503266029355c632cd118adb"), "classid" : 2, "name" : "wade", "age" : 19 }</div><div class="line">{ "_id" : ObjectId("503266029355c632cd118adc"), "classid" : 2, "name" : "bosh", "age" : 19 }</div><div class="line">{ "_id" : ObjectId("503266029355c632cd118add"), "classid" : 2, "name" : "allen", "age" : 25 }</div><div class="line">{ "_id" : ObjectId("503266029355c632cd118ade"), "classid" : 1, "name" : "howard", "age" : 19 }</div><div class="line">{ "_id" : ObjectId("503266029355c632cd118adf"), "classid" : 1, "name" : "paul", "age" : 22 }</div><div class="line">{ "_id" : ObjectId("503266029355c632cd118ae0"), "classid" : 2, "name" : "shane", "age" : 24 }</div><div class="line"><span class="meta">></span><span class="bash"></span></div></pre></td></tr></table></figure><p>说明已经导入成功 </p>]]></content>
<summary type="html">
<p>因为工作环境的切换或是进行备份，我们可能需要将 Mongo 中的数据进行转移，今天我们来看下 Mongo 怎样进行数据导出/导入。<br></p>
</summary>
<category term="MongoDB" scheme="https://yangchenglong11.github.io/categories/MongoDB/"/>
<category term="MongoDB" scheme="https://yangchenglong11.github.io/tags/MongoDB/"/>
</entry>
</feed>