content.json

{"meta":{"title":"跛足的登山者","subtitle":null,"description":null,"author":"xxydliuy","url":"http://www.liuyong520.cn","root":"/"},"pages":[{"title":"toc","date":"2019-04-23T09:31:21.000Z","updated":"2019-06-11T09:37:15.080Z","comments":true,"path":"java.html","permalink":"http://www.liuyong520.cn/java.html","excerpt":"","text":"我媳妇是个老色鬼"},{"title":"书单","date":"2019-06-11T09:37:15.192Z","updated":"2019-06-11T09:37:15.192Z","comments":false,"path":"books/index.html","permalink":"http://www.liuyong520.cn/books/index.html","excerpt":"","text":""},{"title":"关于","date":"2019-06-11T09:37:15.076Z","updated":"2019-06-11T09:37:15.076Z","comments":false,"path":"about/index.html","permalink":"http://www.liuyong520.cn/about/index.html","excerpt":"","text":"欢迎访问我的新博客，会陆续把之前的cnblog上的博客迁移过来，也会不定期更新！1234567891011121314151617181920212223242526272829303132/** * _ooOoo_ * o8888888o * 88\" . \"88 * (| -_- |) * O\\ = /O * ____/`---'\\____ * .' \\\\| |// `. * / \\\\||| : |||// \\ * / _||||| -:- |||||- \\ * | | \\\\\\ - /// | | * | \\_| ''\\---/'' | | * \\ .-\\__ `-` ___/-. / * ___`. .' /--.--\\ `. . __ * .\"\" '< `.___\\_<|>_/___.' >'\"\". * | | : `- \\`.;`\\ _ /`;.`/ - ` : | | * \\ \\ `-. \\_ __\\ /__ _/ .-` / / * ======`-.____`-.___\\_____/___.-`____.-'====== * `=---=' * ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * 佛祖保佑 永无BUG * 佛曰: * 写字楼里写字间，写字间里程序员； * 程序人员写程序，又拿程序换酒钱。 * 酒醒只在网上坐，酒醉还来网下眠； * 酒醉酒醒日复日，网上网下年复年。 * 但愿老死电脑间，不愿鞠躬老板前； * 奔驰宝马贵者趣，公交自行程序员。 * 别人笑我忒疯癫，我笑自己命太贱； * 不见满街漂亮妹，哪个归得程序员？*/"},{"title":"分类","date":"2019-06-11T09:37:15.042Z","updated":"2019-06-11T09:37:15.042Z","comments":false,"path":"categories/index.html","permalink":"http://www.liuyong520.cn/categories/index.html","excerpt":"","text":""},{"title":"标签","date":"2019-06-11T09:37:15.091Z","updated":"2019-06-11T09:37:15.091Z","comments":false,"path":"tags/index.html","permalink":"http://www.liuyong520.cn/tags/index.html","excerpt":"","text":""},{"title":"Repositories","date":"2019-06-11T09:37:16.563Z","updated":"2019-06-11T09:37:16.563Z","comments":false,"path":"repository/index.html","permalink":"http://www.liuyong520.cn/repository/index.html","excerpt":"","text":"qwqwwwwww"},{"title":"友情链接","date":"2019-06-11T09:37:15.142Z","updated":"2019-06-11T09:37:15.142Z","comments":true,"path":"links/index.html","permalink":"http://www.liuyong520.cn/links/index.html","excerpt":"","text":""}],"posts":[{"title":"基于docker的mysql主从同步","slug":"mysql-master-slaver","date":"2019-06-11T08:21:46.000Z","updated":"2019-06-11T09:37:16.621Z","comments":true,"path":"2019/06/11/mysql-master-slaver/","link":"","permalink":"http://www.liuyong520.cn/2019/06/11/mysql-master-slaver/","excerpt":"","text":"环境要求docker环境，mysql:5.7的镜像#1.创建挂载目录123456789--mysql --master --data --conf --my.cnf --slaver --data --conf --my.cnf2、主从配置文件Master: my.cnf12345678910111213[mysqld]server_id=1log-bin= mysql-binbinlog_format=ROWrelay-log=mysql-relay-logreplicate-ignore-db=mysqlreplicate-ignore-db=sysreplicate-ignore-db=information_schemareplicate-ignore-db=performance_schema!includedir /etc/mysql/conf.d/!includedir /etc/mysql/mysql.conf.d/Slaver: my.cnf123456789101112server_id=1log-bin= mysql-binbinlog_format=ROWrelay-log=mysql-relay-logreplicate-ignore-db=mysqlreplicate-ignore-db=sysreplicate-ignore-db=information_schemareplicate-ignore-db=performance_schema!includedir /etc/mysql/conf.d/!includedir /etc/mysql/mysql.conf.d/说明： log-bin ：需要启用二进制日志 server_id : 用于标识不同的数据库服务器，而且唯一binlog-do-db : 需要记录到二进制日志的数据库 binlog-ignore-db : 忽略记录二进制日志的数据库 auto-increment-offset :该服务器自增列的初始值 auto-increment-increment :该服务器自增列增量replicate-do-db ：指定复制的数据库 replicate-ignore-db ：不复制的数据库 relay_log ：从库的中继日志，主库日志写到中继日志，中继日志再重做到从库 log-slave-updates ：该从库是否写入二进制日志，如果需要成为多主则可启用。只读可以不需要如果为多主的话注意设置 auto-increment-offset 和 auto-increment-increment 如上面为双主的设置： 服务器 152 自增列显示为：1,3,5,7,……（offset=1，increment=2） 服务器 153 自增列显示为：2,4,6,8,……（offset=2，increment=2）1234567891011121314151617//获取基础镜像docker pull mysql //创建并启动主从容器;//masterdocker run --name mastermysql -d -p 3307:3306 -e MYSQL_ROOT_PASSWORD=anech -v d:/docker/mysql/master/data:/var/lib/mysql -v d:/docker/mysql/master/conf/my.cnf:/etc/mysql/my.cnf mysql //slaverdocker run --name slavermysql -d -p 3308:3306 -e MYSQL_ROOT_PASSWORD=anech -v d:/docker/mysql/slaver/data:/var/lib/mysql -v d:/docker/mysql/slaver/conf/my.cnf:/etc/mysql/my.cnf mysql这里为了方便查看数据，把Docker的端口都与本机进行了映射,对应的本地master/data文件夹和slaver/data文件夹下也能看到同步的数据库文件3、Master和Slaver设置1234567891011121314151617181920212223242526//进入master容器docker exec -it mastermysql bash //启动mysql命令，刚在创建窗口时我们把密码设置为：anechmysql -u root -p //创建一个用户来同步数据GRANT REPLICATION SLAVE ON *.* to 'backup'@'%' identified by '123456';//这里表示创建一个slaver同步账号backup，允许访问的IP地址为%，%表示通配符//例如：192.168.0.%表示192.168.0.0-192.168.0.255的slaver都可以用backup用户登陆到master上//刷新一下权限flush privileges；//查看状态，记住File、Position的值，在Slaver中将用到show master status;1234567891011121314//进入slaver容器docker exec -it slavermysql bash//启动mysql命令，刚在创建窗口时我们把密码设置为：anechmysql -u root -p//设置主库链接change master to master_host='172.17.0.2',master_user='backup',master_password='123456',master_log_file='mysql-bin.000001',master_log_pos=0,master_port=3306;//启动从库同步start slave;//查看状态show slave status\\G;说明：master_host：主库地址master_user：主库创建的同步账号master_password：主库创建的同步密码master_log_file：主库产生的日志master_log_pos：主库日志记录偏移量master_port：主库使用的端口，默认为33064 主从同步异常解决办法：经常遇到主从同步因为异常导致同步点被破坏，解决方案最常见的两种解决办法是：一种是跳过异常，继续同步，第二种就是将主库数据进行备份导出，然后到从库进行导入，然后重新构造同步点。","categories":[{"name":"mysql","slug":"mysql","permalink":"http://www.liuyong520.cn/categories/mysql/"}],"tags":[{"name":"mysql","slug":"mysql","permalink":"http://www.liuyong520.cn/tags/mysql/"}]},{"title":"基于docker的mysql主主同步","slug":"mysql-master-master","date":"2019-06-11T08:21:46.000Z","updated":"2019-06-11T09:37:15.540Z","comments":true,"path":"2019/06/11/mysql-master-master/","link":"","permalink":"http://www.liuyong520.cn/2019/06/11/mysql-master-master/","excerpt":"","text":"环境要求docker环境，mysql:5.7的镜像#1.创建挂载目录123456789--mysql --mone --data --conf --my.cnf --mtwo --data --conf --my.cnf2、主主配置Mone: my.cnf123456789101112131415161718[mysqld]server_id = 1log-bin= mysql-binreplicate-ignore-db=mysqlreplicate-ignore-db=sysreplicate-ignore-db=information_schemareplicate-ignore-db=performance_schemaread-only=0relay_log=mysql-relay-binlog-slave-updates=onauto-increment-offset=1auto-increment-increment=2!includedir /etc/mysql/conf.d/!includedir /etc/mysql/mysql.conf.d/Mtwo: my.cnf1234567891011121314151617[mysqld]server_id = 2log-bin= mysql-binreplicate-ignore-db=mysqlreplicate-ignore-db=sysreplicate-ignore-db=information_schemareplicate-ignore-db=performance_schemaread-only=0relay_log=mysql-relay-binlog-slave-updates=onauto-increment-offset=2auto-increment-increment=2!includedir /etc/mysql/conf.d/!includedir /etc/mysql/mysql.conf.d/说明： log-bin ：需要启用二进制日志 server_id : 用于标识不同的数据库服务器，而且唯一binlog-do-db : 需要记录到二进制日志的数据库 binlog-ignore-db : 忽略记录二进制日志的数据库 auto-increment-offset :该服务器自增列的初始值 auto-increment-increment :该服务器自增列增量replicate-do-db ：指定复制的数据库 replicate-ignore-db ：不复制的数据库 relay_log ：从库的中继日志，主库日志写到中继日志，中继日志再重做到从库 log-slave-updates ：该从库是否写入二进制日志，如果需要成为多主则可启用。只读可以不需要如果为多主的话注意设置 auto-increment-offset 和 auto-increment-increment 如上面为双主的设置： 服务器 152 自增列显示为：1,3,5,7,……（offset=1，increment=2） 服务器 153 自增列显示为：2,4,6,8,……（offset=2，increment=2）3.启动容器123456//创建并启动主从容器;//monedocker run --name monemysql -d -p 3317:3306 -e MYSQL_ROOT_PASSWORD=root -v ~/test/mysql_test1/mone/data:/var/lib/mysql -v ~/test/mysql_test1/mone/conf/my.cnf:/etc/mysql/my.cnf mysql:5.7//mtwodocker run --name mtwomysql -d -p 3318:3306 -e MYSQL_ROOT_PASSWORD=root -v ~/test/mysql_test1/mtwo/data:/var/lib/mysql -v ~/test/mysql_test1/mtwo/conf/my.cnf:/etc/mysql/my.cnf mysql:5.74.双主配置mone->mtwo123456789101112131415//进入mone容器docker exec -it monemysql mysql -u root -p //启动mysql命令，刚在创建窗口时我们把密码设置为：root //创建一个用户来同步数据//这里表示创建一个slave同步账号slave，允许访问的IP地址为%，%表示通配符GRANT REPLICATION SLAVE ON *.* to 'slave'@'%' identified by '123456';//刷新权限flush privileges；//查看状态，记住File、Position的值，在mtwo中将用到show master status;5.查看容器IP123docker inspect --format='{{.NetworkSettings.IPAddress}}' monemysql我这里查到是172.17.0.26.进入mtwo12345678910111213//进入mtwo容器docker exec -it mtwomysql mysql -u root -p //启动mysql命令，刚在创建窗口时我们把密码设置为：root//设置主库链接，master_host即为容器IP，master_log_file和master_log_pos即为在mone容器中，通过show master status查出来的值；change master to master_host='172.17.0.2',master_user='slave',master_password='123456',master_log_file='mysql-bin.000004',master_log_pos=154,master_port=3306;//启动同步start slave ; //查看状态show slave status\\G;到此 mone->mtwo 的同步已经完成7.配置mtwo->mone的同步1234567891011121314//进入mtwo容器docker exec -it mtwomysql mysql -u root -p //启动mysql命令，刚在创建窗口时我们把密码设置为：root//创建一个用户来同步数据//这里表示创建一个slave同步账号slave，允许访问的IP地址为%，%表示通配符GRANT REPLICATION SLAVE ON *.* to 'slave'@'%' identified by '123456';//刷新权限flush privileges； //查看状态show master status;8.查看mtwo的ip12docker inspect --format='{{.NetworkSettings.IPAddress}}' mtwomysql我这里是172.17.0.39.进入mone中12345678910111213/进入mone容器docker exec -it monemysql mysql -u root -p //启动mysql命令，刚在创建窗口时我们把密码设置为：root//设置主库链接，master_host即为容器IP，master_log_file和master_log_pos即为在mone容器中，通过show master status查出来的值；change master to master_host='172.17.0.3',master_user='slave',master_password='123456',master_log_file='mysql-bin.000004',master_log_pos=154,master_port=3306;//启动同步start slave ; //查看状态show slave status\\G;到此mtwo->mone的同步就配置完了总结：其实主主同步其实就是做了两次主从同步而已。A主B从，B主A从","categories":[{"name":"mysql","slug":"mysql","permalink":"http://www.liuyong520.cn/categories/mysql/"}],"tags":[{"name":"mysql","slug":"mysql","permalink":"http://www.liuyong520.cn/tags/mysql/"}]},{"title":"基于docker的mysql主主同步的跨服务器迁移","slug":"mysql-qianyi","date":"2019-06-11T08:21:46.000Z","updated":"2019-06-11T12:02:34.991Z","comments":true,"path":"2019/06/11/mysql-qianyi/","link":"","permalink":"http://www.liuyong520.cn/2019/06/11/mysql-qianyi/","excerpt":"","text":"为什么要做迁移？1.当前服务器性能瓶颈。2.其他服务器也有同样的数据要求如何迁移？1.导出docker容器查看当前容器状态1234docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESf8fcf4acacd0 mysql:5.7 "docker-entrypoint.s…" 5 hours ago Up 5 hours 33060/tcp, 0.0.0.0:3307->3306/tcp mysqlmasterbd7a5f734bfb mysql:5.7 "docker-entrypoint.s…" 5 hours ago Up 5 hours 33060/tcp, 0.0.0.0:3308->3306/tcp mysqlslaver然后导出到本地12docker export bd7a5f734bfb >/home/mysql/mysqlslaver.tardocker export f8fcf4acacd0 >/home/mysql/mysqlmaster.tar2.同步文件到其他服务器，可以手动上传，也可以使用sync同步命令。我这里的mysql的容器会挂载如下的目录12345678910/data/docker/└── mysql ├── master │ ├── conf │ │ └── my.cnf │ └── data |-- slaver |-- conf | |__my.cnf |__ data所以我这里也会拷贝这些数据挂载目录。3.import 容器12cat /home/mysql/mysqlmaster | docker import - mysqlmastercat /home/mysql/mysqlslaver | docker import - mysqlslaver查看镜像1docker images启动容器12345//启动从库docker run --name mysqlslaver -d -p 3308:3306 -e MYSQL_ROOT_PASSWORD=123456 -v /data/docker/mysql/slaver/data:/var/lib/mysql -v /data/docker/mysql/slaver/conf/my.cnf:/etc/mysql/my.cnf mysqlslaver /entrypoint.sh mysqld//启动主库docker run --name mysqlmaster -d -p 3307:3306 -e MYSQL_ROOT_PASSWORD=123456 -v /data/docker/mysql/master/data:/var/lib/mysql -v /data/docker/mysql/master/conf/my.cnf:/etc/mysql/my.cnf mysqlmaster /entrypoint.sh mysqld然后查看一下1234docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES30cf2e504186 mysqlmaster "/entrypoint.sh mysq…" 30 minutes ago Up 30 minutes 0.0.0.0:3307->3306/tcp mysqlmaster46fac352e00e mysqlslaver "/entrypoint.sh mysq…" 31 minutes ago Up 31 minutes 0.0.0.0:3308->3306/tcp mysqlslaver验证一下1234567//进入容器docker exec -it mysqlmaster bash//进入mysqlmysql -uroot -p123456//因为有主从看一下同步是否正常show slave status\\G这样就把数据库容器和数据迁移到了另外一台服务器上了。","categories":[{"name":"mysql","slug":"mysql","permalink":"http://www.liuyong520.cn/categories/mysql/"}],"tags":[{"name":"mysql","slug":"mysql","permalink":"http://www.liuyong520.cn/tags/mysql/"}]},{"title":"JVM的相关知识","slug":"jvm","date":"2019-06-07T10:13:05.000Z","updated":"2019-06-11T09:37:16.564Z","comments":true,"path":"2019/06/07/jvm/","link":"","permalink":"http://www.liuyong520.cn/2019/06/07/jvm/","excerpt":"","text":"关于JVM介绍网上有很多资料，我这里写的仅仅是我对JVM的理解。是不是只有java编译器才能编译成字节码.class文件?明显是不是的，jruby/scale/groovy/都能编译成.class 文件。Class文件的结构123456789101112package com.climber.jvm;public class Helloword { public static int num = 3; public static void main(String[] args) { Helloword helloword = new Helloword(); num ++; helloword.sum(num,1); } public int sum(int a,int b){ return a + b; }}编译运行得到class文件如图：使用javap -verbose Helloword.class > ./data.txt打开 data.txt123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118Classfile /Users/xxydliuyss/IdeaProjects/jvm/target/classes/com/climber/jvm/Helloword.class Last modified 2019-6-8; size 660 bytes MD5 checksum 3e8014d1d19cb974b06673e17c922039 Compiled from "Helloword.java"public class com.climber.jvm.Helloword minor version: 0 major version: 49 flags: ACC_PUBLIC, ACC_SUPERConstant pool: #1 = Methodref #6.#28 // java/lang/Object."<init>":()V #2 = Class #29 // com/climber/jvm/Helloword #3 = Methodref #2.#28 // com/climber/jvm/Helloword."<init>":()V #4 = Fieldref #2.#30 // com/climber/jvm/Helloword.num:I #5 = Methodref #2.#31 // com/climber/jvm/Helloword.sum:(II)I #6 = Class #32 // java/lang/Object #7 = Utf8 num #8 = Utf8 I #9 = Utf8 <init> #10 = Utf8 ()V #11 = Utf8 Code #12 = Utf8 LineNumberTable #13 = Utf8 LocalVariableTable #14 = Utf8 this #15 = Utf8 Lcom/climber/jvm/Helloword; #16 = Utf8 main #17 = Utf8 ([Ljava/lang/String;)V #18 = Utf8 args #19 = Utf8 [Ljava/lang/String; #20 = Utf8 helloword #21 = Utf8 sum #22 = Utf8 (II)I #23 = Utf8 a #24 = Utf8 b #25 = Utf8 <clinit> #26 = Utf8 SourceFile #27 = Utf8 Helloword.java #28 = NameAndType #9:#10 // "<init>":()V #29 = Utf8 com/climber/jvm/Helloword #30 = NameAndType #7:#8 // num:I #31 = NameAndType #21:#22 // sum:(II)I #32 = Utf8 java/lang/Object{ public static int num; descriptor: I flags: ACC_PUBLIC, ACC_STATIC public com.climber.jvm.Helloword(); descriptor: ()V flags: ACC_PUBLIC Code: stack=1, locals=1, args_size=1 0: aload_0 1: invokespecial #1 // Method java/lang/Object."<init>":()V 4: return LineNumberTable: line 12: 0 LocalVariableTable: Start Length Slot Name Signature 0 5 0 this Lcom/climber/jvm/Helloword; public static void main(java.lang.String[]); descriptor: ([Ljava/lang/String;)V flags: ACC_PUBLIC, ACC_STATIC Code: stack=3, locals=2, args_size=1 0: new #2 // class com/climber/jvm/Helloword 3: dup 4: invokespecial #3 // Method "<init>":()V 7: astore_1 8: getstatic #4 // Field num:I 11: iconst_1 12: iadd 13: putstatic #4 // Field num:I 16: aload_1 17: getstatic #4 // Field num:I 20: iconst_1 21: invokevirtual #5 // Method sum:(II)I 24: pop 25: return LineNumberTable: line 15: 0 line 16: 8 line 17: 16 line 18: 25 LocalVariableTable: Start Length Slot Name Signature 0 26 0 args [Ljava/lang/String; 8 18 1 helloword Lcom/climber/jvm/Helloword; public int sum(int, int); descriptor: (II)I flags: ACC_PUBLIC Code: stack=2, locals=3, args_size=3 0: iload_1 1: iload_2 2: iadd 3: ireturn LineNumberTable: line 20: 0 LocalVariableTable: Start Length Slot Name Signature 0 4 0 this Lcom/climber/jvm/Helloword; 0 4 1 a I 0 4 2 b I static {}; descriptor: ()V flags: ACC_STATIC Code: stack=1, locals=0, args_size=0 0: iconst_3 1: putstatic #4 // Field num:I 4: return LineNumberTable: line 13: 0}SourceFile: "Helloword.java"Class文件格式类型名称数量描述u4magic1魔数u2minor_version1次版本号u2major_version1主版本号u2constant_pool_count1常量池容量cp_infoconstant_poolcostant_pool_count-1常量池u2access_flags1访问标志u2this_class1当前类常量索引u2super_class1超类常量索引u2interfaces_count1接口数量u2interfacesinterfaces_count接口常量索引u2fields_count1字段数量field_infofieldsfields_count字段信息u2methods_count1方法数量method_infomethodsmethods_count方法信息u2attributes_count1属性数量attribute_infoattributesattributes_count属性信息","categories":[{"name":"JVM","slug":"JVM","permalink":"http://www.liuyong520.cn/categories/JVM/"}],"tags":[{"name":"JVM","slug":"JVM","permalink":"http://www.liuyong520.cn/tags/JVM/"}]},{"title":"storm 的分组策略深入理解（二）","slug":"storm-grouping2","date":"2019-05-12T12:10:40.000Z","updated":"2019-06-11T09:37:15.529Z","comments":true,"path":"2019/05/12/storm-grouping2/","link":"","permalink":"http://www.liuyong520.cn/2019/05/12/storm-grouping2/","excerpt":"","text":"上一篇博客提出了一个问题：如果执行1storm jar strom-study-1.0-SNAPSHOT-jar-with-dependencies.jar com.sonly.storm.demo1.grouppings.fieldgrouping.FeildGroupingToplogy FieldGrouping3 FieldGrouping3 4 1 1 4 4bolt 的分配情况是什么样子？这个答案是，只会有两个bolt里面有数据，其他bolt里面是没有数据的。下面接着讲分组策略All grouping这个分组策略其实没什么好说的。spout->bolt，和bolt->bolt之间都是全量的。local or sheffle grouping这个分组策略和sheffle grouping策略是一样的结果，可以完全替代，sheffle grouping这个只有一点不一样就是，当一个work上执行两个同样的task任务时，那么这两个任务间不会再通过RPC远程通信，直接随机分配数据。从而减少了，由于RPC远程通信带来的性能损耗。提高了效率。global grouping直接上代码：12345678910111213141516171819202122232425262728293031323334353637383940414243package com.sonly.storm.demo1.grouppings.globalgrouping;import org.apache.storm.Config;import org.apache.storm.StormSubmitter;import org.apache.storm.generated.AlreadyAliveException;import org.apache.storm.generated.AuthorizationException;import org.apache.storm.generated.InvalidTopologyException;import org.apache.storm.topology.TopologyBuilder;import org.slf4j.Logger;import org.slf4j.LoggerFactory;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:55</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class GlobalGroupingToplogy { public static final Logger LOGGER = LoggerFactory.getLogger(GlobalGroupingToplogy.class); public static void main(String[] args) throws InterruptedException { TopologyBuilder builder = new TopologyBuilder(); Config conf = new Config(); conf.setDebug(true); conf.setNumWorkers(3); String topology = \"GlobalGroupingToplogy\"; builder.setSpout(\"NumberGeneratorSpout\", new NumberGeneratorSpout(), 1); builder.setBolt(\"GlobalGrouppingBolt1\", new GlobalGrouppingBolt1(), 2).globalGrouping(\"NumberGeneratorSpout\"); builder.setBolt(\"GlobalGroupingBolt\", new GlobalGroupingBolt(), 2).globalGrouping(\"NumberGeneratorSpout\"); try { StormSubmitter.submitTopologyWithProgressBar(topology, conf, builder.createTopology()); LOGGER.warn(\"===========================================================\"); LOGGER.warn(\"The Topology {} is Submited \", topology); LOGGER.warn(\"===========================================================\"); } catch (AlreadyAliveException | InvalidTopologyException | AuthorizationException e) { e.printStackTrace(); } }}123456789101112131415161718192021222324252627282930313233343536373839404142434445package com.sonly.storm.demo1.grouppings.globalgrouping;import org.apache.storm.spout.SpoutOutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichSpout;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Values;import java.util.List;import java.util.Map;import java.util.concurrent.atomic.AtomicInteger;/** * <b>package:com.sonly.storm.demo1.grouppings.directgrouping</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-12 23:33</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class NumberGeneratorSpout extends BaseRichSpout { SpoutOutputCollector collector; TopologyContext context; AtomicInteger atomicInteger; @Override public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) { this.collector = collector; this.context = context; atomicInteger = new AtomicInteger(0); } @Override public void nextTuple() { while(atomicInteger.get()<10){ collector.emit(new Values(atomicInteger.incrementAndGet())); } } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"i\")); }}1234567891011121314151617181920212223242526272829303132333435363738394041424344454647package com.sonly.storm.demo1.grouppings.globalgrouping;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichBolt;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Tuple;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.HashMap;import java.util.Map;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:19</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class GlobalGroupingBolt extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(GlobalGroupingBolt.class); private TopologyContext context; private OutputCollector collector; private Map<String,Integer> counts = new HashMap(16); public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"GlobalGroupingBolt->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { Integer i = tuple.getIntegerByField(\"i\"); LOGGER.warn(\"GlobalGroupingBolt->execute:hashcode:{}->ThreadId:{},TaskId:{},value:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId(),i); collector.emit(new Values(i * 2)); collector.ack(tuple); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"double\")); }}1234567891011121314151617181920212223242526272829303132333435363738394041424344package com.sonly.storm.demo1.grouppings.globalgrouping;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichBolt;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Tuple;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.Map;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:29</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class GlobalGrouppingBolt1 extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(GlobalGrouppingBolt1.class); private TopologyContext context; private OutputCollector collector; public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"GlobalGrouppingBolt1->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { Integer i = tuple.getIntegerByField(\"i\"); LOGGER.warn(\"GlobalGrouppingBolt1->execute:hashcode:{}->ThreadId:{},TaskId:{},value:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId(),i); collector.emit(new Values(i*i)); collector.ack(tuple); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"square\")); }}两个bolt组件对接一个spout，按照global分组，实际上的日志情况是，taskId为1的bolt1，和taskId为3的bolt收到了消息。总结：golbal分组会把消息发给同一个bolt中taskId较小的那个，且spout->bolt之间也是全量发送的，只是只会发往同一个bolt组件中的taskID最小的那个direct groupping重点分析一下这个grouping123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051package com.sonly.storm.demo1.grouppings.directgrouping;import org.apache.storm.spout.SpoutOutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichSpout;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Values;import java.util.Collections;import java.util.List;import java.util.Map;import java.util.concurrent.atomic.AtomicInteger;/** * <b>package:com.sonly.storm.demo1.grouppings.directgrouping</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-12 23:33</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class NumberGeneratorSpout extends BaseRichSpout { SpoutOutputCollector collector; TopologyContext context; Integer taskId; AtomicInteger atomicInteger; @Override public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) { this.collector = collector; this.context = context; List<Integer> taskIds = context.getComponentTasks(\"DirectGroupingBolt\"); //拿到DirectGroupingBolt这个组件的最大taskID taskId = taskIds.stream().mapToInt(Integer::intValue).max().getAsInt(); atomicInteger = new AtomicInteger(0); } @Override public void nextTuple() { while(atomicInteger.get()<10){ //直接发往最大的taskId的DirectGroupingBolt的task中 collector.emitDirect(taskId,new Values(atomicInteger.incrementAndGet())); } } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"i\")); }}1234567891011121314151617181920212223242526272829303132333435363738394041424344454647package com.sonly.storm.demo1.grouppings.directgrouping;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichBolt;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Tuple;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.HashMap;import java.util.Map;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:19</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class DirectGroupingBolt extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(DirectGroupingBolt.class); private TopologyContext context; private OutputCollector collector; private Map<String,Integer> counts = new HashMap(16); public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"DirectGroupingBolt->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { Integer i = tuple.getIntegerByField(\"i\"); LOGGER.warn(\"DirectGroupingBolt->execute:hashcode:{}->ThreadId:{},TaskId:{},value:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId(),i); collector.emit(new Values(i * 2)); collector.ack(tuple); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"double\")); }}1234567891011121314151617181920212223242526272829303132333435363738394041424344package com.sonly.storm.demo1.grouppings.directgrouping;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichBolt;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Tuple;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.Map;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:29</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class DirectGrouppingBolt1 extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(DirectGrouppingBolt1.class); private TopologyContext context; private OutputCollector collector; public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"DirectGrouppingBolt1->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { Integer i = tuple.getIntegerByField(\"i\"); LOGGER.warn(\"DirectGrouppingBolt1->execute:hashcode:{}->ThreadId:{},TaskId:{},value:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId(),i); collector.emit(new Values(i*i)); collector.ack(tuple); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"square\")); }}1234567891011121314151617181920212223242526272829303132333435363738394041424344package com.sonly.storm.demo1.grouppings.directgrouping;import com.sonly.storm.demo1.grouppings.spout.WordSpout;import org.apache.storm.Config;import org.apache.storm.StormSubmitter;import org.apache.storm.generated.AlreadyAliveException;import org.apache.storm.generated.AuthorizationException;import org.apache.storm.generated.InvalidTopologyException;import org.apache.storm.topology.TopologyBuilder;import org.slf4j.Logger;import org.slf4j.LoggerFactory;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:55</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class DirectGroupingToplogy { public static final Logger LOGGER = LoggerFactory.getLogger(DirectGroupingToplogy.class); public static void main(String[] args) throws InterruptedException { TopologyBuilder builder = new TopologyBuilder(); Config conf = new Config(); conf.setDebug(true); conf.setNumWorkers(3); String topology = \"DirectGroupingToplogy\"; builder.setSpout(\"NumberGeneratorSpout\", new NumberGeneratorSpout(), 1); builder.setBolt(\"DirectGrouppingBolt1\", new DirectGrouppingBolt1(), 2).directGrouping(\"NumberGeneratorSpout\"); builder.setBolt(\"DirectGroupingBolt\", new DirectGroupingBolt(), 2).directGrouping(\"NumberGeneratorSpout\"); try { StormSubmitter.submitTopologyWithProgressBar(topology, conf, builder.createTopology()); LOGGER.warn(\"===========================================================\"); LOGGER.warn(\"The Topology {} is Submited \", topology); LOGGER.warn(\"===========================================================\"); } catch (AlreadyAliveException | InvalidTopologyException | AuthorizationException e) { e.printStackTrace(); } }}同样是spout->bolt 看看这个消息的分布情况1storm jar strom-study-1.0-SNAPSHOT-jar-with-dependencies.jar com.sonly.storm.demo1.grouppings.directgrouping.DirectGroupingToplogy如图：检查每个节点的日志 发现只有DirectGroupingBolt taskId为2的bolt接收到了消息，其他都没有接收到消息。总结：direct grouping 能够指定bolt发送数据，能够用direct grouping来实现global grouping的功能。custorm grouping这个就是自定义分组的意思，只要继承实现接口 CustomStreamGrouping 就可以对分组自定义了。这里实现一个简单的分组1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556package com.sonly.storm.demo1.grouppings.customgrouping;import org.apache.storm.generated.GlobalStreamId;import org.apache.storm.grouping.CustomStreamGrouping;import org.apache.storm.shade.com.google.common.base.Splitter;import org.apache.storm.shade.com.google.common.collect.ImmutableMap;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.task.WorkerTopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.tuple.Fields;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.Arrays;import java.util.HashMap;import java.util.List;import java.util.Map;import static java.util.concurrent.ThreadLocalRandom.current;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:19</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class CategoriesGrouping implements CustomStreamGrouping { public static final Logger LOGGER = LoggerFactory.getLogger(CategoriesGrouping.class); List<Integer> taskIds; Map<String,Integer> map = new HashMap<>(); @Override public void prepare(WorkerTopologyContext context, GlobalStreamId stream, List<Integer> targetTasks) { this.taskIds = targetTasks; } @Override public List<Integer> chooseTasks(int taskId, List<Object> values) { for (Object value : values) { List<String> strings = Splitter.on(\",\").splitToList(value.toString()); if(map.containsKey(strings.get(0))){ Integer integer = map.get(strings.get(0)); return Arrays.asList(integer); }else { int i = current().nextInt(this.taskIds.size()); map.put(strings.get(0),this.taskIds.get(i)); return Arrays.asList(i); } } return this.taskIds; }}123456789101112131415161718192021222324252627282930313233343536373839404142package com.sonly.storm.demo1.grouppings.customgrouping;import org.apache.storm.Config;import org.apache.storm.StormSubmitter;import org.apache.storm.generated.AlreadyAliveException;import org.apache.storm.generated.AuthorizationException;import org.apache.storm.generated.InvalidTopologyException;import org.apache.storm.topology.TopologyBuilder;import org.slf4j.Logger;import org.slf4j.LoggerFactory;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:55</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class CustomGroupingToplogy { public static final Logger LOGGER = LoggerFactory.getLogger(CustomGroupingToplogy.class); public static void main(String[] args) throws InterruptedException { TopologyBuilder builder = new TopologyBuilder(); Config conf = new Config(); conf.setDebug(true); conf.setNumWorkers(3); String topology = \"GlobalGroupingToplogy\"; builder.setSpout(\"NumberGeneratorSpout\", new NumberGeneratorSpout(), 1); builder.setBolt(\"GlobalGrouppingBolt1\", new CustomGrouppingBolt(), 2).customGrouping(\"NumberGeneratorSpout\",new CategoriesGrouping()); try { StormSubmitter.submitTopologyWithProgressBar(topology, conf, builder.createTopology()); LOGGER.warn(\"===========================================================\"); LOGGER.warn(\"The Topology {} is Submited \", topology); LOGGER.warn(\"===========================================================\"); } catch (AlreadyAliveException | InvalidTopologyException | AuthorizationException e) { e.printStackTrace(); } }}这里有一张图帮助我们理解分组策略：","categories":[{"name":"storm","slug":"storm","permalink":"http://www.liuyong520.cn/categories/storm/"}],"tags":[{"name":"storm","slug":"storm","permalink":"http://www.liuyong520.cn/tags/storm/"}]},{"title":"dubbo rpc 手把手实现","slug":"duboo-rpc","date":"2019-05-12T04:15:31.000Z","updated":"2019-06-11T09:37:16.562Z","comments":true,"path":"2019/05/12/duboo-rpc/","link":"","permalink":"http://www.liuyong520.cn/2019/05/12/duboo-rpc/","excerpt":"","text":"dubbo 简单介绍dubbo 是阿里巴巴开源的一款分布式rpc框架。为什么手写实现一下bubbo？很简单，最近从公司离职了，为了复习一下dubbo原理相关的知识，决定自己手写实现一个tony的dubbo，然后再结合dubbo的源码已达到复习的目的。什么是RPC？rpc 简单的说就是远程调用，以API的方式调用远程的服务器上的方法，像调本地方法一样！创建一个api的包模块，供服务端和消费者端共同使用。接口抽象12345678910package com.nnk.rpc.api;public interface HelloService { /** * 接口服务 * @param name * @return */ String sayHello(String name);}服务端实现服务端server端要实现这个接口。同时要发布这个接口，何谓发布这个接口？其实就是要像注册中心注册一下这个服务。这样，消费者在远程调用的时候可以通过注册中心注册的信息能够感知到服务。服务的实现：1234567891011package com.nnk.rpc.server.provide;import com.nnk.rpc.api.HelloService;public class HelloServiceImpl implements HelloService { public String sayHello(String name) { System.out.println(\"hello,\" + name); return \"hello \" + name; }}服务端抽象：1234567891011121314package com.nnk.rpc.server.protocl;/** * 服务端server */public interface RpcServer { /** * 开启服务 监听hostName：port * @param hostName * @param port */ public void start(String hostName,int port);}http协议的RPCServer实现12345678910111213141516171819202122232425262728293031323334353637383940414243444546package com.nnk.rpc.server.protocl.http;import com.nnk.rpc.server.protocl.RpcServer;import org.apache.catalina.*;import org.apache.catalina.connector.Connector;import org.apache.catalina.core.StandardContext;import org.apache.catalina.core.StandardEngine;import org.apache.catalina.core.StandardHost;import org.apache.catalina.startup.Tomcat;public class HttpServer implements RpcServer { public void start(String hostName,int port){ Tomcat tomcat = new Tomcat(); Server server = tomcat.getServer(); Service service = server.findService(\"Tomcat\"); Connector connector = new Connector(); connector.setPort(port); Engine engine = new StandardEngine(); engine.setDefaultHost(hostName); Host host = new StandardHost(); host.setName(hostName); //设置上下文 String contextPath=\"\"; Context context = new StandardContext(); context.setPath(contextPath); context.addLifecycleListener(new Tomcat.FixContextListener()); host.addChild(context); engine.addChild(host); service.setContainer(engine); service.addConnector(connector); //设置拦截servlet tomcat.addServlet(contextPath,\"dispather\",new DispatcherServlet()); context.addServletMappingDecoded(\"/*\",\"dispather\"); try { //启动tomcat tomcat.start(); tomcat.getServer().await(); } catch (LifecycleException e) { e.printStackTrace(); } }}启动了tomcat并用到DispatcherServlet来拦截我们的请求。12345678910111213141516171819package com.nnk.rpc.server.protocl.http;import javax.servlet.ServletException;import javax.servlet.http.HttpServlet;import javax.servlet.http.HttpServletRequest;import javax.servlet.http.HttpServletResponse;import java.io.IOException;/** * 这个代码大家应该很熟悉吧，这个是sevlet的基本知识。 * 任何请求被进来都会被这个sevlet处理 */public class DispatcherServlet extends HttpServlet { @Override protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException { //把所有的请求交给HttpHandler接口处理 new HttpHandler().handler(req,resp); }}再看一下HttpHandler类：12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152package com.nnk.rpc.server.protocl.http;import com.nnk.rpc.api.entity.Invocation;import com.nnk.rpc.register.RegisterType;import com.nnk.rpc.register.factory.LocalRegisterFactory;import org.apache.commons.io.IOUtils;import javax.servlet.http.HttpServletRequest;import javax.servlet.http.HttpServletResponse;import java.io.IOException;import java.io.InputStream;import java.io.ObjectInputStream;import java.lang.reflect.InvocationTargetException;import java.lang.reflect.Method;public class HttpHandler { public void handler(HttpServletRequest req, HttpServletResponse resp){ // 获取对象 try { //从流里面获取数据 InputStream is = req.getInputStream(); ObjectInputStream objectInputStream = new ObjectInputStream(is); //从流中读取数据反序列话成实体类。 Invocation invocation = (Invocation) objectInputStream.readObject(); //拿到服务的名字 String interfaceName = invocation.getInterfaceName(); //从注册中心里面拿到接口的实现类 Class interfaceImplClass = LocalRegisterFactory.getLocalRegister(RegisterType.LOCAL).get(interfaceName); //获取类的方法 Method method = interfaceImplClass.getMethod(invocation.getMethodName(),invocation.getParamtypes()); //反射调用方法 String result = (String) method.invoke(interfaceImplClass.newInstance(),invocation.getObjects()); //把结果返回给调用者 IOUtils.write(result,resp.getOutputStream()); } catch (IOException e) { e.printStackTrace(); } catch (ClassNotFoundException e) { e.printStackTrace(); } catch (NoSuchMethodException e) { e.printStackTrace(); } catch (IllegalAccessException e) { e.printStackTrace(); } catch (InstantiationException e) { e.printStackTrace(); } catch (InvocationTargetException e) { e.printStackTrace(); } }}我们看看Invocation的实现：123456789101112131415161718192021222324252627package com.nnk.rpc.api.entity;import java.io.Serializable;public class Invocation implements Serializable { private String interfaceName; private String methodName; private Class[] paramtypes; private Object[] objects; /** * * @param interfaceName 接口名字 * @param methodName 方法名字 * @param paramtypes 参数类型列表 * @param objects 参数列表 */ public Invocation(String interfaceName, String methodName, Class[] paramtypes, Object[] objects) { this.interfaceName = interfaceName; this.methodName = methodName; this.paramtypes = paramtypes; this.objects = objects; } .... get set 方法省略掉}到这里服务端先告一段落下面实现一下注册中心注册中心接口抽象：1234567891011121314151617package com.nnk.rpc.register;public interface LocalRegister { /** * * @param interfaceName 接口名称 * @param interfaceImplClass 接口实现类 */ void register(String interfaceName,Class interfaceImplClass); /** * 获取实现类 * @param interfaceName * @return */ Class get(String interfaceName);}LocalRegister 这个主要是供服务端自己在反射调用的时候根据服务名称找到对应的实现。12345678910111213141516171819package com.nnk.rpc.register;import com.nnk.rpc.api.entity.URL;public interface RemoteRegister { /** * 注册到远程注册中心 * @param interfaceName * @param host */ void register(String interfaceName, URL host); /** * 根据服务名称获取调用者的地址信息 * @param interfaceName * @return */ URL getRadomURL(String interfaceName);}这个主要是供消费者端根据服务名字找对应的地址发起远程调用用的。我们分别来看看这两个接口的实现：1234567891011121314151617181920package com.nnk.rpc.register.local;import com.nnk.rpc.register.LocalRegister;import java.util.HashMap;import java.util.Map;public class LocalMapRegister implements LocalRegister { private Map<String, Class> registerMap = new HashMap<String,Class>(1024); public void register(String interfaceName, Class interfaceImplClass) { registerMap.put(interfaceName,interfaceImplClass); } public Class get(String interfaceName) { return registerMap.get(interfaceName); }}很简单就是写在缓存里，map存储。123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566package com.nnk.rpc.register.local;import com.nnk.rpc.api.entity.URL;import com.nnk.rpc.register.RemoteRegister;import java.io.*;import java.util.*;public class RemoterMapRegister implements RemoteRegister { private Map<String, List<URL>> registerMap = new HashMap<String,List<URL>>(1024); public static final String path = \"/data/register\"; public void register(String interfaceName, URL host) { if(registerMap.containsKey(interfaceName)){ List<URL> list = registerMap.get(interfaceName); list.add(host); }else { List<URL> list = new LinkedList<URL>(); list.add(host); registerMap.put(interfaceName,list); } try { saveFile(path,registerMap); } catch (IOException e) { e.printStackTrace(); } } public URL getRadomURL(String interfaceName) { try { registerMap = (Map<String, List<URL>>) readFile(path); } catch (IOException e) { e.printStackTrace(); } catch (ClassNotFoundException e) { e.printStackTrace(); } List<URL> list = registerMap.get(interfaceName); Random random = new Random(); int i = random.nextInt(list.size()); return list.get(i); } /** * 写入文件 * @param path * @param object * @throws IOException */ private void saveFile(String path,Object object) throws IOException { FileOutputStream fileOutputStream = new FileOutputStream(new File(path)); ObjectOutputStream objectOutputStream =new ObjectOutputStream(fileOutputStream); objectOutputStream.writeObject(object); } /** * 从文件中读取 * @param path * @return * @throws IOException * @throws ClassNotFoundException */ private Object readFile(String path) throws IOException, ClassNotFoundException { FileInputStream fileInputStream = new FileInputStream(new File(path)); ObjectInputStream inputStream = new ObjectInputStream(fileInputStream); return inputStream.readObject(); }}这里为什么要写入文件呢？这是因为如果只存在内存中话，消费者和服务者不是同一个程序，消费者不额能感知到服务者程序内存的变化的。所以只能服务端写入文件，消费者从文件里取才能取得到。dubbo注册中心怎么干的呢，dubbo只是把这些信息写到了zookeeper，或者redis.或者其他地方。这里我就不再实现zookeeper的注册中心了。接下来我们开启服务1234567891011121314151617181920212223242526272829package com.nnk.rpc.server.provide;import com.nnk.rpc.api.HelloService;import com.nnk.rpc.api.entity.URL;import com.nnk.rpc.register.LocalRegister;import com.nnk.rpc.register.RegisterType;import com.nnk.rpc.register.RemoteRegister;import com.nnk.rpc.register.factory.LocalRegisterFactory;import com.nnk.rpc.register.factory.RemoteRegisterFactory;import com.nnk.rpc.server.protocl.Protocl;import com.nnk.rpc.server.protocl.ProtoclFactory;import com.nnk.rpc.server.protocl.ProtoclType;public class Provider { public static void main(String[] args) { URL url = new URL("localhost",8021); //远程服务注册地址 RemoteRegister register = RemoteRegisterFactory.getRemoteRegister(RegisterType.ZOOKEEPER); register.register(HelloService.class.getName(),url); //本地注册服务的实现类 LocalRegister localRegister = LocalRegisterFactory.getLocalRegister(RegisterType.LOCAL); localRegister.register(HelloService.class.getName(),HelloServiceImpl.class); //这里我又封装了一层协议层，我们都知道dubbo有基于netty的dubbo协议，有基于http的http协议，还有基于redis的redis协议等等。 Protocl protocl = ProtoclFactory.getProtocl(ProtoclType.HTTP); protocl.start(url); }}消费者端：消费者端其实很简单，就是根据注册中心里的信息远程调用对应服务器上的方法。123456789101112131415package com.nnk.rpc.client.comsummer;import com.nnk.rpc.api.HelloService;import com.nnk.rpc.client.proxy.ProxyFactory;import com.nnk.rpc.register.RegisterType;import com.nnk.rpc.server.protocl.ProtoclType;public class Consumer { public static void main(String[] args) { HelloService helloService = ProxyFactory.getProxy(ProtoclType.HTTP, RegisterType.ZOOKEEPER,HelloService.class); String result = helloService.sayHello(\"liuy\"); System.out.println(result); }}123456789101112131415161718192021222324252627282930package com.nnk.rpc.client.proxy;import com.nnk.rpc.api.entity.Invocation;import com.nnk.rpc.api.entity.URL;import com.nnk.rpc.register.RegisterType;import com.nnk.rpc.register.RemoteRegister;import com.nnk.rpc.register.factory.RemoteRegisterFactory;import com.nnk.rpc.server.protocl.Protocl;import com.nnk.rpc.server.protocl.ProtoclFactory;import com.nnk.rpc.server.protocl.ProtoclType;import java.lang.reflect.InvocationHandler;import java.lang.reflect.Method;import java.lang.reflect.Proxy;public class ProxyFactory { public static <T> T getProxy(final ProtoclType protoclType ,final RegisterType registerType, final Class interfaceClass){ return (T) Proxy.newProxyInstance(interfaceClass.getClassLoader(), new Class[]{interfaceClass}, new InvocationHandler() { public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { Protocl protocl = ProtoclFactory.getProtocl(protoclType); Invocation invocation = new Invocation(interfaceClass.getName(),method.getName(),method.getParameterTypes(),args); RemoteRegister remoteRegister = RemoteRegisterFactory.getRemoteRegister(registerType); URL radomURL = remoteRegister.getRadomURL(interfaceClass.getName()); System.out.println(\"调用地址host:\"+ radomURL.getHost()+ \",port:\"+radomURL.getPort()); return protocl.invokeProtocl(radomURL,invocation); } }); }}至此Dubbo的RPC调用核心框架就已经基本实现了。涉及到的东西其实挺多的，有tomcat的知识（http协议实现），协议的序列化和反序列化（远程调用消息的传递），netty的知识（dubbo协议的实现），动态代理的知识（消费者端实现）。反射（远程调用的核心）。再深入点就是负载均衡算法（在远程获取服务者的地址时可以抽象）。更完整的代码请去我的github上下载 Dubbo-tony如果有什么不清楚的地方，欢迎大家留言，咱们可以一起交流讨论。","categories":[],"tags":[{"name":"duboo","slug":"duboo","permalink":"http://www.liuyong520.cn/tags/duboo/"},{"name":"rpc","slug":"rpc","permalink":"http://www.liuyong520.cn/tags/rpc/"}]},{"title":"storm 的分组策略深入理解（-）","slug":"storm-groupping","date":"2019-05-11T12:10:40.000Z","updated":"2019-06-11T09:37:15.537Z","comments":true,"path":"2019/05/11/storm-groupping/","link":"","permalink":"http://www.liuyong520.cn/2019/05/11/storm-groupping/","excerpt":"","text":"storm的分组策略洗牌分组(Shuffle grouping): 随机分配元组到Bolt的某个任务上，这样保证同一个Bolt的每个任务都能够得到相同数量的元组。字段分组(Fields grouping): 按照指定的分组字段来进行流的分组。例如，流是用字段“user-id”来分组的，那有着相同“user-id”的元组就会分到同一个任务里，但是有不同“user-id”的元组就会分到不同的任务里。这是一种非常重要的分组方式，通过这种流分组方式，我们就可以做到让Storm产出的消息在这个”user-id”级别是严格有序的，这对一些对时序敏感的应用(例如，计费系统)是非常重要的。Partial Key grouping: 跟字段分组一样，流也是用指定的分组字段进行分组的，但是在多个下游Bolt之间是有负载均衡的，这样当输入数据有倾斜时可以更好的利用资源。这篇论文很好的解释了这是如何工作的，有哪些优势。All grouping: 流会复制给Bolt的所有任务。小心使用这种分组方式。在拓扑中，如果希望某类元祖发送到所有的下游消费者，就可以使用这种All grouping的流分组策略。Global grouping: 整个流会分配给Bolt的一个任务。具体一点，会分配给有最小ID的任务。不分组(None grouping): 说明不关心流是如何分组的。目前，None grouping等价于洗牌分组。Direct grouping：一种特殊的分组。对于这样分组的流，元组的生产者决定消费者的哪个任务会接收处理这个元组。只能在声明做直连的流(direct streams)上声明Direct groupings分组方式。只能通过使用emitDirect系列函数来吐元组给直连流。一个Bolt可以通过提供的TopologyContext来获得消费者的任务ID，也可以通过OutputCollector对象的emit函数(会返回元组被发送到的任务的ID)来跟踪消费者的任务ID。在ack的实现中，Spout有两个直连输入流，ack和ackFail，使用了这种直连分组的方式。Local or shuffle grouping：如果目标Bolt在同一个worker进程里有一个或多个任务，元组就会通过洗牌的方式分配到这些同一个进程内的任务里。否则，就跟普通的洗牌分组一样。这种方式的好处是可以提高拓扑的处理效率，因为worker内部通信就是进程内部通信了，相比拓扑间的进程间通信要高效的多。worker进程间通信是通过使用Netty来进行网络通信的。根据实例来分析分组策略common配置：12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.sonly.strom</groupId> <artifactId>strom-study</artifactId> <version>1.0-SNAPSHOT</version> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>7</source> <target>7</target> </configuration> </plugin> <plugin> <artifactId>maven-assembly-plugin</artifactId> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> <archive> <manifest> <mainClass>com.sonly.storm.demo1.HelloToplogy</mainClass> </manifest> </archive> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build> <dependencies> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-core</artifactId> <version>1.2.2</version> <scope>provided</scope> </dependency> </dependencies></project>Shuffle groupingshuffle grouping的实例代码123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354package com.sonly.storm.demo1.grouppings.spout;import org.apache.storm.spout.SpoutOutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichSpout;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.Map;import java.util.Random;import java.util.concurrent.atomic.AtomicInteger;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 20:27</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class WordSpout extends BaseRichSpout { public static final Logger LOGGER = LoggerFactory.getLogger(WordSpout.class); //拓扑上下文 private TopologyContext context; private SpoutOutputCollector collector; private Map config; private AtomicInteger atomicInteger = new AtomicInteger(0); public void open(Map conf, TopologyContext topologyContext, SpoutOutputCollector collector) { this.config = conf; this.context = topologyContext; this.collector = collector; LOGGER.warn(\"WordSpout->open:hashcode:{}->ThreadId:{},TaskId:{}\", this.hashCode(), Thread.currentThread().getId(), context.getThisTaskId()); } public void nextTuple() { String[] sentences = new String[]{\"zhangsan\",\"zhangsan\",\"zhangsan\",\"zhangsan\",\"zhangsan\",\"zhangsan\",\"zhangsan\",\"zhangsan\",\"lisi\",\"lisi\"}; int i = atomicInteger.get(); if(i<10){ atomicInteger.incrementAndGet(); final String sentence = sentences[i]; collector.emit(new Values(sentence)); LOGGER.warn(\"WordSpout->nextTuple:hashcode:{}->ThreadId:{},TaskId:{},Values:{}\", this.hashCode(), Thread.currentThread().getId(), context.getThisTaskId(), sentence); } } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"sentence\")); }}bolt112345678910111213141516171819202122232425262728293031323334353637383940414243444546474849package com.sonly.storm.demo1.grouppings;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichBolt;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Tuple;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.HashMap;import java.util.Map;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:19</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class SheffleGroupingBolt extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(SheffleGroupingBolt.class); private TopologyContext context; private Map conf; private OutputCollector collector; private Map<String,Integer> counts = new HashMap(16); public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.conf=map; this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"SheffleGroupingBolt->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { String word = tuple.getString(0); LOGGER.warn(\"SheffleGroupingBolt->execute:hashcode:{}->ThreadId:{},TaskId:{},value:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId(),word); collector.emit(new Values(word)); collector.ack(tuple); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"bolt1\")); }}bolt12345678910111213141516171819202122232425262728293031323334353637383940414243444546package com.sonly.storm.demo1.grouppings;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichBolt;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Tuple;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.Map;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:29</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class SheffleGrouppingBolt1 extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(SheffleGrouppingBolt1.class); private TopologyContext context; private Map conf; private OutputCollector collector; public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.conf=map; this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"SheffleGrouppingBolt1->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { String word = tuple.getStringByField(\"sentence\"); LOGGER.warn(\"SheffleGroupingBolt1->execute:hashcode:{}->ThreadId:{},TaskId:{},value:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId(),word); collector.emit(new Values(word)); collector.ack(tuple); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"bolt\")); }}topology123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161package com.sonly.storm.demo1.grouppings;import com.sonly.storm.demo1.grouppings.spout.WordSpout;import org.apache.storm.Config;import org.apache.storm.LocalCluster;import org.apache.storm.StormSubmitter;import org.apache.storm.generated.AlreadyAliveException;import org.apache.storm.generated.AuthorizationException;import org.apache.storm.generated.InvalidTopologyException;import org.apache.storm.topology.TopologyBuilder;import org.apache.storm.tuple.Fields;import org.slf4j.Logger;import org.slf4j.LoggerFactory;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:55</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class ShuffleGroupingToplogy { public static final Logger LOGGER = LoggerFactory.getLogger(ShuffleGroupingToplogy.class); //Topology Name //component prefix //workers //spout executor (parallelism_hint) //spout task size //bolt executor (parallelism_hint) //bolt task size public static void main(String[] args) throws InterruptedException { TopologyBuilder builder = new TopologyBuilder(); Config conf = new Config(); conf.setDebug(true); if (args==null || args.length < 7) { conf.setNumWorkers(3); builder.setSpout(\"spout\", new WordSpout(), 4).setNumTasks(4); builder.setBolt(\"split-bolt\", new SheffleGrouppingBolt1(), 4).shuffleGrouping(\"spout\").setNumTasks(8); builder.setBolt(\"count-bolt\", new SheffleGroupingBolt(), 8).fieldsGrouping(\"split-bolt\", new Fields(\"word\")).setNumTasks(8); LocalCluster cluster = new LocalCluster(); cluster.submitTopology(\"word-count\", conf, builder.createTopology()); Thread.sleep(10000); cluster.killTopology(\"word-count\"); cluster.shutdown(); } else { Options options = Options.builder(args); LOGGER.warn(\"The Topology Options {} is Submited \",options.toString()); conf.setNumWorkers(options.getWorkers()); builder.setSpout(options.getPrefix()+\"-spout\", new WordSpout(), options.getSpoutParallelismHint()).setNumTasks(options.getSpoutTaskSize()); builder.setBolt(\"bolt1\", new SheffleGrouppingBolt1(), options.getBoltParallelismHint()).shuffleGrouping(options.getPrefix()+\"-spout\").setNumTasks(options.getBoltTaskSize()); builder.setBolt(\"bolt\", new SheffleGroupingBolt(), options.getBoltParallelismHint()).shuffleGrouping(options.getPrefix()+\"-spout\").setNumTasks(options.getBoltTaskSize()); try { StormSubmitter.submitTopologyWithProgressBar(options.getTopologyName(), conf, builder.createTopology()); LOGGER.warn(\"===========================================================\"); LOGGER.warn(\"The Topology {} is Submited \",options.getTopologyName()); LOGGER.warn(\"===========================================================\"); } catch (AlreadyAliveException | InvalidTopologyException | AuthorizationException e) { e.printStackTrace(); } } } public static class Options{ private String topologyName; private String prefix; private Integer workers; private Integer spoutParallelismHint; private Integer spoutTaskSize; private Integer boltParallelismHint; private Integer boltTaskSize; public Options(String topologyName, String prefix, Integer workers, Integer spoutParallelismHint, Integer spoutTaskSize, Integer boltParallelismHint, Integer boltTaskSize) { this.topologyName = topologyName; this.prefix = prefix; this.workers = workers; this.spoutParallelismHint = spoutParallelismHint; this.spoutTaskSize = spoutTaskSize; this.boltParallelismHint = boltParallelismHint; this.boltTaskSize = boltTaskSize; } public static Options builder(String[] args){ return new Options(args[0],args[1],Integer.parseInt(args[2]) ,Integer.parseInt(args[3]),Integer.parseInt(args[4]),Integer.parseInt(args[5]),Integer.parseInt(args[6]) ); } public String getTopologyName() { return topologyName; } public void setTopologyName(String topologyName) { this.topologyName = topologyName; } public String getPrefix() { return prefix; } public void setPrefix(String prefix) { this.prefix = prefix; } public Integer getWorkers() { return workers; } public void setWorkers(Integer workers) { this.workers = workers; } public Integer getSpoutParallelismHint() { return spoutParallelismHint; } public void setSpoutParallelismHint(Integer spoutParallelismHint) { this.spoutParallelismHint = spoutParallelismHint; } public Integer getSpoutTaskSize() { return spoutTaskSize; } public void setSpoutTaskSize(Integer spoutTaskSize) { this.spoutTaskSize = spoutTaskSize; } public Integer getBoltParallelismHint() { return boltParallelismHint; } public void setBoltParallelismHint(Integer boltParallelismHint) { this.boltParallelismHint = boltParallelismHint; } public Integer getBoltTaskSize() { return boltTaskSize; } public void setBoltTaskSize(Integer boltTaskSize) { this.boltTaskSize = boltTaskSize; } @Override public String toString() { return \"Options{\" + \"topologyName='\" + topologyName + '\\'' + \", prefix='\" + prefix + '\\'' + \", workers=\" + workers + \", spoutParallelismHint=\" + spoutParallelismHint + \", spoutTaskSize=\" + spoutTaskSize + \", boltParallelismHint=\" + boltParallelismHint + \", boltTaskSize=\" + boltTaskSize + '}'; } }}mvn package 打包，上传到storm服务器ShuffleGrouping 样例分析1)样例11.执行：1storm jar strom-study-1.0-SNAPSHOT-jar-with-dependencies.jar com.sonly.storm.demo1.grouppings.ShuffleGroupingToplogy ShuffleGrouping ShuffleGrouping 1 2 1 2 12.参数：topologyName=’ShuffleGrouping’, prefix=’ShuffleGrouping’, workers=1, spoutParallelismHint=2, spoutTaskSize=1, boltParallelismHint=2, boltTaskSize=13.拓扑图：一个spout接了两个bolt4.查看一下这个bolt分布情况：5.进入服务器去看每一个bolt的日志123456789102019-05-07 18:09:13.109 c.s.s.d.g.SheffleGrouppingBolt1 Thread-11-bolt1-executor[5 5] [WARN] SheffleGroupingBolt1->execute:hashcode:1393282516->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 18:09:13.110 c.s.s.d.g.SheffleGrouppingBolt1 Thread-11-bolt1-executor[5 5] [WARN] SheffleGroupingBolt1->execute:hashcode:1393282516->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 18:09:13.110 c.s.s.d.g.SheffleGrouppingBolt1 Thread-11-bolt1-executor[5 5] [WARN] SheffleGroupingBolt1->execute:hashcode:1393282516->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 18:09:13.111 c.s.s.d.g.SheffleGrouppingBolt1 Thread-11-bolt1-executor[5 5] [WARN] SheffleGroupingBolt1->execute:hashcode:1393282516->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 18:09:13.112 c.s.s.d.g.SheffleGrouppingBolt1 Thread-11-bolt1-executor[5 5] [WARN] SheffleGroupingBolt1->execute:hashcode:1393282516->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 18:09:13.115 c.s.s.d.g.SheffleGrouppingBolt1 Thread-11-bolt1-executor[5 5] [WARN] SheffleGroupingBolt1->execute:hashcode:1393282516->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 18:09:13.116 c.s.s.d.g.SheffleGrouppingBolt1 Thread-11-bolt1-executor[5 5] [WARN] SheffleGroupingBolt1->execute:hashcode:1393282516->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 18:09:13.117 c.s.s.d.g.SheffleGrouppingBolt1 Thread-11-bolt1-executor[5 5] [WARN] SheffleGroupingBolt1->execute:hashcode:1393282516->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 18:09:13.118 c.s.s.d.g.SheffleGrouppingBolt1 Thread-11-bolt1-executor[5 5] [WARN] SheffleGroupingBolt1->execute:hashcode:1393282516->ThreadId:45,TaskId:5,value:lisi2019-05-07 18:09:13.119 c.s.s.d.g.SheffleGrouppingBolt1 Thread-11-bolt1-executor[5 5] [WARN] SheffleGroupingBolt1->execute:hashcode:1393282516->ThreadId:45,TaskId:5,value:lisi6.进入另外一个bolt的日志 10条信息被处理了123456789102019-05-07 18:09:00.791 c.s.s.d.g.SheffleGroupingBolt Thread-9-bolt-executor[4 4] [WARN] SheffleGroupingBolt->execute:hashcode:1430296959->ThreadId:43,TaskId:4,value:zhangsan2019-05-07 18:09:00.793 c.s.s.d.g.SheffleGroupingBolt Thread-9-bolt-executor[4 4] [WARN] SheffleGroupingBolt->execute:hashcode:1430296959->ThreadId:43,TaskId:4,value:zhangsan2019-05-07 18:09:00.794 c.s.s.d.g.SheffleGroupingBolt Thread-9-bolt-executor[4 4] [WARN] SheffleGroupingBolt->execute:hashcode:1430296959->ThreadId:43,TaskId:4,value:zhangsan2019-05-07 18:09:00.795 c.s.s.d.g.SheffleGroupingBolt Thread-9-bolt-executor[4 4] [WARN] SheffleGroupingBolt->execute:hashcode:1430296959->ThreadId:43,TaskId:4,value:zhangsan2019-05-07 18:09:00.795 c.s.s.d.g.SheffleGroupingBolt Thread-9-bolt-executor[4 4] [WARN] SheffleGroupingBolt->execute:hashcode:1430296959->ThreadId:43,TaskId:4,value:zhangsan2019-05-07 18:09:00.796 c.s.s.d.g.SheffleGroupingBolt Thread-9-bolt-executor[4 4] [WARN] SheffleGroupingBolt->execute:hashcode:1430296959->ThreadId:43,TaskId:4,value:zhangsan2019-05-07 18:09:00.797 c.s.s.d.g.SheffleGroupingBolt Thread-9-bolt-executor[4 4] [WARN] SheffleGroupingBolt->execute:hashcode:1430296959->ThreadId:43,TaskId:4,value:zhangsan2019-05-07 18:09:00.805 c.s.s.d.g.SheffleGroupingBolt Thread-9-bolt-executor[4 4] [WARN] SheffleGroupingBolt->execute:hashcode:1430296959->ThreadId:43,TaskId:4,value:zhangsan2019-05-07 18:09:00.805 c.s.s.d.g.SheffleGroupingBolt Thread-9-bolt-executor[4 4] [WARN] SheffleGroupingBolt->execute:hashcode:1430296959->ThreadId:43,TaskId:4,value:lisi2019-05-07 18:09:00.806 c.s.s.d.g.SheffleGroupingBolt Thread-9-bolt-executor[4 4] [WARN] SheffleGroupingBolt->execute:hashcode:1430296959->ThreadId:43,TaskId:4,value:lisi也是一样10条被处理了总结：对于spout直接对接两个bolt，sheffgrouping 分组不会随机给两个bolt分配消息，而是全量发给两个BOlT2）样例21.修改一下参数看一下：topologyName=’ShuffleGrouping1’, prefix=’ShuffleGrouping1’, workers=2, spoutParallelismHint=1, spoutTaskSize=2, boltParallelismHint=2, boltTaskSize=2总共4个bolt，两个spout，总共发送了40条消息，spout产生消息20条。transfer了40次。看看4个bolt的消息分配的情况。因为只有两个worker所以会有两个bolt在同一个work上，日志会打在一起，但是从名字可以可以区分开来，同样每个bolt都是10条。2.修改拓扑结构为：3.修改代码：bolt1String word = tuple.getStringByField(\"bolt\");topoloy：12 builder.setBolt(\"bolt1\", new SheffleGrouppingBolt1(), 1).shuffleGrouping(options.getPrefix()+\"-spout\");builder.setBolt(\"bolt\", new SheffleGroupingBolt(), options.getBoltParallelismHint()).shuffleGrouping(\"bolt1\").setNumTasks(options.getBoltTaskSize());4.参数：topologyName=’ShuffleGrouping2’, prefix=’ShuffleGrouping2’, workers=2, spoutParallelismHint=1, spoutTaskSize=1, boltParallelismHint=2, boltTaskSize=25.查看日志：k8s-n2 这个节点只有bolt bolt1这个节点在k8s-n3上12345678[root@k8s-n2 6706]# grep "SheffleGroupingBolt->execute" worker.log |wc -l3[root@k8s-n2 6706]# grep "SheffleGroupingBolt1->execute" worker.log |wc -l0[root@k8s-n3 6706]# grep "SheffleGroupingBolt->execute" worker.log |wc -l7[root@k8s-n3 6706]# grep "SheffleGroupingBolt1->execute" worker.log |wc -l10可以看出来bolt1->bolt这条线上的数据被随机分配了一个三条一个两条。总结：对于bolt 连接bolt的shuffingGrouping，消息是随机分配到多个bolt上面的Fields groupingFields grouping 的实例代码:123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130public class FeildGroupingToplogy { public static final Logger LOGGER = LoggerFactory.getLogger(FeildGroupingToplogy.class); //Topology Name //component prefix //workers //spout executor (parallelism_hint) //spout task size //bolt executor (parallelism_hint) //bolt task size public static void main(String[] args) throws InterruptedException { TopologyBuilder builder = new TopologyBuilder(); Config conf = new Config(); conf.setDebug(true); Options options = Options.builder(args); LOGGER.warn(\"The Topology Options {} is Submited \", options.toString()); conf.setNumWorkers(options.getWorkers()); String spoutName = options.getPrefix() + \"-spout\"; builder.setSpout(spoutName, new WordSpout(), options.getSpoutParallelismHint()).setNumTasks(options.getSpoutTaskSize()); builder.setBolt(options.getPrefix() + \"bolt1\", new FieldGrouppingBolt1(), options.getBoltParallelismHint()).fieldsGrouping(spoutName, new Fields(\"sentence\")).setNumTasks(options.getBoltTaskSize()); builder.setBolt(options.getPrefix() + \"bolt\", new FieldGroupingBolt(), options.getBoltParallelismHint()).fieldsGrouping(spoutName, new Fields(\"sentence\")).setNumTasks(options.getBoltTaskSize());// builder.setBolt(\"bolt1\", new FieldGrouppingBolt1(), 1).shuffleGrouping(options.getPrefix()+\"-spout\");// builder.setBolt(\"bolt\", new FieldGroupingBolt(), options.getBoltParallelismHint()).fieldsGrouping(\"bolt1\",new Fields(\"bolt\")).setNumTasks(options.getBoltTaskSize()); try { StormSubmitter.submitTopologyWithProgressBar(options.getTopologyName(), conf, builder.createTopology()); LOGGER.warn(\"===========================================================\"); LOGGER.warn(\"The Topology {} is Submited \", options.getTopologyName()); LOGGER.warn(\"===========================================================\"); } catch (AlreadyAliveException | InvalidTopologyException | AuthorizationException e) { e.printStackTrace(); } } public static class Options { private String topologyName; private String prefix; private Integer workers; private Integer spoutParallelismHint; private Integer spoutTaskSize; private Integer boltParallelismHint; private Integer boltTaskSize; public Options(String topologyName, String prefix, Integer workers, Integer spoutParallelismHint, Integer spoutTaskSize, Integer boltParallelismHint, Integer boltTaskSize) { this.topologyName = topologyName; this.prefix = prefix; this.workers = workers; this.spoutParallelismHint = spoutParallelismHint; this.spoutTaskSize = spoutTaskSize; this.boltParallelismHint = boltParallelismHint; this.boltTaskSize = boltTaskSize; } public static Options builder(String[] args) { return new Options(args[0], args[1], Integer.parseInt(args[2]) , Integer.parseInt(args[3]), Integer.parseInt(args[4]), Integer.parseInt(args[5]), Integer.parseInt(args[6]) ); } public String getTopologyName() { return topologyName; } public void setTopologyName(String topologyName) { this.topologyName = topologyName; } public String getPrefix() { return prefix; } public void setPrefix(String prefix) { this.prefix = prefix; } public Integer getWorkers() { return workers; } public void setWorkers(Integer workers) { this.workers = workers; } public Integer getSpoutParallelismHint() { return spoutParallelismHint; } public void setSpoutParallelismHint(Integer spoutParallelismHint) { this.spoutParallelismHint = spoutParallelismHint; } public Integer getSpoutTaskSize() { return spoutTaskSize; } public void setSpoutTaskSize(Integer spoutTaskSize) { this.spoutTaskSize = spoutTaskSize; } public Integer getBoltParallelismHint() { return boltParallelismHint; } public void setBoltParallelismHint(Integer boltParallelismHint) { this.boltParallelismHint = boltParallelismHint; } public Integer getBoltTaskSize() { return boltTaskSize; } public void setBoltTaskSize(Integer boltTaskSize) { this.boltTaskSize = boltTaskSize; } @Override public String toString() { return \"Options{\" + \"topologyName='\" + topologyName + '\\'' + \", prefix='\" + prefix + '\\'' + \", workers=\" + workers + \", spoutParallelismHint=\" + spoutParallelismHint + \", spoutTaskSize=\" + spoutTaskSize + \", boltParallelismHint=\" + boltParallelismHint + \", boltTaskSize=\" + boltTaskSize + '}'; } }}12345678910111213141516171819202122232425public class FieldGroupingBolt extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(FieldGroupingBolt.class); private TopologyContext context; private Map conf; private OutputCollector collector; private Map<String,Integer> counts = new HashMap(16); public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.conf=map; this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"FieldGroupingBolt->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { String word = tuple.getStringByField(\"bolt\"); LOGGER.warn(\"FieldGroupingBolt->execute:hashcode:{}->ThreadId:{},TaskId:{},value:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId(),word); collector.emit(new Values(word)); collector.ack(tuple); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"bolt1\")); }}123456789101112131415161718192021public class FieldGrouppingBolt1 extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(FieldGrouppingBolt1.class); private TopologyContext context; private OutputCollector collector; public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"FieldGrouppingBolt1->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { String word = tuple.getStringByField(\"sentence\"); LOGGER.warn(\"SheffleGroupingBolt1->execute:hashcode:{}->ThreadId:{},TaskId:{},value:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId(),word); collector.emit(new Values(word)); collector.ack(tuple); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"bolt\")); }}2.打包上传到服务器3.执行：1storm jar strom-study-1.0-SNAPSHOT-jar-with-dependencies.jar com.sonly.storm.demo1.grouppings.fieldgrouping.FeildGroupingToplogy FieldGrouping1 FieldGrouping1 2 1 1 2 24.参数topologyName=’FieldGrouping1’, prefix=’FieldGrouping1’, workers=2, spoutParallelismHint=1, spoutTaskSize=1, boltParallelismHint=2, boltTaskSize=25。拓扑图：6.并发度以及组件分布图：同样看图可以看到消息被发送了20次，但是被transfer40次。这是因为spout对bolt，对消息进行了复制，全量发送到了每个bolt，所以每个bolt都会有10条消息。总结：和sheffleGrouping 一样，spout->bolt是全量广播发送，每个bolt都会spout的全量消息。样例21.修改拓扑的代码123456789101112131415161718192021222324public static void main(String[] args) throws InterruptedException { TopologyBuilder builder = new TopologyBuilder(); Config conf = new Config(); conf.setDebug(true); Options options = Options.builder(args); LOGGER.warn("The Topology Options {} is Submited ", options.toString()); conf.setNumWorkers(options.getWorkers()); String spoutName = options.getPrefix() + "-spout"; builder.setSpout(spoutName, new WordSpout(), options.getSpoutParallelismHint()).setNumTasks(options.getSpoutTaskSize());// builder.setBolt(options.getPrefix() + "bolt1", new FieldGrouppingBolt1(), options.getBoltParallelismHint()).fieldsGrouping(spoutName, new Fields("sentence")).setNumTasks(options.getBoltTaskSize());// builder.setBolt(options.getPrefix() + "bolt", new FieldGroupingBolt(), options.getBoltParallelismHint()).fieldsGrouping(spoutName, new Fields("sentence")).setNumTasks(options.getBoltTaskSize()); builder.setBolt("bolt1", new FieldGrouppingBolt1(), 1).fieldsGrouping(spoutName, new Fields("sentence")); builder.setBolt("bolt", new FieldGroupingBolt(), options.getBoltParallelismHint()).fieldsGrouping("bolt1",new Fields("bolt")).setNumTasks(options.getBoltTaskSize()); try { StormSubmitter.submitTopologyWithProgressBar(options.getTopologyName(), conf, builder.createTopology()); LOGGER.warn("==========================================================="); LOGGER.warn("The Topology {} is Submited ", options.getTopologyName()); LOGGER.warn("==========================================================="); } catch (AlreadyAliveException | InvalidTopologyException | AuthorizationException e) { e.printStackTrace(); } }FieldGrouping 样例分析1）样例12.将上面代码打包上传服务器执行命令1storm jar strom-study-1.0-SNAPSHOT-jar-with-dependencies.jar com.sonly.storm.demo1.grouppings.fieldgrouping.FeildGroupingToplogy FieldGrouping2 FieldGrouping2 2 1 1 2 23.参数topologyName=’FieldGrouping2’, prefix=’FieldGrouping2’, workers=2, spoutParallelismHint=1, spoutTaskSize=1, boltParallelismHint=2, boltTaskSize=4.拓扑图：6.并发度以及组件分布图：7.根据分布情况检查各个work的日志查看消息的发送情况k8s-n3节点上：1234[root@k8s-n3 6701]# grep "SheffleGroupingBolt1->execute" worker.log|wc -l10[root@k8s-n3 6701]# grep "FieldGroupingBolt->execute" worker.log|wc -l2k8s-n2节点：1234[root@k8s-n2 6701]# grep "SheffleGroupingBolt1->execute" worker.log|wc -l0[root@k8s-n2 6701]# grep "FieldGroupingBolt->execute" worker.log|wc -l8再看一下详情如何k8s-n3：bolt1有10条消息应为bolt只有一个所以Fied分组是不会生效的。1234567891011[root@k8s-n3 6701]# grep "SheffleGroupingBolt1->execute" worker.log2019-05-07 21:59:35.805 c.s.s.d.g.f.FieldGrouppingBolt1 Thread-7-bolt1-executor[6 6] [WARN] SheffleGroupingBolt1->execute:hashcode:107880849->ThreadId:41,TaskId:6,value:zhangsan2019-05-07 21:59:35.810 c.s.s.d.g.f.FieldGrouppingBolt1 Thread-7-bolt1-executor[6 6] [WARN] SheffleGroupingBolt1->execute:hashcode:107880849->ThreadId:41,TaskId:6,value:zhangsan2019-05-07 21:59:35.810 c.s.s.d.g.f.FieldGrouppingBolt1 Thread-7-bolt1-executor[6 6] [WARN] SheffleGroupingBolt1->execute:hashcode:107880849->ThreadId:41,TaskId:6,value:zhangsan2019-05-07 21:59:35.811 c.s.s.d.g.f.FieldGrouppingBolt1 Thread-7-bolt1-executor[6 6] [WARN] SheffleGroupingBolt1->execute:hashcode:107880849->ThreadId:41,TaskId:6,value:zhangsan2019-05-07 21:59:35.811 c.s.s.d.g.f.FieldGrouppingBolt1 Thread-7-bolt1-executor[6 6] [WARN] SheffleGroupingBolt1->execute:hashcode:107880849->ThreadId:41,TaskId:6,value:zhangsan2019-05-07 21:59:35.812 c.s.s.d.g.f.FieldGrouppingBolt1 Thread-7-bolt1-executor[6 6] [WARN] SheffleGroupingBolt1->execute:hashcode:107880849->ThreadId:41,TaskId:6,value:zhangsan2019-05-07 21:59:35.813 c.s.s.d.g.f.FieldGrouppingBolt1 Thread-7-bolt1-executor[6 6] [WARN] SheffleGroupingBolt1->execute:hashcode:107880849->ThreadId:41,TaskId:6,value:zhangsan2019-05-07 21:59:35.814 c.s.s.d.g.f.FieldGrouppingBolt1 Thread-7-bolt1-executor[6 6] [WARN] SheffleGroupingBolt1->execute:hashcode:107880849->ThreadId:41,TaskId:6,value:zhangsan2019-05-07 21:59:35.815 c.s.s.d.g.f.FieldGrouppingBolt1 Thread-7-bolt1-executor[6 6] [WARN] SheffleGroupingBolt1->execute:hashcode:107880849->ThreadId:41,TaskId:6,value:lisi2019-05-07 21:59:35.838 c.s.s.d.g.f.FieldGrouppingBolt1 Thread-7-bolt1-executor[6 6] [WARN] SheffleGroupingBolt1->execute:hashcode:107880849->ThreadId:41,TaskId:6,value:lisik8s-n3:bolt 有两个实例，按照field分组。里面有两条消息。都是lisi123[root@k8s-n3 6701]# grep "FieldGroupingBolt->execute" worker.log2019-05-07 21:59:35.855 c.s.s.d.g.f.FieldGroupingBolt Thread-11-bolt-executor[4 4] [WARN] FieldGroupingBolt->execute:hashcode:281792799->ThreadId:45,TaskId:4,value:lisi2019-05-07 21:59:35.856 c.s.s.d.g.f.FieldGroupingBolt Thread-11-bolt-executor[4 4] [WARN] FieldGroupingBolt->execute:hashcode:281792799->ThreadId:45,TaskId:4,value:lisik8s-n2: bolt 应该就是8条消息，验证一下123456789[root@k8s-n2 6701]# grep "FieldGroupingBolt->execute" worker.log2019-05-07 21:59:48.315 c.s.s.d.g.f.FieldGroupingBolt Thread-11-bolt-executor[5 5] [WARN] FieldGroupingBolt->execute:hashcode:1858735164->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 21:59:48.317 c.s.s.d.g.f.FieldGroupingBolt Thread-11-bolt-executor[5 5] [WARN] FieldGroupingBolt->execute:hashcode:1858735164->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 21:59:48.317 c.s.s.d.g.f.FieldGroupingBolt Thread-11-bolt-executor[5 5] [WARN] FieldGroupingBolt->execute:hashcode:1858735164->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 21:59:48.318 c.s.s.d.g.f.FieldGroupingBolt Thread-11-bolt-executor[5 5] [WARN] FieldGroupingBolt->execute:hashcode:1858735164->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 21:59:48.318 c.s.s.d.g.f.FieldGroupingBolt Thread-11-bolt-executor[5 5] [WARN] FieldGroupingBolt->execute:hashcode:1858735164->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 21:59:48.319 c.s.s.d.g.f.FieldGroupingBolt Thread-11-bolt-executor[5 5] [WARN] FieldGroupingBolt->execute:hashcode:1858735164->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 21:59:48.319 c.s.s.d.g.f.FieldGroupingBolt Thread-11-bolt-executor[5 5] [WARN] FieldGroupingBolt->execute:hashcode:1858735164->ThreadId:45,TaskId:5,value:zhangsan2019-05-07 21:59:48.320 c.s.s.d.g.f.FieldGroupingBolt Thread-11-bolt-executor[5 5] [WARN] FieldGroupingBolt->execute:hashcode:1858735164->ThreadId:45,TaskId:5,value:zhangsan总结：bolt->bolt节点时，feild分组会按照field字段的key值进行分组，key相同的会被分配到一个bolt里面。如果执行1storm jar strom-study-1.0-SNAPSHOT-jar-with-dependencies.jar com.sonly.storm.demo1.grouppings.fieldgrouping.FeildGroupingToplogy FieldGrouping3 FieldGrouping3 4 1 1 4 4bolt 的分配情况是什么样子？这个留给大家去思考一下。例子中按照field分组后只有两种数据，但是这两种数据要分配给4个bolt，那这个是怎么分配的？我将在下一篇博客里揭晓答案！下一篇我会继续分析这个分组策略，我会把我在学习这个storm的时候当时的自己思考的一个过程展现给大家，如果有什么错误的，或者没有讲清楚的地方，欢迎大家给我留言，咱们可以一起交流讨论。","categories":[{"name":"storm","slug":"storm","permalink":"http://www.liuyong520.cn/categories/storm/"}],"tags":[{"name":"storm","slug":"storm","permalink":"http://www.liuyong520.cn/tags/storm/"}]},{"title":"理解storm并发度","slug":"storm-parallelism","date":"2019-05-10T03:46:25.000Z","updated":"2019-06-11T09:37:15.349Z","comments":true,"path":"2019/05/10/storm-parallelism/","link":"","permalink":"http://www.liuyong520.cn/2019/05/10/storm-parallelism/","excerpt":"","text":"什么是storm的并发度一个topology（拓扑）在storm集群上最总是以executor和task的形式运行在suppervisor管理的worker节点上。而worker进程都是运行在jvm虚拟机上面的，每个拓扑都会被拆开多个组件分布式的运行在worker节点上。1.worker2.executor3.task这三个简单关系图：一个worker工作进程运行一个拓扑的子集（其实就是拓扑的组件），每个组件的都会以executor（线程）在worker进程上执行，一个worker进程可以同时运行多个拓扑的组件也就是线程。一个executor线程可以运行同一个组件的一个或者多个taskstask是实际处理数据的执行者，每一个spout或者bolt会在集群上执行很多个task。在拓扑的生命周期内拓扑结构相同的拓扑的组件任务task数量总是相同的。但是每个组件的执行的线程（executor）数是可以变化的。这就意味着以下条件总是成立的：#threads ≤ #tasks 也就是task的数量总是大于线程数，一般情况下，任务task的数量往往设置成和线程（executor）的数量一致，这样，每个线程执行一个task。在storm拓扑的并发度其实就是集群上拓扑组件在集群上运行的executor（线程）的数量。如何设置拓扑的并发度“并行度”如何配置？其实不仅仅是设置executor线程的数量,同时也要从worker工作进程和task任务的数量的方面考虑。可以用以下几种方式配置并发度：1.通过storm的配置文件配置。storm配置文件的加载优先级是：defaults.yaml < storm.yaml < topology-specific configuration < internal component-specific configuration < external component-specific configuration.工作进程数描述：为群集中的计算机上的拓扑创建多少个工作进程。配置选项：TOPOLOGY_WORKERS如何设置代码（示例）：配置＃setNumWorkers执行者数（线程数）描述：每个组件生成多少个执行程序。配置选项：无（将parallelism_hint参数传递给setSpout或setBolt）如何设置代码（示例）：TopologyBuilder＃setSpout（）TopologyBuilder＃setBolt（）请注意，从Storm 0.8开始，parallelism_hint参数现在指定该螺栓的执行者的初始数量（不是任务！）。任务数量描述：每个组件创建多少个任务。配置选项：TOPOLOGY_TASKS如何设置代码（示例）：ComponentConfigurationDeclarer＃setNumTasks（）以下是在实践中显示这些设置的示例代码段：1234567891011121314151617181920212223242526272829topologyBuilder.setBolt(\"green-bolt\", new GreenBolt(), 2) .setNumTasks(4) .shuffleGrouping(\"blue-spout\");``` 在上面的代码中，我们配置了Storm来运行GreenBolt带有初始数量为两个执行器和四个相关任务的bolt 。Storm将为每个执行程序（线程）运行两个任务。如果您没有明确配置任务数，Storm将默认运行每个执行程序一个任务。#官方例子下图显示了简单拓扑在操作中的外观。拓扑结构由三个部分组成：一个叫做spout BlueSpout，两个叫做GreenBolt和YellowBolt。组件被链接，以便BlueSpout将其输出发送到GreenBolt，然后将其自己的输出发送到YellowBolt。![官方图](https://www.github.com/liuyong520/pic/raw/master/小书匠/1557460241883.png)在GreenBolt被配置为每代码段以上而BlueSpout和YellowBolt仅设置并行提示（执行人数）。这是相关代码：```javaConfig conf = new Config();conf.setNumWorkers(2); // use two worker processestopologyBuilder.setSpout(\"blue-spout\", new BlueSpout(), 2); // set parallelism hint to 2topologyBuilder.setBolt(\"green-bolt\", new GreenBolt(), 2) .setNumTasks(4) .shuffleGrouping(\"blue-spout\");topologyBuilder.setBolt(\"yellow-bolt\", new YellowBolt(), 6) .shuffleGrouping(\"green-bolt\");StormSubmitter.submitTopology( \"mytopology\", conf, topologyBuilder.createTopology() );当然，Storm附带了额外的配置设置来控制拓扑的并行性，包括：TOPOLOGY_MAX_TASK_PARALLELISM：此设置为可以为单个组件生成的执行程序数量设置上限。它通常在测试期间用于限制在本地模式下运行拓扑时产生的线程数。您可以通过例如Config＃setMaxTaskParallelism（）设置此选项。从实际运行的拓扑的角度理解storm的并发度自己写一个拓扑实现一个可以设置worker数量，设置spout 、bolt 的Parallelism Hint的拓扑然后打包上传到storm集群运行。123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146package com.sonly.storm.demo1;import org.apache.storm.Config;import org.apache.storm.LocalCluster;import org.apache.storm.StormSubmitter;import org.apache.storm.generated.AlreadyAliveException;import org.apache.storm.generated.AuthorizationException;import org.apache.storm.generated.InvalidTopologyException;import org.apache.storm.topology.TopologyBuilder;import org.apache.storm.tuple.Fields;import org.slf4j.Logger;import org.slf4j.LoggerFactory;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)HelloToplogy</b> * <b>creat date(创建时间):2019-05-09 21:55</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class HelloToplogy { public static final Logger LOGGER = LoggerFactory.getLogger(HelloToplogy.class); //Topology Name //component prefix //workers //spout executor (parallelism_hint) //spout task size //bolt executor (parallelism_hint) //bolt task size public static void main(String[] args) throws InterruptedException { TopologyBuilder builder = new TopologyBuilder(); Config conf = new Config(); conf.setDebug(true); if (args==null || args.length < 7) { conf.setNumWorkers(3); builder.setSpout(\"spout\", new HellowordSpout(), 4).setNumTasks(4); builder.setBolt(\"split-bolt\", new SplitBolt(), 4).shuffleGrouping(\"spout\").setNumTasks(8); builder.setBolt(\"count-bolt\", new HellowordBolt(), 8).fieldsGrouping(\"split-bolt\", new Fields(\"word\")).setNumTasks(8); LocalCluster cluster = new LocalCluster(); cluster.submitTopology(\"word-count\", conf, builder.createTopology()); Thread.sleep(10000); cluster.killTopology(\"word-count\"); cluster.shutdown(); } else { Options options = Options.builder(args); conf.setNumWorkers(options.getWorkers()); builder.setSpout(options.getPrefix()+\"-spout\", new HellowordSpout(), options.getSpoutParallelismHint()).setNumTasks(options.getSpoutTaskSize()); builder.setBolt(options.getPrefix()+\"-split-bolt\", new SplitBolt(), options.getBoltParallelismHint()).shuffleGrouping(options.getPrefix()+\"-spout\").setNumTasks(options.getBoltTaskSize()); builder.setBolt(options.getPrefix()+\"-count-bolt\", new HellowordBolt(), options.getBoltParallelismHint()).fieldsGrouping(options.getPrefix()+\"-split-bolt\", new Fields(\"word\")).setNumTasks(options.getBoltTaskSize()); try { StormSubmitter.submitTopologyWithProgressBar(options.getTopologyName(), conf, builder.createTopology()); LOGGER.warn(\"===========================================================\"); LOGGER.warn(\"The Topology {} is Submited \",options.getTopologyName()); LOGGER.warn(\"===========================================================\"); } catch (AlreadyAliveException | InvalidTopologyException | AuthorizationException e) { e.printStackTrace(); } } } public static class Options{ private String topologyName; private String prefix; private Integer workers; private Integer spoutParallelismHint; private Integer spoutTaskSize; private Integer boltParallelismHint; private Integer boltTaskSize; public Options(String topologyName, String prefix, Integer workers, Integer spoutParallelismHint, Integer spoutTaskSize, Integer boltParallelismHint, Integer boltTaskSize) { this.topologyName = topologyName; this.prefix = prefix; this.workers = workers; this.spoutParallelismHint = spoutParallelismHint; this.spoutTaskSize = spoutTaskSize; this.boltParallelismHint = boltParallelismHint; this.boltTaskSize = boltTaskSize; } public static Options builder(String[] args){ return new Options(args[0],args[1],Integer.parseInt(args[2]) ,Integer.parseInt(args[3]),Integer.parseInt(args[4]),Integer.parseInt(args[5]),Integer.parseInt(args[6]) ); } public String getTopologyName() { return topologyName; } public void setTopologyName(String topologyName) { this.topologyName = topologyName; } public String getPrefix() { return prefix; } public void setPrefix(String prefix) { this.prefix = prefix; } public Integer getWorkers() { return workers; } public void setWorkers(Integer workers) { this.workers = workers; } public Integer getSpoutParallelismHint() { return spoutParallelismHint; } public void setSpoutParallelismHint(Integer spoutParallelismHint) { this.spoutParallelismHint = spoutParallelismHint; } public Integer getSpoutTaskSize() { return spoutTaskSize; } public void setSpoutTaskSize(Integer spoutTaskSize) { this.spoutTaskSize = spoutTaskSize; } public Integer getBoltParallelismHint() { return boltParallelismHint; } public void setBoltParallelismHint(Integer boltParallelismHint) { this.boltParallelismHint = boltParallelismHint; } public Integer getBoltTaskSize() { return boltTaskSize; } public void setBoltTaskSize(Integer boltTaskSize) { this.boltTaskSize = boltTaskSize; } }}spout 类：123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657package com.sonly.storm.demo1;import org.apache.storm.spout.SpoutOutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichSpout;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.Currency;import java.util.Map;import java.util.Random;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${HellowordSpout}</b> * <b>creat date(创建时间):2019-05-09 20:27</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class HellowordSpout extends BaseRichSpout { public static final Logger LOGGER = LoggerFactory.getLogger(HellowordSpout.class); //拓扑上下文 private TopologyContext context; private SpoutOutputCollector collector; private Map config; private Random random; public void open(Map conf, TopologyContext topologyContext, SpoutOutputCollector collector) { this.config = conf; this.context = topologyContext; this.collector = collector; this.random = new Random(); LOGGER.warn(\"HellowordSpout->open:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void nextTuple() { String[] sentences = new String[]{\"hello world !\", \"hello Storm !\", \"hello apache flink !\", \"hello apache kafka stream !\", \"hello apache spark !\"}; final String sentence = sentences[random.nextInt(sentences.length)]; collector.emit(new Values(sentence)); LOGGER.warn(\"HellowordSpout->nextTuple:hashcode:{}->ThreadId:{},TaskId:{},Values:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId(),sentence); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"sentence\")); } @Override public void close() { LOGGER.warn(\"HellowordSpout->close:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); super.close(); }}实现两个bolt一个用来统计单词出现个数，一个用来拆分语句。12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152package com.sonly.storm.demo1;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichBolt;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Tuple;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.HashMap;import java.util.Map;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:19</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class HellowordBolt extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(HellowordBolt.class); private TopologyContext context; private Map conf; private OutputCollector collector; private Map<String,Integer> counts = new HashMap(16); public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.conf=map; this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"HellowordBolt->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { LOGGER.warn(\"HellowordBolt->execute:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); String word = tuple.getString(0); Integer count = counts.get(word); if (count == null) count = 0; count++; counts.put(word, count); collector.emit(new Values(word, count)); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"word\", \"count\")); }}12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849package com.sonly.storm.demo1;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichBolt;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Tuple;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.Map;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:29</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class SplitBolt extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(SplitBolt.class); private TopologyContext context; private Map conf; private OutputCollector collector; public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.conf=map; this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"SplitBolt->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { String words = tuple.getStringByField(\"sentence\"); String[] contents = words.split(\" +\"); for (String content : contents) { collector.emit(new Values(content)); collector.ack(tuple); } LOGGER.warn(\"SplitBolt->execute:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"word\")); }}local模式启动运行在pom文件中添加打包插件12345678910111213141516171819202122<plugin> <artifactId>maven-assembly-plugin</artifactId> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> <archive> <manifest> <mainClass>com.sonly.storm.demo1.HelloToplogy</mainClass> </manifest> </archive> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions></plugin>同时修改dependency 的scope为provide1<scope>provide</scope>原因是服务器上storm相关包都已经存在了，防止重复打包导致冲突。1234567//Topology Name//component prefix//workers//spout executor (parallelism_hint)//spout task size//bolt executor (parallelism_hint)//bolt task size在storm集群提交拓扑修改日志级别修改worker的工作进程的日志级别，修改成只输出warn日志，避免其他日志对我的干扰。进入${your_storm_path}/log4j2/目录修改worker.xml文件。先把worker.xml备份把Info级别改成warn1$ cp worker.xml worker.xml.bak修改成：1234567891011121314151617<loggers> <root level="warn"> <!-- We log everything --> <appender-ref ref="A1"/> <appender-ref ref="syslog"/> </root> <Logger name="org.apache.storm.metric.LoggingMetricsConsumer" level="info" additivity="false"> <appender-ref ref="METRICS"/> </Logger> <Logger name="STDERR" level="INFO"> <appender-ref ref="STDERR"/> <appender-ref ref="syslog"/> </Logger> <Logger name="STDOUT" level="INFO"> <appender-ref ref="STDOUT"/> <appender-ref ref="syslog"/> </Logger></loggers>同步到另外两台supervisor的工作节点服务器。为了跟清晰的理解并发度，我会通过这个demo 拓扑，修改参数观察stormUI的exector数量和tasks数量。参数说明1234567// topologyName='count' ## Topology Name 拓扑的名字// prefix='tp1' ## component prefix 即为每个spout，bolt的前缀名称// workers=1 ## worker number 即为工作进程jvm数量// spoutParallelismHint=2 ## spout executor (parallelism_hint) 即spout的线程数量// spoutTaskSize=1 ## spout task size 即spout的运行实例数// boltParallelismHint=2 ## bolt executor (parallelism_hint) 即bolt的线程数量// boltTaskSize=1 ##bolt task size 即bolt的运行实例数根据样例分析理解storm的并发度例子1执行：1storm jar storm-demo1.jar com.sonly.storm.demo1.HelloToplogy tp1 tp1 1 2 0 2 0参数详情：1{topologyName='tp1', prefix='tp1', workers=1, spoutParallelismHint=2, spoutTaskSize=0, boltParallelismHint=2, boltTaskSize=0}这时候task都被设置成0了。如下图：excutors为1，task为1。接着往下看：此时我们的bolt的task都被设置0了，所以我们是没有创建spout，bolt的，但是你会发现一个_acker的bolt，这是storm的acker机制，storm自己给我们创建的bolt，并且每一个worker都会必须有一个_acker的bolt，如果我们没有取消ack机制的话。所以worker上只用了一个excutor来跑这个_acker的bolt。例子21storm jar storm-demo1.jar com.sonly.storm.demo1.HelloToplogy tp2 tp2 1 2 1 2 1参数详情：1{topologyName='tp2', prefix='tp2', workers=1, spoutParallelismHint=2, spoutTaskSize=1, boltParallelismHint=2, boltTaskSize=1}此时 task的值都被设置成1了。如下图：excutors为4，task为4。接着看一下spout，bolt 以及组件的分布情况见下图：此时已经我们的有tp2-spout 一个spout，除了系统的acker 还有我们自己创建的两个bolt。因为只有一个worker所以全部分布在一个worker里面。尽管我们设置了spout的线程数为2，bolt的线程数为2，但是task都被设置成1，只有一个任务需要被两个excutor执行，所以有一个线程实际上是没有任务执行的。所以线程数，就是这几个task的值的和，一个spout，两个自己的创建的bolt以及acker的task数量的和。例子31storm jar storm-demo1.jar com.sonly.storm.demo1.HelloToplogy tp3 tp3 2 2 1 2 1参数详情：1{topologyName='tp3', prefix='tp3', workers=2, spoutParallelismHint=2, spoutTaskSize=1, boltParallelismHint=2, boltTaskSize=1}此时worker已经被设置成2了，如下图：executor为5，task为5.接着看一下spout，bolt以及组件的分布情况如下图：此时，task任务数依然是1，spout和bolt都是1份，acker每个worker都必须有一份的，所以，executor的数就是task实例数也就是：一个spout 两个系统acker bolt，和两个我们自己的bolt。也就是5.这个5个task不均匀的分配到了两个worker进程上。例子41storm jar storm-demo1.jar com.sonly.storm.demo1.HelloToplogy tp4 tp4 2 2 2 2 2参数详情：1{topologyName='tp4', prefix='tp4', workers=2, spoutParallelismHint=2, spoutTaskSize=2, boltParallelismHint=2, boltTaskSize=2}此时参数已经taks 数量被设置成2了，如下图：executor为8，task为8.再看一下spout，bolt的分布情况：如下图：此时，我们task都被设置成了2，那spout实例和bolt的实例都是2，也就是2+2+2=6 这个是我们自己的创建的task，再加上acker两个task，所以task参数就是8.而这里设置时executor也是8个 被均分到两个worker上面。例子51storm jar storm-demo1.jar com.sonly.storm.demo1.HelloToplogy tp5 tp5 2 2 4 2 4参数详情：1{topologyName='tp5', prefix='tp5', workers=2, spoutParallelismHint=2, spoutTaskSize=4, boltParallelismHint=2, boltTaskSize=4}此时task设置成4，excutor设置成2，那这样的话，一个excutor会跑两个task ，executor=8,task=14 如下图：继续看一下spout和bolt的分布情况：此时task设置成4，executor是2，那就是bolt和spout实例就是 4+4+4=12 再加上两个worker的Acker就是14个task。exector 是bolt的设置的值2+2+2=6个再加上两个acker的值，就是8个。同时，一个executor执行了两个task。8个executor平均分配到两个worker上面了。总结exector和task的值，和拓扑结构有关系，拓扑中的spout 和bolt设置的parallelism_hint都会影响到exector和task的数量。task和exectuor之间的关系在设置上就已经确定了，最好exector和task之间，task 的数量最好设置成executor的倍数，这样每个executor执行的task才是一样的。说到这里，相信大家对并发度，有了比较清晰的理解。","categories":[{"name":"storm","slug":"storm","permalink":"http://www.liuyong520.cn/categories/storm/"}],"tags":[{"name":"storm","slug":"storm","permalink":"http://www.liuyong520.cn/tags/storm/"}]},{"title":"storm 基本知识","slug":"storm-base","date":"2019-05-02T14:00:40.000Z","updated":"2019-06-11T09:37:15.528Z","comments":true,"path":"2019/05/02/storm-base/","link":"","permalink":"http://www.liuyong520.cn/2019/05/02/storm-base/","excerpt":"","text":"引言介绍storm之前，我先抛出这两个问题：1.实时计算需要解决些什么问题？2.storm作为实时计算到底有何优势？storm简介官方介绍：Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!翻译：Apache Storm是一个免费的开源分布式实时计算系统。Storm可以轻松可靠地处理无限数据流，实时处理Hadoop为批处理所做的工作。storm很简单，可以与任何编程语言一起使用，并且使用起来很有趣！简单的说就是：storm是一个分布式实时计算系统。1.实时计算需要解决些什么问题？伴随着信息科技的日新月异的发展，信息呈现出爆发式的膨胀，人们获取信息的渠道也更加多元化，获取信息的方式也更加便捷，对信息的实效性也越来越高，举个电商系统一个搜索的简单例子，当卖家发布一条宝贝信息时，买家在搜索的时候需要能够马上呈现出来，同时买家购买后，能够需要统计该商品的卖出的数量以及该商品总营业额，利润等等。而且可以根据最近的购买的记录，可以给用户推荐同类型的产品。诸如此类的都需要以大数据为基础的，通过离线或者实时计算，获取相关的信息，从而获得商机。在Storm之前，进行实时处理是非常痛苦的事情: 需要维护一堆消息队列和消费者，他们构成了非常复杂的图结构。消费者进程从队列里取消息，处理完成后，去更新数据库，或者给其他队列发新消息。这样进行实时处理是非常痛苦的。我们主要的时间都花在关注往哪里发消息，从哪里接收消息，消息如何序列化，真正的业务逻辑只占了源代码的一小部分。一个应用程序的逻辑运行在很多worker上，但这些worker需要各自单独部署，还需要部署消息队列。最大问题是系统很脆弱，而且不是容错的：需要自己保证消息队列和worker进程工作正常。例如上面的例子，简单的统计数量，统计营业额，计算毛利润，根据购买记录，推荐相似商品等等，就是通过，开启多个工作线程，实时去扫表，或者是从消息对列中拿出数据，计算统计值，写入对应的表。这种定时任务，往往数据处理也不够及时，实效性比较差。2.实时计算系统要考虑哪些问题？低延迟 处理消息一定要及时，延迟高就不叫实时了。高性能 性能不高，那么就要浪费机器，这样浪费机器就是浪费资源分布式 系统数据和来源可能有多个，处理数据结果也有可能作为基础数据给其他系统。如果你的系统应用单机就能搞定，那么不需要考虑这么复杂了，实时计算系统就是为了解决这种单机系统无法解决的问题的。可扩展 伴随业务的的发展，我们的业务量，及计算量可能会越来越大，系统是要求可以扩展的，容错性 这个是分布式系统的通用问题了，一个节点挂了，不能影响到整个系统。3.storm的优势简单的编程模型。类似于MapReduce降低了批处理的复杂性，storm降低了进行实时处理的复杂性服务化，一个服务框架，支持热部署，即时上线或者下线app支持多种语言，你可以在storm之上使用各种编程语言，默认支持的有clojure,java,ruby,python容错性，storm会管理工作进程和节点故障水平扩展性，计算是在多个系统、进程、服务器之间进行的。可以水平扩展机器，进程，或者线程等。可靠的消息处理 storm 能够保证消息至少能够得到一次完整的消息处理。任务失败时，它会负责从消息源头重新处理。快速 系统的设计保证消息的快速处理，低版本的storm使用的zeroMQ作为内部消息系统。高版本中使用netty完全替代了ZeroMQ作为内部消息系统本地模式 storm 又一个本地模式，能够模拟集群，使我们的开发测试变得更加简单。storm的基本概念拓扑(Topologies)实时应用程序的逻辑被打包到Storm拓扑中。Storm拓扑类似于MapReduce作业。一个关键的区别是MapReduce作业最终完成，而拓扑结构永远运行（当然，直到你杀死它）。个拓扑是一个通过流分组(stream grouping)把Spout和Bolt连接到一起的拓扑结构。图的每条边代表一个Bolt订阅了其他Spout或者Bolt的输出流。一个拓扑就是一个复杂的多阶段的流计算。元组(Tuple)元组是Storm提供的一个轻量级的数据格式，可以用来包装你需要实际处理的数据。元组是一次消息传递的基本单元。一个元组是一个命名的值列表，其中的每个值都可以是任意类型的。元组是动态地进行类型转化的–字段的类型不需要事先声明。在Storm中编程时，就是在操作和转换由元组组成的流。通常，元组包含整数，字节，字符串，浮点数，布尔值和字节数组等类型。要想在元组中使用自定义类型，就需要实现自己的序列化方式。流(Streams)流是Storm中的核心抽象。一个流由无限的元组序列组成，这些元组会被分布式并行地创建和处理。通过流中元组包含的字段名称来定义这个流。每个流声明时都被赋予了一个ID。只有一个流的Spout和Bolt非常常见，所以OutputFieldsDeclarer提供了不需要指定ID来声明一个流的函数(Spout和Bolt都需要声明输出的流)。这种情况下，流的ID是默认的“default”。Spouts(喷嘴)Spout(喷嘴，这个名字很形象)是Storm中流的来源。通常Spout从外部数据源，如消息队列中读取元组数据并吐到拓扑里。Spout可以是可靠的(reliable)或者不可靠(unreliable)的。可靠的Spout能够在一个元组被Storm处理失败时重新进行处理，而非可靠的Spout只是吐数据到拓扑里，不关心处理成功还是失败了。Spout可以一次给多个流吐数据。此时需要通过OutputFieldsDeclarer的declareStream函数来声明多个流并在调用SpoutOutputCollector提供的emit方法时指定元组吐给哪个流。Spout中最主要的函数是nextTuple，Storm框架会不断调用它去做元组的轮询。如果没有新的元组过来，就直接返回，否则把新元组吐到拓扑里。nextTuple必须是非阻塞的，因为Storm在同一个线程里执行Spout的函数。Spout中另外两个主要的函数是ack和fail。当Storm检测到一个从Spout吐出的元组在拓扑中成功处理完时调用ack,没有成功处理完时调用fail。只有可靠型的Spout会调用ack和fail函数。Bolts在拓扑中所有的计算逻辑都是在Bolt中实现的。一个Bolt可以处理任意数量的输入流，产生任意数量新的输出流。Bolt可以做函数处理，过滤，流的合并，聚合，存储到数据库等操作。Bolt就是流水线上的一个处理单元，把数据的计算处理过程合理的拆分到多个Bolt、合理设置Bolt的task数量，能够提高Bolt的处理能力，提升流水线的并发度。Bolt可以给多个流吐出元组数据。此时需要使用OutputFieldsDeclarer的declareStream方法来声明多个流并在使用OutputColletor的emit方法时指定给哪个流吐数据。当你声明了一个Bolt的输入流，也就订阅了另外一个组件的某个特定的输出流。如果希望订阅另一个组件的所有流，需要单独挨个订阅。InputDeclarer有语法糖来订阅ID为默认值的流。例如declarer.shuffleGrouping(“redBolt”)订阅了redBolt组件上的默认流，跟declarer.shuffleGrouping(“redBolt”, DEFAULT_STREAM_ID)是相同的。在Bolt中最主要的函数是execute函数，它使用一个新的元组当作输入。Bolt使用OutputCollector对象来吐出新的元组。Bolts必须为处理的每个元组调用OutputCollector的ack方法以便于Storm知道元组什么时候被各个Bolt处理完了（最终就可以确认Spout吐出的某个元组处理完了）。通常处理一个输入的元组时，会基于这个元组吐出零个或者多个元组，然后确认(ack)输入的元组处理完了，Storm提供了IBasicBolt接口来自动完成确认。必须注意OutputCollector不是线程安全的，所以所有的吐数据(emit)、确认(ack)、通知失败(fail)必须发生在同一个线程里任务(Tasks)每个Spout和Bolt会以多个任务(Task)的形式在集群上运行。每个任务对应一个执行线程，流分组定义了如何从一组任务(同一个Bolt)发送元组到另外一组任务(另外一个Bolt)上。可以在调用TopologyBuilder的setSpout和setBolt函数时设置每个Spout和Bolt的并发数。组件(Component)组件(component)是对Bolt和Spout的统称流分组(Stream groupings)定义拓扑的时候，一部分工作是指定每个Bolt应该消费哪些流。流分组定义了一个流在一个消费它的Bolt内的多个任务(task)之间如何分组。流分组跟计算机网络中的路由功能是类似的，决定了每个元组在拓扑中的处理路线。在Storm中有七个内置的流分组策略，你也可以通过实现CustomStreamGrouping接口来自定义一个流分组策略:洗牌分组(Shuffle grouping): 随机分配元组到Bolt的某个任务上，这样保证同一个Bolt的每个任务都能够得到相同数量的元组。字段分组(Fields grouping): 按照指定的分组字段来进行流的分组。例如，流是用字段“user-id”来分组的，那有着相同“user-id”的元组就会分到同一个任务里，但是有不同“user-id”的元组就会分到不同的任务里。这是一种非常重要的分组方式，通过这种流分组方式，我们就可以做到让Storm产出的消息在这个”user-id”级别是严格有序的，这对一些对时序敏感的应用(例如，计费系统)是非常重要的。Partial Key grouping: 跟字段分组一样，流也是用指定的分组字段进行分组的，但是在多个下游Bolt之间是有负载均衡的，这样当输入数据有倾斜时可以更好的利用资源。这篇论文很好的解释了这是如何工作的，有哪些优势。All grouping: 流会复制给Bolt的所有任务。小心使用这种分组方式。在拓扑中，如果希望某类元祖发送到所有的下游消费者，就可以使用这种All grouping的流分组策略。Global grouping: 整个流会分配给Bolt的一个任务。具体一点，会分配给有最小ID的任务。不分组(None grouping): 说明不关心流是如何分组的。目前，None grouping等价于洗牌分组。Direct grouping：一种特殊的分组。对于这样分组的流，元组的生产者决定消费者的哪个任务会接收处理这个元组。只能在声明做直连的流(direct streams)上声明Direct groupings分组方式。只能通过使用emitDirect系列函数来吐元组给直连流。一个Bolt可以通过提供的TopologyContext来获得消费者的任务ID，也可以通过OutputCollector对象的emit函数(会返回元组被发送到的任务的ID)来跟踪消费者的任务ID。在ack的实现中，Spout有两个直连输入流，ack和ackFail，使用了这种直连分组的方式。Local or shuffle grouping：如果目标Bolt在同一个worker进程里有一个或多个任务，元组就会通过洗牌的方式分配到这些同一个进程内的任务里。否则，就跟普通的洗牌分组一样。这种方式的好处是可以提高拓扑的处理效率，因为worker内部通信就是进程内部通信了，相比拓扑间的进程间通信要高效的多。worker进程间通信是通过使用Netty来进行网络通信的。可靠性(Reliability)Storm保证了拓扑中Spout产生的每个元组都会被处理。Storm是通过跟踪每个Spout所产生的所有元组构成的树形结构并得知这棵树何时被完整地处理来达到可靠性。每个拓扑对这些树形结构都有一个关联的“消息超时”。如果在这个超时时间里Storm检测到Spout产生的一个元组没有被成功处理完，那Sput的这个元组就处理失败了，后续会重新处理一遍。为了发挥Storm的可靠性，需要你在创建一个元组树中的一条边时告诉Storm，也需要在处理完每个元组之后告诉Storm。这些都是通过Bolt吐元组数据用的OutputCollector对象来完成的。标记是在emit函数里完成，完成一个元组后需要使用ack函数来告诉Storm。Workers(工作进程)拓扑以一个或多个Worker进程的方式运行。每个Worker进程是一个物理的Java虚拟机，执行拓扑的一部分任务。例如，如果拓扑的并发设置成了300，分配了50个Worker，那么每个Worker执行6个任务(作为Worker内部的线程）。Storm会尽量把所有的任务均分到所有的Worker上。storm的工作流程Storm实现了一个数据流(data flow)的模型，在这个模型中数据持续不断地流经一个由很多转换实体构成的网络。一个数据流的抽象叫做流(stream)，流是无限的元组(Tuple)序列。元组就像一个可以表示标准数据类型（例如int，float和byte数组）和用户自定义类型（需要额外序列化代码的）的数据结构。每个流由一个唯一的ID来标示的，这个ID可以用来构建拓扑中各个组件的数据源。如下图所示，其中的水龙头代表了数据流的来源，一旦水龙头打开，数据就会源源不断地流经Bolt而被处理。图中有三个流，用不同的颜色来表示，每个数据流中流动的是元组(Tuple)，它承载了具体的数据。元组通过流经不同的转换实体而被处理。Storm对数据输入的来源和输出数据的去向没有做任何限制。像Hadoop，是需要把数据放到自己的文件系统HDFS里的。在Storm里，可以使用任意来源的数据输入和任意的数据输出，只要你实现对应的代码来获取/写入这些数据就可以。典型场景下，输入/输出数据来是基于类似Kafka或者ActiveMQ这样的消息队列，但是数据库，文件系统或者web服务也都是可以的。如图：storm 测试用例并运行往往我在学习这些开源框架的时候，查看官方文档和源码中的例子是入门上手比较快的一种方式。这里我也是从官方文档和github上的代码入手的1.clone 下来1.2.2的源码1git clone --branch v1.2.2 https://github.com/apache/storm.git2.进入examples目录下有很多例子，如storm-starter这个项目。同时在github上进入这个例子的目录下有README.md文件，介绍如何运行我们的测试例子。我们可以先感官体验一下storm运行拓扑是怎样的。详情请看storm-starterstep 1：用idea 打开storm源码 然后用maven打包或者进入storm-starter项目里面执行1mvn package会在target目录里面生成一个start-storm-{version}.jar包上传到storm的服务器。step 2: 提交运行实例拓扑，有本地和集群两种模式。是不是本地模式🉐️看代码如何实现的。12storm jar stormlib/storm-starter-1.2.2.jar org.apache.storm.starter.ExclamationTopology ##本地模式storm jar stormlib/storm-starter-1.2.2.jar org.apache.storm.starter.ExclamationTopology ExclamationTopology ## ExclamationTopology为Topology的名字step 3: org.apache.storm.starter.ExclamationTopology 这个拓扑类如下：1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * \"License\"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an \"AS IS\" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */package org.apache.storm.starter;import org.apache.storm.Config;import org.apache.storm.LocalCluster;import org.apache.storm.StormSubmitter;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.testing.TestWordSpout;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.TopologyBuilder;import org.apache.storm.topology.base.BaseRichBolt;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Tuple;import org.apache.storm.tuple.Values;import org.apache.storm.utils.Utils;import java.util.Map;/** * This is a basic example of a Storm topology. */public class ExclamationTopology { public static class ExclamationBolt extends BaseRichBolt { OutputCollector _collector; @Override public void prepare(Map conf, TopologyContext context, OutputCollector collector) { _collector = collector; } @Override public void execute(Tuple tuple) { _collector.emit(tuple, new Values(tuple.getString(0) + \"!!!\")); _collector.ack(tuple); } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"word\")); } } public static void main(String[] args) throws Exception { //构建拓扑类 TopologyBuilder builder = new TopologyBuilder(); //设置数据获取来源spout builder.setSpout(\"word\", new TestWordSpout(), 10); //设置数据处理bolt类按照word分组 builder.setBolt(\"exclaim1\", new ExclamationBolt(), 3).shuffleGrouping(\"word\"); //设置数据处理bolt类按照exclaim1分组 builder.setBolt(\"exclaim2\", new ExclamationBolt(), 2).shuffleGrouping(\"exclaim1\"); //设置配置参数，打开debug模式 Config conf = new Config(); conf.setDebug(true); // 根据传入参数创建对应拓扑，如果参数大于0，就提交到storm集群，集群模式运行 if (args != null && args.length > 0) { conf.setNumWorkers(3); //通过nimbus提交拓扑到suppervisisor工作节点运行 StormSubmitter.submitTopologyWithProgressBar(args[0], conf, builder.createTopology()); } else { //否则就是local本地模式运行，本地模式运行所有的日志都会打印到本地，可以用来调试 LocalCluster cluster = new LocalCluster(); cluster.submitTopology(\"test\", conf, builder.createTopology()); Utils.sleep(10000); //运行10秒后杀掉拓扑 cluster.killTopology(\"test\"); //同时释放资源，关掉storm cluster.shutdown(); } }}org.apache.storm.testing.TestWordSpout;123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * \"License\"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an \"AS IS\" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */package org.apache.storm.testing;import org.apache.storm.Config;import org.apache.storm.topology.OutputFieldsDeclarer;import java.util.Map;import org.apache.storm.spout.SpoutOutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.base.BaseRichSpout;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Values;import org.apache.storm.utils.Utils;import java.util.HashMap;import java.util.Random;import org.slf4j.Logger;import org.slf4j.LoggerFactory;public class TestWordSpout extends BaseRichSpout { public static Logger LOG = LoggerFactory.getLogger(TestWordSpout.class); boolean _isDistributed; SpoutOutputCollector _collector; public TestWordSpout() { this(true); } public TestWordSpout(boolean isDistributed) { _isDistributed = isDistributed; } public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) { _collector = collector; } public void close() { } public void nextTuple() { Utils.sleep(100); final String[] words = new String[] {\"nathan\", \"mike\", \"jackson\", \"golda\", \"bertels\"}; final Random rand = new Random(); final String word = words[rand.nextInt(words.length)]; //随机从这些字符串中获取数据 _collector.emit(new Values(word)); } public void ack(Object msgId) { } public void fail(Object msgId) { } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"word\")); } @Override public Map<String, Object> getComponentConfiguration() { if(!_isDistributed) { Map<String, Object> ret = new HashMap<String, Object>(); ret.put(Config.TOPOLOGY_MAX_TASK_PARALLELISM, 1); return ret; } else { return null; } } }storm API 简单介绍1.拓扑构建TopologyBuilder公开了Java API，用于指定要执行的Storm拓扑。拓扑结构最终是Thrift结构，但由于Thrift API非常冗长，TopologyBuilder极大地简化了创建拓扑的过程。用于创建和提交拓扑的模板类似于：12345678910111213TopologyBuilder builder = new TopologyBuilder();builder.setSpout(\"1\", new TestWordSpout(true), 5);builder.setSpout(\"2\", new TestWordSpout(true), 3);builder.setBolt(\"3\", new TestWordCounter(), 3) .fieldsGrouping(\"1\", new Fields(\"word\")) .fieldsGrouping(\"2\", new Fields(\"word\"));builder.setBolt(\"4\", new TestGlobalCount()) .globalGrouping(\"1\");Map conf = new HashMap();conf.put(Config.TOPOLOGY_WORKERS, 4);StormSubmitter.submitTopology(\"mytopology\", conf, builder.createTopology());在本地模式（正在处理）中运行完全相同的拓扑，并将其配置为记录所有发出的元组，如下所示。请注意，在关闭本地群集之前，它允许拓扑运行10秒。123456789101112131415161718TopologyBuilder builder = new TopologyBuilder();builder.setSpout(\"1\", new TestWordSpout(true), 5);builder.setSpout(\"2\", new TestWordSpout(true), 3);builder.setBolt(\"3\", new TestWordCounter(), 3) .fieldsGrouping(\"1\", new Fields(\"word\")) .fieldsGrouping(\"2\", new Fields(\"word\"));builder.setBolt(\"4\", new TestGlobalCount()) .globalGrouping(\"1\");Map conf = new HashMap();conf.put(Config.TOPOLOGY_WORKERS, 4);conf.put(Config.TOPOLOGY_DEBUG, true);LocalCluster cluster = new LocalCluster();cluster.submitTopology(\"mytopology\", conf, builder.createTopology());Utils.sleep(10000);cluster.shutdown();模式TopologyBuilder是使用setSpout和setBolt方法将组件ID映射到组件。这些方法返回的对象随后用于声明该组件的输入。详情可以查看TopologyBuilder2.创建spout相关ISpout接口：123456789101112public interface ISpout extends Serializable { //初始化方法 void open(Map conf, TopologyContext context, SpoutOutputCollector collector); //关闭 void close(); //遍历元组的方法，会一直执行这个方法 void nextTuple(); //成功时调用的确认方法 void ack(Object msgId); //失败时调用的失败方法 void fail(Object msgId);}storm已经给我们提供的很多的spout接口和实现类，我们只需要实现或者对应的实现类就能和其他技术集成在一起。基本的spout，我们可以实现IRichSpout或者继承BasicRichSpout完成spout的创建。3.创建Bolt类storm已经给我们提供的很多的Bolt接口和实现类，我们只需要实现或者对应的实现类就能和其他技术集成在一起。基本的spout，我们可以实现IRichBolt或者继承BasicRichBolt如1234567891011121314151617181920212223/*** BaseRichBolt 是一个不需要实现的ACK确认方法和fail（）失败方法* */public class ExampleBolt extends BaseRichBolt { OutputCollector _collector; @Override public void prepare(Map conf, TopologyContext context, OutputCollector collector) { _collector = collector; } @Override public void execute(Tuple tuple) { _collector.emit(tuple, new Values(tuple.getString(0) + \"!!!\")); _collector.ack(tuple); } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"word\")); } }Storm使用入门起来是非常简单的。下面我将简单的自己实现一个拓扑123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146package com.sonly.storm.demo1;import org.apache.storm.Config;import org.apache.storm.LocalCluster;import org.apache.storm.StormSubmitter;import org.apache.storm.generated.AlreadyAliveException;import org.apache.storm.generated.AuthorizationException;import org.apache.storm.generated.InvalidTopologyException;import org.apache.storm.topology.TopologyBuilder;import org.apache.storm.tuple.Fields;import org.slf4j.Logger;import org.slf4j.LoggerFactory;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)HelloToplogy</b> * <b>creat date(创建时间):2019-05-09 21:55</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class HelloToplogy { public static final Logger LOGGER = LoggerFactory.getLogger(HelloToplogy.class); //Topology Name //component prefix //workers //spout executor (parallelism_hint) //spout task size //bolt executor (parallelism_hint) //bolt task size public static void main(String[] args) throws InterruptedException { TopologyBuilder builder = new TopologyBuilder(); Config conf = new Config(); conf.setDebug(true); if (args==null || args.length < 7) { conf.setNumWorkers(3); builder.setSpout(\"spout\", new HellowordSpout(), 4).setNumTasks(4); builder.setBolt(\"split-bolt\", new SplitBolt(), 4).shuffleGrouping(\"spout\").setNumTasks(8); builder.setBolt(\"count-bolt\", new HellowordBolt(), 8).fieldsGrouping(\"split-bolt\", new Fields(\"word\")).setNumTasks(8); LocalCluster cluster = new LocalCluster(); cluster.submitTopology(\"word-count\", conf, builder.createTopology()); Thread.sleep(10000); cluster.killTopology(\"word-count\"); cluster.shutdown(); } else { Options options = Options.builder(args); conf.setNumWorkers(options.getWorkers()); builder.setSpout(options.getPrefix()+\"-spout\", new HellowordSpout(), options.getSpoutParallelismHint()).setNumTasks(options.getSpoutTaskSize()); builder.setBolt(options.getPrefix()+\"-split-bolt\", new SplitBolt(), options.getBoltParallelismHint()).shuffleGrouping(options.getPrefix()+\"-spout\").setNumTasks(options.getBoltTaskSize()); builder.setBolt(options.getPrefix()+\"-count-bolt\", new HellowordBolt(), options.getBoltParallelismHint()).fieldsGrouping(options.getPrefix()+\"-split-bolt\", new Fields(\"word\")).setNumTasks(options.getBoltTaskSize()); try { StormSubmitter.submitTopologyWithProgressBar(options.getTopologyName(), conf, builder.createTopology()); LOGGER.warn(\"===========================================================\"); LOGGER.warn(\"The Topology {} is Submited \",options.getTopologyName()); LOGGER.warn(\"===========================================================\"); } catch (AlreadyAliveException | InvalidTopologyException | AuthorizationException e) { e.printStackTrace(); } } } public static class Options{ private String topologyName; private String prefix; private Integer workers; private Integer spoutParallelismHint; private Integer spoutTaskSize; private Integer boltParallelismHint; private Integer boltTaskSize; public Options(String topologyName, String prefix, Integer workers, Integer spoutParallelismHint, Integer spoutTaskSize, Integer boltParallelismHint, Integer boltTaskSize) { this.topologyName = topologyName; this.prefix = prefix; this.workers = workers; this.spoutParallelismHint = spoutParallelismHint; this.spoutTaskSize = spoutTaskSize; this.boltParallelismHint = boltParallelismHint; this.boltTaskSize = boltTaskSize; } public static Options builder(String[] args){ return new Options(args[0],args[1],Integer.parseInt(args[2]) ,Integer.parseInt(args[3]),Integer.parseInt(args[4]),Integer.parseInt(args[5]),Integer.parseInt(args[6]) ); } public String getTopologyName() { return topologyName; } public void setTopologyName(String topologyName) { this.topologyName = topologyName; } public String getPrefix() { return prefix; } public void setPrefix(String prefix) { this.prefix = prefix; } public Integer getWorkers() { return workers; } public void setWorkers(Integer workers) { this.workers = workers; } public Integer getSpoutParallelismHint() { return spoutParallelismHint; } public void setSpoutParallelismHint(Integer spoutParallelismHint) { this.spoutParallelismHint = spoutParallelismHint; } public Integer getSpoutTaskSize() { return spoutTaskSize; } public void setSpoutTaskSize(Integer spoutTaskSize) { this.spoutTaskSize = spoutTaskSize; } public Integer getBoltParallelismHint() { return boltParallelismHint; } public void setBoltParallelismHint(Integer boltParallelismHint) { this.boltParallelismHint = boltParallelismHint; } public Integer getBoltTaskSize() { return boltTaskSize; } public void setBoltTaskSize(Integer boltTaskSize) { this.boltTaskSize = boltTaskSize; } }}spout 类：123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657package com.sonly.storm.demo1;import org.apache.storm.spout.SpoutOutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichSpout;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.Currency;import java.util.Map;import java.util.Random;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${HellowordSpout}</b> * <b>creat date(创建时间):2019-05-09 20:27</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class HellowordSpout extends BaseRichSpout { public static final Logger LOGGER = LoggerFactory.getLogger(HellowordSpout.class); //拓扑上下文 private TopologyContext context; private SpoutOutputCollector collector; private Map config; private Random random; public void open(Map conf, TopologyContext topologyContext, SpoutOutputCollector collector) { this.config = conf; this.context = topologyContext; this.collector = collector; this.random = new Random(); LOGGER.warn(\"HellowordSpout->open:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void nextTuple() { String[] sentences = new String[]{\"hello world !\", \"hello Storm !\", \"hello apache flink !\", \"hello apache kafka stream !\", \"hello apache spark !\"}; final String sentence = sentences[random.nextInt(sentences.length)]; collector.emit(new Values(sentence)); LOGGER.warn(\"HellowordSpout->nextTuple:hashcode:{}->ThreadId:{},TaskId:{},Values:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId(),sentence); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"sentence\")); } @Override public void close() { LOGGER.warn(\"HellowordSpout->close:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); super.close(); }}实现两个bolt一个用来统计单词出现个数，一个用来拆分语句。12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152package com.sonly.storm.demo1;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichBolt;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Tuple;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.HashMap;import java.util.Map;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:19</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class HellowordBolt extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(HellowordBolt.class); private TopologyContext context; private Map conf; private OutputCollector collector; private Map<String,Integer> counts = new HashMap(16); public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.conf=map; this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"HellowordBolt->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { LOGGER.warn(\"HellowordBolt->execute:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); String word = tuple.getString(0); Integer count = counts.get(word); if (count == null) count = 0; count++; counts.put(word, count); collector.emit(new Values(word, count)); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"word\", \"count\")); }}12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849package com.sonly.storm.demo1;import org.apache.storm.task.OutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichBolt;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Tuple;import org.apache.storm.tuple.Values;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.util.Map;/** * <b>package:com.sonly.storm.demo1</b> * <b>project(项目):stormstudy</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-09 21:29</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class SplitBolt extends BaseRichBolt { public static final Logger LOGGER = LoggerFactory.getLogger(SplitBolt.class); private TopologyContext context; private Map conf; private OutputCollector collector; public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { this.conf=map; this.context = topologyContext; this.collector = outputCollector; LOGGER.warn(\"SplitBolt->prepare:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void execute(Tuple tuple) { String words = tuple.getStringByField(\"sentence\"); String[] contents = words.split(\" +\"); for (String content : contents) { collector.emit(new Values(content)); collector.ack(tuple); } LOGGER.warn(\"SplitBolt->execute:hashcode:{}->ThreadId:{},TaskId:{}\",this.hashCode(),Thread.currentThread().getId(),context.getThisTaskId()); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields(\"word\")); }}local模式启动运行在pom文件中添加打包插件12345678910111213<plugin> <artifactId>maven-assembly-plugin</artifactId> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> <archive> <manifest> <mainClass>com.sonly.storm.demo1.HelloToplogy</mainClass> </manifest> </archive> </configuration></plugin>同时修改dependency 的scope为provide1<scope>provide</scope>原因是服务器上storm相关包都已经存在了，防止重复打包导致冲突。1234567//Topology Name//component prefix//workers//spout executor (parallelism_hint)//spout task size//bolt executor (parallelism_hint)//bolt task size打包上传后，storm jar jarName arg0 arg1 arg2 args3 …后面跟参数运行即可。","categories":[{"name":"storm","slug":"storm","permalink":"http://www.liuyong520.cn/categories/storm/"}],"tags":[{"name":"storm","slug":"storm","permalink":"http://www.liuyong520.cn/tags/storm/"}]},{"title":"kafka API的使用","slug":"kafka-API","date":"2019-05-02T13:37:40.000Z","updated":"2019-06-11T09:37:15.530Z","comments":true,"path":"2019/05/02/kafka-API/","link":"","permalink":"http://www.liuyong520.cn/2019/05/02/kafka-API/","excerpt":"","text":"kafka APIkafka Consumer提供两套Java API：高级Consumer API、和低级Consumer API。高级Consumer API 优点：高级API写起来简单，易用。不需要自行去管理offset，API已经封装好了offset这块的东西，会通过zookeeper自行管理不需要管理分区，副本等情况，系统自动管理消费者断线后会自动根据上次记录在zookeeper中的offset接着消费消息。高级Consumer API 缺点：不能自行控制offset。不能自行管理分区，副本，zk等相关信息。低级API 优点：能够让开发者自己维护offset.想从哪里消费就从哪里消费自行控制连接分区，对分区自定义负载均衡对zookeeper的依赖性降低（如 offset 不一定要用zk来存储，可以存在缓存里或者内存中）缺点：过于复杂，需要自行控制offset，连接哪个分区，找分区leader等。简单入门使用引入maven依赖12345dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>2.2.0</version></dependency>Producer简单使用1234567891011121314151617181920212223242526272829303132package com.sonly.kafka;import org.apache.kafka.clients.producer.KafkaProducer;import org.apache.kafka.clients.producer.ProducerConfig;import org.apache.kafka.clients.producer.ProducerRecord;import java.util.Properties;/** * <b>package:com.sonly.kafka</b> * <b>project(项目):kafkaAPIdemo</b> * <b>class(类)demo</b> * <b>creat date(创建时间):2019-05-03 12:17</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class demo { public static void main(String[] args) { Properties properties = new Properties(); properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,\"k8s-n1:9092\"); properties.put(ProducerConfig.ACKS_CONFIG,\"1\"); properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,\"org.apache.kafka.common.serialization.StringSerializer\"); properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,\"org.apache.kafka.common.serialization.StringSerializer\"); KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties); for (int i = 0; i < 100; i++) producer.send(new ProducerRecord<String, String>(\"mytest\", Integer.toString(i), Integer.toString(i))); producer.close(); }}带回调函数的生产者123456789101112131415161718192021222324252627282930313233343536373839404142434445464748package com.sonly.kafka;import org.apache.kafka.clients.producer.*;import java.util.Properties;/** * <b>package:com.sonly.kafka</b> * <b>project(项目):kafkaAPIdemo</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-03 12:58</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class demo1 { public static void main(String[] args) { Properties properties = new Properties(); //设置kafka集群 properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,\"k8s-n1:9092\"); //设置brokeACK应答机制 properties.put(ProducerConfig.ACKS_CONFIG,\"1\"); //设置key序列化 properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,\"org.apache.kafka.common.serialization.StringSerializer\"); //设置value序列化 properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,\"org.apache.kafka.common.serialization.StringSerializer\"); //设置批量大小 properties.put(ProducerConfig.BATCH_SIZE_CONFIG,\"6238\"); //设置提交延时 properties.put(ProducerConfig.LINGER_MS_CONFIG,\"1\"); //设置producer缓存 properties.put(ProducerConfig.BUFFER_MEMORY_CONFIG,Long.MAX_VALUE); KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties); for ( int i = 0; i < 12; i++) { final int finalI = i; producer.send(new ProducerRecord<String, String>(\"mytest\", Integer.toString(i), Integer.toString(i)), new Callback() { public void onCompletion(RecordMetadata metadata, Exception exception) { if(exception==null){ System.out.println(\"发送成功: \" + finalI +\",\"+metadata.partition()+\",\"+ metadata.offset()); } } }); } producer.close(); }}结果：123456789101112发送成功: 0,0,170发送成功: 2,0,171发送成功: 11,0,172发送成功: 4,1,101发送成功: 5,2,116发送成功: 6,2,117发送成功: 10,2,118发送成功: 1,3,175发送成功: 3,3,176发送成功: 7,3,177发送成功: 8,3,178发送成功: 9,3,179数据不均等的分配到0-3 号分区上自定义分区发送12345678910111213141516171819202122232425262728package com.sonly.kafka;import org.apache.kafka.clients.producer.Partitioner;import org.apache.kafka.common.Cluster;import java.util.Map;/** * <b>package:com.sonly.kafka</b> * <b>project(项目):kafkaAPIdemo</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-03 13:43</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class CustomProducer implements Partitioner { public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) { return 0; } public void close() { } public void configure(Map<String, ?> configs) { }}设置分区12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849package com.sonly.kafka;import org.apache.kafka.clients.producer.*;import java.util.Properties;/** * <b>package:com.sonly.kafka</b> * <b>project(项目):kafkaAPIdemo</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-03 13:46</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class demo2 { public static void main(String[] args) { Properties properties = new Properties(); //设置kafka集群 properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,\"k8s-n1:9092\"); //设置brokeACK应答机制 properties.put(ProducerConfig.ACKS_CONFIG,\"1\"); //设置key序列化 properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,\"org.apache.kafka.common.serialization.StringSerializer\"); //设置value序列化 properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,\"org.apache.kafka.common.serialization.StringSerializer\"); //设置批量大小 properties.put(ProducerConfig.BATCH_SIZE_CONFIG,\"6238\"); //设置提交延时 properties.put(ProducerConfig.LINGER_MS_CONFIG,\"1\"); //设置producer缓存 properties.put(ProducerConfig.BUFFER_MEMORY_CONFIG,Long.MAX_VALUE); //设置partition properties.put(ProducerConfig.PARTITIONER_CLASS_CONFIG,\"com.sonly.kafka.CustomProducer\"); KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties); for ( int i = 0; i < 12; i++) { final int finalI = i; producer.send(new ProducerRecord<String, String>(\"mytest\", Integer.toString(i), Integer.toString(i)), new Callback() { public void onCompletion(RecordMetadata metadata, Exception exception) { if(exception==null){ System.out.println(\"发送成功: \" + finalI +\",\"+metadata.partition()+\",\"+ metadata.offset()); } } }); } producer.close(); }}消费者高级API：1234567891011121314151617181920212223242526272829303132333435363738394041424344454647package com.sonly.kafka.consumer;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients.consumer.KafkaConsumer;import org.apache.kafka.clients.producer.ProducerConfig;import java.util.Arrays;import java.util.Properties;/** * <b>package:com.sonly.kafka.consumer</b> * <b>project(项目):kafkaAPIdemo</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-03 13:59</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class ConsumerDemo { public static void main(String[] args) { Properties properties = new Properties(); //设置kafka集群 properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,\"k8s-n1:9092\"); //设置brokeACK应答机制 properties.put(ConsumerConfig.GROUP_ID_CONFIG,\"teste3432\"); //设置key反序列化 properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,\"org.apache.kafka.common.serialization.StringDeserializer\"); //设置value反序列化 properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,\"org.apache.kafka.common.serialization.StringDeserializer\"); //设置拿取大小 properties.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG,100*1024*1024); //设置自动提交offset properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,true); //设置自动提交延时 properties.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG,1000); KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(properties); consumer.subscribe(Arrays.asList(\"mytest\",\"test\")); while (true){ ConsumerRecords<String, String> records = consumer.poll(10); for (ConsumerRecord<String, String> record : records) { System.out.println(record.topic()+\"--\"+record.partition()+\"--\"+record.value()); } } }}低级API：1.消费者使用低级API的主要步骤步骤主要工作1根据指定分区从topic元数据中找到leader2获取分区最新的消费进度3从主副本中拉取分区消息4识别主副本的变化，重试2.方法描述：方法描述findLeader()客户端向种子阶段发送主题元数据，将副本加入备用节点getLastOffset()消费者客户端发送偏移量请求，获取分区最近的偏移量run()消费者低级API拉取消息的方法findNewLeader()当分区主副本节点发生故障时，客户端将要找出新的主副本修改pom12345678910<dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.11</artifactId> <version>1.1.1</version></dependency><dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>1.1.1</version></dependency>123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106package com.sonly.kafka.consumer;import kafka.api.FetchRequest;import kafka.api.FetchRequestBuilder;import kafka.api.KAFKA_0_8_1$;import kafka.cluster.BrokerEndPoint;import kafka.javaapi.*;import kafka.javaapi.consumer.SimpleConsumer;import kafka.javaapi.message.ByteBufferMessageSet;import kafka.message.MessageAndOffset;import org.apache.kafka.clients.consumer.Consumer;import java.nio.ByteBuffer;import java.util.*;import java.util.concurrent.TimeUnit;/** * <b>package:com.sonly.kafka.consumer</b> * <b>project(项目):kafkaAPIdemo</b> * <b>class(类)${CLASS_NAME}</b> * <b>creat date(创建时间):2019-05-03 15:21</b> * <b>author(作者):</b>xxydliuyss</br> * <b>note(备注)):</b> * If you want to change the file header,please modify zhe File and Code Templates. */public class LowerConsumer { //保存offset private long offset; //保存分区副本 private Map<Integer,List<BrokerEndPoint>> partitionsMap = new HashMap<Integer, List<BrokerEndPoint>>(1024); public static void main(String[] args) throws InterruptedException { List<String> brokers = Arrays.asList(\"k8s-n1\", \"k8s-n2\",\"k8s-n3\"); int port = 9092; int partition = 1; long offset=2; LowerConsumer lowerConsumer = new LowerConsumer(); while(true){// offset = lowerConsumer.getOffset(); lowerConsumer.getData(brokers,port,\"mytest\",partition,offset); TimeUnit.SECONDS.sleep(1); } } public long getOffset() { return offset; } private BrokerEndPoint findLeader(Collection<String> brokers,int port,String topic,int partition){ for (String broker : brokers) { //创建消费者对象操作每一台服务器 SimpleConsumer getLeader = new SimpleConsumer(broker, port, 10000, 1024 * 24, \"getLeader\"); //构造元数据请求 TopicMetadataRequest topicMetadataRequest = new TopicMetadataRequest(Collections.singletonList(topic)); //发送元数据请求 TopicMetadataResponse response = getLeader.send(topicMetadataRequest); //解析元数据 List<TopicMetadata> topicMetadatas = response.topicsMetadata(); //遍历数据 for (TopicMetadata topicMetadata : topicMetadatas) { //获取分区元数据信息 List<PartitionMetadata> partitionMetadatas = topicMetadata.partitionsMetadata(); //遍历分区元数据 for (PartitionMetadata partitionMetadata : partitionMetadatas) { if(partition == partitionMetadata.partitionId()){ //保存，分区对应的副本，如果需要主副本挂掉重新获取leader只需要遍历这个缓存即可 List<BrokerEndPoint> isr = partitionMetadata.isr(); this.partitionsMap.put(partition,isr); return partitionMetadata.leader(); } } } } return null; } private void getData(Collection<String> brokers,int port,String topic,int partition,long offset){ //获取leader BrokerEndPoint leader = findLeader(brokers, port, topic, partition); if(leader==null) return; String host = leader.host(); //获取数据的消费者对象 SimpleConsumer getData = new SimpleConsumer(host, port, 10000, 1024 * 10, \"getData\"); //构造获取数据request 这里一次可以添加多个topic addFecth 添加即可 FetchRequest fetchRequestBuilder = new FetchRequestBuilder().addFetch(topic, partition, offset, 1024 * 10).build(); //发送获取数据请求 FetchResponse fetchResponse = getData.fetch(fetchRequestBuilder); //解析元数据返回，这是message的一个set集合 ByteBufferMessageSet messageAndOffsets = fetchResponse.messageSet(topic, partition); //遍历消息集合 for (MessageAndOffset messageAndOffset : messageAndOffsets) { long offset1 = messageAndOffset.offset(); this.setOffset(offset); ByteBuffer payload = messageAndOffset.message().payload(); byte[] buffer = new byte[payload.limit()]; payload.get(buffer); String message = new String(buffer); System.out.println(\"offset:\"+ offset1 +\"--message:\"+ message); } } private void setOffset(long offset) { this.offset = offset; }}这个低级API在最新的kafka版本中已经不再提供了。","categories":[{"name":"消息队列","slug":"消息队列","permalink":"http://www.liuyong520.cn/categories/消息队列/"}],"tags":[{"name":"kafka","slug":"kafka","permalink":"http://www.liuyong520.cn/tags/kafka/"}]},{"title":"storm 安装使用","slug":"storm-install","date":"2019-05-02T13:37:40.000Z","updated":"2019-06-11T09:37:15.238Z","comments":true,"path":"2019/05/02/storm-install/","link":"","permalink":"http://www.liuyong520.cn/2019/05/02/storm-install/","excerpt":"","text":"环境准备zookeeper集群环境storm是依赖于zookeeper注册中心的一款分布式消息对列，所以需要有zookeeper单机或者集群环境。准备三台服务器：123172.16.18.198 k8s-n1172.16.18.199 k8s-n2172.16.18.200 k8s-n3下载storm安装包http://storm.apache.org/downloads.html 中下载，目前最新版本的strom已经到1.2.2,我这里之前下载的是1.1.3版本的。安装storm集群上传压缩包到三台服务器解压缩到/opt/目录下12tar -zxf apache-storm-1.1.3.tar.gz -C /optln -sf apache-storm-1.1.3/ storm修改 conf目录下的storm.yml文件Storm包含一个conf/storm.yaml配置Storm守护进程的文件。这个文件里面配置的值会覆盖掉default.yml里面的值，同时里面有一些配置是必须填的注意：yml文件的前面的空格必须有，不然就会出问题，yml配置文件有严格的格式1)storm.zookeeper.servers：这是Storm集群的Zookeeper集群中的主机列表。它应该看起来像：1234storm.zookeeper.servers: - "k8s-n1" - "k8s-n2 - "k8s-n3"2)如果zookeeper的默认端口不是2181的话还需要配置storm.zookeeper.port,如果是2181，此选项可以不用配置1storm.zookeeper.port: 21813）storm.local.dir：Nimbus和Supervisor守护进程需要本地磁盘上的一个目录来存储少量状态（如jar，confs和类似的东西）。您应该在每台计算机上创建该目录，为其提供适当的权限，然后使用此配置填写目录位置。例如：1storm.local.dir: "/data/storm"4）nimbus.seeds：工作节点需要知道哪些机器是主机的候选者才能下载拓扑罐和confs。例如：1nimbus.seeds: ["k8s-n1","k8s-n2","k8s-n3"]4）supervisor.slots.ports：对于每个工作者计算机，您可以使用此配置配置在该计算机上运行的工作程序数。每个工作人员使用单个端口接收消息，此设置定义哪些端口可以使用。如果您在此处定义了五个端口，那么Storm将分配最多五个工作人员在此计算机上运行。如果您定义了三个端口，Storm最多只能运行三个端口。默认情况下，此设置配置为在端口6700,6701,6702和6703上运行4个工作程序。例如：12345supervisor.slots.ports: - 6700 - 6701 - 6702 - 67035)开启监督机制监督健康状况Storm提供了一种机制，管理员可以通过该机制定期管理员定期运行管理员提供的脚本，以确定节点是否健康。管理员可以让主管通过在storm.health.check.dir中的脚本中执行他们选择的任何检查来确定节点是否处于健康状态。如果脚本检测到节点处于不健康状态，则必须从标准输出打印一行，以字符串ERROR开头。主管将定期运行运行状况检查目录中的脚本并检查输出。如果脚本的输出包含字符串ERROR，如上所述，主管将关闭所有工作人员并退出。如果主管在监督下运行，则可以调用“/ bin / storm node-health-check”来确定是否应该启动主管或节点是否运行状况不佳。运行状况检查目录位置可以配置为：1storm.health.check.dir: "healthchecks"脚本必须具有执行权限。允许任何给定的运行状况检查脚本在由于超时而标记为失败之前运行的时间可以配置为：1storm.health.check.timeout.ms: 5000启动Nimbus：在主机监督下运行命令“bin / storm nimbus”。主管：在每台工作机器的监督下运行命令“bin / storm supervisor”。管理程序守护程序负责启动和停止该计算机上的工作进程。UI：通过在监督下运行命令“bin / storm ui”，运行Storm UI（您可以从浏览器访问的站点，该站点提供对群集和拓扑的诊断）。可以通过将Web浏览器导航到http：// {ui host}：8080来访问UI。如您所见，运行守护进程非常简单。守护程序将在您解压缩Storm版本的任何位置登录到logs /目录。后台启动1nohup storm ui >/dev/null 2>&1 &","categories":[{"name":"storm","slug":"storm","permalink":"http://www.liuyong520.cn/categories/storm/"}],"tags":[{"name":"storm","slug":"storm","permalink":"http://www.liuyong520.cn/tags/storm/"}]},{"title":"kafka基本介绍","slug":"kafka-intruduce","date":"2019-04-29T02:56:51.000Z","updated":"2019-06-11T09:37:16.614Z","comments":true,"path":"2019/04/29/kafka-intruduce/","link":"","permalink":"http://www.liuyong520.cn/2019/04/29/kafka-intruduce/","excerpt":"","text":"kafka是什么？Kafka是一个分布式流式存储并处理的消息队列。由scale+java语言编写，它提供了类似于JMS的特性，但是在设计实现上又完全不同，因为kafka并不是按照JMS规范实现的。kafka集群由多个broke（Kafka实例称之为broke）组成，在集群里，kafka通过消息订阅和发布将消息以topic的形式发布出来，同时，消息也是存储在topic中的，消息的发送者成为producer，消息接受者成为Consummer。同时，topic 是根据分区partitions，和副本replications来实现的数据的分布式存储，和加强数据的可靠性。何为topic？一个topic可以认为是一类消息，每个topic将被分成多个partitions，每个partition在存储append log的形式存在文件里的。任何发布到partition的消息都会直接被追加到log文件的末尾，每条消息在文件中的位置称之为offset偏移量，offset为一个long型数字，它唯一标识一条消息，kafka并没有提供其他索引来存储offset，因此kafka不支持消息的随机读写。kafka和JMS（Java Message Service）实现(activeMQ)不同的是:即使消息被消费,消息仍然不会被立即删除.日志文件将会根据broker中的配置要求,保留一定的时间之后(默认是7天)删除;比如log文件保留2天,那么两天后,文件会被清除,无论其中的消息是否被消费.kafka通过这种简单的手段,来释放磁盘空间,以及减少消息消费之后对文件内容改动的磁盘IO开支.kafka消息如何消费的？对于consumer而言,它需要保存消费消息的offset,对于offset的保存和使用,有consumer来控制;当consumer正常消费消息时,offset将会”线性”的向前驱动,即消息将依次顺序被消费.事实上consumer可以使用任意顺序消费消息,它只需要将offset重置为任意值..(kafka 老版本中offset将会保存在zookeeper中,1.x之后也会存储在broke集群里,参见下文)kafka 集群里consumer和producer的状态信息是如何保存的？kafka集群几乎不需要维护任何consumer和producer状态信息,这些信息由zookeeper保存;因此producer和consumer的客户端实现非常轻量级,它们可以随意离开,而不会对集群造成额外的影响.kafka为何要引入分区的概念，有何好处？partitions的设计目的有多个.最根本原因是kafka基于文件存储.通过分区,可以将日志内容分散到多个kafka实例上,来避免文件尺寸达到单机磁盘的上限,每个partiton都会被当前server(kafka实例)保存;可以将一个topic切分多任意多个partitions,来消息保存/消费的效率.此外越多的partitions意味着可以容纳更多的consumer,有效提升并发消费的能力.有负载均衡的功效(具体原理参见下文).kakfa数据是如何写入到磁盘的？一个Topic的多个partitions,被分布在kafka集群中的多个server上;每个server(kafka实例)负责partitions中消息的读写操作;此外kafka还可以配置partitions需要备份的个数(replicas),每个partition将会被备份到多台机器上,以提高可用性.基于replicated方案,那么就意味着需要对多个备份进行调度;每个partition都有一个server为”leader”;leader负责所有的读写操作,如果leader失效,那么将会有其他follower来接管(成为新的leader);follower只是单调的和leader跟进,同步消息即可..由此可见作为leader的server承载了全部的请求压力,因此从集群的整体考虑,有多少个partitions就意味着有多少个”leader”,kafka会将”leader”均衡的分散在每个实例上,来确保整体的性能稳定.这和zookeeper的follower是有区别的：zookeeper的follower是可以读到数据的，而kafka的follower是读不到数据的。kafka使用文件存储消息,这就直接决定kafka在性能上严重依赖文件系统的本身特性.且无论任何OS下,对文件系统本身的优化几乎没有可能.文件缓存/直接内存映射等是常用的手段.因为kafka是对日志文件进行append操作,因此磁盘检索的开支是较小的;同时为了减少磁盘写入的次数,broker会将消息暂时buffer起来,当消息的个数(或尺寸)达到一定阀值时,再flush到磁盘,这样减少了磁盘IO调用的次数.kafka中消费者组如何理解？Producer将消息发布到指定的Topic中,同时Producer也能决定将此消息归属于哪个partition;比如基于”round-robin”方式或者通过其他的一些算法等.本质上kafka只支持Topic.每个consumer属于一个consumer group;反过来说,每个group中可以有多个consumer.发送到Topic的消息,只会被订阅此Topic的每个group中的一个consumer消费.如果所有的consumer都具有相同的group,这种情况和queue模式很像;消息将会在consumers之间负载均衡.如果所有的consumer都具有不同的group,那这就是”发布-订阅”;消息将会广播给所有的消费者.在kafka中,一个partition中的消息只会被group中的一个consumer消费;每个group中consumer消息消费互相独立;我们可以认为一个group是一个”订阅”者,一个Topic中的每个partions,只会被一个”订阅者”中的一个consumer消费,不过一个consumer可以消费多个partitions中的消息.kafka只能保证一个partition中的消息被某个consumer消费时,消息是顺序的.事实上,从Topic角度来说,消息仍不是有序的. 因为消费者消费消息的时候是按照分区依次读取的，所以无法保证消息的全局顺序性，只能保证在同一个分区内的消息是顺序的。如果想要所有的消息都是顺序的，可以把分区数设置为1.kafka中如何保证数据一段时间内不丢失？kafka 的producer有ACK机制。可以由用户自行设定是否开启确认机制，如果开启确认机制，kafka会等发送消息到kafka集群时，当leader服务器，会返回元数据给producer客户端，ACK机制也在元数据里，这里的ACK有两种，一种就是leader只要接收成功，就返回确认，另外一种就是：要等所有follower都收到了之后才返回确认。producer在接收到确认之后，才会发下一条消息。而所有的消息最终都是存储在磁盘一段时间的，所以一段时间内消息是不会丢失的。kafka 的应用场景主要有哪些？官方介绍是讲可以用作message queue，数据采集，简单流式计算等。用作消息队列message queue有哪些优缺点？对于一些常规的消息系统,kafka是个不错的选择;partitons/replication和容错,可以使kafka具有良好的扩展性和性能优势.不过到目前为止,我们应该很清楚认识到,kafka并没有提供JMS中的”事务性””消息传输担保(消息确认机制)””消息分组”等企业级特性;kafka只能使用作为”常规”的消息系统,在一定程度上,尚未确保消息的发送与接收绝对可靠(比如,消息重发,消息发送丢失等)kafka是如何保持高性能的？需要考虑的影响性能点很多,除磁盘IO之外,我们还需要考虑网络IO,这直接关系到kafka的吞吐量问题.kafka并没有提供太多高超的技巧;对于producer端,可以将消息buffer起来,当消息的条数达到一定阀值时,批量发送给broker;对于consumer端也是一样,批量fetch多条消息.不过消息量的大小可以通过配置文件来指定.对于kafka broker端,似乎有个sendfile系统调用可以潜在的提升网络IO的性能:将文件的数据映射到系统内存中,socket直接读取相应的内存区域即可,而无需进程再次copy和交换. 其实对于producer/consumer/broker三者而言,CPU的开支应该都不大,因此启用消息压缩机制是一个良好的策略;压缩需要消耗少量的CPU资源,不过对于kafka而言,网络IO更应该需要考虑.可以将任何在网络上传输的消息都经过压缩.kafka支持gzip/snappy等多种压缩方式.kafka在消费者端有哪些异常处理策略？对于JMS实现,消息传输担保非常直接:有且只有一次(exactly once).在kafka中稍有不同:1) at most once: 最多一次,这个和JMS中”非持久化”消息类似.发送一次,无论成败,将不会重发.2) at least once: 消息至少发送一次,如果消息未能接受成功,可能会重发,直到接收成功.3) exactly once: 消息只会发送一次.at most once: 消费者fetch消息,然后保存offset,然后处理消息;当client保存offset之后,但是在消息处理过程中出现了异常,导致部分消息未能继续处理.那么此后”未处理”的消息将不能被fetch到,这就是”at most once”.at least once: 消费者fetch消息,然后处理消息,然后保存offset.如果消息处理成功之后,但是在保存offset阶段zookeeper异常导致保存操作未能执行成功,这就导致接下来再次fetch时可能获得上次已经处理过的消息,这就是”at least once”，原因offset没有及时的提交给zookeeper，zookeeper恢复正常还是之前offset状态.exactly once: kafka中并没有严格的去实现基于2阶段提交,事务),我们认为这种策略在kafka中是没有必要的.通常情况下”at-least-once”是我们搜选.(相比at most once而言,重复接收数据总比丢失数据要好).kafka 工作流程是怎样的？主要结构图：大体可以从三个方面分析：生产者产生消息、消费者消费消息、Broker cluster保存消息。生产者产生消息过程分析写入方式：producer 采用push的方式将消息发送到broker cluster，每条消息都被追加到分区中，属于顺序写磁盘（顺序写磁盘效率比随机写内存效率要高，能提高Kafka吞吐率）而且broker集群并不是每一条消息都及时写磁盘，而是先写buffer，达到一定大小或者每隔一段时间再flush到磁盘上。多个producer可以给同一个topic 发布消息，而且可以指定分区发布。分区Partition每个Topic可以有多个分区，而消息最终是存储在磁盘的文件里的，Partition在磁盘上是文件夹的形式存在的。如12345678cd /var/applog/kafka/ ## 赚到kafka数据目录 即log.dir=配置的目录lscleaner-offset-checkpoint __consumer_offsets-22 __consumer_offsets-4 log-start-offset-checkpoint recovery-point-offset-checkpoint__consumer_offsets-1 __consumer_offsets-25 __consumer_offsets-40 meta.properties replication-offset-checkpoint__consumer_offsets-10 __consumer_offsets-28 __consumer_offsets-43 mytest-0 test-0__consumer_offsets-13 __consumer_offsets-31 __consumer_offsets-46 mytest-1__consumer_offsets-16 __consumer_offsets-34 __consumer_offsets-49 mytest-2__consumer_offsets-19 __consumer_offsets-37 __consumer_offsets-7 mytest-3其中mytest-0 mytest-1 mytest-2 mytest-3 即为分区Partition，里面的文件就是分区里面存放的数据。broker cluster 保存消息broker 收到消息后，首先会去找topic对应分区的leader，找到leader后，先将数据写入buffer，再flush到磁盘。然后zookeeper会协调follower自动同步leader分区的数据，以达到replication备份的目的，同时leader会按照备份完成的先后顺序给follower作一次排序，作为leader发生意外时选举时选举为leader的顺序。消费者消费消息消费者消费消息，同一个分区里的数据不能够被一个消费组里面的多个消费者同时消费，同一个消费组里的消费者只能消费不同分区的数据。不同消费者组可以消费同一个分区里的数据。消费者消费数据时是按照分区的一个一个分区数据进行消费的。zookeeper在kafka中的具体作用是什么？kafka是依赖于zookeeper注册中心的，主要来协调各个broker的分区备份，broker的选举，以及消费者相关状信息的存储。kafka使用zookeeper来存储一些meta信息,并使用了zookeeper watch机制来发现meta信息的变更并作出相应的动作(比如consumer失效,触发负载均衡等)1) Broker node registry: 当一个kafkabroker启动后,首先会向zookeeper注册自己的节点信息(临时znode),同时当broker和zookeeper断开连接时,此znode也会被删除.格式: /broker/ids/[0…N] –>host:port;其中[0..N]表示broker id,每个broker的配置文件中都需要指定一个数字类型的id(全局不可重复),znode的值为此broker的host:port信息.123456789101112131415161718$ zkCli -server k8s-n1:2181$ ls /brokers[ids, topics, seqid]$ ls /brokers/ids[0, 1, 2]$ get /brokers/ids/0{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://k8s-n1:9092"],"jmx_port":-1,"host":"k8s-n1","timestamp":"1556568752340","port":9092,"version":4}cZxid = 0xd0000003cctime = Wed Apr 24 16:10:19 CST 2019mZxid = 0xd0000003cmtime = Wed Apr 24 16:10:19 CST 2019pZxid = 0xd0000003ccversion = 0dataVersion = 1aclVersion = 0ephemeralOwner = 0x26a4e173fc40002dataLength = 182numChildren = 02) Broker Topic Registry: 当一个broker启动时,会向zookeeper注册自己持有的topic和partitions信息,仍然是一个临时znode.格式: /broker/topics/[topic]/[0…N] 其中[0..N]表示partition索引号.12$ ls /brokers/topics[test, __consumer_offsets]__consumer_offsets 是消费端的offset12345678910111213141516171819$ ls /brokers/topics/test[partitions] ##test的分区信息$ ls /brokers/topics/test/partitions[0]$ ls /brokers/topics/test/partitions/0[state]$ get /brokers/topics/test/partitions/0/state {"controller_epoch":19,"leader":0,"version":1,"leader_epoch":3,"isr":[0]}cZxid = 0x2000000b6ctime = Wed Apr 24 07:53:42 CST 2019mZxid = 0xd00000044mtime = Wed Apr 24 16:10:19 CST 2019pZxid = 0x2000000b6cversion = 0dataVersion = 3aclVersion = 0ephemeralOwner = 0x0dataLength = 73numChildren = 03) Consumer and Consumer group: 每个consumer客户端被创建时,会向zookeeper注册自己的信息;此作用主要是为了”负载均衡”.一个group中的多个consumer可以交错的消费一个topic的所有partitions;简而言之,保证此topic的所有partitions都能被此group所消费,且消费时为了性能考虑,让partition相对均衡的分散到每个consumer上.4) Consumer id Registry: 每个consumer都有一个唯一的ID(host:uuid,可以通过配置文件指定,也可以由系统生成),此id用来标记消费者信息.格式:/consumers/[group_id]/ids/[consumer_id]仍然是一个临时的znode,此节点的值为{“topic_name”:#streams…},即表示此consumer目前所消费的topic + partitions列表.启动消费者：1$ kafka-console-consumer.sh --bootstrap-server k8s-n2:9092 --topic test启动生成者：1kafka-console-producer.sh --broker-list k8s-n1:9092 --topic test查看zookeeper信息：1234$ ls /[cluster, controller_epoch, controller, brokers, zookeeper, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config]$ ls /consumers[]发现consummer下啥也没有？这是因为新版本的kafka，consumer中offset不是放在这个位置的，而是放在__consumer_offset 这个topic下的。那么该如何验证呢？启动消费者：1$ kafka-console-consumer.sh --bootstrap-server k8s-n2:9092 --topic test启动生成者：1kafka-console-producer.sh --broker-list k8s-n1:9092 --topic test验证消息生产成功12345kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list k8s-n1:9092 --topic mytest --time -1mytest:0:15mytest:1:16mytest:2:16mytest:3:15mytest topic 上 0号分区有15条消息。很好理解。再创建一个消费者组1kafka-console-consumer.sh --bootstrap-server k8s-n1:9092 --topic mytest --from-beginning查询一下消费者组信息123kafka-consumer-groups.sh --bootstrap-server k8s-n1:9092 --listconsole-consumer-24766console-consumer-52794查询一下topic里的内容：1kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server k8s-n1:9092 --formatter "kafka.coordinator.group.GroupMetadataManager\\$OffsetsMessageFormatter" --consumer.config config/consumer.properties --from-beginning结果：1234567891011 [console-consumer-52794,__consumer_offsets,12]::OffsetAndMetadata(offset=0, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1556122524504, expireTimestamp=None)[console-consumer-52794,__consumer_offsets,45]::OffsetAndMetadata(offset=0, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1556122524504, expireTimestamp=None)[console-consumer-52794,__consumer_offsets,1]::OffsetAndMetadata(offset=0, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1556122524504, expireTimestamp=None)[console-consumer-52794,__consumer_offsets,5]::OffsetAndMetadata(offset=0, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1556122524504, expireTimestamp=None)[console-consumer-52794,__consumer_offsets,26]::OffsetAndMetadata(offset=0, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1556122524504, expireTimestamp=None)[console-consumer-52794,__consumer_offsets,29]::OffsetAndMetadata(offset=0, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1556122524504, expireTimestamp=None)[console-consumer-52794,__consumer_offsets,34]::OffsetAndMetadata(offset=0, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1556122524504, expireTimestamp=None)[console-consumer-52794,__consumer_offsets,10]::OffsetAndMetadata(offset=0, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1556122524504, expireTimestamp=None)[console-consumer-52794,__consumer_offsets,32]::OffsetAndMetadata(offset=5, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1556122524504, expireTimestamp=None)[console-consumer-52794,__consumer_offsets,40]::OffsetAndMetadata(offset=3, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1556122524504, expireTimestamp=None)^CProcessed a total of 1674 messages参考了 http://www.cnblogs.com/huxi2b/p/6061110.html这篇blog的作法，但是我的版本是kafka_2.2.0里面并没有找offset的命令。5) Consumer offset Tracking: 用来跟踪每个consumer目前所消费的partition中最大的offset.格式:/consumers/[group_id]/offsets/[topic]/[broker_id-partition_id]–>offset_value此znode为持久节点,可以看出offset跟group_id有关,以表明当group中一个消费者失效,其他consumer可以继续消费.6) Partition Owner registry: 用来标记partition被哪个consumer消费.临时znode格式:/consumers/[group_id]/owners/[topic]/[broker_id-partition_id]–>consumer_node_id当consumer启动时,所触发的操作:A) 首先进行”Consumer id Registry”;B) 然后在”Consumer id Registry”节点下注册一个watch用来监听当前group中其他consumer的”leave”和”join”;只要此znode path下节点列表变更,都会触发此group下consumer的负载均衡.(比如一个consumer失效,那么其他consumer接管partitions).C) 在”Broker id registry”节点下,注册一个watch用来监听broker的存活情况;如果broker列表变更,将会触发所有的groups下的consumer重新balance.","categories":[{"name":"消息队列","slug":"消息队列","permalink":"http://www.liuyong520.cn/categories/消息队列/"}],"tags":[{"name":"kafka","slug":"kafka","permalink":"http://www.liuyong520.cn/tags/kafka/"},{"name":"linux","slug":"linux","permalink":"http://www.liuyong520.cn/tags/linux/"}]},{"title":"Linux下kafka集群搭建","slug":"kafka-install","date":"2019-04-29T02:56:51.000Z","updated":"2019-06-11T09:37:16.613Z","comments":true,"path":"2019/04/29/kafka-install/","link":"","permalink":"http://www.liuyong520.cn/2019/04/29/kafka-install/","excerpt":"","text":"环境准备zookeeper集群环境kafka是依赖于zookeeper注册中心的一款分布式消息对列，所以需要有zookeeper单机或者集群环境。三台服务器：123172.16.18.198 k8s-n1172.16.18.199 k8s-n2172.16.18.200 k8s-n3下载kafka安装包http://kafka.apache.org/downloads 中下载，目前最新版本的kafka已经到2.2.0,我这里之前下载的是kafka_2.11-2.2.0.tgz.安装kafka集群上传压缩包到三台服务器解压缩到/opt/目录下12tar -zxvf kafka_2.11-2.2.0.tgz -C /opt/ls -s kafka_2.11-2.2.0 kafka修改 server.properties123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121############################# Server Basics ############################## The id of the broker. This must be set to a unique integer for each broker.broker.id=0############################# Socket Server Settings ############################## The address the socket server listens on. It will get the value returned from # java.net.InetAddress.getCanonicalHostName() if not configured.# FORMAT:# listeners = listener_name://host_name:port# EXAMPLE:# listeners = PLAINTEXT://your.host.name:9092listeners=PLAINTEXT://k8s-n1:9092# Hostname and port the broker will advertise to producers and consumers. If not set, # it uses the value for "listeners" if configured. Otherwise, it will use the value# returned from java.net.InetAddress.getCanonicalHostName().advertised.listeners=PLAINTEXT://k8s-n1:9092# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL# The number of threads that the server uses for receiving requests from the network and sending responses to the networknum.network.threads=3# The number of threads that the server uses for processing requests, which may include disk I/Onum.io.threads=8# The send buffer (SO_SNDBUF) used by the socket serversocket.send.buffer.bytes=102400# The receive buffer (SO_RCVBUF) used by the socket serversocket.receive.buffer.bytes=102400# The maximum size of a request that the socket server will accept (protection against OOM)socket.request.max.bytes=104857600############################# Log Basics ############################## A comma separated list of directories under which to store log fileslog.dirs=/var/applog/kafka/# The default number of log partitions per topic. More partitions allow greater# parallelism for consumption, but this will also result in more files across# the brokers.num.partitions=5# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.# This value is recommended to be increased for installations with data dirs located in RAID array.num.recovery.threads.per.data.dir=1############################# Internal Topic Settings ############################## The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.offsets.topic.replication.factor=1transaction.state.log.replication.factor=1transaction.state.log.min.isr=1############################# Log Flush Policy ############################## Messages are immediately written to the filesystem but by default we only fsync() to sync# the OS cache lazily. The following configurations control the flush of data to disk.# There are a few important trade-offs here:# 1. Durability: Unflushed data may be lost if you are not using replication.# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.# The settings below allow one to configure the flush policy to flush data after a period of time or# every N messages (or both). This can be done globally and overridden on a per-topic basis.# The number of messages to accept before forcing a flush of data to disklog.flush.interval.messages=10000# The maximum amount of time a message can sit in a log before we force a flushlog.flush.interval.ms=1000############################# Log Retention Policy ############################## The following configurations control the disposal of log segments. The policy can# be set to delete segments after a period of time, or after a given size has accumulated.# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens# from the end of the log.# The minimum age of a log file to be eligible for deletion due to agelog.retention.hours=24# A size-based retention policy for logs. Segments are pruned from the log unless the remaining# segments drop below log.retention.bytes. Functions independently of log.retention.hours.#log.retention.bytes=1073741824# The maximum size of a log segment file. When this size is reached a new log segment will be created.log.segment.bytes=1073741824# The interval at which log segments are checked to see if they can be deleted according# to the retention policieslog.retention.check.interval.ms=300000############################# Zookeeper ############################## Zookeeper connection string (see zookeeper docs for details).# This is a comma separated host:port pairs, each corresponding to a zk# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".# You can also append an optional chroot string to the urls to specify the# root directory for all kafka znodes.zookeeper.connect=k8s-n1:2181,k8s-n2:2181,k8s-n3:2181# Timeout in ms for connecting to zookeeperzookeeper.connection.timeout.ms=6000############################# Group Coordinator Settings ############################## The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.# The default value for this is 3 seconds.# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.group.initial.rebalance.delay.ms=0delete.topic.enable=true拷贝两份到k8s-n2,k8s-n3123456789[root@k8s-n2 config]# cat server.properties broker.id=1listeners=PLAINTEXT://k8s-n2:9092advertised.listeners=PLAINTEXT://k8s-n2:9092[root@k8s-n3 config]# cat server.propertiesbroker.id=2listeners=PLAINTEXT://k8s-n3:9092advertised.listeners=PLAINTEXT://k8s-n3:9092添加环境变量 在/etc/profile 中添加12export ZOOKEEPER_HOME=/opt/kafka_2.11-2.2.0export PATH=$PATH:$ZOOKEEPER_HOME/binsource /etc/profile 重载生效启动kafka1kafka-server-start.sh config/server.properties &Zookeeper+Kafka集群测试创建topic:1kafka-topics.sh --create --zookeeper k8s-n1:2181, k8s-n2:2181, k8s-n3:2181 --replication-factor 3 --partitions 3 --topic test显示topic1kafka-topics.sh --describe --zookeeper k8s-n1:2181, k8s-n2:2181, k8s-n3:2181 --topic test列出topic12kafka-topics.sh --list --zookeeper k8s-n1:2181, k8s-n2:2181, k8s-n3:2181test创建 producer(生产者);12kafka-console-producer.sh --broker-list k8s-n1:9092 --topic testhello创建 consumer（消费者）12kafka-console-consumer.sh --bootstrap-server k8s-n1:9092 --topic test --from-beginninghello至此，kafka集群搭建就已经完成了。","categories":[{"name":"消息队列","slug":"消息队列","permalink":"http://www.liuyong520.cn/categories/消息队列/"}],"tags":[{"name":"kafka","slug":"kafka","permalink":"http://www.liuyong520.cn/tags/kafka/"},{"name":"linux","slug":"linux","permalink":"http://www.liuyong520.cn/tags/linux/"}]},{"title":"Linux下zookeeper集群搭建","slug":"zookeeper-install","date":"2019-04-29T02:56:51.000Z","updated":"2019-06-11T09:37:16.607Z","comments":true,"path":"2019/04/29/zookeeper-install/","link":"","permalink":"http://www.liuyong520.cn/2019/04/29/zookeeper-install/","excerpt":"","text":"部署前准备下载zookeeper的安装包http://zookeeper.apache.org/releases.html 我下载的版本是zookeeper-3.4.10。准备三台服务器ip地址为：123172.16.18.198172.16.18.199172.16.18.200检查jdk版本，安装jdk环境，jdk需要1.7以上。安装zookeeper三台服务器分别上传zookeeper安装包，上传到/opt/目录下，然后tar zxvf zookeeper-3.4.10.tar.gz拷贝zoo_sample.cfg 为zoo.cfg 修改/opt/zookeeper-3.4.10/conf/zoo.cfg配置文件，添加如下内容：123server.1=172.16.18.198:2888:3888server.2=172.16.18.199:2888:3888server.3=172.16.18.200:2888:3888修改zookeeper数据文件存放目录1dataDir=/data/zookeeper此时zoo.cfg 配置文件内容为：12345678910111213141516171819202122232425262728293031# The number of milliseconds of each ticktickTime=2000 ##zookeeper单位时间为2ms# The number of ticks that the initial # synchronization phase can takeinitLimit=10 ##对于从节点最初连接到主节点时的超时时间，单位为tick值的倍数。即20ms# The number of ticks that can pass between # sending a request and getting an acknowledgementsyncLimit=5 ##对于主节点与从节点进行同步操作时的超时时间，单位为tick值的倍数。即10ms# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just # example sakes.dataDir=/data/zookeeper# the port at which the clients will connectclientPort=2181 ##客户端链接端口# the maximum number of client connections.# increase this if you need to handle more clientsmaxClientCnxns=60 ##客户端最大链接数## Be sure to read the maintenance section of the # administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to "0" to disable auto purge feature#autopurge.purgeInterval=1server.1=172.16.18.198:2888:3888 server.2=172.16.18.199:2888:3888server.3=172.16.18.200:2888:3888新建myid文件在三台服务器的数据存放目录下新建myid文件，并写入对应的server.num 中的num数字如：在172.16.18.198上将server.1中1写入myid1echo 1 >/data/zookeeper/myid添加环境变量，方便我们执行脚本命令vi etc/profile 在最后添加如下两个。12export ZOOKEEPER_HOME=/opt/zookeeper-3.4.9export PATH=$PATH:$ZOOKEEPER_HOME/bin:$ZOOKEEPER_HOME/conf保存后重新加载一下：1source /etc/profile修改日志存放目录（可选）vi /opt/zookeeper/bin/zkEnv.sh 找到ZOO_LOG_DIR 和 ZOO_LOG4J_PROP位置1234567891011if [ "x${ZOO_LOG_DIR}" = "x" ] then #配置zookeeper日志输出存放路径 ZOO_LOG_DIR="/var/applog/zookeeper" fi if [ "x${ZOO_LOG4J_PROP}" = "x" ] then #配置日志输出级别,这里把几个级别一并配上 ZOO_LOG4J_PROP="INFO,CONSOLE,ROLLINGFILE,TRACEFILE" fi编辑conf目录下log4j.properties123456789# Define some default values that can be overridden by system properties zookeeper.root.logger=INFO, CONSOLE, ROLLINGFILE, TRACEFILE zookeeper.console.threshold=INFO zookeeper.log.dir=. zookeeper.log.file=zookeeper.log zookeeper.log.threshold=ERROR zookeeper.tracelog.dir=. zookeeper.tracelog.file=zookeeper_trace.log log4j.rootLogger=${zookeeper.root.logger}完成log的日志目录的修改。7.启动zookeeper服务zkServer.sh start来启动。zkServer.sh restart (重启)zkServer.sh status (查看状态)zkServer.sh stop (关闭)zkServer.sh start-foreground (以打印日志方式启动)三台服务器分别执行：1zkServer.sh start然后用 status 检查下状态 如果出现 Mode：leader 或者Mode:follower 表示搭建成功。否则前台执行看一下日志。1234$ zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: /opt/zookeeper-3.4.10/bin/../conf/zoo.cfgMode: follower如出现：123456789102019-04-29 14:04:05,992 [myid:3] - INFO [ListenerThread:QuorumCnxManager$Listener@739] - My election bind port: /172.16.18.200:38882019-04-29 14:04:06,019 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumPeer@865] - LOOKING2019-04-29 14:04:06,025 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@818] - New election. My id = 3, proposed zxid=0x02019-04-29 14:04:06,056 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@588] - Cannot open channel to 1 at election address /172.16.18.198:3888java.net.NoRouteToHostException: 没有到主机的路由 at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)"zookeeper.log" 303L, 35429C报这种异常一般有三种情况：1）：zoo.cfg配置文件中，server.x:2888:3888配置出现错误；2）：myid文件内容和server.x不对应，或者myid不在data目录下；3）：系统防火墙是否在启动。我检查了三种原因后发现是防火墙running。centos7下查看防火墙状态的命令：1firewall-cmd --state关闭防火墙的命令：12systemctl stop firewalld.servicesystemctl disable firewalld.service （禁止开机启动，永久关闭防火墙）关闭防火墙后重启即可。验证是否成功在命令行中输入：zkCli.sh -server 172.16.18.198:2181（由于本人在不同的办公地点在修改该文章，所以ip地址也在变化，知道原理即可）即可连接到其中一台ZooKeeper服务器。其他自动实现同步，客户端只需要和一台保持连接即可。出现如下表示链接成功1234WATCHER::WatchedEvent state:SyncConnected type:None path:null[zk: 172.16.18.198:2181(CONNECTED) 0]","categories":[{"name":"分布式集群","slug":"分布式集群","permalink":"http://www.liuyong520.cn/categories/分布式集群/"}],"tags":[{"name":"linux","slug":"linux","permalink":"http://www.liuyong520.cn/tags/linux/"},{"name":"ZooKeeper","slug":"ZooKeeper","permalink":"http://www.liuyong520.cn/tags/ZooKeeper/"}]},{"title":"mysql 快速安装","slug":"mysql-install","date":"2018-06-09T01:12:23.000Z","updated":"2019-06-11T09:37:15.280Z","comments":true,"path":"2018/06/09/mysql-install/","link":"","permalink":"http://www.liuyong520.cn/2018/06/09/mysql-install/","excerpt":"","text":"废话少说直接贴脚本123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384#!/usr/bin/bash##install_mysql.sh mysql-8.0.15-linux-glibc2.12-x86_64.tar.xzyum install -y libaiomysqltar=$1extension=\"${mysqltar#*x86_64}\"filename=`basename $1 ${extension}`tar -xvf ${mysqltar} && mv -f ${filename} /usr/local/mysql##添加权限groupadd mysql##添加用户mysqluseradd -r -g mysql mysqlmkdir -p /etc/my.cnf.d/mkdir -p /data/log/mysql-log/mkdir -p /usr/local/mysql/datacd /data/log/mysql-log/ && touch error.logchown -R mysql:mysql /data/log/mysql-log/MYSQL_HOME=`cd -P /usr/local/mysql; pwd`echo \"修改权限目录：${MYSQL_HOME}\"export MYSQL_HOME=${MYSQL_HOME}export PATH=$PATH:$MYSQL_HOME/bin##修改目录权限chown -R mysql:mysql ${MYSQL_HOME}mysqlpath=`cd ${MYSQL_HOME} && pwd`cat > /etc/my.cnf <<EOF[client]port=3306 # 设置mysql客户端连接服务端时默认使用的端口default-character-set=utf8socket=/usr/local/mysql/data/mysql.sock[mysqld]# 设置mysql的安装目录basedir=/usr/local/mysql#datadir=/usr/local/mysql/datasocket=/usr/local/mysql/data/mysql.sock# 设置3306端口port=3306# 允许最大连接数max_connections=10000# 允许连接失败的次数。这是为了防止有人从该主机试图攻击数据库系统max_connect_errors=10# 服务端使用的字符集默认为UTF8character-set-server=UTF8MB4# 创建新表时将使用的默认存储引擎default-storage-engine=INNODB# 默认使用“mysql_native_password”插件认证default_authentication_plugin=mysql_native_password# Settings user and group are ignored when systemd is used.# If you need to run mysqld under a different user or group,# customize your systemd unit file for mariadb according to the# instructions in http://fedoraproject.org/wiki/Systemd[mysqld_safe]log-error=/data/log/mysql-log/error.logpid-file=/usr/local/mysql/data/mysql.pid## include all files from the config directory#!includedir /etc/my.cnf.dEOF##初始化mysqlecho \"初始化mysql${mysqlpath}\"${MYSQL_HOME}/bin/mysqld --user=mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/dataecho \"添加启动服务\"cp ${MYSQL_HOME}/support-files/mysql.server /etc/init.d/mysqlecho \"启动服务\"service mysql start && service mysql statuschkconfig --add mysqlecho \"设置root用户密码\"${MYSQL_HOME}/bin/mysqladmin -u root password '123456'${MYSQL_HOME}/bin/mysql -uroot -p123456 -e \"use mysql;update user set host = '%' where user = 'root';FLUSH PRIVILEGES;\"安装包地址：链接: https://pan.baidu.com/s/1nUZNjWrvJSqGXT55CUcj7w 提取码: nc3i 复制这段内容后打开百度网盘手机App，操作更方便哦mysql多实例运行利用mysqld_muti来运行mysqld_multi多实例启动工具我们往往喜欢在一台服务器上安装多个实例，一般使用不同的端口，如3306，3307等，那么怎么才能启动这些实例呢？怎么才能一起启动呢？又怎么才能一个一个启动呢？MySQL很人性化的提供了一款自带的工具：mysqld_multi，可以毫无压力地满足我们对多实例启动的方式。mysqld_multi准备知识mysqld_multi启动会查找my.cnf文件中的[mysqldN]组，N为mysqld_multi后携带的整数值。mysqld_multi的固定选项可在配置文件my.cnf中进行配置，在[mysqld_multi]组下配置（如果没有该组，可自行建立）。mysqld_multi使用方式如下：1mysqld_multi [options] {start|stop|reload|report} [GNR[,GNR] ...]mysqld_multi选项start 开启MySQL实例stop 关闭MySQL实例reload 重新加载MySQL实例report 返回MySQL当前状态12345678910111213141516171819202122# report返回当前MySQL状态[root@mysql log]# mysqld_multi report 3307Reporting MySQL serversMySQL server from group: mysqld3307 is not running# start开启MySQL实例[root@mysql log]# mysqld_multi start 3307[root@mysql log]# mysqld_multi report 3307Reporting MySQL serversMySQL server from group: mysqld3307 is running# reload重新加载MySQL实例[root@mysql log]# mysqld_multi reload 3307[root@mysql log]# mysqld_multi report 3307Reporting MySQL serversMySQL server from group: mysqld3307 is running# stop关闭MySQL实例，注意此处是需要一个具有shutdown权限的用户，且密码并被是加密的，也不可以交互式输入密码，Linux又具有history功能，所以为了数据库的安全，还是不要用mysqld_multi stop的方式关闭数据库了吧[root@mysql log]# mysqld_multi stop 3307 --user=root --password=******[root@mysql log]# mysqld_multi report 3307Reporting MySQL serversMySQL server from group: mysqld3307 is not running–example 输出一个mysqld_multi配置文件中的配置示例123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122 mysqld_multi --example# This is an example of a my.cnf file for mysqld_multi.# Usually this file is located in home dir ~/.my.cnf or /etc/my.cnf## SOME IMPORTANT NOTES FOLLOW:## 1.COMMON USER## Make sure that the MySQL user, who is stopping the mysqld services, has# the same password to all MySQL servers being accessed by mysqld_multi.# This user needs to have the 'Shutdown_priv' -privilege, but for security# reasons should have no other privileges. It is advised that you create a# common 'multi_admin' user for all MySQL servers being controlled by# mysqld_multi. Here is an example how to do it:## GRANT SHUTDOWN ON *.* TO multi_admin@localhost IDENTIFIED BY 'password'## You will need to apply the above to all MySQL servers that are being# controlled by mysqld_multi. 'multi_admin' will shutdown the servers# using 'mysqladmin' -binary, when 'mysqld_multi stop' is being called.## 2.PID-FILE## If you are using mysqld_safe to start mysqld, make sure that every# MySQL server has a separate pid-file. In order to use mysqld_safe# via mysqld_multi, you need to use two options:## mysqld=/path/to/mysqld_safe# ledir=/path/to/mysqld-binary/## ledir (library executable directory), is an option that only mysqld_safe# accepts, so you will get an error if you try to pass it to mysqld directly.# For this reason you might want to use the above options within [mysqld#]# group directly.## 3.DATA DIRECTORY## It is NOT advised to run many MySQL servers within the same data directory.# You can do so, but please make sure to understand and deal with the# underlying caveats. In short they are:# - Speed penalty# - Risk of table/data corruption# - Data synchronising problems between the running servers# - Heavily media (disk) bound# - Relies on the system (external) file locking# - Is not applicable with all table types. (Such as InnoDB)# Trying so will end up with undesirable results.## 4.TCP/IP Port## Every server requires one and it must be unique.## 5.[mysqld#] Groups## In the example below the first and the fifth mysqld group was# intentionally left out. You may have 'gaps' in the config file. This# gives you more flexibility.## 6.MySQL Server User## You can pass the user=... option inside [mysqld#] groups. This# can be very handy in some cases, but then you need to run mysqld_multi# as UNIX root.## 7.A Start-up Manage Script for mysqld_multi## In the recent MySQL distributions you can find a file called# mysqld_multi.server.sh. It is a wrapper for mysqld_multi. This can# be used to start and stop multiple servers during boot and shutdown.## You can place the file in /etc/init.d/mysqld_multi.server.sh and# make the needed symbolic links to it from various run levels# (as per Linux/Unix standard). You may even replace the# /etc/init.d/mysql.server script with it.## Before using, you must create a my.cnf file either in /usr/my.cnf# or /root/.my.cnf and add the [mysqld_multi] and [mysqld#] groups.## The script can be found from support-files/mysqld_multi.server.sh# in MySQL distribution. (Verify the script before using)#[mysqld_multi]mysqld = /usr/bin/mysqld_safemysqladmin = /usr/bin/mysqladminuser = multi_adminpassword = my_password[mysqld2]socket = /tmp/mysql.sock2port = 3307pid-file = /var/lib/mysql2/hostname.pid2datadir = /var/lib/mysql2language = /usr/share/mysql/mysql/englishuser = unix_user1[mysqld3]mysqld = /path/to/mysqld_safeledir = /path/to/mysqld-binary/mysqladmin = /path/to/mysqladminsocket = /tmp/mysql.sock3port = 3308pid-file = /var/lib/mysql3/hostname.pid3datadir = /var/lib/mysql3language = /usr/share/mysql/mysql/swedishuser = unix_user2[mysqld4]socket = /tmp/mysql.sock4port = 3309pid-file = /var/lib/mysql4/hostname.pid4datadir = /var/lib/mysql4language = /usr/share/mysql/mysql/estoniauser = unix_user3 [mysqld6]socket = /tmp/mysql.sock6port = 3311pid-file = /var/lib/mysql6/hostname.pid6datadir = /var/lib/mysql6language = /usr/share/mysql/mysql/japaneseuser = unix_user4–log=file_name 指定一个日志输出文件，如果文件存在则在文件末尾处添加日志信息–mysqladmin=pro_name 用于指定一个程序来实现mysqladmin的功能–mysqld=pro_name 用于指定一个程序来实现mysqld的功能，如mysqld_safe–no-log 将日志信息输出到屏幕上，而不是输入日志文件中–password= mysqladmin用户的密码–silent 简要信息–user= mysqladmin用户，默认为mysql–tcp-ip 连接到每个服务器的tcp/ip端口，有时候sock文件丢失，但仍然可以通过tcp/ip端口连接服务器上面是介绍例子：在/etc/mysql/目录下新建mysqld_muti.cnf 内容如下：123456789101112131415161718192021222324252627282930313233343536373839[mysqld_multi]mysqld=/usr/local/mysql/bin/mysqld_safemysqladmin=/usr/local/mysql/bin/mysqladminuser=rootpassword=123456log=/var/log/mysqld_multi.log[mysqld3307]port=3307pid-file=/var/lib/mysql3307/mysql3307.pidsocket=/var/lib/mysql3307/mysql3307.sockdatadir=/var/lib/mysql3307user=mysqllog_bin=mysql-binserver_id=3307[mysqld3308]port=3308pid-file=/var/lib/mysql3308/mysql3308.pidsocket=/var/lib/mysql3308/mysql3308.sockdatadir=/var/lib/mysql3308user=mysqllog_bin=mysql-binserver_id=3308relay_log=/var/lib/mysql3308/mysql-relay-binlog_slave_updates=1read_only=1[mysqld3309]port=3309pid-file=/var/lib/mysql3309/mysql3309.pidsocket=/var/lib/mysql3309/mysql3309.sockdatadir=/var/lib/mysql3309user=mysqllog_bin=mysql-binserver_id=3309relay_log=/var/lib/mysql3309/mysql-relay-binlog_slave_updates=1read_only=1启动全部实例：1mysqld_multi --defaults-file=/etc/mysql/mysqld_muti.cnf start查看实例状态：12345mysqld_multi --defaults-file=/etc/mysql/mysqld_muti.cnf reportReporting MySQL serversMySQL server from group: mysqld3307 is runningMySQL server from group: mysqld3308 is runningMySQL server from group: mysqld3309 is running","categories":[{"name":"mysql","slug":"mysql","permalink":"http://www.liuyong520.cn/categories/mysql/"}],"tags":[{"name":"mysql","slug":"mysql","permalink":"http://www.liuyong520.cn/tags/mysql/"}]},{"title":"分布式系统之冗余数据一致性","slug":"distributed-base-redundant-data","date":"2018-05-04T05:34:47.000Z","updated":"2019-06-11T09:37:15.539Z","comments":true,"path":"2018/05/04/distributed-base-redundant-data/","link":"","permalink":"http://www.liuyong520.cn/2018/05/04/distributed-base-redundant-data/","excerpt":"","text":"一，为什么要冗余数据互联网数据量很大的业务场景，往往数据库需要进行水平切分来降低单库数据量。水平切分会有一个patition key，通过patition key的查询能够直接定位到库，但是非patition key上的查询可能就需要扫描多个库了。此时常见的架构设计方案，是使用数据冗余这种反范式设计来满足分库后不同维度的查询需求。例如：订单业务，对用户和商家都有查询需求：Order(oid, info_detail);T(buyer_id, seller_id, oid);如果用buyer_id来分库，seller_id的查询就需要扫描多库。如果用seller_id来分库，buyer_id的查询就需要扫描多库。此时可以使用数据冗余来分别满足buyer_id和seller_id上的查询需求：T1(buyer_id, seller_id, oid)T2(seller_id, buyer_id, oid)同一个数据，冗余两份，一份以buyer_id来分库，满足买家的查询需求；一份以seller_id来分库，满足卖家的查询需求。如何实施数据的冗余，以及如何保证数据的一致性，是今天将要讨论的内容。二，如何进行数据冗余服务同步双写顾名思义，由服务层同步写冗余数据，如上图1-4流程：业务方调用服务，新增数据服务先插入DB1数据服务再插入DB2数据服务返回业务方新增数据成功优点：不复杂，服务层由单次写，变两次写数据一致性相对较高（因为双写成功才返回）缺点：请求的处理时间增加（要插入两次，时间加倍）数据仍可能不一致，例如第二步写入T1完成后服务重启，则数据不会写入T2如果系统对处理时间比较敏感，引出常用的第二种方案。服务异步双写数据的双写并不再由服务来完成，服务层异步发出一个消息，通过消息总线发送给一个专门的数据复制服务来写入冗余数据，如上图1-6流程：业务方调用服务，新增数据服务先插入DB1数据服务向消息总线发送一个异步消息（发出即可，不用等返回，通常很快就能完成）服务返回业务方新增数据成功消息总线将消息投递给数据同步中心数据同步中心插入DB2数据优点：请求处理时间短（只插入1次）缺点：系统的复杂性增加了，多引入了一个组件（消息总线）和一个服务（专用的数据复制服务）因为返回业务线数据插入成功时，数据还不一定插入到T2中，因此数据有一个不一致时间窗口（这个窗口很短，最终是一致的）在消息总线丢失消息时，冗余表数据会不一致不管是服务同步双写，还是服务异步双写，服务都需要关注“冗余数据”带来的复杂性。如果想解除“数据冗余”对系统的耦合，引出常用的第三种方案。线下异步双写为了屏蔽“冗余数据”对服务带来的复杂性，数据的双写不再由服务层来完成，而是由线下的一个服务或者任务来完成，如上图1-6流程：业务方调用服务，新增数据服务先插入DB1数据服务返回业务方新增数据成功数据会被写入到数据库的log中线下服务或者任务读取数据库的log线下服务或者任务插入DB2数据优点：数据双写与业务完全解耦请求处理时间短（只插入1次）缺点：返回业务线数据插入成功时，数据还不一定插入到T2中，因此数据有一个不一致时间窗口（这个窗口很短，最终是一致的）数据的一致性依赖于线下服务或者任务的可靠性不管哪种方案，毕竟不是分布式事务，万一出现数据不一致，怎么办呢？高并发的情况下，实时一致性很难，方法论是：最终一致性。实现方式是：异步检测，异步修复。三，如何保证数据的一致性线下扫描全量数据法如上图所示，线下启动一个离线的扫描工具，不停的比对正表DB1和反表DB2，如果发现数据不一致，就进行补偿修复。优点：比较简单，开发代价小线上服务无需修改，修复工具与线上服务解耦缺点：扫描效率低，会扫描大量的“已经能够保证一致”的数据由于扫描的数据量大，扫描一轮的时间比较长，即数据如果不一致，不一致的时间窗口比较长有没有只扫描“可能存在不一致可能性”的数据，而不是每次扫描全部数据，以提高效率的优化方法呢？线下扫描增量数据法每次只扫描增量的日志数据，就能够极大提高效率，缩短数据不一致的时间窗口，如上图2-8流程所示：写入正表DB1第一步成功后，写入日志log1写入反表DB2第二步成功后，写入日志log2当然，我们还是需要一个离线的扫描工具，不停的比对日志log1和日志log2，如果发现数据不一致，就进行补偿修复优点：虽比方法一复杂，但仍然是比较简单的数据扫描效率高，只扫描增量数据缺点：线上服务略有修改（代价不高，多写了2条日志）虽然比方法一更实时，但时效性还是不高，不一致窗口取决于扫描的周期有没有实时检测一致性并进行修复的方法呢？线上实时检测“消息对”法这次不是写日志了，而是向消息总线发送消息，如上图2-7流程所示：写入正表DB1第一步成功后，发送消息msg1写入反表DB2第二步成功后，发送消息msg2这次不是需要一个周期扫描的离线工具了，而是一个实时订阅消息的服务不停的收消息。假设正常情况下，msg1和msg2的接收时间应该在3s以内，如果检测服务在收到msg1后没有收到msg2，就尝试检测数据的一致性，不一致时进行补偿修复。优点：效率高实时性高缺点：方案比较复杂，上线引入了消息总线这个组件线下多了一个订阅总线的检测服务however，技术方案本身就是一个投入产出比的折衷，可以根据业务对一致性的需求程度决定使用哪一种方法。我曾经做过IM系统，好友关系链上亿，好友数据正反表的数据冗余，使用的就是方法二。四，总结互联网数据量大的业务场景，常常:使用水平切分来降低单库数据量使用数据冗余的反范式设计来满足不同维度的查询需求冗余数据三种方案：(1)服务同步双写法能够很容易的实现数据冗余(2)为了降低时延，可以优化为服务异步双写法(3)为了屏蔽“冗余数据”对服务带来的复杂性，可以优化为线下异步双写法保证数据一致性的方案：(1)最简单的方式，线下脚本扫全量数据比对(2)提高效率的方式，线下脚本扫增量数据比对(3)最实时的方式，线上检测“消息对”","categories":[{"name":"分布式系统","slug":"分布式系统","permalink":"http://www.liuyong520.cn/categories/分布式系统/"}],"tags":[{"name":"分布式系统","slug":"分布式系统","permalink":"http://www.liuyong520.cn/tags/分布式系统/"}]},{"title":"分布式系统之分布式事务处理","slug":"distributed-base-transaction","date":"2018-04-05T09:23:25.000Z","updated":"2019-06-11T09:37:15.347Z","comments":true,"path":"2018/04/05/distributed-base-transaction/","link":"","permalink":"http://www.liuyong520.cn/2018/04/05/distributed-base-transaction/","excerpt":"","text":"为保障系统的可用性、可靠性以及性能，在分布式系统中，往往会设置数据冗余，即对数据进行复制。举例来说，当一个数据库的副本被破环以后，那么系统只需要转换到其他数据副本就能继续运行下去。另外一个例子，当访问单一服务器管理的数据的进程数不断增加时，系统就需要对服务器的数量进行扩充，此时，对服务器进行复制，随后让它们分担工作负荷，就可以提高性能。但同时，如何保障多个数据节点之间数据的一致以及如何处理分布式事务，将成为为一个复杂的话题。本文将介绍常用的事务处理机制。举个例子：用户下了一个订单，需要修改余额表，订单表，流水表，于是会有类似的伪代码：12345start transaction; CURD table t_account; any Exception rollback; CURD table t_order; any Exception rollback; CURD table t_flow; any Exception rollback;commit;如果对余额表，订单表，流水表的SQL操作全部成功，则全部提交如果任何一个出现问题，则全部回滚事务，以保证数据的完整性以及一致性。事务的方案会有什么潜在问题？答：互联网的业务特点是海量数据，大并发的数据访问量，所以经常使用拆库的方式提升系统的性能。如果进行了拆库，余额、订单、流水可能分布在不同的数据库上，甚至不同的数据库实例上，此时就不能用数据库原生事务来保证数据的一致性了，就需要用到分布式事务才能解决这类问题了。那么常见的分布式事务解决方案有哪些呢？补偿事务后置提交优化两阶段提交（2pc）三阶段提交TCC事务解决方案可靠消息一致性解决方案什么是补偿事务？补偿事务，是一种在业务端实施业务逆向操作事务。举个栗子：修改余额，事务为：12345678int Do_AccountT(uid, money){ start transaction; //余额改变money这么多 CURD table t_account with money for uid; anyException rollback return NO; commit; return YES;}那么，修改余额，补偿事务可以是：123456789101112131415161718192021222324252627282930313233343536373839404142434445int Compensate_AccountT(uid, money){ //做一个money的反向操作 return Do_AccountT(uid, -1*money){}``` 同理，订单操作，事务是：Do_OrderT，新增一个订单；订单操作，补偿事务是：Compensate_OrderT，删除一个订单。 要保证余额与订单的一致性，伪代码：```java// 执行第一个事务int flag = Do_AccountT();if(flag=YES){ //第一个事务成功，则执行第二个事务 flag= Do_OrderT(); if(flag=YES){ // 第二个事务成功，则成功 return YES; } else{ // 第二个事务失败，执行第一个事务的补偿事务 Compensate_AccountT(); }}``` # 补偿事务有什么缺点？- 不同的业务要写不同的补偿事务，不具备通用性；- 没有考虑补偿事务的失败；- 如果业务流程很复杂，if/else会嵌套非常多层；**上面的例子还只考虑了余额+订单的一致性，就有2*2=4个分支，如果要考虑余额+订单+流水的一致性，则会有2*2*2=8个if/else分支，复杂性呈指数级增长。**# 后置提交优化举个例子：单库是用这样一个大事务保证一致性：```javastart transaction; CURD table t_account; any Exception rollback; CURD table t_order; any Exception rollback; CURD table t_flow; any Exception rollback;commit;拆分成了多个库后，大事务会变成三个小事务：123456start transaction1; //第一个库事务执行 CURD table t_account; any Exception rollback; …// 第一个库事务提交commit1;123456start transaction2; //第二个库事务执行 CURD table t_order; any Exception rollback; …// 第二个库事务提交commit2;123456start transaction3; //第三个库事务执行 CURD table t_flow; any Exception rollback; …// 第三个库事务提交commit3;再次提醒，这三个事务发生在三个库，甚至3个不同实例的数据库上。一个事务，分成执行与提交两个阶段：执行(CURD)的时间很长提交(commit)的执行很快于是整个执行过程的时间轴如下：第一个事务执行200ms，提交1ms；第二个事务执行200ms，提交1ms；第三个事务执行250ms，提交1ms；在什么时候，会出现不一致？第一个事务成功提交之后，最后一个事务成功提交之前，如果出现问题（例如服务器重启，数据库异常等），都可能导致数据不一致。如上图，最后452ms可能会出现数据不一致的问题何谓后置提交优化？如果改变事务执行与提交的时序，变成事务先执行，最后一起提交。就变成了下图第一个事务执行200ms，第二个事务执行200ms，第三个事务执行250ms；第一个事务提交1ms，第二个事务提交1ms，第三个事务提交1ms；后置提交优化后，在什么时候，会出现不一致？第一个事务成功提交之后，最后一个事务成功提交之前，如果出现问题（例如服务器重启，数据库异常等），都可能导致数据不一致如上图最后2ms可能会出现数据不一致。有什么区别和差异？串行事务方案，总执行时间是653ms，最后452ms内出现异常都可能导致不一致；后置提交优化方案，总执行时间也是653ms，但最后2ms内出现异常才会导致不一致；虽然没有彻底解决数据的一致性问题，但不一致出现的概率大大降低了。上面这个例子，概率降低了250倍。后置提交优化，有什么不足？对事务吞吐量会有影响：串行事务方案，第一个库事务提交，数据库连接就释放了；后置提交优化方案，所有库的连接，要等到所有事务执行完才释放；这就意味着，数据库连接占用的时间增长了，系统整体的吞吐量降低了。两阶段提交（2PC）两阶段提交协议 （Two-phase commit protocol，2PC）的过程涉及到协调者和参与者。协调者可以看做成事务的发起者，同时也是事务的一个参与者。对于一个分布式事务来说，一个事务是涉及到多个参与者的。具体的两阶段提交的过程如下：第一阶段（准备阶段）协调者节点向所有参与者节点询问是否可以执行提交操作(vote)，并开始等待各参与者节点的响应。参与者节点执行询问发起为止的所有事务操作，并将 Undo 信息和 Redo 信息写入日志。（注意：若成功这里其实每个参与者已经执行了事务操作）各参与者节点响应协调者节点发起的询问。如果参与者节点的事务操作实际执行成功，则它返回一个“同意”消息；如果参与者节点的事务操作实际执行失败，则它返回一个“中止”消息。第二阶段（提交阶段）如果协调者收到了参与者的失败消息或者超时，直接给每个参与者发送回滚(Rollback)消息；否则，发送提交(Commit)消息；参与者根据协调者的指令执行提交或者回滚操作，释放所有事务处理过程中使用的锁资源。(注意:必须在最后阶段释放锁资源)当协调者节点从所有参与者节点获得的相应消息都为“同意”时:协调者节点向所有参与者节点发出“正式提交(commit)”的请求。参与者节点正式完成操作，并释放在整个事务期间内占用的资源。参与者节点向协调者节点发送“完成”消息。如果任一参与者节点在第一阶段返回的响应消息为”中止”，或者 协调者节点在第一阶段的询问超时之前无法获取所有参与者节点的响应消息时：协调者节点向所有参与者节点发出”回滚操作(rollback)”的请求。参与者节点利用之前写入的Undo信息执行回滚，并释放在整个事务期间内占用的资源。参与者节点向协调者节点发送”回滚完成”消息。协调者节点受到所有参与者节点反馈的”回滚完成”消息后，取消事务。协调者节点受到所有参与者节点反馈的”完成”消息后，完成事务不管最后结果如何，第二阶段都会结束当前事务。二段式提交协议的优缺点：优点：原理简单，实现方便；缺点：同步阻塞问题。执行过程中，所有参与节点都是事务阻塞型的。单点故障。由于协调者的重要性，一旦协调者发生故障，参与者会一直阻塞下去。尤其在第二阶段，协调者发生故障，那么所有的参与者还都处于锁定事务资源的状态中，而无法继续完成事务操作。数据不一致。在阶段二中，当协调者向参与者发送 commit 请求之后，发生了局部网络异常或者在发送 commit 请求过程中协调者发生了故障，这回导致只有一部分参与者接受到了 commit 请求。而在这部分参与者接到 commit 请求之后就会执行 commit 操作。但是其他部分未接到 commit 请求的机器则无法执行事务提交。于是整个分布式系统便出现了数据部一致性的现象。二阶段无法解决的问题：协调者再发出 commit 消息之后宕机，而唯一接收到这条消息的参与者同时也宕机了。那么即使协调者通过选举协议产生了新的协调者，这条事务的状态也是不确定的，没人知道事务是否被已经提交。为了解决两阶段提交协议的种种问题，研究者们在二阶段提交的基础上做了改进，提出了三阶段提交。三阶段提交（3PC）三阶段提交协议（Three-phase commit protocol，3PC），是二阶段提交（2PC）的改进版本。与两阶段提交不同的是，三阶段提交有两个改动点：引入超时机制。同时在协调者和参与者中都引入超时机制。在第一阶段和第二阶段中插入一个准备阶段。保证了在最后提交阶段之前各参与节点的状态是一致的。即 3PC 把 2PC 的准备阶段再次一分为二，这样三阶段提交就有 CanCommit、PreCommit、DoCommit 三个阶段。CanCommit 阶段CanCommit 阶段其实和 2PC 的准备阶段很像。协调者向参与者发送 commit 请求，参与者如果可以提交就返回 Yes 响应，否则返回 No 响应。事务询问：协调者向参与者发送 CanCommit 请求。询问是否可以执行事务提交操作。然后开始等待参与者的响应。响应反馈：参与者接到 CanCommit 请求之后，正常情况下，如果其自身认为可以顺利执行事务，则返回 Yes 响应，并进入预备状态。否则反馈 NoPreCommit 阶段协调者根据参与者的反应情况来决定是否可以记性事务的 PreCommit 操作。根据响应情况，有以下两种可能。假如协调者从所有的参与者获得的反馈都是 Yes 响应，那么就会执行事务的预执行。发送预提交请求：协调者向参与者发送 PreCommit 请求，并进入Prepared 阶段。事务预提交：参与者接收到 PreCommit 请求后，会执行事务操作，并将undo 和 redo 信息记录到事务日志中。响应反馈：如果参与者成功的执行了事务操作，则返回 ACK 响应，同时开始等待最终指令。假如有任何一个参与者向协调者发送了 No 响应，或者等待超时之后，协调者都没有接到参与者的响应，那么就执行事务的中断。发送中断请求：协调者向所有参与者发送 abort 请求。中断事务：参与者收到来自协调者的 abort 请求之后（或超时之后，仍未收到协调者的请求），执行事务的中断。doCommit 阶段该阶段进行真正的事务提交，也可以分为以下两种情况。执行提交发送提交请求：协调接收到参与者发送的 ACK 响应，那么他将从预提交状态进入到提交状态。并向所有参与者发送 doCommit 请求。事务提交：参与者接收到 doCommit 请求之后，执行正式的事务提交。并在完成事务提交之后释放所有事务资源。响应反馈：事务提交完之后，向协调者发送 ACK 响应。完成事务：协调者接收到所有参与者的 ACK 响应之后，完成事务。中断事务：协调者没有接收到参与者发送的 ACK 响应（可能是接受者发送的不是 ACK 响应，也可能响应超时），那么就会执行中断事务。发送中断请求：协调者向所有参与者发送 abort 请求事务回滚：参与者接收到 abort 请求之后，利用其在阶段二记录的undo 信息来执行事务的回滚操作，并在完成回滚之后释放所有的事务资源。反馈结果：参与者完成事务回滚之后，向协调者发送 ACK 消息中断事务：协调者接收到参与者反馈的 ACK 消息之后，执行事务的中断。在 doCommit 阶段，如果参与者无法及时接收到来自协调者的 doCommit 或者 rebort 请求时，会在等待超时之后，会继续进行事务的提交。即当进入第三阶段时，由于网络超时等原因，虽然参与者没有收 到 commit 或者 abort 响应，事务仍然会提交。三阶段提交不会一直持有事务资源并处于阻塞状态。但是这种机制也会导致数据一致性问题，因为，由于网络原因，协调者发送的 abort 响应没有及时被参与者接收到，那么参与者在等待超时之后执行了 commit 操作，这样就和其他接到 abort 命令并执行回滚的参与者之间存在数据不一致的情况。TCC事务解决方案在08年的软件开发2.0技术大会上，支付宝程立在PPT大规模SOA系统中的分布事务处理，提出TCC概念。在网络上搜索分布式事务相关的博客，基本都会提及这个PPT，目前很多分布式事务开源项目也都是基于TCC的思想实现。TCC 将事务提交分为 Try - Confirm - Cancel 3个操作。Try：预留业务资源/数据效验Confirm：确认执行业务操作Cancel：取消执行业务操作TCC事务处理流程和 2PC 二阶段提交类似，不过 2PC通常都是在跨库的DB层面，而TCC本质就是一个应用层面的2PC。事务开始时，业务应用会向事务协调器注册启动事务。之后业务应用会调用所有服务的try接口，完成一阶段准备。之后事务协调器会根据try接口返回情况，决定调用confirm接口或者cancel接口。如果接口调用失败，会进行重试。TCC事务的优缺点优点：让应用自己定义数据库操作的粒度，使得降低锁冲突、提高吞吐量成为可能。缺点：对应用的侵入性强。业务逻辑的每个分支都需要实现try、confirm、cancel三个操作，应用侵入性较强，改造成本高。实现难度较大。需要按照网络状态、系统故障等不同的失败原因实现不同的回滚策略。为了满足一致性的要求，confirm和cancel接口必须实现幂等。举个例子：支付宝红包+余额支付的例子：用户下一个订单等待支付，会修改余额账户和红包账户系统的钱。假设用户下单操作来自3个系统下单系统、资金账户系统、红包账户系统，下单成功需要同时调用资金账户服务和红包服务完成支付假设购买商品1000元，使用账户红包200元，余额800元，确认支付。伪代码：1234567891011121314Try操作tryX create tb_order //下单系统创建待支付订单tryY freeze tb_balance //update tb_balance set freezebanlance=freezebalance+200 冻结账户红包200元tryZ freeze tb_account //update tb_balance set freezeaccount=freezeaccount+800 冻结资金账户800元Confirm操作confirmX update tb_order set state="success" // 订单更新为支付成功confirmY update tb_balance set balance= balance-200//扣减账户红包200元confirmZ update tb_account set account= account-800 //扣减资金账户800元Cancel操作cancelX 订单处理异常，资金红包退回，订单支付失败cancelY 冻结红包失败，账户余额退回，订单支付失败cancelZ 冻结余额失败，账户红包退回，订单支付失败可靠消息一致性解决方案消息一致性方案是通过消息中间件保证上、下游应用数据操作的一致性。基本思路是将本地操作和发送消息放在一个事务中，保证本地操作和消息发送要么两者都成功或者都失败。下游应用向消息系统订阅该消息，收到消息后执行相应操作。消息方案从本质上讲是将分布式事务转换为两个本地事务，然后依靠下游业务的重试机制达到最终一致性。基于消息的最终一致性方案对应用侵入性也很高，应用需要进行大量业务改造，成本较高。","categories":[{"name":"分布式系统","slug":"分布式系统","permalink":"http://www.liuyong520.cn/categories/分布式系统/"}],"tags":[{"name":"分布式系统","slug":"分布式系统","permalink":"http://www.liuyong520.cn/tags/分布式系统/"}]},{"title":"分布式系统之CAP定理","slug":"distributed-base","date":"2018-04-02T08:04:25.000Z","updated":"2019-06-11T09:37:16.616Z","comments":true,"path":"2018/04/02/distributed-base/","link":"","permalink":"http://www.liuyong520.cn/2018/04/02/distributed-base/","excerpt":"","text":"分布式系统非常关注三个指标：数据一致性系统可用性节点连通性与扩展性关于一致性数据“强一致性”，是希望系统只读到最新写入的数据，例如：通过单点串行化的方式，就能够达到这个效果。关于session一致性，DB主从一致性，DB双主一致性，DB与Cache一致性，数据冗余一致性，消息时序一致性，分布式事务一致性，库存扣减一致性，详见文章《究竟啥才是互联网架构“一致性”》。关于可用性如果系统每运行100个时间单位，会有1个时间单位无法提供服务，则说系统的可用性是99%。可用性和可靠性是比较容易搞混的两个指标，以一台取款机为例：正确的输入，能够取到正确的钱，表示系统可靠取款机7*24小时提供服务，表示系统可用保证系统高可用的方法是：冗余故障自动转移反向代理层，站点层，服务层，缓存层，数据库层各层保证系统高可用的方法，详见文章《究竟啥才是互联网架构“高可用”》。关于连通性与扩展性分布式系统，往往有多个节点，每个节点之间，都不是完全独立的，需要相互通信，当发生节点无法联通时，数据是否还能保持一致，系统要如何进行容错处理，是需要考虑的。同时，连通性和扩展性紧密相关，想要加机器扩展性能，必须有良好的连通性。当一个节点脱离系统，系统就出现问题，往往意味着系统是无法扩展的。什么是CAP定理？CAP定理，是对上述分布式系统的三个特性，进行了归纳：一致性(Consistency)可用性(Availability)分区容忍性(Partition Tolerance)并且，定理指出，在系统实现时，这三者最多兼顾两点。一致性，可用性，多节点扩展性三者只能取其二，既然加锁已经加上，常见的最佳工程架构实践是什么呢？互联网，最常见的实践是这样的：节点连通性，多节点扩展性，连通性异常的处理必须保证，满足P一致性C与可用性A一般二选一选择一致性C，举例：传统单库水平切分，就是这类选型的典型选择可用性A，举例：双主库同步高可用，就是这类选型的典型强一致很难怎么办？单点串行化，虽然能保证“强一致”，但对系统的并发性能，以及高可用有较大影响，互联网的玩法，更多的是“最终一致性”，短期内未必读到最新的数据，但在一个可接受的时间窗口之后，能够读到最新的数据。例如：数据库主从同步，从库上的数据，就是一个最终的一致。总结CAP可以理解为一致性，可用性，联通与扩展性CAP三者只能取其二最常见的实践是AP+最终一致性思路比结论重要。","categories":[{"name":"分布式系统","slug":"分布式系统","permalink":"http://www.liuyong520.cn/categories/分布式系统/"}],"tags":[{"name":"分布式系统","slug":"分布式系统","permalink":"http://www.liuyong520.cn/tags/分布式系统/"}]},{"title":"6张图读懂大型网站架构","slug":"distributed","date":"2018-03-02T07:08:52.000Z","updated":"2019-06-11T09:37:16.159Z","comments":true,"path":"2018/03/02/distributed/","link":"","permalink":"http://www.liuyong520.cn/2018/03/02/distributed/","excerpt":"","text":"大型网站架构演化大型网站架构方法网站的安全架构网站的高可用架构网站的伸缩性架构大型网站核心架构要素","categories":[],"tags":[{"name":"分布式架构","slug":"分布式架构","permalink":"http://www.liuyong520.cn/tags/分布式架构/"}]},{"title":"Docker 容器技术","slug":"docker","date":"2018-02-12T04:15:31.000Z","updated":"2019-06-11T09:37:16.606Z","comments":true,"path":"2018/02/12/docker/","link":"","permalink":"http://www.liuyong520.cn/2018/02/12/docker/","excerpt":"","text":"一、Docker 简介Docker是一个新的容器化的技术，它轻巧，且易移植。Docker 组件与元素Docker有三个组件和三个基本元素。三个组件分别是：Docker Client 是用户界面，它支持用户与Docker Daemon之间通信。Docker Daemon运行于主机上，处理服务请求。Docker Index是中央registry，支持拥有公有与私有访问权限的Docker容器镜像的备份。三个基本要素分别是：Docker Containers负责应用程序的运行，包括操作系统、用户添加的文件以及元数据。Docker Images是一个只读模板，用来运行Docker容器。DockerFile是文件指令集，用来说明如何自动创建Docker镜像。1.1 Docker 守护进程如上图所示，Docker 守护进程运行在一台主机上。用户并不直接和守护进程进行交互，而是通过 Docker 客户端间接和其通信。1.2 Docker 客户端Docker 客户端，实际上是 docker 的二进制程序，是主要的用户与 Docker 交互方式。它接收用户指令并且与背后的 Docker 守护进程通信，如此来回往复。1.3 Docker 内部要理解 Docker 内部构建，需要理解以下三种部件：Docker 镜像 - Docker imagesDocker 仓库 - Docker registeriesDocker 容器 - Docker containersDocker 镜像Docker 镜像是 Docker 容器运行时的只读模板，每一个镜像由一系列的层 (layers) 组成。Docker 使用 UnionFS 来将这些层联合到单独的镜像中。UnionFS 允许独立文件系统中的文件和文件夹(称之为分支)被透明覆盖，形成一个单独连贯的文件系统。正因为有了这些层的存在，Docker 是如此的轻量。当你改变了一个 Docker 镜像，比如升级到某个程序到新的版本，一个新的层会被创建。因此，不用替换整个原先的镜像或者重新建立(在使用虚拟机的时候你可能会这么做)，只是一个新 的层被添加或升级了。现在你不用重新发布整个镜像，只需要升级，层使得分发 Docker 镜像变得简单和快速。Docker 仓库Docker 仓库用来保存镜像，可以理解为代码控制中的代码仓库。同样的，Docker 仓库也有公有和私有的概念。公有的 Docker 仓库名字是 Docker Hub。Docker Hub 提供了庞大的镜像集合供使用。这些镜像可以是自己创建，或者在别人的镜像基础上创建。Docker 仓库是 Docker 的分发部分。Docker 容器Docker 容器和文件夹很类似，一个Docker容器包含了所有的某个应用运行所需要的环境。每一个 Docker 容器都是从 Docker 镜像创建的。Docker 容器可以运行、开始、停止、移动和删除。每一个 Docker 容器都是独立和安全的应用平台，Docker 容器是 Docker 的运行部分。二、Docker 安装docker 的相关安装方法这里不作介绍，具体安装参考 官档获取当前 docker 版本12345678910$ sudo docker versionClient version: 1.3.2Client API version: 1.15Go version (client): go1.3.3Git commit (client): 39fa2fa/1.3.2OS/Arch (client): linux/amd64Server version: 1.3.2Server API version: 1.15Go version (server): go1.3.3Git commit (server): 39fa2fa/1.3.2三、Docker 基础用法Docker HUB : Docker镜像首页，包括官方镜像和其它公开镜像因为国情的原因，国内下载 Docker HUB 官方的相关镜像比较慢，可以使用 docker.cn 镜像，镜像保持和官方一致，关键是速度块，推荐使用。3.1 Search images1$ sudo docker search ubuntu3.2 Pull images1$ sudo docker pull ubuntu # 获取 ubuntu 官方镜像 $ sudo docker images # 查看当前镜像列表3.3 Running an interactive shell1$ sudo docker run -i -t ubuntu:14.04 /bin/bashdocker run - 运行一个容器-t - 分配一个（伪）tty (link is external)-i - 交互模式 (so we can interact with it)ubuntu:14.04 - 使用 ubuntu 基础镜像 14.04/bin/bash - 运行命令 bash shell注: ubuntu 会有多个版本，通过指定 tag 来启动特定的版本 [image]:[tag]12$ sudo docker ps # 查看当前运行的容器, ps -a 列出当前系统所有的容器 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES6c9129e9df10 ubuntu:14.04 /bin/bash 6 minutes ago Up 6 minutes cranky_babbage四、Docker 命令帮助4.1 docker helpdocker command1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950$ sudo docker # docker 命令帮助Commands: attach Attach to a running container # 当前 shell 下 attach 连接指定运行镜像 build Build an image from a Dockerfile # 通过 Dockerfile 定制镜像 commit Create a new image from a container's changes # 提交当前容器为新的镜像 cp Copy files/folders from the containers filesystem to the host path # 从容器中拷贝指定文件或者目录到宿主机中 create Create a new container # 创建一个新的容器，同 run，但不启动容器 diff Inspect changes on a container's filesystem # 查看 docker 容器变化 exec Run a command in an existing container # 在已存在的容器上运行命令 export Stream the contents of a container as a tar archive # 导出容器的内容流作为一个 tar 归档文件[对应 import ] history Show the history of an image # 展示一个镜像形成历史 images List images # 列出系统当前镜像 import Create a new filesystem image from the contents of a tarball # 从tar包中的内容创建一个新的文件系统映像[对应 export] info Display system-wide information # 显示系统相关信息 inspect Return low-level information on a container # 查看容器详细信息 kill Kill a running container # kill 指定 docker 容器 load Load an image from a tar archive # 从一个 tar 包中加载一个镜像[对应 save] login Register or Login to the docker registry server # 注册或者登陆一个 docker 源服务器 logout Log out from a Docker registry server # 从当前 Docker registry 退出 logs Fetch the logs of a container # 输出当前容器日志信息 port Lookup the public-facing port which is NAT-ed to PRIVATE_PORT # 查看映射端口对应的容器内部源端口 pause Pause all processes within a container # 暂停容器 ps List containers # 列出容器列表 pull Pull an image or a repository from the docker registry server # 从docker镜像源服务器拉取指定镜像或者库镜像 push Push an image or a repository to the docker registry server # 推送指定镜像或者库镜像至docker源服务器 restart Restart a running container # 重启运行的容器 rm Remove one or more containers # 移除一个或者多个容器 rmi Remove one or more images # 移除一个或多个镜像[无容器使用该镜像才可删除，否则需删除相关容器才可继续或 -f 强制删除] run Run a command in a new container # 创建一个新的容器并运行一个命令 save Save an image to a tar archive # 保存一个镜像为一个 tar 包[对应 load] search Search for an image on the Docker Hub # 在 docker hub 中搜索镜像 start Start a stopped containers # 启动容器 stop Stop a running containers # 停止容器 tag Tag an image into a repository # 给源中镜像打标签 top Lookup the running processes of a container # 查看容器中运行的进程信息 unpause Unpause a paused container # 取消暂停容器 version Show the docker version information # 查看 docker 版本号 wait Block until a container stops, then print its exit code # 截取容器停止时的退出状态值Run 'docker COMMAND --help' for more information on a command.docker option12345678910111213141516171819202122232425262728293031323334353637Usage of docker: --api-enable-cors=false Enable CORS headers in the remote API # 远程 API 中开启 CORS 头 -b, --bridge=\"\" Attach containers to a pre-existing network bridge # 桥接网络 use 'none' to disable container networking --bip=\"\" Use this CIDR notation address for the network bridge's IP, not compatible with -b # 和 -b 选项不兼容，具体没有测试过 -d, --daemon=false Enable daemon mode # daemon 模式 -D, --debug=false Enable debug mode # debug 模式 --dns=[] Force docker to use specific DNS servers # 强制 docker 使用指定 dns 服务器 --dns-search=[] Force Docker to use specific DNS search domains # 强制 docker 使用指定 dns 搜索域 -e, --exec-driver=\"native\" Force the docker runtime to use a specific exec driver # 强制 docker 运行时使用指定执行驱动器 --fixed-cidr=\"\" IPv4 subnet for fixed IPs (ex: 10.20.0.0/16) this subnet must be nested in the bridge subnet (which is defined by -b or --bip) -G, --group=\"docker\" Group to assign the unix socket specified by -H when running in daemon mode use '' (the empty string) to disable setting of a group -g, --graph=\"/var/lib/docker\" Path to use as the root of the docker runtime # 容器运行的根目录路径 -H, --host=[] The socket(s) to bind to in daemon mode # daemon 模式下 docker 指定绑定方式[tcp or 本地 socket] specified using one or more tcp://host:port, unix:///path/to/socket, fd://* or fd://socketfd. --icc=true Enable inter-container communication # 跨容器通信 --insecure-registry=[] Enable insecure communication with specified registries (no certificate verification for HTTPS and enable HTTP fallback) (e.g., localhost:5000 or 10.20.0.0/16) --ip=\"0.0.0.0\" Default IP address to use when binding container ports # 指定监听地址，默认所有 ip --ip-forward=true Enable net.ipv4.ip_forward # 开启转发 --ip-masq=true Enable IP masquerading for bridge's IP range --iptables=true Enable Docker's addition of iptables rules # 添加对应 iptables 规则 --mtu=0 Set the containers network MTU # 设置网络 mtu if no value is provided: default to the default route MTU or 1500 if no default route is available -p, --pidfile=\"/var/run/docker.pid\" Path to use for daemon PID file # 指定 pid 文件位置 --registry-mirror=[] Specify a preferred Docker registry mirror -s, --storage-driver=\"\" Force the docker runtime to use a specific storage driver # 强制 docker 运行时使用指定存储驱动 --selinux-enabled=false Enable selinux support # 开启 selinux 支持 --storage-opt=[] Set storage driver options # 设置存储驱动选项 --tls=false Use TLS; implied by tls-verify flags # 开启 tls --tlscacert=\"/root/.docker/ca.pem\" Trust only remotes providing a certificate signed by the CA given here --tlscert=\"/root/.docker/cert.pem\" Path to TLS certificate file # tls 证书文件位置 --tlskey=\"/root/.docker/key.pem\" Path to TLS key file # tls key 文件位置 --tlsverify=false Use TLS and verify the remote (daemon: verify client, client: verify daemon) # 使用 tls 并确认远程控制主机 -v, --version=false Print version information and quit # 输出 docker 版本信息4.2 docker search1234567$ sudo docker search --helpUsage: docker search TERMSearch the Docker Hub for images # 从 Docker Hub 搜索镜像 --automated=false Only show automated builds --no-trunc=false Don't truncate output -s, --stars=0 Only displays with at least xxx stars示例：12$ sudo docker search -s 100 ubuntu # 查找 star 数至少为 100 的镜像，找出只有官方镜像 start 数超过 100，默认不加 s 选项找出所有相关 ubuntu 镜像 NAME DESCRIPTION STARS OFFICIAL AUTOMATEDubuntu Official Ubuntu base image 425 [OK]4.3 docker info1234567891011$ sudo docker info Containers: 1 # 容器个数 Images: 22 # 镜像个数 Storage Driver: devicemapper # 存储驱动 Pool Name: docker-8:17-3221225728-pool Pool Blocksize: 65.54 kB Data file: /data/docker/devicemapper/devicemapper/data Metadata file: /data/docker/devicemapper/devicemapper/metadata Data Space Used: 1.83 GB Data Space Total: 107.4 GB Metadata Space Used: 2.191 MB Metadata Space Total: 2.147 GB Library Version: 1.02.84-RHEL7 (2014-03-26) Execution Driver: native-0.2 # 存储驱动 Kernel Version: 3.10.0-123.el7.x86_64Operating System: CentOS Linux 7 (Core)4.4 docker pull && docker push123$ sudo docker pull --help # pull 拉取镜像 Usage: docker pull [OPTIONS] NAME[:TAG] Pull an image or a repository from the registry -a, --all-tags=false Download all tagged images in the repository $ sudo docker push # push 推送指定镜像 Usage: docker push NAME[:TAG] Push an image or a repository to the registry示例：1$ sudo docker pull ubuntu # 下载官方 ubuntu docker 镜像，默认下载所有 ubuntu 官方库镜像 $ sudo docker pull ubuntu:14.04 # 下载指定版本 ubuntu 官方镜像1$ sudo docker push 192.168.0.100:5000/ubuntu # 推送镜像库到私有源[可注册 docker 官方账户，推送到官方自有账户] $ sudo docker push 192.168.0.100:5000/ubuntu:14.04 # 推送指定镜像到私有源4.5 docker images列出当前系统镜像123456$ sudo docker images --helpUsage: docker images [OPTIONS] [NAME] List images -a, --all=false Show all images (by default filter out the intermediate image layers) # -a 显示当前系统的所有镜像，包括过渡层镜像，默认 docker images 显示最终镜像，不包括过渡层镜像 -f, --filter=[] Provide filter values (i.e. 'dangling=true') --no-trunc=false Don't truncate output -q, --quiet=false Only show numeric IDs示例：1234$ sudo docker images # 显示当前系统镜像，不包括过渡层镜像 $ sudo docker images -a # 显示当前系统所有镜像，包括过渡层镜像 $ sudo docker images ubuntu # 显示当前系统 docker ubuntu 库中的所有镜像 REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZEubuntu 12.04 ebe4be4dd427 4 weeks ago 210.6 MBubuntu 14.04 e54ca5efa2e9 4 weeks ago 276.5 MBubuntu 14.04-ssh 6334d3ac099a 7 weeks ago 383.2 MB4.6 docker rmi删除一个或者多个镜像1234567$ sudo docker rmi --helpUsage: docker rmi IMAGE [IMAGE...] Remove one or more images -f, --force=false Force removal of the image # 强制移除镜像不管是否有容器使用该镜像 --no-prune=false Do not delete untagged parents # 不要删除未标记的父镜像 $ sudo docker rm `docker ps -a -q` !!!批量清除所有容器4.7 docker run1234567891011121314151617181920212223242526$ sudo docker run --helpUsage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...] Run a command in a new container -a, --attach=[] Attach to stdin, stdout or stderr. -c, --cpu-shares=0 CPU shares (relative weight) # 设置 cpu 使用权重 --cap-add=[] Add Linux capabilities --cap-drop=[] Drop Linux capabilities --cidfile="" Write the container ID to the file # 把容器 id 写入到指定文件 --cpuset="" CPUs in which to allow execution (0-3, 0,1) # cpu 绑定 -d, --detach=false Detached mode: Run container in the background, print new container id # 后台运行容器 --device=[] Add a host device to the container (e.g. --device=/dev/sdc:/dev/xvdc) --dns=[] Set custom dns servers # 设置 dns --dns-search=[] Set custom dns search domains # 设置 dns 域搜索 -e, --env=[] Set environment variables # 定义环境变量 --entrypoint="" Overwrite the default entrypoint of the image # ？ --env-file=[] Read in a line delimited file of ENV variables # 从指定文件读取变量值 --expose=[] Expose a port from the container without publishing it to your host # 指定对外提供服务端口 -h, --hostname="" Container host name # 设置容器主机名 -i, --interactive=false Keep stdin open even if not attached # 保持标准输出开启即使没有 attached --link=[] Add link to another container (name:alias) # 添加链接到另外一个容器 --lxc-conf=[] (lxc exec-driver only) Add custom lxc options --lxc-conf="lxc.cgroup.cpuset.cpus = 0,1" -m, --memory="" Memory limit (format: <number><optional unit>, where unit = b, k, m or g) # 内存限制 --name="" Assign a name to the container # 设置容器名 --net="bridge" Set the Network mode for the container # 设置容器网络模式 'bridge': creates a new network stack for the container on the docker bridge 'none': no networking for this container 'container:<name|id>': reuses another container network stack 'host': use the host network stack inside the container. Note: the host mode gives the container full access to local system services such as D-bus and is therefore considered insecure. -P, --publish-all=false Publish all exposed ports to the host interfaces # 自动映射容器对外提供服务的端口 -p, --publish=[] Publish a container's port to the host # 指定端口映射 format: ip:hostPort:containerPort | ip::containerPort | hostPort:containerPort (use 'docker port' to see the actual mapping) --privileged=false Give extended privileges to this container # 提供更多的权限给容器 --restart="" Restart policy to apply when a container exits (no, on-failure[:max-retry], always) --rm=false Automatically remove the container when it exits (incompatible with -d) # 如果容器退出自动移除和 -d 选项冲突 --security-opt=[] Security Options --sig-proxy=true Proxify received signals to the process (even in non-tty mode). SIGCHLD is not proxied. -t, --tty=false Allocate a pseudo-tty # 分配伪终端 -u, --user="" Username or UID # 指定运行容器的用户 uid 或者用户名 -v, --volume=[] Bind mount a volume (e.g., from the host: -v /host:/container, from docker: -v /container) # 挂载卷 --volumes-from=[] Mount volumes from the specified container(s) # 从指定容器挂载卷 -w, --workdir="" Working directory inside the container # 指定容器工作目录示例：1234567891011$ sudo docker images ubuntuREPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZEubuntu 14.04 e54ca5efa2e9 4 weeks ago 276.5 MB... ... $ sudo docker run -t -i -c 100 -m 512MB -h test1 -d --name="docker_test1" ubuntu /bin/bash # 创建一个 cpu 优先级为 100，内存限制 512MB，主机名为 test1，名为 docker_test1 后台运行 bash 的容器 a424ca613c9f2247cd3ede95adfbaf8d28400cbcb1d5f9b69a7b56f97b2b52e5 $ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESa424ca613c9f ubuntu:14.04 /bin/bash 6 seconds ago Up 5 seconds docker_test1 $ sudo docker attach docker_test1root@test1:/# pwd /root@test1:/# exit exit4.8 docker start|stop|kill… …dockerstart|stop|kill|restart|pause|unpause|rm|commit|inspect|logsdocker start CONTAINER [CONTAINER…]# 运行一个或多个停止的容器docker stop CONTAINER [CONTAINER…]​# 停掉一个或多个运行的容器-t选项可指定超时时间docker kill [OPTIONS] CONTAINER [CONTAINER…]# 默认 kill 发送 SIGKILL 信号-s可以指定发送 kill 信号类型docker restart [OPTIONS] CONTAINER [CONTAINER…]# 重启一个或多个运行的容器-t选项可指定超时时间docker pause CONTAINER# 暂停一个容器，方便 commitdocker unpause CONTAINER# 继续暂停的容器docker rm [OPTIONS] CONTAINER [CONTAINER…]​# 移除一个或多个容器-f, –force=false Force removal of running container-l, –link=false Remove the specified link and not the underlying container-v, –volumes=false Remove the volumes associated with the containerdocker commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]​# 提交指定容器为镜像-a, –author=”” Author (e.g., “John Hannibal Smith hannibal@a-team.com“)-m, –message=”” Commit message-p, –pause=true Pause container during commit# 默认 commit 是暂停状态docker inspect CONTAINER|IMAGE [CONTAINER|IMAGE…]​# 查看容器或者镜像的详细信息docker logs CONTAINER# 输出指定容器日志信息-f, –follow=false Follow log output# 类似 tail -f-t, –timestamps=false Show timestamps–tail=”all” Output the specified number of lines at the end of logs (defaults to all logs)参考文档：Docker Run Reference五、Docker 端口映射12# Find IP address of container with ID <container_id> 通过容器 id 获取 ip $ sudo docker inspect <container_id> | grep IPAddress | cut -d ’"’ -f 4无论如何，这些 ip 是基于本地系统的并且容器的端口非本地主机是访问不到的。此外，除了端口只能本地访问外，对于容器的另外一个问题是这些 ip 在容器每次启动的时候都会改变。Docker 解决了容器的这两个问题，并且给容器内部服务的访问提供了一个简单而可靠的方法。Docker 通过端口绑定主机系统的接口，允许非本地客户端访问容器内部运行的服务。为了简便的使得容器间通信，Docker 提供了这种连接机制。5.1 自动映射端口-P使用时需要指定–expose选项，指定需要对外提供服务的端口1$ sudo docker run -t -P --expose 22 --name server ubuntu:14.04使用docker run -P自动绑定所有对外提供服务的容器端口，映射的端口将会从没有使用的端口池中 (49000..49900) 自动选择，你可以通过docker ps、docker inspect <container_id>或者docker port <container_id>确定具体的绑定信息。5.2 绑定端口到指定接口基本语法1$ sudo docker run -p [([<host_interface>:[host_port]])|(<host_port>):]<container_port>[/udp] <image> <cmd>默认不指定绑定 ip 则监听所有网络接口。绑定 TCP 端口12345# Bind TCP port 8080 of the container to TCP port 80 on 127.0.0.1 of the host machine. $ sudo docker run -p 127.0.0.1:80:8080 <image> <cmd> # Bind TCP port 8080 of the container to a dynamically allocated TCP port on 127.0.0.1 of the host machine. $ sudo docker run -p 127.0.0.1::8080 <image> <cmd> # Bind TCP port 8080 of the container to TCP port 80 on all available interfaces of the host machine. $ sudo docker run -p 80:8080 <image> <cmd> # Bind TCP port 8080 of the container to a dynamically allocated TCP port on all available interfaces$ sudo docker run -p 8080 <image> <cmd>绑定 UDP 端口12# Bind UDP port 5353 of the container to UDP port 53 on 127.0.0.1 of the host machine. $ sudo docker run -p 127.0.0.1:53:5353/udp <image> <cmd>六、Docker 网络配置图: Docker - container and lightweight virtualizationDokcer 通过使用 Linux 桥接提供容器之间的通信，docker0 桥接接口的目的就是方便 Docker 管理。当 Docker daemon 启动时需要做以下操作：creates the docker0 bridge if not present# 如果 docker0 不存在则创建searches for an IP address range which doesn’t overlap with an existing route# 搜索一个与当前路由不冲突的 ip 段picks an IP in the selected range# 在确定的范围中选择 ipassigns this IP to the docker0 bridge# 绑定 ip 到 docker06.1 Docker 四种网络模式四种网络模式摘自 Docker 网络详解及 pipework 源码解读与实践docker run 创建 Docker 容器时，可以用 –net 选项指定容器的网络模式，Docker 有以下 4 种网络模式：host 模式，使用 –net=host 指定。container 模式，使用 –net=container:NAMEorID 指定。none 模式，使用 –net=none 指定。bridge 模式，使用 –net=bridge 指定，默认设置。host 模式如果启动容器的时候使用 host 模式，那么这个容器将不会获得一个独立的 Network Namespace，而是和宿主机共用一个 Network Namespace。容器将不会虚拟出自己的网卡，配置自己的 IP 等，而是使用宿主机的 IP 和端口。例如，我们在 10.10.101.105/24 的机器上用 host 模式启动一个含有 web 应用的 Docker 容器，监听 tcp 80 端口。当我们在容器中执行任何类似 ifconfig 命令查看网络环境时，看到的都是宿主机上的信息。而外界访问容器中的应用，则直接使用 10.10.101.105:80 即可，不用任何 NAT 转换，就如直接跑在宿主机中一样。但是，容器的其他方面，如文件系统、进程列表等还是和宿主机隔离的。container 模式这个模式指定新创建的容器和已经存在的一个容器共享一个 Network Namespace，而不是和宿主机共享。新创建的容器不会创建自己的网卡，配置自己的 IP，而是和一个指定的容器共享 IP、端口范围等。同样，两个容器除了网络方面，其他的如文件系统、进程列表等还是隔离的。两个容器的进程可以通过 lo 网卡设备通信。none模式这个模式和前两个不同。在这种模式下，Docker 容器拥有自己的 Network Namespace，但是，并不为 Docker容器进行任何网络配置。也就是说，这个 Docker 容器没有网卡、IP、路由等信息。需要我们自己为 Docker 容器添加网卡、配置 IP 等。bridge模式图:The Container World | Part 2 Networkingbridge 模式是 Docker 默认的网络设置，此模式会为每一个容器分配 Network Namespace、设置 IP 等，并将一个主机上的 Docker 容器连接到一个虚拟网桥上。当 Docker server 启动时，会在主机上创建一个名为 docker0 的虚拟网桥，此主机上启动的 Docker 容器会连接到这个虚拟网桥上。虚拟网桥的工作方式和物理交换机类似，这样主机上的所有容器就通过交换机连在了一个二层网络中。接下来就要为容器分配 IP 了，Docker 会从 RFC1918 所定义的私有 IP 网段中，选择一个和宿主机不同的IP地址和子网分配给 docker0，连接到 docker0 的容器就从这个子网中选择一个未占用的 IP 使用。如一般 Docker 会使用 172.17.0.0/16 这个网段，并将 172.17.42.1/16 分配给 docker0 网桥（在主机上使用 ifconfig 命令是可以看到 docker0 的，可以认为它是网桥的管理接口，在宿主机上作为一块虚拟网卡使用）6.2 列出当前主机网桥12$ sudo brctl show # brctl 工具依赖 bridge-utils 软件包 bridge name bridge id STP enabled interfacesdocker0 8000.000000000000 no6.3 查看当前 docker0 ip123$ sudo ifconfig docker0docker0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xxinet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0在容器运行时，每个容器都会分配一个特定的虚拟机口并桥接到 docker0。每个容器都会配置同 docker0 ip 相同网段的专用 ip 地址，docker0 的 IP 地址被用于所有容器的默认网关。6.4 运行一个容器12345$ sudo docker run -t -i -d ubuntu /bin/bash52f811c5d3d69edddefc75aff5a4525fc8ba8bcfa1818132f9dc7d4f7c7e78b4$ sudo brctl showbridge name bridge id STP enabled interfacesdocker0 8000.fef213db5a66 no vethQCDY1N以上, docker0 扮演着 52f811c5d3d6 container 这个容器的虚拟接口 vethQCDY1N interface 桥接的角色。使用特定范围的 IPDocker 会尝试寻找没有被主机使用的 ip 段，尽管它适用于大多数情况下，但是它不是万能的，有时候我们还是需要对 ip 进一步规划。Docker 允许你管理 docker0 桥接或者通过-b选项自定义桥接网卡，需要安装bridge-utils软件包。基本步骤如下：ensure Docker is stopped​# 确保 docker 的进程是停止的create your own bridge (bridge0 for example)# 创建自定义网桥assign a specific IP to this bridge​# 给网桥分配特定的 ipstart Docker with the -b=bridge0 parameter# 以 -b 的方式指定网桥12345# Stopping Docker and removing docker0 $ sudo service docker stop $ sudo ip link set dev docker0 down $ sudo brctl delbr docker0 # Create our own bridge $ sudo brctl addbr bridge0 $ sudo ip addr add 192.168.5.1/24 dev bridge0 $ sudo ip link set dev bridge0 up # Confirming that our bridge is up and running $ ip addr show bridge04: bridge0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state UP group default link/ether 66:38:d0:0d:76:18 brd ff:ff:ff:ff:ff:ff inet 192.168.5.1/24 scope global bridge0 valid_lft forever preferred_lft forever # Tell Docker about it and restart (on Ubuntu) $ echo 'DOCKER_OPTS="-b=bridge0"' >> /etc/default/docker $ sudo service docker start参考文档: Network Configuration6.5 不同主机间容器通信不同容器之间的通信可以借助于 pipework 这个工具：12$ git clone https://github.com/jpetazzo/pipework.git$ sudo cp -rp pipework/pipework /usr/local/bin/安装相应依赖软件1$ sudo apt-get install iputils-arping bridge-utils -y桥接网络桥接网络可以参考 日常问题处理 Tips 关于桥接的配置说明，这里不再赘述。1234# brctl showbridge name bridge id STP enabled interfacesbr0 8000.000c291412cd no eth0docker0 8000.56847afe9799 no vetheb48029可以删除 docker0，直接把 docker 的桥接指定为 br0。也可以保留使用默认的配置，这样单主机容器之间的通信可以通过 docker0，而跨主机不同容器之间通过 pipework 新建 docker 容器的网卡桥接到 br0，这样跨主机容器之间就可以通信了。ubuntu12345$ sudo service docker stop$ sudo ip link set dev docker0 down$ sudo brctl delbr docker0$ echo 'DOCKER_OPTS="-b=br0"' >> /etc/default/docker$ sudo service docker startCentOS 7/RHEL 7123456$ sudo systemctl stop docker$ sudo ip link set dev docker0 down$ sudo brctl delbr docker0$ cat /etc/sysconfig/docker | grep 'OPTIONS='OPTIONS=--selinux-enabled -b=br0 -H fd://$ sudo systemctl start dockerpipework不同容器之间的通信可以借助于 pipework 这个工具给 docker 容器新建虚拟网卡并绑定 IP 桥接到 br01234567$ git clone https://github.com/jpetazzo/pipework.git$ sudo cp -rp pipework/pipework /usr/local/bin/$ pipework Syntax:pipework <hostinterface> [-i containerinterface] <guest> <ipaddr>/<subnet>[@default_gateway] [macaddr][@vlan]pipework <hostinterface> [-i containerinterface] <guest> dhcp [macaddr][@vlan]pipework --wait [-i containerinterface]如果删除了默认的 docker0 桥接，把 docker 默认桥接指定到了 br0，则最好在创建容器的时候加上–net=none，防止自动分配的 IP 在局域网中有冲突。1234567891011121314151617181920212223242526$ sudo docker run --rm -ti --net=none ubuntu:14.04 /bin/bashroot@a46657528059:/#$ # Ctrl-P + Ctrl-Q 回到宿主机 shell，容器 detach 状态$ sudo docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESa46657528059 ubuntu:14.04 "/bin/bash" 4 minutes ago Up 4 minutes hungry_lalande$ sudo pipework br0 -i eth0 a46657528059 192.168.115.10/24@192.168.115.2 # 默认不指定网卡设备名，则默认添加为 eth1# 另外 pipework 不能添加静态路由，如果有需求则可以在 run 的时候加上 --privileged=true 权限在容器中手动添加，# 但这种安全性有缺陷，可以通过 ip netns 操作$ sudo docker attach a46657528059root@a46657528059:/# ifconfig eth0eth0 Link encap:Ethernet HWaddr 86:b6:6b:e8:2e:4d inet addr:192.168.115.10 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::84b6:6bff:fee8:2e4d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:9 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:648 (648.0 B) TX bytes:690 (690.0 B)root@a46657528059:/# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0 192.168.115.2 0.0.0.0 UG 0 0 0 eth0192.168.115.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0使用ip netns添加静态路由，避免创建容器使用–privileged=true选项造成一些不必要的安全问题：1234567$ docker inspect --format="{{ .State.Pid }}" a46657528059 # 获取指定容器 pid6350$ sudo ln -s /proc/6350/ns/net /var/run/netns/6350$ sudo ip netns exec 6350 ip route add 192.168.0.0/16 dev eth0 via 192.168.115.2$ sudo ip netns exec 6350 ip route # 添加成功192.168.0.0/16 via 192.168.115.2 dev eth0 ... ...在其它宿主机进行相应的配置，新建容器并使用 pipework 添加虚拟网卡桥接到 br0，测试通信情况即可。另外，pipework 可以创建容器的 vlan 网络，这里不作过多的介绍了，官方文档已经写的很清楚了，可以查看以下两篇文章：Pipework 官方文档Docker 网络详解及 pipework 源码解读与实践七、DockerfileDocker 可以通过 Dockerfile 的内容来自动构建镜像。Dockerfile 是一个包含创建镜像所有命令的文本文件，通过docker build命令可以根据 Dockerfile 的内容构建镜像，在介绍如何构建之前先介绍下 Dockerfile 的基本语法结构。Dockerfile 有以下指令选项:FROMMAINTAINERRUNCMDEXPOSEENVADDCOPYENTRYPOINTVOLUMEUSERWORKDIRONBUILD7.1 FROM用法:1FROM <image>或者1FROM <image>FROM指定构建镜像的基础源镜像，如果本地没有指定的镜像，则会自动从 Docker 的公共库 pull 镜像下来。FROM必须是 Dockerfile 中非注释行的第一个指令，即一个 Dockerfile 从FROM语句开始。FROM可以在一个 Dockerfile 中出现多次，如果有需求在一个 Dockerfile 中创建多个镜像。如果FROM语句没有指定镜像标签，则默认使用latest标签。7.2 MAINTAINER用法:1MAINTAINER <name>指定创建镜像的用户RUN 有两种使用方式RUNRUN “executable”, “param1”, “param2”每条RUN指令将在当前镜像基础上执行指定命令，并提交为新的镜像，后续的RUN都在之前RUN提交后的镜像为基础，镜像是分层的，可以通过一个镜像的任何一个历史提交点来创建，类似源码的版本控制。exec 方式会被解析为一个 JSON 数组，所以必须使用双引号而不是单引号。exec 方式不会调用一个命令 shell，所以也就不会继承相应的变量，如：1RUN [ "echo", "$HOME" ]这种方式是不会达到输出 HOME 变量的，正确的方式应该是这样的1RUN [ "sh", "-c", "echo", "$HOME" ]RUN产生的缓存在下一次构建的时候是不会失效的，会被重用，可以使用–no-cache选项，即docker build –no-cache，如此便不会缓存。7.3 CMDCMD有三种使用方式:CMD “executable”,”param1”,”param2”CMD “param1”,”param2”CMD command param1 param2 (shell form)CMD指定在 Dockerfile 中只能使用一次，如果有多个，则只有最后一个会生效。CMD的目的是为了在启动容器时提供一个默认的命令执行选项。如果用户启动容器时指定了运行的命令，则会覆盖掉CMD指定的命令。CMD会在启动容器的时候执行，build 时不执行，而RUN只是在构建镜像的时候执行，后续镜像构建完成之后，启动容器就与RUN无关了，这个初学者容易弄混这个概念，这里简单注解一下。7.4 EXPOSE1EXPOSE <port> [<port>...]告诉 Docker 服务端容器对外映射的本地端口，需要在 docker run 的时候使用-p或者-P选项生效。7.5 ENV12ENV <key> <value> # 只能设置一个变量ENV <key>=<value> ... # 允许一次设置多个变量指定一个环节变量，会被后续RUN指令使用，并在容器运行时保留。例子:12ENV myName="John Doe" myDog=Rex\\ The\\ Dog \\ myCat=fluffy等同于123ENV myName John DoeENV myDog Rex The DogENV myCat fluffy7.6 ADD1ADD <src>... <dest>ADD复制本地主机文件、目录或者远程文件 URLS 从 并且添加到容器指定路径中 。支持通过 GO 的正则模糊匹配，具体规则可参见 Go filepath.Match12ADD hom* /mydir/ # adds all files starting with "hom"ADD hom?.txt /mydir/ # ? is replaced with any single character路径必须是绝对路径，如果 不存在，会自动创建对应目录路径必须是 Dockerfile 所在路径的相对路径如果是一个目录，只会复制目录下的内容，而目录本身则不会被复制7.7 COPY1COPY <src>... <dest>COPY复制新文件或者目录从 并且添加到容器指定路径中 。用法同ADD，唯一的不同是不能指定远程文件 URLS。7.8 ENTRYPOINTENTRYPOINT “executable”, “param1”, “param2”ENTRYPOINT command param1 param2 (shell form)配置容器启动后执行的命令，并且不可被 docker run 提供的参数覆盖，而CMD是可以被覆盖的。如果需要覆盖，则可以使用docker run –entrypoint选项。每个 Dockerfile 中只能有一个ENTRYPOINT，当指定多个时，只有最后一个生效。Exec form ENTRYPOINT 例子通过ENTRYPOINT使用 exec form 方式设置稳定的默认命令和选项，而使用CMD添加默认之外经常被改动的选项。123FROM ubuntuENTRYPOINT ["top", "-b"]CMD ["-c"]通过 Dockerfile 使用ENTRYPOINT展示前台运行 Apache 服务12345FROM debian:stableRUN apt-get update && apt-get install -y --force-yes apache2EXPOSE 80 443VOLUME ["/var/www", "/var/log/apache2", "/etc/apache2"]ENTRYPOINT ["/usr/sbin/apache2ctl", "-D", "FOREGROUND"]Shell form ENTRYPOINT 例子这种方式会在/bin/sh -c中执行，会忽略任何CMD或者docker run命令行选项，为了确保docker stop能够停止长时间运行ENTRYPOINT的容器，确保执行的时候使用exec选项。12FROM ubuntuENTRYPOINT exec top -b如果在ENTRYPOINT忘记使用exec选项，则可以使用CMD补上:123FROM ubuntuENTRYPOINT top -bCMD --ignored-param1 # --ignored-param2 ... --ignored-param3 ... 依此类推7.9 VOLUME1VOLUME ["/data"]创建一个可以从本地主机或其他容器挂载的挂载点，后续具体介绍。7.10 USER1USER daemon指定运行容器时的用户名或 UID，后续的RUN、CMD、ENTRYPOINT也会使用指定用户。7.11 WORKDIR1WORKDIR /path/to/workdir为后续的RUN、CMD、ENTRYPOINT指令配置工作目录。可以使用多个WORKDIR指令，后续命令如果参数是相对路径，则会基于之前命令指定的路径。1234WORKDIR /aWORKDIR bWORKDIR cRUN pwd最终路径是/a/b/c。WORKDIR指令可以在ENV设置变量之后调用环境变量:12ENV DIRPATH /pathWORKDIR $DIRPATH/$DIRNAME最终路径则为 /path/$DIRNAME。7.12 ONBUILD1ONBUILD [INSTRUCTION]配置当所创建的镜像作为其它新创建镜像的基础镜像时，所执行的操作指令。例如，Dockerfile 使用如下的内容创建了镜像 image-A：1234[...]ONBUILD ADD . /app/srcONBUILD RUN /usr/local/bin/python-build --dir /app/src[...]如果基于 image-A 创建新的镜像时，新的 Dockerfile 中使用 FROM image-A 指定基础镜像时，会自动执行 ONBUILD 指令内容，等价于在后面添加了两条指令。123# Automatically run the followingADD . /app/srcRUN /usr/local/bin/python-build --dir /app/src使用ONBUILD指令的镜像，推荐在标签中注明，例如 ruby:1.9-onbuild。7.13 Dockerfile Examples12345678910111213141516171819202122232425262728293031323334353637383940# Nginx## VERSION 0.0.1FROM ubuntuMAINTAINER Victor Vieux <victor@docker.com>RUN apt-get update && apt-get install -y inotify-tools nginx apache2 openssh-server# Firefox over VNC## VERSION 0.3FROM ubuntu# Install vnc, xvfb in order to create a 'fake' display and firefoxRUN apt-get update && apt-get install -y x11vnc xvfb firefoxRUN mkdir ~/.vnc# Setup a passwordRUN x11vnc -storepasswd 1234 ~/.vnc/passwd# Autostart firefox (might not be the best way, but it does the trick)RUN bash -c 'echo "firefox" >> /.bashrc'EXPOSE 5900CMD ["x11vnc", "-forever", "-usepw", "-create"]# Multiple images example## VERSION 0.1FROM ubuntuRUN echo foo > bar# Will output something like ===> 907ad6c2736fFROM ubuntuRUN echo moo > oink# Will output something like ===> 695d7793cbe4# You᾿ll now have two images, 907ad6c2736f with /bar, and 695d7793cbe4 with# /oink.7.14 docker build1234567891011$ docker build --helpUsage: docker build [OPTIONS] PATH | URL | -Build a new image from the source code at PATH --force-rm=false Always remove intermediate containers, even after unsuccessful builds # 移除过渡容器，即使构建失败 --no-cache=false Do not use cache when building the image # 不实用 cache -q, --quiet=false Suppress the verbose output generated by the containers --rm=true Remove intermediate containers after a successful build # 构建成功后移除过渡层容器 -t, --tag="" Repository name (and optionally a tag) to be applied to the resulting image in case of success参考文档:Dockerfile Reference7.15 dockerfile 最佳实践使用.dockerignore文件为了在docker build过程中更快上传和更加高效，应该使用一个.dockerignore文件用来排除构建镜像时不需要的文件或目录。例如,除非.git在构建过程中需要用到，否则你应该将它添加到.dockerignore文件中，这样可以节省很多时间。避免安装不必要的软件包为了降低复杂性、依赖性、文件大小以及构建时间，应该避免安装额外的或不必要的包。例如，不需要在一个数据库镜像中安装一个文本编辑器。每个容器都跑一个进程在大多数情况下，一个容器应该只单独跑一个程序。解耦应用到多个容器使其更容易横向扩展和重用。如果一个服务依赖另外一个服务，可以参考 Linking Containers Together。最小化层我们知道每执行一个指令，都会有一次镜像的提交，镜像是分层的结构，对于Dockerfile，应该找到可读性和最小化层之间的平衡。多行参数排序如果可能，通过字母顺序来排序，这样可以避免安装包的重复并且更容易更新列表，另外可读性也会更强，添加一个空行使用\\换行:123456RUN apt-get update && apt-get install -y \\ bzr \\ cvs \\ git \\ mercurial \\ subversion创建缓存镜像构建过程中会按照Dockerfile的顺序依次执行，每执行一次指令 Docker 会寻找是否有存在的镜像缓存可复用，如果没有则创建新的镜像。如果不想使用缓存，则可以在docker build时添加–no-cache=true选项。从基础镜像开始就已经在缓存中了，下一个指令会对比所有的子镜像寻找是否执行相同的指令，如果没有则缓存失效。在大多数情况下只对比Dockerfile指令和子镜像就足够了。ADD和COPY指令除外，执行ADD和COPY时存放到镜像的文件也是需要检查的，完成一个文件的校验之后再利用这个校验在缓存中查找，如果检测的文件改变则缓存失效。RUN apt-get -y update命令只检查命令是否匹配，如果匹配就不会再执行更新了。为了有效地利用缓存，你需要保持你的 Dockerfile 一致，并且尽量在末尾修改。Dockerfile 指令FROM: 只要可能就使用官方镜像库作为基础镜像RUN: 为保持可读性、方便理解、可维护性，把长或者复杂的RUN语句使用\\分隔符分成多行不建议RUN apt-get update独立成行，否则如果后续包有更新，那么也不会再执行更新避免使用RUN apt-get upgrade或者dist-upgrade，很多必要的包在一个非privileged权限的容器里是无法升级的。如果知道某个包更新，使用apt-get install -y xxx标准写法RUN apt-get update && apt-get install -y package-bar package-foo例子:12345678910111213141516171819RUN apt-get update && apt-get install -y \\ aufs-tools \\ automake \\ btrfs-tools \\ build-essential \\ curl \\ dpkg-sig \\ git \\ iptables \\ libapparmor-dev \\ libcap-dev \\ libsqlite3-dev \\ lxc=1.0* \\ mercurial \\ parallel \\ reprepro \\ ruby1.9.1 \\ ruby1.9.1-dev \\ s3cmd=1.1.0*CMD: 推荐使用CMD [“executable”, “param1”, “param2”…]这种格式，CMD [“param”, “param”]则配合ENTRYPOINT使用EXPOSE: Dockerfile 指定要公开的端口，使用docker run时指定映射到宿主机的端口即可ENV: 为了使新的软件更容易运行，可以使用ENV更新PATH变量。如ENV PATH /usr/local/nginx/bin:$PATH确保CMD [“nginx”]即可运行ENV也可以这样定义变量：1234ENV PG_MAJOR 9.3ENV PG_VERSION 9.3.4RUN curl -SL http://example.com/postgres-$PG_VERSION.tar.xz | tar -xJC /usr/src/postgress && …ENV PATH /usr/local/postgres-$PG_MAJOR/bin:$PATHADDorCOPY:ADD比COPY多一些特性「tar 文件自动解包和支持远程 URL」，不推荐添加远程 URL如不推荐这种方式:123ADD http://example.com/big.tar.xz /usr/src/things/RUN tar -xJf /usr/src/things/big.tar.xz -C /usr/src/thingsRUN make -C /usr/src/things all推荐使用 curl 或者 wget 替换，使用如下方式:1234RUN mkdir -p /usr/src/things \\ && curl -SL http://example.com/big.tar.gz \\ | tar -xJC /usr/src/things \\ && make -C /usr/src/things all如果不需要添加 tar 文件，推荐使用COPY。参考文档:Best practices for writing DockerfilesDockerfile最佳实践（一）Dockerfile最佳实践（二）八、容器数据管理docker管理数据的方式有两种：数据卷数据卷容器8.1 数据卷数据卷是一个或多个容器专门指定绕过Union File System的目录，为持续性或共享数据提供一些有用的功能：数据卷可以在容器间共享和重用数据卷数据改变是直接修改的数据卷数据改变不会被包括在容器中数据卷是持续性的，直到没有容器使用它们添加一个数据卷你可以使用-v选项添加一个数据卷，或者可以使用多次-v选项为一个 docker 容器运行挂载多个数据卷。123456$ sudo docker run --name data -v /data -t -i ubuntu:14.04 /bin/bash # 创建数据卷绑定到到新建容器，新建容器中会创建 /data 数据卷 bash-4.1# ls -ld /data/drwxr-xr-x 2 root root 4096 Jul 23 06:59 /data/bash-4.1# df -ThFilesystem Type Size Used Avail Use% Mounted on... ... ext4 91G 4.6G 82G 6% /data创建的数据卷可以通过docker inspect获取宿主机对应路径12$ sudo docker inspect data... ... "Volumes": { "/data": "/var/lib/docker/vfs/dir/151de401d268226f96d824fdf444e77a4500aed74c495de5980c807a2ffb7ea9" }, # 可以看到创建的数据卷宿主机路径 ... ...或者直接指定获取12$ sudo docker inspect --format="{{ .Volumes }}" datamap[/data: /var/lib/docker/vfs/dir/151de401d268226f96d824fdf444e77a4500aed74c495de5980c807a2ffb7ea9]挂载宿主机目录为一个数据卷-v选项除了可以创建卷，也可以挂载当前主机的一个目录到容器中。1234567$ sudo docker run --name web -v /source/:/web -t -i ubuntu:14.04 /bin/bashbash-4.1# ls -ld /web/drwxr-xr-x 2 root root 4096 Jul 23 06:59 /web/bash-4.1# df -Th... ... ext4 91G 4.6G 82G 6% /webbash-4.1# exit默认挂载卷是可读写的，可以在挂载时指定只读1$ sudo docker run --rm --name test -v /source/:/test:ro -t -i ubuntu:14.04 /bin/bash8.2 创建和挂载一个数据卷容器如果你有一些持久性的数据并且想在容器间共享，或者想用在非持久性的容器上，最好的方法是创建一个数据卷容器，然后从此容器上挂载数据。创建数据卷容器1$ sudo docker run -t -i -d -v /test --name test ubuntu:14.04 echo hello使用–volumes-from选项在另一个容器中挂载 /test 卷。不管 test 容器是否运行，其它容器都可以挂载该容器数据卷，当然如果只是单独的数据卷是没必要运行容器的。1$ sudo docker run -t -i -d --volumes-from test --name test1 ubuntu:14.04 /bin/bash添加另一个容器1$ sudo docker run -t -i -d --volumes-from test --name test2 ubuntu:14.04 /bin/bash也可以继承其它挂载有 /test 卷的容器1$ sudo docker run -t -i -d --volumes-from test1 --name test3 ubuntu:14.04 /bin/bash8.3 备份、恢复或迁移数据卷备份1234567$ sudo docker run --rm --volumes-from test -v $(pwd):/backup ubuntu:14.04 tar cvf /backup/test.tar /testtar: Removing leading `/' from member names/test//test/b/test/d/test/c/test/a启动一个新的容器并且从test容器中挂载卷，然后挂载当前目录到容器中为 backup，并备份 test 卷中所有的数据为 test.tar，执行完成之后删除容器–rm，此时备份就在当前的目录下，名为test.tar。1$ ls # 宿主机当前目录下产生了 test 卷的备份文件 test.tar test.tar恢复你可以恢复给同一个容器或者另外的容器，新建容器并解压备份文件到新的容器数据卷1$ sudo docker run -t -i -d -v /test --name test4 ubuntu:14.04 /bin/bash $ sudo docker run --rm --volumes-from test4 -v $(pwd):/backup ubuntu:14.04 tar xvf /backup/test.tar -C / # 恢复之前的文件到新建卷中，执行完后自动删除容器 test/ test/b test/d test/c test/a8.4 删除 VolumesVolume 只有在下列情况下才能被删除：docker rm -v删除容器时添加了-v选项docker run –rm运行容器时添加了–rm选项否则，会在/var/lib/docker/vfs/dir目录中遗留很多不明目录。参考文档：Managing Data in Containers深入理解Docker Volume（一）深入理解Docker Volume（二）九、链接容器docker 允许把多个容器连接在一起，相互交互信息。docker 链接会创建一种容器父子级别的关系，其中父容器可以看到其子容器提供的信息。9.1 容器命名在创建容器时，如果不指定容器的名字，则默认会自动创建一个名字，这里推荐给容器命名：1、给容器命名方便记忆，如命名运行 web 应用的容器为 web2、为 docker 容器提供一个参考，允许方便其他容器调用，如把容器 web 链接到容器 db可以通过–name选项给容器自定义命名：123$ sudo docker run -d -t -i --name test ubuntu:14.04 bash $ sudo docker inspect --format="{{ .Nmae }}" test/test注：容器名称必须唯一，即你只能命名一个叫test的容器。如果你想复用容器名，则必须在创建新的容器前通过docker rm删除旧的容器或者创建容器时添加–rm选项。9.2 链接容器链接允许容器间安全通信，使用–link选项创建链接。1$ sudo docker run -d --name db training/postgres基于 training/postgres 镜像创建一个名为 db 的容器，然后下面创建一个叫做 web 的容器，并且将它与 db 相互连接在一起1$ sudo docker run -d -P --name web --link db:db training/webapp python app.py–link:alias选项指定链接到的容器。查看 web 容器的链接关系:12$ sudo docker inspect -f "{{ .HostConfig.Links }}" web[/db:/web/db]可以看到 web 容器被链接到 db 容器为/web/db，这允许 web 容器访问 db 容器的信息。容器之间的链接实际做了什么？一个链接允许一个源容器提供信息访问给一个接收容器。在本例中，web 容器作为一个接收者，允许访问源容器 db 的相关服务信息。Docker 创建了一个安全隧道而不需要对外公开任何端口给外部容器，因此不需要在创建容器的时候添加-p或-P指定对外公开的端口，这也是链接容器的最大好处，本例为 PostgreSQL 数据库。Docker 主要通过以下两个方式提供连接信息给接收容器：环境变量更新/etc/hosts文件环境变量当两个容器链接，Docker 会在目标容器上设置一些环境变量，以获取源容器的相关信息。首先，Docker 会在每个通过–link选项指定别名的目标容器上设置一个_NAME环境变量。如果一个名为 web 的容器通过–link db:webdb被链接到一个名为 db 的数据库容器，那么 web 容器上会设置一个环境变量为WEBDB_NAME=/web/webdb.以之前的为例，Docker 还会设置端口变量:123456789$ sudo docker run --rm --name web2 --link db:db training/webapp env. . .DB_NAME=/web2/dbDB_PORT=tcp://172.17.0.5:5432 DB_PORT_5432_TCP=tcp://172.17.0.5:5432 # <name>_PORT_<port>_<protocol> 协议可以是 TCP 或 UDPDB_PORT_5432_TCP_PROTO=tcpDB_PORT_5432_TCP_PORT=5432DB_PORT_5432_TCP_ADDR=172.17.0.5. . .注：这些环境变量只设置给容器中的第一个进程，类似一些守护进程 (如 sshd ) 当他们派生 shells 时会清除这些变量更新/etc/hosts文件除了环境变量，Docker 会在目标容器上添加相关主机条目到/etc/hosts中，上例中就是 web 容器。12345$ sudo docker run -t -i --rm --link db:db training/webapp /bin/bashroot@aed84ee21bde:/opt/webapp# cat /etc/hosts172.17.0.7 aed84ee21bde. . .172.17.0.5 db/etc/host文件在源容器被重启之后会自动更新 IP 地址，而环境变量中的 IP 地址则不会自动更新的。十、构建私有库Docker 官方提供了 docker registry 的构建方法 docker-registry10.1 快速构建快速构建 docker registry 通过以下两步:安装 docker运行 registry:docker run -p 5000:5000 registry这种方法通过 Docker hub 使用官方镜像 official image from the Docker hub10.2 不使用容器构建 registry安装必要的软件1$ sudo apt-get install build-essential python-dev libevent-dev python-pip liblzma-dev配置 docker-registry1sudo pip install docker-registry或者 使用 github clone 手动安装12345$ git clone https://github.com/dotcloud/docker-registry.git$ cd docker-registry/$ cp config/config_sample.yml config/config.yml$ mkdir /data/registry -p$ pip install .运行1docker-registry高级启动方式 [不推荐]使用gunicorn控制:1gunicorn -c contrib/gunicorn_config.py docker_registry.wsgi:application或者对外监听开放1gunicorn --access-logfile - --error-logfile - -k gevent -b 0.0.0.0:5000 -w 4 --max-requests 100 docker_registry.wsgi:application10.3 提交指定容器到私有库12$ docker tag ubuntu:12.04 私有库IP:5000/ubuntu:12.04$ docker push 私有库IP:5000/ubuntu十一、docker 启动redis一.docker运行单个redis拉取官方镜像，镜像地址：https://hub.docker.com/_/redis/拉取镜像：docker pull redis执行指令启动Redisdocker run –name redis -d -p 6379:6379 redis二.docker运行单个redis1.拉取镜像: docker pull redis2.运行容器(本地image是:docker.io/redis latest 8f2e175b3bd1 2 weeks ago 106.6 MB):docker run -d --name my_redis -p 6379:6379 -v /data/redis/data/:/data redis redis-server-d是后台运行;–name是设置别名-v /data/redis/data/:/data是将 /data/redis/data/挂载到容器的/data(数据默认存储在VOLUME /data目录下,可以使用$PWD/data代替/data/redis/data/)3.运行客户端:docker run -it --link my_redis --rm docker.io/redis redis-cli -h my_redis -p 6379-it是交互模式(-i: 以交互模式运行容器,-t: 为容器重新分配一个伪输入终端)–link 连接另一个容器,这样就可以使用容器名作为host了–rm 自动清理容器,因为这里是测试,属于前台程序二.docker运行redis主从复制模式（以两个数据库为例）主从复制模式:主数据库(master)可以读写,从数据库(slave)只能读;主数据库的写会同步到从数据库,从数据库主要负责读操作。一个主数据库可以拥有多个从数据库，一个从数据库只能拥有一个主数据库。1.启动两个服务端:12docker run -d --name redis-master -v /data/redis/data/:/data redis redis-serverdocker run -d --name redis-slave --link redis-master redis redis-server --port 6380 --slaveof redis-master 6379122.启动对应的客户端:12docker run -it --link redis-master --rm redis redis-cli -h redis-master -p 6379docker run -it --link redis-slave --rm redis redis-cli -h redis-slave -p 638012使用：slaveof no one退出主从关系十二、docker 部署zookeeper镜像下载hub.docker.com 上有不少 ZK 镜像, 不过为了稳定起见, 我们就使用官方的 ZK 镜像吧.首先执行如下命令:1docker pull zookeeper当出现如下结果时, 表示镜像已经下载完成了:12345678910111213>>> docker pull zookeeperUsing default tag: latestlatest: Pulling from library/zookeepere110a4a17941: Pull completea696cba1f6e8: Pull completebc427bd93e95: Pull completec72391ae24f6: Pull complete40ab409b6b34: Pull completed4bb8183b85d: Pull complete0600755f1470: Pull completeDigest: sha256:12458234bb9f01336df718b7470cabaf5c357052cbcb91f8e80be07635994464Status: Downloaded newer image for zookeeper:latest2.启动 ZK 镜像1>>> docker run --name my_zookeeper -d zookeeper:latest这个命令会在后台运行一个 zookeeper 容器, 名字是 my_zookeeper, 并且它默认会导出 2181 端口.接着我们使用:1docker logs -f my_zookeeper这个命令查看 ZK 的运行情况, 输出类似如下内容时, 表示 ZK 已经成功启动了:12345>>> docker logs -f my_zookeeperZooKeeper JMX enabled by defaultUsing config: /conf/zoo.cfg...2016-09-14 06:40:03,445 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:21813.使用 ZK 命令行客户端连接 ZK因为刚才我们启动的那个 ZK 容器并没有绑定宿主机的端口, 因此我们不能直接访问它. 但是我们可以通过 Docker 的 link 机制来对这个 ZK 容器进行访问. 执行如下命令:1docker run -it --rm --link my_zookeeper:zookeeper zookeeper zkCli.sh -server zookeeper如果对 Docker 有过了解的话, 那么对上面的命令一定不会陌生了.这个命令的含义是:启动一个 zookeeper 镜像, 并运行这个镜像内的 zkCli.sh 命令, 命令参数是 “-server zookeeper”将我们先前启动的名为 my_zookeeper 的容器连接(link) 到我们新建的这个容器上, 并将其主机名命名为 zookeeper当我们执行了这个命令后, 就可以像正常使用 ZK 命令行客户端一样操作 ZK 服务了.ZK 集群的搭建因为一个一个地启动 ZK 太麻烦了, 所以为了方便起见, 我直接使用 docker-compose 来启动 ZK 集群.首先创建一个名为 docker-compose.yml 的文件, 其内容如下:12345678910111213141516171819202122232425262728293031version: '2'services: zoo1: image: zookeeper restart: always container_name: zoo1 ports: - "2181:2181" environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 zoo2: image: zookeeper restart: always container_name: zoo2 ports: - "2182:2181" environment: ZOO_MY_ID: 2 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 zoo3: image: zookeeper restart: always container_name: zoo3 ports: - "2183:2181" environment: ZOO_MY_ID: 3 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888这个配置文件会告诉 Docker 分别运行三个 zookeeper 镜像, 并分别将本地的 2181, 2182, 2183 端口绑定到对应的容器的2181端口上.ZOO_MY_ID 和 ZOO_SERVERS 是搭建 ZK 集群需要设置的两个环境变量, 其中 ZOO_MY_ID 表示 ZK 服务的 id, 它是1-255 之间的整数, 必须在集群中唯一. ZOO_SERVERS 是ZK 集群的主机列表.接着我们在 docker-compose.yml 当前目录下运行:1COMPOSE_PROJECT_NAME=zk_test docker-compose up即可启动 ZK 集群了.执行上述命令成功后, 接着在另一个终端中运行 docker-compose ps 命令可以查看启动的 ZK 容器:123456>>> COMPOSE_PROJECT_NAME=zk_test docker-compose psName Command State Ports----------------------------------------------------------------------zoo1 /docker-entrypoint.sh zkSe ... Up 0.0.0.0:2181->2181/tcpzoo2 /docker-entrypoint.sh zkSe ... Up 0.0.0.0:2182->2181/tcpzoo3 /docker-entrypoint.sh zkSe ... Up 0.0.0.0:2183->2181/tcp注意, 我们在 “docker-compose up” 和 “docker-compose ps” 前都添加了 COMPOSE_PROJECT_NAME=zk_test 这个环境变量, 这是为我们的 compose 工程起一个名字, 以免与其他的 compose 混淆.使用 Docker 命令行客户端连接 ZK 集群通过 docker-compose ps 命令, 我们知道启动的 ZK 集群的三个主机名分别是 zoo1, zoo2, zoo3, 因此我们分别 link 它们即可:123456docker run -it --rm \\ --link zoo1:zk1 \\ --link zoo2:zk2 \\ --link zoo3:zk3 \\ --net zktest_default \\ zookeeper zkCli.sh -server zk1:2181,zk2:2181,zk3:2181通过本地主机连接 ZK 集群因为我们分别将 zoo1, zoo2, zoo3 的 2181 端口映射到了 本地主机的2181, 2182, 2183 端口上, 因此我们使用如下命令即可连接 ZK 集群了:1zkCli.sh -server localhost:2181,localhost:2182,localhost:2183查看集群我们可以通过 nc 命令连接到指定的 ZK 服务器, 然后发送 stat 可以查看 ZK 服务的状态, 例如:12345678910111213>>> echo stat | nc 127.0.0.1 2181Zookeeper version: 3.4.9-1757313, built on 08/23/2016 06:50 GMTClients: /172.18.0.1:49810[0](queued=0,recved=1,sent=0)Latency min/avg/max: 5/39/74Received: 4Sent: 3Connections: 1Outstanding: 0Zxid: 0x200000002Mode: followerNode count: 412345678910111213>>> echo stat | nc 127.0.0.1 2182Zookeeper version: 3.4.9-1757313, built on 08/23/2016 06:50 GMTClients: /172.18.0.1:50870[0](queued=0,recved=1,sent=0)Latency min/avg/max: 0/0/0Received: 2Sent: 1Connections: 1Outstanding: 0Zxid: 0x200000002Mode: followerNode count: 412345678910111213>>> echo stat | nc 127.0.0.1 2183Zookeeper version: 3.4.9-1757313, built on 08/23/2016 06:50 GMTClients: /172.18.0.1:51820[0](queued=0,recved=1,sent=0)Latency min/avg/max: 0/0/0Received: 2Sent: 1Connections: 1Outstanding: 0Zxid: 0x200000002Mode: leaderNode count: 4十三、docker 部署 rabbitmq拉取官方镜像，镜像地址：https://hub.docker.com/_/rabbitmq/拉取镜像：docker pull rabbitmq，如需要管理界面：docker pull rabbitmq:management执行指令启动RabbitMQ无管理界面：docker run –localhost rabbit-host –name my_rabbit -d -p 5672:5672 rabbitmq有管理界面：docker run –localhost rabbit-host –name my_rabbit -d -p 5672:5672 -p 15672:15672 rabbitmq:management账号：guest 密码：guest 十四 、启动Eureka拉取官方镜像，镜像地址：https://hub.docker.com/r/springcloud/eureka/拉取镜像：docker pull springcloud/eureka执行指令启动Eurekadocker run –name my_eureka -d -p 8761:8761 springcloud/eureka十五 、启动Config Server拉取官方镜像，镜像地址：https://hub.docker.com/r/hyness/spring-cloud-config-server/拉取镜像：docker pull hyness/spring-cloud-config-server在GitHub上准备配置文件：https://github.com/ErikXu/.NetCore-Spring-Clould/tree/master/Configs准备启动资源文件application.yml:info:`component: config service`server:`port: 8888`spring:`application:``name: git-config``profiles:``active: dev``cloud:``config:``server:``git:``uri: https://github.com/ErikXu/.NetCore-Spring-Clould``searchPaths: Configs`执行指令启动Config Server，注：该指令前无空格docker run –name my_config-server -it -d -p 8888:8888 \\-v /home/erikxu/config/application.yml:/config/application.yml \\hyness/spring-cloud-config-server十六 、启动 mysql拉取官方镜像，镜像地址：https://hub.docker.com/_/mysql/拉取镜像：docker pull mysql准备Mysql数据存放目录，我这里是：/data/mysql执行指令启动Mysqldocker run –name my_mysql -v /data/mysql:/var/lib/mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=root -d mysql:latest十七、启动nginx拉取官方镜像，镜像地址：https://hub.docker.com/_/nginx/拉取镜像：docker pull nginx准备配置文件4、执行指令启动Nginxdocker run –name my_nginx -p 80:80 -v /data/etc/nginx.conf:/etc/nginx/nginx.conf:ro -d nginx","categories":[],"tags":[{"name":"docker","slug":"docker","permalink":"http://www.liuyong520.cn/tags/docker/"}]},{"title":"hexo博客主题优化","slug":"hexo-promise","date":"2017-08-29T02:56:51.000Z","updated":"2019-06-11T09:37:16.610Z","comments":true,"path":"2017/08/29/hexo-promise/","link":"","permalink":"http://www.liuyong520.cn/2017/08/29/hexo-promise/","excerpt":"","text":"在介绍博客主题优化这个话题之前，我想先介绍hexo主题的大体结构，便于后面将主题优化方面的东西。hexo主题结构我这里选用pure主题为例进行讲解。进入themes/pure文件夹下执行如下命令123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158$ tree.├── LICENSE├── README.cn.md├── README.md├── _config.yml #主题主配置文件├── _config.yml.example #主题配置文件例子├── _source #博客页面例子文件夹│ ├── 404 #博客404页面只要拷贝到站点soure就行│ │ └── index.md│ ├── _data #博客友情链接页面│ │ ├── gallery.yml│ │ └── links.yml│ ├── about #博客关于页面│ │ └── index.md│ ├── books #博客书单页面│ │ └── index.md│ ├── categories #博客分类页面│ │ └── index.md│ ├── links #博客友情链接│ │ └── index.md│ ├── repository #博客仓库模版页面│ │ └── index.md│ └── tags #博客标签页面│ └── index.md├── languages #博客语言切换配置文件夹│ ├── default.yml│ ├── en.yml│ ├── zh-CN.yml│ └── zh-TW.yml├── layout #博客布局文件夹 这里就是生成页面的精华部分了│ ├── _common│ │ ├── footer.ejs│ │ ├── head.ejs│ │ ├── header.ejs│ │ ├── script.ejs│ │ └── social.ejs│ ├── _partial│ │ ├── archive-book.ejs│ │ ├── archive-category.ejs│ │ ├── archive-link.ejs│ │ ├── archive-list.ejs│ │ ├── archive-post.ejs│ │ ├── archive-repository.ejs│ │ ├── archive-tag.ejs│ │ ├── archive.ejs│ │ ├── article-about.ejs│ │ ├── article.ejs│ │ ├── item-post.ejs│ │ ├── pagination.ejs│ │ ├── post│ │ │ ├── category.ejs│ │ │ ├── comment.ejs│ │ │ ├── copyright.ejs│ │ │ ├── date.ejs│ │ │ ├── donate.ejs│ │ │ ├── gallery.ejs│ │ │ ├── nav.ejs│ │ │ ├── pv.ejs│ │ │ ├── tag.ejs│ │ │ ├── thumbnail.ejs│ │ │ ├── title.ejs│ │ │ └── wordcount.ejs│ │ ├── sidebar-about.ejs│ │ ├── sidebar-toc.ejs│ │ └── sidebar.ejs│ ├── _script│ │ ├── _analytics│ │ │ ├── baidu-analytics.ejs│ │ │ ├── google-analytics.ejs│ │ │ └── tencent-analytics.ejs│ │ ├── _comment│ │ │ ├── disqus.ejs│ │ │ ├── gitalk.ejs│ │ │ ├── gitment.ejs│ │ │ ├── livere.ejs│ │ │ ├── valine.ejs│ │ │ └── youyan.ejs│ │ ├── _search│ │ │ ├── baidu.ejs│ │ │ └── insight.ejs│ │ ├── analytics.ejs│ │ ├── comment.ejs│ │ ├── douban.ejs│ │ ├── fancybox.ejs│ │ ├── mathjax.ejs│ │ ├── pv.ejs│ │ ├── repository.ejs│ │ └── search.ejs│ ├── _search│ │ ├── baidu.ejs│ │ ├── index-mobile.ejs│ │ ├── index.ejs│ │ ├── insight.ejs│ │ └── swiftype.ejs│ ├── _widget│ │ ├── archive.ejs│ │ ├── board.ejs│ │ ├── category.ejs│ │ ├── recent_posts.ejs│ │ ├── tag.ejs│ │ └── tagcloud.ejs│ ├── about.ejs│ ├── archive.ejs│ ├── books.ejs│ ├── categories.ejs│ ├── category.ejs│ ├── index.ejs│ ├── layout.ejs│ ├── links.ejs│ ├── page.ejs│ ├── post.ejs│ ├── repository.ejs│ ├── tag.ejs│ └── tags.ejs├── package.json├── screenshot #主题颜色切换背景│ ├── pure-theme-black.png│ ├── pure-theme-blue.png│ ├── pure-theme-green.png│ ├── pure-theme-purple.png│ ├── pure.png│ └── pure.psd├── scripts│ └── thumbnail.js└── source #主题静态资源文件目录 ├── css │ ├── style.css │ └── style.min.css ├── favicon.png ├── fonts │ ├── README.md │ ├── iconfont.eot │ ├── iconfont.svg │ ├── iconfont.ttf │ └── iconfont.woff ├── images │ ├── avatar.jpg │ ├── avatar.jpg1 │ ├── donate │ │ ├── alipayimg.png │ │ └── wechatpayimg.png │ ├── favatar │ │ ├── SzsFox-logo.png │ │ ├── chuangzaoshi-logo.png │ │ └── idesign-logo.png │ ├── thumb-default.png │ └── xingqiu-qrcode.jpg └── js ├── application.js ├── application.min.js ├── insight.js ├── jquery.min.js ├── plugin.js ├── plugin.js.map └── plugin.min.js29 directories, 125 fileslayout里面的文件使用ejs （js模版语言）ejs官网实现的，里面把整个页面通过js抽取各个小的模块模版文件，同时数据和标签页面是分离的，所以在页面里面可以加载config.yml 里面的配置。整个页面入口文件就是layout.js12345678910111213141516171819202122232425262728<!DOCTYPE html><html<%= config.language ? " lang=" + config.language.substring(0, 2) : ""%>><%- partial('_common/head', {post: page}) %>##这里会判断是否启用layout配置<% var bodyClass = 'main-center'; if (theme.config.layout) { bodyClass = theme.config.layout; } if (theme.config.skin) { bodyClass += ' ' + theme.config.skin; } bodyClass = page.sidebar === 'none' ? (bodyClass + ' no-sidebar') : bodyClass;%><body class="<%= bodyClass %>" itemscope itemtype="http://schema.org/WebPage"> <%- partial('_common/header', null, {cache: !config.relative_link}) %> <% if (theme.sidebar && (page.sidebar!='none' || page.sidebar!='custom')){ %> <% if (theme.config.toc && page.toc){ %> <%- partial('_partial/sidebar-toc', {post: page}) %> <% }else{ %> <%- partial('_partial/sidebar', null, {cache: !config.relative_link}) %> <% } %> <% } %> <%- body %> <%- partial('_common/footer', null, {cache: !config.relative_link}) %> <%- partial('_common/script', {post: page}) %></body></html>其中<%- partial(‘_common/footer’, null, {cache: !config.relative_link}) %> 表示引入子模块_common/footer.ejs文件，{cache: !config.relative_link}表示参数我们的创建的博客文章都会加载这个布局文件。我们新创建的博客文章有如下的配置：123456789title: 文章标题categories: - 文章分类tags: - 文章标签toc: true # 是否启用内容索引comment:true #是否启用评论layout:模版文件，如果没有默认不加载任何模版sidebar: none # 是否启用sidebar侧边栏，none：不启用，不配置默认启动以上配置属于page 域的配置文件属于单个页面的，而config.language 这种是全局配置文件（也就是站点配置文件config.yml），每个页面都能使用。theme.config 加载的就是主题的配置文件config.yml 文件。主题配置文件config.yml123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236 # menumenu: Home: . Archives: archives # 归档 Categories: categories # 分类 Tags: tags # 标签 Repository: repository # github repositories Books: books # 豆瓣书单 Links: links # 友链 About: about # 关于# Enable/Disable menu iconsmenu_icons: enable: true # 是否启用导航菜单图标 home: icon-home-fill archives: icon-archives-fill categories: icon-folder tags: icon-tags repository: icon-project books: icon-book-fill links: icon-friendship about: icon-cup-fill# rssrss: /atom.xml# Sitesite: logo: enabled: true width: 40 height: 40 url: ../images/logo.png title: Hexo # 页面title favicon: /favicon.png board: <p>欢迎交流与分享经验!</p> # 站点公告 copyright: false # 底部版权信息# configconfig: skin: theme-black # 主题颜色 theme-black theme-blue theme-green theme-purple layout: main-center # 布局方式 main-left main-center main-right toc: true # 是否开启文章章节目录导航 menu_highlight: false # 是否开启当前菜单高亮显示 thumbnail: false # enable posts thumbnail, options: true, false excerpt_link: Read More# Pagination 分页pagination: number: false #是否开启数字 prev: alwayShow: true next: alwayShow: true# Sidebarsidebar: rightwidgets: - board - category - tag - tagcloud - archive - recent_posts# display widgets at the bottom of index pages (pagination == 2)index_widgets:# - category# - tagcloud# - archive # widget behaviorarchive_type: 'monthly'show_count: true# Fancyboxfancybox: false# Searchsearch: insight: true # you need to install `hexo-generator-json-content` before using Insight Search baidu: false # you need to disable other search engines to use Baidu search, options: true, false# Donatedonate: enable: true # 微信打赏 wechatpay: qrcode: images/donate/wechatpayimg.png title: 微信支付 # 支付宝打赏 alipay: qrcode: images/donate/alipayimg.png title: 支付宝# Share# weibo,qq,qzone,wechat,tencent,douban,diandian,facebook,twitter,google,linkedinshare: enable: true # 是否启用分享 sites: weibo,qq,wechat,facebook,twitter # PC端显示的分享图标 mobile_sites: weibo,qq,qzone # 移动端显示的分享图标# Githubgithub: username: ***# Comment# Gitment# Introduction: https://imsun.net/posts/gitment-introduction/comment: type: youyan disqus: # enter disqus shortname here youyan: uid: 1783844 # enter youyan uid livere: uid: # enter youyan uid gitment: githubID: repo: ClientID: ClientSecret: lazy: false gitalk: # gitalk. https://gitalk.github.io/ owner: #必须. GitHub repository 所有者，可以是个人或者组织。 admin: #必须. GitHub repository 的所有者和合作者 (对这个 repository 有写权限的用户)。 repo: #必须. GitHub repository. ClientID: #必须. GitHub Application Client ID. ClientSecret: #必须. GitHub Application Client Secret. valine: # Valine. https://valine.js.org appid: # your leancloud application appid appkey: # your leancloud application appkey notify: false # mail notifier , https://github.com/xCss/Valine/wiki verify: false # Verification code placeholder: Just go go # comment box placeholder avatar: mm # gravatar style meta: nick,mail,link # custom comment header pageSize: 10 # pagination size visitor: false # Article reading statistic https://valine.js.org/visitor.html# douban 豆瓣书单# Api： # https://developers.douban.com/wiki/?title=book_v2 图书 # https://developers.douban.com/wiki/?title=movie_v2 电影# books： # https://api.douban.com/v2/book/user/:name/collections?start=0&count=100 个人书单列表# movies: # https://api.douban.com/v2/movie/in_theaters 正在上映的电影 # https://api.douban.com/v2/movie/coming_soon 即将上映的电影 # https://api.douban.com/v2/movie/subject/:id 单个电影信息 # https://api.douban.com/v2/movie/search?q={text} 电影搜索douban: user: # 豆瓣用户名 start: 0 # 从哪一条记录开始 count: 100 # 获取豆瓣书单数据条数 # PVpv: busuanzi: enable: false # 不蒜子统计 leancloud: enable: false # leancloud统计 app_id: # leancloud <AppID> app_key: # leancloud <AppKey> # wordcountpostCount: enable: false wordcount: true # 文章字数统计 min2read: true # 阅读时长预计 # Pluginsplugins: google_analytics: # enter the tracking ID for your Google Analytics google_site_verification: # enter Google site verification code baidu_analytics: # enter Baidu Analytics hash key tencent_analytics: # Miscellaneoustwitter:google_plus:fb_admins:fb_app_id: # profileprofile: enabled: true # Whether to show profile bar avatar: images/avatar.jpg gravatar: # Gravatar email address, if you enable Gravatar, your avatar config will be overriden author: 昵称 author_title: Web Developer & Designer author_description: 个人简介。 location: Shenzhen, China follow: https://github.com/cofess # Social Links social: links: github: https://github.com/cofess weibo: http://weibo.com/cofess twitter: https://twitter.com/iwebued # facebook: / # dribbble: / behance: https://www.behance.net/cofess rss: atom.xml link_tooltip: true # enable the social link tooltip, options: true, false # My Skills skills: Git: ★★★☆☆ Gulp: ★★★☆☆ Javascript: ★★★☆☆ HTML+CSS: ★★★☆☆ Bootstrap: ★★★☆☆ ThinkPHP: ★★★☆☆ 平面设计: ★★★☆☆ # My Personal Links links: Github: https://github.com/cofess Blog: http://blog.cofess.com 微博: http://weibo.com/cofess 花瓣: http://huaban.com/cofess Behance: https://www.behance.net/cofess # My Personal Labels labels: - 前端 - 前端开发 - 前端重构 - Web前端 - 网页重构 # My Personal Works works: name: link: http://www.example.com date: 2016 # My Personal Projects projects: cofess/gulp-startpro: https://github.com/cofess/gulp-startpro cofess/hexo-theme-pure: https://github.com/cofess/hexo-theme-pure基本上每个配置做什么用的，配置文件里面基本写了注解。也很容易理解。如果还不是很能理解配置项。可以查看https://github.com/cofess/hexo-theme-pure/blob/master/README.cn.md 文件。至此，hexo模版的大体结构已经清楚了。主题优化修改主题在config.yml 文件中修改1234 # Extensions## Plugins: https://hexo.io/plugins/## Themes: https://hexo.io/themes/theme: pure修改语言在config.yml 文件中修改12# Sitelanguage: zh-CN #修改成中文添加Rss订阅安装feed插件1npm install hexo-generator-feed --save在config.yml添加12345 # Extensions## Plugins: https://hexo.io/plugins/#RSS订阅plugin:- hexo-generator-feed设置feed插件参数12345 #Feed Atomfeed: type: atom path: atom.xml limit: 20生成预览12hexo ghexo d预览下就是如下添加站点地图站点地图是一种文件，您可以通过该文件列出您网站上的网页，从而将您网站内容的组织架构告知Google和其他搜索引擎。Googlebot等搜索引擎网页抓取工具会读取此文件，以便更加智能地抓取您的网站分别安装百度和google插件12npm install hexo-generator-sitemap --savenpm install hexo-generator-baidu-sitemap --save在博客目录的_config.yml中添加如下代码12345# 自动生成sitemapsitemap:path: sitemap.xmlbaidusitemap:path: baidusitemap.xml编译你的博客12hexo ghexo s如果你在你的博客根目录的public下面发现生成了sitemap.xml以及baidusitemap.xml就表示成功了,在本地访问 http://127.0.0.4000/sitemap.xml 和 http://127.0.0.4000/baidusitemap.xml 就能正确的展示出两个sitemap 文件了。注册百度站长平台4.1 访问：https://ziyuan.baidu.com/linksubmit/index4.2 提交链接提交链接方式有主动推送、自动推送、sitemap、手动上传等。4.3主动推送安装对应提交插件1npm install hexo-baidu-url-submit --save修改配置：123456789101112131415##配置插件plugin:- hexo-generator-baidu-sitemap- hexo-generator-sitemap- hexo-baidu-url-submitbaidu_url_submit: ## 比如3，代表提交最新的三个链接 count: 3 # 在百度站长平台中注册的域名 host: www.liuyong520.cn ## 请注意这是您的秘钥， 请不要发布在公众仓库里! token: upR0BjzCYxTC2CPq ## 文本文档的地址， 新链接会保存在此文本文档里 path: baidu_urls.txt编译博客12hexo ghexo d如果出现下图即表示成功了4.4 自动推送将如下代码添加到head.ejs中即可生效1234567891011121314<script> (function(){ var bp = document.createElement('script'); var curProtocol = window.location.protocol.split(':')[0]; if (curProtocol === 'https') { bp.src = 'https://zz.bdstatic.com/linksubmit/push.js'; } else { bp.src = 'http://push.zhanzhang.baidu.com/push.js'; } var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(bp, s); })(); </script>4.5 sitemap 提交方式打开百度站长平台，点击sitemap，填入我们的sitemap文件路径：<域名>/<sitemap名字>如下提交即可.但是此时你的域名其实并没有被百度站长所收录：百度依然检索不到你的网站，需要10多个工作日之后才能审核通过。绑定站点到熊掌ID，这样熊掌ID站点管理里面就能看到相关站点数据了登录站长平台，注册熊掌ID，提交审核过后点击站点收录：静态资源压缩hexo 的文章是通过md格式的文件经过swig转换成的html，生成的html会有很多空格，而且自己写的js以及css中会有很多的空格和注释。js和java不一样，注释也会影响一部分的性能，空格同样是的。静态资源压缩也有多种手段：有gulp插件和hexo自带的neat插件。1.hexo-neat 插件：安装hexo-neat插件1npm install hexo-neat --save修改站点配置文件_config.yml：12345678910111213141516171819202122 # hexo-neat# 博文压缩neat_enable: true# 压缩htmlneat_html: enable: true exclude:# 压缩css neat_css: enable: true exclude: - '**/*.min.css'# 压缩jsneat_js: enable: true mangle: true output: compress: exclude: - '**/*.min.js' - '**/jquery.fancybox.pack.js' - '**/index.js'编译博客12hexo g hexo dgulp插件方式安装gulp及相关插件123456npm install gulp -gnpm install gulp-minify-css --savenpm install gulp-uglify --savenpm install gulp-htmlmin --savenpm install gulp-htmlclean --savenpm install gulp-imagemin --save在 Hexo 站点下新建 gulpfile.js文件，文件内容如下：123456789101112131415161718192021222324252627282930313233343536373839404142434445var gulp = require('gulp');var minifycss = require('gulp-minify-css');var uglify = require('gulp-uglify');var htmlmin = require('gulp-htmlmin');var htmlclean = require('gulp-htmlclean');var imagemin = require('gulp-imagemin');// 压缩css文件gulp.task('minify-css', function() { return gulp.src('./public/**/*.css') .pipe(minifycss()) .pipe(gulp.dest('./public'));});// 压缩html文件gulp.task('minify-html', function() { return gulp.src('./public/**/*.html') .pipe(htmlclean()) .pipe(htmlmin({ removeComments: true, minifyJS: true, minifyCSS: true, minifyURLs: true, })) .pipe(gulp.dest('./public'))});// 压缩js文件gulp.task('minify-js', function() { return gulp.src(['./public/**/.js','!./public/js/**/*min.js']) .pipe(uglify()) .pipe(gulp.dest('./public'));});// 压缩 public/demo 目录内图片gulp.task('minify-images', function() { gulp.src('./public/demo/**/*.*') .pipe(imagemin({ optimizationLevel: 5, //类型：Number 默认：3 取值范围：0-7（优化等级） progressive: true, //类型：Boolean 默认：false 无损压缩jpg图片 interlaced: false, //类型：Boolean 默认：false 隔行扫描gif进行渲染 multipass: false, //类型：Boolean 默认：false 多次优化svg直到完全优化 })) .pipe(gulp.dest('./public/uploads'));});// 默认任务gulp.task('default', [ 'minify-html','minify-css','minify-js','minify-images']);只需要每次在执行 generate 命令后执行 gulp 就可以实现对静态资源的压缩，压缩完成后执行 deploy 命令同步到服务器：123hexo ggulphexo d修改访问URL路径默认情况下访问URL路径为：domain/2018/10/18/关于本站,修改为 domain/About/关于本站。 编辑 Hexo 站点下的 _config.yml 文件，修改其中的 permalink字段：1permalink: :category/:title/博文置顶安装插件12npm uninstall hexo-generator-index --savenpm install hexo-generator-index-pin-top --save然后在需要置顶的文章的Front-matter中加上top即可：12345--title: 2018date: 2018-10-25 16:10:03top: 10---设置置顶标志打开：/themes/*/layout/_macro/post.swig，定位到12345{% if post.top %} <i class="fa fa-thumb-tack"></i> <font color=7D26CD>置顶</font> <span class="post-meta-divider">|</span>{% endif %}","categories":[{"name":"hexo","slug":"hexo","permalink":"http://www.liuyong520.cn/categories/hexo/"}],"tags":[{"name":"hexo","slug":"hexo","permalink":"http://www.liuyong520.cn/tags/hexo/"}]},{"title":"利用hexo搭建博客","slug":"creatblog","date":"2017-08-27T02:56:51.000Z","updated":"2019-06-11T09:37:15.532Z","comments":true,"path":"2017/08/27/creatblog/","link":"","permalink":"http://www.liuyong520.cn/2017/08/27/creatblog/","excerpt":"","text":"如果你和我一样是小白，那么恭喜你！看完这篇文章，你也可以拥有一个这样的博客前面已经介绍过如何搭建hexo环境，现在我将介绍如何用hexo搭建自己的blog博客搭建实施方案方案一：GithubPages创建Github账号创建仓库 ，仓库名为：<Github账号名称>.github.io点击settings往下翻就能看到githubPages，我这里是已经配置过了的，没有配置可以是select themes ，点击能够选择SkyII主题。（SkyII主题也是和hexo类似的blog的框架，这里不与介绍）将本地Hexo博客推送到GithubPages3.1. 安装hexo-deployer-git插件。在命令行（即Git Bash）运行以下命令即可：1$ npm install hexo-deployer-git --save3.2. 添加SSH key。创建一个 SSH key 。在命令行（即Git Bash）输入以下命令， 回车三下即可：1$ ssh-keygen -t rsa -C "邮箱地址"添加到 github。 复制密钥文件内容（路径形如C:\\Users\\Administrator.ssh\\id_rsa.pub），粘贴到New SSH Key即可。测试是否添加成功。在命令行（即Git Bash）依次输入以下命令，返回“You’ve successfully authenticated”即成功：1ssh -T git@github.com3.3. 修改_config.yml（在站点目录下）。文件末尾修改为：123456# Deployment## Docs: https://hexo.io/docs/deployment.htmldeploy: type: git repo: git@github.com:<Github账号名称>/<Github账号名称>.github.io.git branch: master注意：上面仓库地址写ssh地址，不写http地址。3.4. 推送到GithubPages。在命令行（即Git Bash）依次输入以下命令， 返回INFO Deploy done: git即成功推送：12$ hexo g$ hexo d等待1分钟左右，浏览器访问网址： https://<Github账号名称>.github.io至此，您的Hexo博客已经搭建在GithubPages, 域名为https://<Github账号名称>.github.io。方案二：GithubPages + 域名在方案一的基础上，添加自定义域名（您购买的域名）。我的是从阿里云购买的。域名解析类型选择为 CNAME；主机记录即域名前缀，填写为www；记录值填写为<Github账号名称>.github.io；解析线路，TTL 默认即可点击 liuyong520.cn仓库设置。2.1. 打开博客仓库设置：https://github.com/<Github账号名称>/<Github账号名称>.github.io/settings2.2. 在Custom domain下，填写自定义域名，点击save。2.3. 在站点目录的source文件夹下，创建并打开CNAME.txt，写入你的域名（如www.liuyong520.cn），保存，并重命名为CNAME。如图等待10分钟左右。浏览器访问自定义域名。http://www.liuyong520.cn至此，您的Hexo博客已经解析到自定义域名，https://<Github账号名称>.github.io依然可用。方案三：GithubPages + CodingPages + 域名GithubPages 在国内较慢，百度不收录，而CodingPages 在国外较快。所以在方案二的基础上，添加CodingPages 。创建Coding账号创建仓库， 仓库名为：<Coding账号名称>进入项目里『代码』页面，点击『一键开启静态 Pages』，稍等片刻CodingPages即可部署成功。将本地Hexo博客推送到CodingPages4.1. 鉴于创建GithubPages 时，已经生成过公钥。可直接复制密钥文件内容（路径形如C:\\Users\\Administrator.ssh\\id_rsa.pub）， 粘贴到新增公钥。4.2. 测试是否添加成功。在命令行（即Git Bash）依次输入以下命令，返回“You’ve successfully authenticated”即成功：12$ ssh -T git@git.coding.net$ yes4.3. 修改_config.yml（在存放Hexo初始化文件的路径下）。文件末尾修改为：123456789# Deployment## Docs: https://hexo.io/docs/deployment.htmldeploy:- type: git repo: git@github.com:<Github账号名称>/<Github账号名称>.github.io.git branch: master- type: git repo: git@git.dev.tencent.com:<Coding账号名称>/<Coding账号名称>.git branch: master4.4. 推送到GithubPages。在命令行（即Git Bash）依次输入以下命令， 返回INFO Deploy done: git即成功推送：12$ hexo g$ hexo d域名解析添加 CNAME 记录指向 <Coding账号名称>.coding.me类型选择为 CNAME；主机记录即域名前缀，填写为www；记录值填写为<Github账号名称>.coding.me；解析线路，TTL 默认即可。添加 两条A 记录指向 192.30.252.153和192.30.252.154类型选择为 A；主机记录即域名前缀，填写为@；记录值填写为192.30.252.153和192.30.252.154；解析线路，境外或谷歌。在『Pages 服务』设置页（https://dev.tencent.com/u/<Coding账号名称>/p/<Coding账号名称>/git/pages/settings）中绑定自定义域名至此，您的Hexo博客已经解析到自定义域名，https://<Github账号名称>.github.io和https://<Coding账号名称>.coding.me依然可用。切换主题选择主题hexo主题是非常多的，默认的主题是landscape，您可以自主的在hexo官方网站上挑选自己喜欢的主题，网站：https://hexo.io/themes/推荐以下主题：snippetHieroJsimpleBlueLakePureNextHueman我这里选择的是Pure。1git clone https://github.com/cofess/hexo-theme-pure.git themes/pure此时会在themes 目录下生成 pure目录应用主题更改站点配置_config.yml 修改成1234# Extensions## Plugins: https://hexo.io/plugins/## Themes: https://hexo.io/themes/theme: <主题文件夹的名称>主题优化以上主题都有比较详细的说明文档，本节主要解决主题优化的常见问题。主题优化一般包括：设置「RSS」添加「标签」页面添加「分类」页面设置「字体」设置「代码高亮主题」侧边栏社交链接开启打赏功能设置友情链接腾讯公益404页面站点建立时间订阅微信公众号设置「动画效果」设置「背景动画」下一次我将针对Pure进行主题方面的相关配置，以及讲解一下hexo主题的的实现原理的。这样你们针对不同的主题也就都能配置了。","categories":[{"name":"hexo","slug":"hexo","permalink":"http://www.liuyong520.cn/categories/hexo/"}],"tags":[{"name":"hexo","slug":"hexo","permalink":"http://www.liuyong520.cn/tags/hexo/"}]},{"title":"Hexo之环境搭建","slug":"hexo-install","date":"2017-08-27T02:56:51.000Z","updated":"2019-06-11T09:37:16.619Z","comments":true,"path":"2017/08/27/hexo-install/","link":"","permalink":"http://www.liuyong520.cn/2017/08/27/hexo-install/","excerpt":"","text":"如果你和我一样是小白，那么恭喜你！看完这篇文章，你也可以拥有一个这样的博客啦！前言在以前我们要维护一个专属于自己的blog，是比较麻烦的，要购买服务器，部署博客程序到服务器，还要维护相关数据和网络。这一类blog最为典型的例子就是WordPress。而今天我们要介绍的是如何基于Hexo博客快速的搭建我们自己服务器系列。hexo介绍Hexo 是一个快速、简洁且高效的博客框架。Hexo 使用 Markdown（或其他渲染引擎）解析文章，在几秒内，即可利用靓丽的主题生成静态网页。hexo安装hexo 是基于node.js环境的，所以安装前，您必须检查电脑中是否已安装下列应用程序：node.js如果您的电脑中已经安装上述必备程序，那么恭喜您！接下来只需要使用 npm 即可完成 Hexo 的安装。1$ npm install -g hexo-cli如果您的电脑中未安装Node，那么就需要安装Node.js详细安装步骤参考：http://www.liuyong520.cn/2017/08/26/nodejs-install/再安装Hexo，在命令行（即Git Bash）运行以下命令：1npm install -g hexo-cli至此Hexo的环境就搭建好了，下一步验证一下hexo123456789101112131415161718MacBook-Pro:_posts xxydliuyss$ hexo versionhexo: 3.8.0hexo-cli: 1.1.0os: Darwin 18.5.0 darwin x64http_parser: 2.8.0node: 10.15.3v8: 6.8.275.32-node.51uv: 1.23.2zlib: 1.2.11ares: 1.15.0modules: 64nghttp2: 1.34.0napi: 3openssl: 1.1.0jicu: 62.1unicode: 11.0cldr: 33.1tz: 2018e这样hexo就安装完成了hexo命令介绍官网已经介绍的比较详细了这里就不再赘述了详情请看官方命令地址：https://hexo.io/zh-cn/docs/commandshexo快速新建博客初始化Hexo，在命令行（即Git Bash）依次运行以下命令即可：以下，即存放Hexo初始化文件的路径， 即站点目录。123$ hexo init myproject$ cd myproject$ npm install新建完成后，在路径下，会产生这些文件和文件夹：123456789$ tree.├── _config.yml├── package.json├── scaffolds├── source| ├── _drafts| └── _posts└── themes目录名或者文件名详情介绍_config.ymlhexo 全局配置文件package.jsonnodejs 包配置文件scaffoldshexo模版文件夹hexo new filename 会对应根据模版文件生成文件source项目源代码文件目录_drafts为草稿原文件目录_posts项目发布文件目录 项目最终会根据这个目录下的文件生成htmlthemes博客主题存放目录注：hexo相关命令均在站点目录下，用Git Bash运行。站点配置文件：站点目录下的_config.yml。​ 路径为_config.yml主题配置文件：站点目录下的themes文件夹下的，主题文件夹下的_config.yml。​ 路径为\\themes\\<主题文件夹>_config.yml启动服务器。在路径下，命令行（即Git Bash）输入以下命令，运行即可：1hexo server浏览器访问网址： http://localhost:4000/ 就可以预览博客了下一篇 我将介绍如何搭建自己的blog","categories":[{"name":"hexo","slug":"hexo","permalink":"http://www.liuyong520.cn/categories/hexo/"}],"tags":[{"name":"hexo","slug":"hexo","permalink":"http://www.liuyong520.cn/tags/hexo/"}]},{"title":"node.js环境搭建","slug":"nodejs-install","date":"2017-08-26T01:56:51.000Z","updated":"2019-06-11T09:37:15.387Z","comments":true,"path":"2017/08/26/nodejs-install/","link":"","permalink":"http://www.liuyong520.cn/2017/08/26/nodejs-install/","excerpt":"","text":"安装node.js登录官网下载对应的exe安装包。下载地址为：你可以根据不同平台系统选择你需要的Node.js安装包。Node.js 历史版本下载地址：https://nodejs.org/dist/注意：Linux上安装Node.js需要安装Python 2.6 或 2.7 ，不建议安装Python 3.0以上版本。windows 上安装 node.js你可以采用以下两种方式来安装。1、Windows 安装包(.msi)32 位安装包下载地址 : https://nodejs.org/dist/v4.4.3/node-v4.4.3-x86.msi64 位安装包下载地址 : https://nodejs.org/dist/v4.4.3/node-v4.4.3-x64.msi本文实例以 v0.10.26 版本为例，其他版本类似， 安装步骤：步骤 1 : 双击下载后的安装包 v0.10.26，如下所示：步骤 2: 点击以上的Run(运行)，将出现如下界面：步骤 3 : 勾选接受协议选项，点击 next（下一步） 按钮 :步骤 4 : Node.js默认安装目录为 “C:\\Program Files\\nodejs\\” , 你可以修改目录，并点击 next（下一步）：步骤 5 : 点击树形图标来选择你需要的安装模式 , 然后点击下一步 next（下一步）步骤 6 :点击 Install（安装） 开始安装Node.js。你也可以点击 Back（返回）来修改先前的配置。 然后并点击 next（下一步）：点击 Finish（完成）按钮退出安装向导。检测PATH环境变量是否配置了Node.js，点击开始=》运行=》输入”cmd” => 输入命令”path”，输出如下结果：12345PATH=C:\\oraclexe\\app\\oracle\\product\\10.2.0\\server\\bin;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;c:\\python32\\python;C:\\MinGW\\bin;C:\\Program Files\\GTK2-Runtime\\lib;C:\\Program Files\\MySQL\\MySQL Server 5.5\\bin;C:\\Program Files\\nodejs\\;C:\\Users\\rg\\AppData\\Roaming\\npm我们可以看到环境变量中已经包含了C:\\Program Files\\nodejs\\检查Node.js版本12E:\\> node --versionv0.10.262、Windows 二进制文件 (.exe)安装32 位安装包下载地址 : http://nodejs.org/dist/v0.10.26/node.exe64 位安装包下载地址 : http://nodejs.org/dist/v0.10.26/x64/node.exe安装步骤步骤 1 : 双击下载的安装包 Node.exe ，将出现如下界面 :点击 Run（运行）按钮将出现命令行窗口：版本测试进入 node.exe 所在的目录，如下所示如果你获得以上输出结果，说明你已经成功安装了Node.js。linux安装node.js直接使用已编译好的包Node 官网已经把 linux 下载版本更改为已编译好的版本了，我们可以直接下载解压后使用：12345# wget https://nodejs.org/dist/v10.9.0/node-v10.9.0-linux-x64.tar.xz // 下载# tar xf node-v10.9.0-linux-x64.tar.xz // 解压# cd node-v10.9.0-linux-x64/ // 进入解压目录# ./bin/node -v // 执行node命令 查看版本v10.9.0解压文件的 bin 目录底下包含了 node、npm 等命令，我们可以使用 ln 命令来设置软连接12ln -s /usr/software/nodejs/bin/npm /usr/local/bin/ ln -s /usr/software/nodejs/bin/node /usr/local/bin/Ubuntu 源码安装 Node.js以下部分我们将介绍在 Ubuntu Linux 下使用源码安装 Node.js 。 其他的 Linux 系统，如 Centos 等类似如下安装步骤。在 Github 上获取 Node.js 源码：12$ sudo git clone https://github.com/nodejs/node.gitCloning into 'node'...修改目录权限：1$ sudo chmod -R 755 node使用 ./configure 创建编译文件，并按照：1234$ cd node$ sudo ./configure$ sudo make$ sudo make install查看 node 版本：12$ node --versionv0.10.25Ubuntu apt-get命令安装命令格式如下：12sudo apt-get install nodejssudo apt-get install npmCentOS 下源码安装 Node.js1、下载源码，你需要在https://nodejs.org/en/download/ 下载最新的Nodejs版本，本文以v0.10.24为例:12cd /usr/local/src/wget http://nodejs.org/dist/v0.10.24/node-v0.10.24.tar.gz2、解压源码1tar zxvf node-v0.10.24.tar.gz3、 编译安装1234cd node-v0.10.24./configure --prefix=/usr/local/node/0.10.24makemake install4、 配置NODE_HOME，进入profile编辑环境变量1vim /etc/profile设置nodejs环境变量，在 export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL 一行的上面添加如下内容:123#set for nodejsexport NODE_HOME=/usr/local/node/0.10.24export PATH=$NODE_HOME/bin:$PATH:wq保存并退出，编译/etc/profile 使配置生效1source /etc/profile验证是否安装配置成功1node -v输出 v0.10.24 表示配置成功npm模块安装路径:1/usr/local/node/0.10.24/lib/node_modules/注：Nodejs 官网提供了编译好的Linux二进制包，你也可以下载下来直接应用。Mac OS 上安装你可以通过以下两种方式在 Mac OS 上来安装 node：1、在官方下载网站下载 pkg 安装包，直接点击安装即可。2、使用 brew 命令来安装：1brew install node","categories":[{"name":"node","slug":"node","permalink":"http://www.liuyong520.cn/categories/node/"}],"tags":[{"name":"node","slug":"node","permalink":"http://www.liuyong520.cn/tags/node/"}]},{"title":"跟我阅读spring源码之spring core","slug":"spring","date":"2017-05-20T13:03:44.000Z","updated":"2019-06-11T09:37:15.693Z","comments":true,"path":"2017/05/20/spring/","link":"","permalink":"http://www.liuyong520.cn/2017/05/20/spring/","excerpt":"","text":"这么多的源代码，这么多包，往往不知道从何处开始下手阅读的这个源代码。我们在接触spring的时候，首先介绍的都是按照IOC、MVC、AOP、这种顺序介绍的。ClassPathXmlApplicationContext类的继承图谱废话不多说，就从这个启动类开始看吧。先看一下这个类的继承图谱Classpath应用上下文最顶层接口Beanfactory。Beanfactory就是springIOC的容器。构造方法123456789101112public ClassPathXmlApplicationContext(String[] configLocations, boolean refresh, ApplicationContext parent) throws BeansException { //最终会调用的是AbstractApplicationContext的构造方法 super(parent); //根据配置的路径生成classpath的加载路径 setConfigLocations(configLocations); if (refresh) { //刷新容器完成容器的初始化工作 refresh(); } }看一下AbstractApplicationContext 构造方法作的是什么事情123456789101112131415161718192021222324252627282930313233public AbstractApplicationContext(ApplicationContext parent) { this(); //设置父容器 setParent(parent);}public AbstractApplicationContext() { //获取配置资源的解析器 this.resourcePatternResolver = getResourcePatternResolver();}protected ResourcePatternResolver getResourcePatternResolver() { //直接new一个PathMatchingResourcePatternResolver解析器 等一下再看这个PathMatchingResourcePatternResolver return new PathMatchingResourcePatternResolver(this);}@Overridepublic void setParent(ApplicationContext parent) { this.parent = parent; if (parent != null) { //如果父容器不为空且是ConfigurableEnvironment就把环境合并在一起 Environment parentEnvironment = parent.getEnvironment(); if (parentEnvironment instanceof ConfigurableEnvironment) { getEnvironment().merge((ConfigurableEnvironment) parentEnvironment); } }}//getEnvironment方法来自于ConfigurableApplicationContext接口，源码很简单，如果为空就调用createEnvironment创建一个。AbstractApplicationContext.createEnvironment:public ConfigurableEnvironment getEnvironment() { if (this.environment == null) { this.environment = createEnvironment(); } return this.environment;}setConfigLocations123456789101112131415161718//此方法的目的在于将占位符(placeholder)解析成实际的地址。比如可以这么写: new ClassPathXmlApplicationContext(\"classpath:config.xml\");那么classpath:就是需要被解析的public void setConfigLocations(String... locations) { if (locations != null) { Assert.noNullElements(locations, \"Config locations must not be null\"); this.configLocations = new String[locations.length]; for (int i = 0; i < locations.length; i++) { //解析成classpath的路径 this.configLocations[i] = resolvePath(locations[i]).trim(); } } else { this.configLocations = null; }}protected String resolvePath(String path) { return getEnvironment().resolveRequiredPlaceholders(path);}refresh重点介绍这个refresh方法：12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061@Overridepublic void refresh() throws BeansException, IllegalStateException { synchronized (this.startupShutdownMonitor) { //准备刷新容器这里干的是：初始化资源，初始化spring事件容器，验证一下系统环境配置是否正确 这个 prepareRefresh(); //刷新内部bean工厂，同时拿到内部工厂beanfactory。 ConfigurableListableBeanFactory beanFactory = obtainFreshBeanFactory(); // prepareBeanFactory(beanFactory); try { // Allows post-processing of the bean factory in context subclasses. postProcessBeanFactory(beanFactory); // Invoke factory processors registered as beans in the context. invokeBeanFactoryPostProcessors(beanFactory); // Register bean processors that intercept bean creation. registerBeanPostProcessors(beanFactory); // Initialize message source for this context. initMessageSource(); // Initialize event multicaster for this context. initApplicationEventMulticaster(); // Initialize other special beans in specific context subclasses. onRefresh(); // Check for listener beans and register them. registerListeners(); // Instantiate all remaining (non-lazy-init) singletons. finishBeanFactoryInitialization(beanFactory); // Last step: publish corresponding event. finishRefresh(); } catch (BeansException ex) { if (logger.isWarnEnabled()) { logger.warn(\"Exception encountered during context initialization - \" + \"cancelling refresh attempt: \" + ex); } // Destroy already created singletons to avoid dangling resources. destroyBeans(); // Reset 'active' flag. cancelRefresh(ex); // Propagate exception to caller. throw ex; } finally { // Reset common introspection caches in Spring's core, since we // might not ever need metadata for singleton beans anymore... resetCommonCaches(); } }prepareRefresh看看prepareRefresh 这个方法：12345678910111213141516171819202122protected void prepareRefresh() { this.startupDate = System.currentTimeMillis(); this.closed.set(false); this.active.set(true); if (logger.isInfoEnabled()) { logger.info(\"Refreshing \" + this); } // 初始化properties配置信息，这个方法其实是个空方法，让子类去复写的。子类可以继承这个类，实现这个方法自行去加载properties配置 initPropertySources(); //验证环境配置的properties是否是require的 //如果是key=value value 为空的话， //就会存到一个MissingRequiredPropertiesException（这是一个异常的集合） //类里，同时抛出MissingRequiredPropertiesException getEnvironment().validateRequiredProperties(); //初始化spring事件的容器 this.earlyApplicationEvents = new LinkedHashSet<ApplicationEvent>();}obtainFreshBeanFactory接着继续看：ConfigurableListableBeanFactory beanFactory = obtainFreshBeanFactory();123456789101112protected ConfigurableListableBeanFactory obtainFreshBeanFactory() { //刷新内部工厂 为嘛要刷新？这里其实做了三部操作 //1。关闭原来创建的的容器，同时释放bean对象资源 //2.重新加载beans配置文件，存到DefaultListableBeanFactory 容器里 refreshBeanFactory(); //获取beanFactory的实例。这里是调用的AbstractXmlApplicationContext里面的 getBeanFactory获取到DefaultListableBeanFactory的实例。 ConfigurableListableBeanFactory beanFactory = getBeanFactory(); if (logger.isDebugEnabled()) { logger.debug(\"Bean factory for \" + getDisplayName() + \": \" + beanFactory); } return beanFactory;}继续看refreshBeanFactory：这个方法是它的父类AbstractRefreshableApplicationContext实现的。12345678910111213141516171819202122@Overrideprotected final void refreshBeanFactory() throws BeansException { //如果已经有beanfactory了就把所以的bean给销毁掉，同时关闭beanfactory if (hasBeanFactory()) { destroyBeans(); closeBeanFactory(); } try { //重新创建一个beanfactory。待会我们再分析这个bean factory DefaultListableBeanFactory beanFactory = createBeanFactory(); beanFactory.setSerializationId(getId()); customizeBeanFactory(beanFactory); //加载所有的工厂实例 这个方法在classpathApplicationContext中是由AbstractXmlApplicationContext实现的。加载配置文件将所有的bean以beanDefinition的描述存在DefaultListableBeanFactory这个IOC容器里 loadBeanDefinitions(beanFactory); synchronized (this.beanFactoryMonitor) { this.beanFactory = beanFactory; } } catch (IOException ex) { throw new ApplicationContextException(\"I/O error parsing bean definition source for \" + getDisplayName(), ex); }}prepareBeanFactory接着往下看prepareBeanFactory 方法很长。12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364protected void prepareBeanFactory(ConfigurableListableBeanFactory beanFactory) { // 设置beanFactory的ClassLoader为当前的ClassLoader beanFactory.setBeanClassLoader(getClassLoader()); // 设置表达式解析器(解析bean定义中的一些表达式)这里是spel表达式解析器 beanFactory.setBeanExpressionResolver(new StandardBeanExpressionResolver(beanFactory.getBeanClassLoader())); // 添加属性编辑注册器(注册属性编辑器)，属性编辑器实际上是属性的类型转换器，编辑器注册器里面其实是map结构 // 因为bean的属性配置都是字符串类型的 实例化的时候要将这些属性转换为实际类型 beanFactory.addPropertyEditorRegistrar(new ResourceEditorRegistrar(this, getEnvironment())); //// 添加BeanPostProcessor(Bean后置处理器)：ApplicationContextAwareProcessor // 在BEAN初始化之前，调用ApplicationContextAwareProcessor的postProcessBeforeInitialization // postProcessBeforeInitialization有如下功能 // 处理所有的Aware接口，进行如下操作： // 如果bean实现了EnvironmentAware接口，调用bean.setEnvironment // 如果bean实现了EmbeddedValueResolverAware接口，调用bean.setEmbeddedValueResolver // 如果bean实现了ResourceLoaderAware接口，调用bean.setResourceLoader // 如果bean实现了ApplicationEventPublisherAware接口，调用bean.setApplicationEventPublisher // 如果bean实现了MessageSourceAware接口，调用bean.setMessageSource // 如果bean实现了ApplicationContextAware接口，调用bean.setApplicationContext beanFactory.addBeanPostProcessor(new ApplicationContextAwareProcessor(this)); // 取消ResourceLoaderAware // 、ApplicationEventPublisherAware // 、MessageSourceAware // 、ApplicationContextAware // 、EnvironmentAware这5个接口的自动注入 // 因为ApplicationContextAwareProcessor把这5个接口的实现工作做了 beanFactory.ignoreDependencyInterface(ResourceLoaderAware.class); beanFactory.ignoreDependencyInterface(ApplicationEventPublisherAware.class); beanFactory.ignoreDependencyInterface(MessageSourceAware.class); beanFactory.ignoreDependencyInterface(ApplicationContextAware.class); beanFactory.ignoreDependencyInterface(EnvironmentAware.class); // 注入一些特殊的bean，不需要在bean文件里面定义。 beanFactory.registerResolvableDependency(BeanFactory.class, beanFactory); beanFactory.registerResolvableDependency(ResourceLoader.class, this); beanFactory.registerResolvableDependency(ApplicationEventPublisher.class, this); beanFactory.registerResolvableDependency(ApplicationContext.class, this); // 检查容器中是否包含名称为loadTimeWeaver的bean，实际上是增加Aspectj的支持 // AspectJ采用编译期织入、类加载期织入两种方式进行切面的织入 // 类加载期织入简称为LTW（Load Time Weaving）,通过特殊的类加载器来代理JVM默认的类加载器实现 if (beanFactory.containsBean(LOAD_TIME_WEAVER_BEAN_NAME)) { // 添加BEAN后置处理器：LoadTimeWeaverAwareProcessor // 在BEAN初始化之前检查BEAN是否实现了LoadTimeWeaverAware接口， // 如果是，则进行加载时织入，即静态代理。 beanFactory.addBeanPostProcessor(new LoadTimeWeaverAwareProcessor(beanFactory)); //设置特殊的类加载器 beanFactory.setTempClassLoader(new ContextTypeMatchClassLoader(beanFactory.getBeanClassLoader())); } // 注册环境的environment bean if (!beanFactory.containsLocalBean(ENVIRONMENT_BEAN_NAME)) { beanFactory.registerSingleton(ENVIRONMENT_BEAN_NAME, getEnvironment()); } //注册systemProperties的bean 其实就是map if (!beanFactory.containsLocalBean(SYSTEM_PROPERTIES_BEAN_NAME)) { beanFactory.registerSingleton(SYSTEM_PROPERTIES_BEAN_NAME, getEnvironment().getSystemProperties()); } 注册系统环境bean，其实就是map if (!beanFactory.containsLocalBean(SYSTEM_ENVIRONMENT_BEAN_NAME)) { beanFactory.registerSingleton(SYSTEM_ENVIRONMENT_BEAN_NAME, getEnvironment().getSystemEnvironment()); }}看看ApplicationContextAwareProcessor 的postProcessBeforeInitialization这个方法,看完这个方法就知道上面为嘛写这么多东西了123456789101112131415161718192021222324252627282930public Object postProcessBeforeInitialization(final Object bean, String beanName) throws BeansException { AccessControlContext acc = null; //如果bean 实现了EmbeddedValueResolverAware、ResourceLoaderAware、 //ApplicationEventPublisherAware、ApplicationContextAware接口。 if (System.getSecurityManager() != null && (bean instanceof EnvironmentAware || bean instanceof EmbeddedValueResolverAware || bean instanceof ResourceLoaderAware || bean instanceof ApplicationEventPublisherAware || bean instanceof MessageSourceAware || bean instanceof ApplicationContextAware)) { //获取权限控制上下文 acc = this.applicationContext.getBeanFactory().getAccessControlContext(); } //权限控制上下文非空 if (acc != null) { //用权限控制器去调invokeAwareInterfaces方法 AccessController.doPrivileged(new PrivilegedAction<Object>() { @Override public Object run() { invokeAwareInterfaces(bean); return null; } }, acc); } else { //否则就直接调用了 invokeAwareInterfaces(bean); } return bean;}上面的方法始终都会调用invokeAwareInterfaces这个方法。1234567891011121314151617181920212223242526272829private void invokeAwareInterfaces(Object bean) { if (bean instanceof Aware) { if (bean instanceof EnvironmentAware) { //setEnvironment ((EnvironmentAware) bean).setEnvironment(this.applicationContext.getEnvironment()); } if (bean instanceof EmbeddedValueResolverAware) { //调用setEmbeddedValueResolver ((EmbeddedValueResolverAware) bean).setEmbeddedValueResolver( new EmbeddedValueResolver(this.applicationContext.getBeanFactory())); } if (bean instanceof ResourceLoaderAware) { //调用setResourceLoader ((ResourceLoaderAware) bean).setResourceLoader(this.applicationContext); } if (bean instanceof ApplicationEventPublisherAware) { //调用setApplicationEventPublisher ((ApplicationEventPublisherAware) bean).setApplicationEventPublisher(this.applicationContext); } if (bean instanceof MessageSourceAware) { //调用setMessageSource ((MessageSourceAware) bean).setMessageSource(this.applicationContext); } if (bean instanceof ApplicationContextAware) { //调用setApplicationContext ((ApplicationContextAware) bean).setApplicationContext(this.applicationContext); } }}同理LoadTimeWeaverAwareProcessor里面实现也可以从postProcessBeforeInitialization的方法。这里就不介绍了。postProcessBeanFactory继续介绍refresh方法里的方法postProcessBeanFactory(beanFactory);进去一看，一个未实现的空方法。干嘛用的？这个spring的提供的扩展，如果我们需要在容器所有bean定义被加载未实例化之前，我们可以注册一些BeanPostProcessors来实现在一些bean实例化之后做一些操作。12protected void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) { }继续往下走：invokeBeanFactoryPostProcessors123456protected void invokeBeanFactoryPostProcessors(ConfigurableListableBeanFactory beanFactory) { //这是一个比较复杂的方法了比较长。后面再看这个方法。 //调用所有的BeanFactoryPostProcessors //getBeanFactoryPostProcessors()这里获取的是一个list的集合。 PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(beanFactory, getBeanFactoryPostProcessors());}PostProcessorRegistrationDelegate 包含了beanPostProcessors的注册，和BeanFactoryPostProcessors的调用123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136public static void invokeBeanFactoryPostProcessors( ConfigurableListableBeanFactory beanFactory, List<BeanFactoryPostProcessor> beanFactoryPostProcessors) { // Invoke BeanDefinitionRegistryPostProcessors first, if any. Set<String> processedBeans = new HashSet<String>(); // 如果bean if (beanFactory instanceof BeanDefinitionRegistry) { BeanDefinitionRegistry registry = (BeanDefinitionRegistry) beanFactory; List<BeanFactoryPostProcessor> regularPostProcessors = new LinkedList<BeanFactoryPostProcessor>(); List<BeanDefinitionRegistryPostProcessor> registryPostProcessors = new LinkedList<BeanDefinitionRegistryPostProcessor>(); for (BeanFactoryPostProcessor postProcessor : beanFactoryPostProcessors) { //如果是BeanDefinitionRegistryPostProcessor的后置处理器就调用postProcessBeanDefinitionRegistry方法。然后加入registryPostProcessors集合 if (postProcessor instanceof BeanDefinitionRegistryPostProcessor) { BeanDefinitionRegistryPostProcessor registryPostProcessor = (BeanDefinitionRegistryPostProcessor) postProcessor; registryPostProcessor.postProcessBeanDefinitionRegistry(registry); registryPostProcessors.add(registryPostProcessor); } else { //否则就加入到寻常的后置处理器集合 regularPostProcessors.add(postProcessor); } } // Do not initialize FactoryBeans here: We need to leave all regular beans // uninitialized to let the bean factory post-processors apply to them! // Separate between BeanDefinitionRegistryPostProcessors that implement // PriorityOrdered, Ordered, and the rest. //从容器中获取所有的BeanDefinitionRegistryPostProcessor后置处理器 String[] postProcessorNames = beanFactory.getBeanNamesForType(BeanDefinitionRegistryPostProcessor.class, true, false); // First, invoke the BeanDefinitionRegistryPostProcessors that implement PriorityOrdered. // 获取@PriorityOrdered标记的BeanDefinitionRegistryPostProcessors List<BeanDefinitionRegistryPostProcessor> priorityOrderedPostProcessors = new ArrayList<BeanDefinitionRegistryPostProcessor>(); for (String ppName : postProcessorNames) { if (beanFactory.isTypeMatch(ppName, PriorityOrdered.class)) { priorityOrderedPostProcessors.add(beanFactory.getBean(ppName, BeanDefinitionRegistryPostProcessor.class)); processedBeans.add(ppName); } } //排序 sortPostProcessors(beanFactory, priorityOrderedPostProcessors); registryPostProcessors.addAll(priorityOrderedPostProcessors); //按照顺序调用BeanDefinitionRegistryPostProcessor invokeBeanDefinitionRegistryPostProcessors(priorityOrderedPostProcessors, registry); //获取@Order标记的BeanDefinitionRegistryPostProcessor postProcessorNames = beanFactory.getBeanNamesForType(BeanDefinitionRegistryPostProcessor.class, true, false); List<BeanDefinitionRegistryPostProcessor> orderedPostProcessors = new ArrayList<BeanDefinitionRegistryPostProcessor>(); for (String ppName : postProcessorNames) { //去除已经@PriorityOrdered标记的类，防止两个注解，同时找到，调用多次 if (!processedBeans.contains(ppName) && beanFactory.isTypeMatch(ppName, Ordered.class)) { orderedPostProcessors.add(beanFactory.getBean(ppName, BeanDefinitionRegistryPostProcessor.class)); processedBeans.add(ppName); } } sortPostProcessors(beanFactory, orderedPostProcessors); registryPostProcessors.addAll(orderedPostProcessors); //按照顺序调用BeanDefinitionRegistryPostProcessor invokeBeanDefinitionRegistryPostProcessors(orderedPostProcessors, registry); // boolean reiterate = true; while (reiterate) { reiterate = false; postProcessorNames = beanFactory.getBeanNamesForType(BeanDefinitionRegistryPostProcessor.class, true, false); for (String ppName : postProcessorNames) { if (!processedBeans.contains(ppName)) { BeanDefinitionRegistryPostProcessor pp = beanFactory.getBean(ppName, BeanDefinitionRegistryPostProcessor.class); registryPostProcessors.add(pp); processedBeans.add(ppName); pp.postProcessBeanDefinitionRegistry(registry); reiterate = true; } } } // 调用BeanDefinitionRegistryPostProcessor类的回调方法postProcessBeanFactory() invokeBeanFactoryPostProcessors(registryPostProcessors, beanFactory); // 寻常bean的回调方法postProcessBeanFactory invokeBeanFactoryPostProcessors(regularPostProcessors, beanFactory); } else { // 调用回调方法postProcessBeanFactory() invokeBeanFactoryPostProcessors(beanFactoryPostProcessors, beanFactory); } String[] postProcessorNames = beanFactory.getBeanNamesForType(BeanFactoryPostProcessor.class, true, false); // Separate between BeanFactoryPostProcessors that implement PriorityOrdered, // Ordered, and the rest. List<BeanFactoryPostProcessor> priorityOrderedPostProcessors = new ArrayList<BeanFactoryPostProcessor>(); List<String> orderedPostProcessorNames = new ArrayList<String>(); List<String> nonOrderedPostProcessorNames = new ArrayList<String>(); for (String ppName : postProcessorNames) { if (processedBeans.contains(ppName)) { // skip - already processed in first phase above } else if (beanFactory.isTypeMatch(ppName, PriorityOrdered.class)) { priorityOrderedPostProcessors.add(beanFactory.getBean(ppName, BeanFactoryPostProcessor.class)); } else if (beanFactory.isTypeMatch(ppName, Ordered.class)) { orderedPostProcessorNames.add(ppName); } else { nonOrderedPostProcessorNames.add(ppName); } } sortPostProcessors(beanFactory, priorityOrderedPostProcessors); invokeBeanFactoryPostProcessors(priorityOrderedPostProcessors, beanFactory); List<BeanFactoryPostProcessor> orderedPostProcessors = new ArrayList<BeanFactoryPostProcessor>(); for (String postProcessorName : orderedPostProcessorNames) { orderedPostProcessors.add(beanFactory.getBean(postProcessorName, BeanFactoryPostProcessor.class)); } sortPostProcessors(beanFactory, orderedPostProcessors); invokeBeanFactoryPostProcessors(orderedPostProcessors, beanFactory); List<BeanFactoryPostProcessor> nonOrderedPostProcessors = new ArrayList<BeanFactoryPostProcessor>(); for (String postProcessorName : nonOrderedPostProcessorNames) { nonOrderedPostProcessors.add(beanFactory.getBean(postProcessorName, BeanFactoryPostProcessor.class)); } invokeBeanFactoryPostProcessors(nonOrderedPostProcessors, beanFactory); // 清理元数据缓存 beanFactory.clearMetadataCache();}registerBeanPostProcessors继续往下走：registerBeanPostProcessors1234protected void registerBeanPostProcessors(ConfigurableListableBeanFactory beanFactory) { //注册BeanPostProcessors后面统一看这个PostProcessorRegistrationDelegate PostProcessorRegistrationDelegate.registerBeanPostProcessors(beanFactory, this);}initMessageSource继续往下看：initMessageSource 用以支持Spring国际化。12345678910111213141516171819202122232425262728293031323334protected void initMessageSource() { //拿到当前的beanFactory ConfigurableListableBeanFactory beanFactory = getBeanFactory(); //如果已经存在MessageSource了 if (beanFactory.containsLocalBean(MESSAGE_SOURCE_BEAN_NAME)) { this.messageSource = beanFactory.getBean(MESSAGE_SOURCE_BEAN_NAME, MessageSource.class); // Make MessageSource aware of parent MessageSource. if (this.parent != null && this.messageSource instanceof HierarchicalMessageSource) { //HierarchicalMessageSource采用的职责链的设计模式。 //如果消息当前对象处理不了，就将消息给上级父对象处理，把消息分层次处理。 HierarchicalMessageSource hms = (HierarchicalMessageSource) this.messageSource; if (hms.getParentMessageSource() == null) { //如果父消息源不为空，就设置父消息源， hms.setParentMessageSource(getInternalParentMessageSource()); } } if (logger.isDebugEnabled()) { logger.debug(\"Using MessageSource [\" + this.messageSource + \"]\"); } } else { // Use empty MessageSource to be able to accept getMessage calls. // 包装一个空的消息源可以用getMessage方法调用。 DelegatingMessageSource dms = new DelegatingMessageSource(); // 设置父消息源 dms.setParentMessageSource(getInternalParentMessageSource()); this.messageSource = dms; beanFactory.registerSingleton(MESSAGE_SOURCE_BEAN_NAME, this.messageSource); if (logger.isDebugEnabled()) { logger.debug(\"Unable to locate MessageSource with name '\" + MESSAGE_SOURCE_BEAN_NAME + \"': using default [\" + this.messageSource + \"]\"); } }}initApplicationEventMulticaster继续：initApplicationEventMulticaster 初始化事件广播器。可以通过multicastEvent方法广播消息1234567891011121314151617181920protected void initApplicationEventMulticaster() { ConfigurableListableBeanFactory beanFactory = getBeanFactory(); //如果容器里面有就直接拿出来用，如果没有就初始化一个。 if (beanFactory.containsLocalBean(APPLICATION_EVENT_MULTICASTER_BEAN_NAME)) { this.applicationEventMulticaster = beanFactory.getBean(APPLICATION_EVENT_MULTICASTER_BEAN_NAME, ApplicationEventMulticaster.class); if (logger.isDebugEnabled()) { logger.debug(\"Using ApplicationEventMulticaster [\" + this.applicationEventMulticaster + \"]\"); } } else { this.applicationEventMulticaster = new SimpleApplicationEventMulticaster(beanFactory); beanFactory.registerSingleton(APPLICATION_EVENT_MULTICASTER_BEAN_NAME, this.applicationEventMulticaster); if (logger.isDebugEnabled()) { logger.debug(\"Unable to locate ApplicationEventMulticaster with name '\" + APPLICATION_EVENT_MULTICASTER_BEAN_NAME + \"': using default [\" + this.applicationEventMulticaster + \"]\"); } }}onRefresh继续：onRefresh方法也是一个模版方法，空方法，目的也是为了给子类继承用的。AbstractRefreshableWebApplicationContext、StaticWebApplicationContext用这个方法来刷新初始化主题源。继续：registerListeners 注册监听器123456789101112131415161718192021protected void registerListeners() { //把监听者加入到事件广播器 for (ApplicationListener<?> listener : getApplicationListeners()) { getApplicationEventMulticaster().addApplicationListener(listener); } //获取所有的ApplicationListener的bean的名字，然后把bean名字加入到事件广播器 String[] listenerBeanNames = getBeanNamesForType(ApplicationListener.class, true, false); for (String listenerBeanName : listenerBeanNames) { getApplicationEventMulticaster().addApplicationListenerBean(listenerBeanName); } //拿到所有的earlyApplicationEvents事件消息，直接广播发送事件给所有的监听者。 Set<ApplicationEvent> earlyEventsToProcess = this.earlyApplicationEvents; this.earlyApplicationEvents = null; if (earlyEventsToProcess != null) { for (ApplicationEvent earlyEvent : earlyEventsToProcess) { getApplicationEventMulticaster().multicastEvent(earlyEvent); } }}finishBeanFactoryInitialization继续：finishBeanFactoryInitialization初始化非延迟加载的单例Bean， 实例化BeanFactory中已经被注册但是未实例化的所有实例(@Lazy注解的Bean不在此实例化)。1234567891011121314151617181920212223protected void finishBeanFactoryInitialization(ConfigurableListableBeanFactory beanFactory) { // 初始化类型转换器 if (beanFactory.containsBean(CONVERSION_SERVICE_BEAN_NAME) && beanFactory.isTypeMatch(CONVERSION_SERVICE_BEAN_NAME, ConversionService.class)) { beanFactory.setConversionService( beanFactory.getBean(CONVERSION_SERVICE_BEAN_NAME, ConversionService.class)); } //获取LoadTimeWeaverAware.class的单例bean String[] weaverAwareNames = beanFactory.getBeanNamesForType(LoadTimeWeaverAware.class, false, false); for (String weaverAwareName : weaverAwareNames) { getBean(weaverAwareName); } // 停止使用零时加载器 beanFactory.setTempClassLoader(null); // 允许缓存所有的bean的定义，不允许修改 beanFactory.freezeConfiguration(); // 初始化所有的单例bean，@lazy bean不在这里初始化 beanFactory.preInstantiateSingletons();}finishRefresh继续：finishRefreshrefresh结束之前需要做善后工作。包括生命周期组件LifecycleProcessor的初始化和调用、事件发布、JMX组件的处理等。12345678910111213protected void finishRefresh() { // 初始化生命周期组件LifecycleProcessor initLifecycleProcessor(); // 调用一次生命周期组件LifecycleProcessor getLifecycleProcessor().onRefresh(); // 发布容器刷新事件 publishEvent(new ContextRefreshedEvent(this)); // 向MBeanServer注册LiveBeansView，可以通过JMX来监控此ApplicationContext。 LiveBeansView.registerApplicationContext(this);}这个类refresh方法干的活也是有很多，其中就包括BeanFactory的设置、Configuration类解析、Bean实例化、属性和依赖注入、事件监听器注册。下面会继续去分析一下每一步是怎样实现的。Environment 接口继承图谱往上继承了PropertyResolver 属性解析器，Environment接口里面有三个独立的方法如下：123String[] getDefaultProfiles();String[] getActiveProfiles();boolean acceptsProfiles(String... profiles);都和Profile有关系。Spring Profile特性是从3.1开始的，其主要是为了解决这样一种问题: 线上环境和测试环境使用不同的配置或是数据库或是其它。有了Profile便可以在 不同环境之间无缝切换。Spring容器管理的所有bean都是和一个profile绑定在一起的。使用了Profile的配置文件示例:123456789<beans profile=\"develop\"> <context:property-placeholder location=\"classpath*:jdbc-develop.properties\"/> </beans> <beans profile=\"production\"> <context:property-placeholder location=\"classpath*:jdbc-production.properties\"/> </beans> <beans profile=\"test\"> <context:property-placeholder location=\"classpath*:jdbc-test.properties\"/> </beans>可以通过context.getEnvironment().setActiveProfiles(“dev”);或者spring.profiles.active=dev 进行设置。spring 中Environment 默认就是 StandardEnvironment实例。123456789101112131415161718public class StandardEnvironment extends AbstractEnvironment { /** System environment property source name: {@value} */ //系统级环境参数可以通过{@systemEnvironment[xxx]},或者{#systemEnvironment[xxx]}获取 public static final String SYSTEM_ENVIRONMENT_PROPERTY_SOURCE_NAME = \"systemEnvironment\"; /** JVM system properties property source name: {@value} */ //jvm层面级参数可以通过{@systemProperties[xxx]},或者{#systemProperties[xxx]}获取 public static final String SYSTEM_PROPERTIES_PROPERTY_SOURCE_NAME = \"systemProperties\"; //在实例化的时候，会调用父类里面的构造方法，而父类的构造方法里会调用此方法。 protected void customizePropertySources(MutablePropertySources propertySources) { //MapPropertySource其实就是MaP对象的封装 propertySources.addLast(new MapPropertySource(SYSTEM_PROPERTIES_PROPERTY_SOURCE_NAME, getSystemProperties())); //SystemEnvironmentPropertySource继承的MapPropertySource，其实里面也是map对象 propertySources.addLast(new SystemEnvironmentPropertySource(SYSTEM_ENVIRONMENT_PROPERTY_SOURCE_NAME, getSystemEnvironment())); }}MutablePropertySources是PropertySources的实现类。里面封装了一个Log对象，和用一个CopyOnWriteArrayList实现的一个PropertySource的一个集合，里面有一个PropertySourcesPropertyResolver解析器，这个解析器在PropertyResolver章节分析。在StandardEnvironment实例化的时调用AbstractEnvironment构造方法。123456789101112public AbstractEnvironment() { //此时这里就会被子类的customizePropertySources给复写掉。会调用子类的方法。 //此时的this.propertySources=new MutablePropertySources(this.logger); //此时MutablePropertySources对象只有龙对象，PropertySource集合是空的 //通过子类的propertySources.addLast往里面加入PropertySource对象。 customizePropertySources(this.propertySources); if (this.logger.isDebugEnabled()) { this.logger.debug(format( \"Initialized %s with PropertySources %s\", getClass().getSimpleName(), this.propertySources)); }}再看看StandardEnvironment#getSystemProperties函数:这个函数就是调用System.getProperties获取所有的系统配置，如果系统管理说没有权限获取，就一条一条的获取，这个地方我不甚理解。why？1234567891011121314151617181920212223public Map<String, Object> getSystemProperties() { try { return (Map) System.getProperties(); } catch (AccessControlException ex) { return (Map) new ReadOnlySystemAttributesMap() { @Override protected String getSystemAttribute(String attributeName) { try { return System.getProperty(attributeName); } catch (AccessControlException ex) { if (logger.isInfoEnabled()) { logger.info(format("Caught AccessControlException when accessing system " + "property [%s]; its value will be returned [null]. Reason: %s", attributeName, ex.getMessage())); } return null; } } }; }}同样的getSystemEnvironment函数：是调用的System.getenv获取jvm级系统参数，包活jdk版本，os参数等。1234567891011121314151617181920212223242526272829public Map<String, Object> getSystemEnvironment() { //这一句会从spring.properties文件里找spring.getenv.ignore标识 //如果spring.getenv.ignore=true就返回空， //如果不为空就调用System.getenv获取jvm系统级参数。 if (suppressGetenvAccess()) { return Collections.emptyMap(); } try { return (Map) System.getenv(); } catch (AccessControlException ex) { return (Map) new ReadOnlySystemAttributesMap() { @Override protected String getSystemAttribute(String attributeName) { try { return System.getenv(attributeName); } catch (AccessControlException ex) { if (logger.isInfoEnabled()) { logger.info(format(\"Caught AccessControlException when accessing system \" + \"environment variable [%s]; its value will be returned [null]. Reason: %s\", attributeName, ex.getMessage())); } return null; } } }; }}再看看Environment接口里的三个私有方法的实现：123456789101112131415161718192021222324@Overridepublic String[] getActiveProfiles() { return StringUtils.toStringArray(doGetActiveProfiles());}@Overridepublic String[] getDefaultProfiles() { return StringUtils.toStringArray(doGetDefaultProfiles());}public boolean acceptsProfiles(String... profiles) { Assert.notEmpty(profiles, \"Must specify at least one profile\"); for (String profile : profiles) { //这里判断的是以！开头的profile配置。 if (StringUtils.hasLength(profile) && profile.charAt(0) == '!') { //双重否定 if (!isProfileActive(profile.substring(1))) { return true; } } else if (isProfileActive(profile)) { return true; } } return false;}1234567891011121314protected Set<String> doGetActiveProfiles() { synchronized (this.activeProfiles) { if (this.activeProfiles.isEmpty()) { //拿到spring.profiles.active的配置。 //spring.profiles.active=dev,prod 如果有多个可以用逗号分割。实际应用估计也很少用到多个吧。 String profiles = getProperty(ACTIVE_PROFILES_PROPERTY_NAME); if (StringUtils.hasText(profiles)) { //拿到值去除空白字符串按照逗号分割成一个数组 setActiveProfiles(commaDelimitedListToStringArray(trimAllWhitespace(profiles))); } } return this.activeProfiles; }}12345678910111213protected Set<String> doGetDefaultProfiles() { synchronized (this.defaultProfiles) { //如果是default就拿到spring.profiles.default的配置的值， //同样spring.profiles.default也是可以配置多个的，按照逗号分隔。 if (this.defaultProfiles.equals(getReservedDefaultProfiles())) { String profiles = getProperty(DEFAULT_PROFILES_PROPERTY_NAME); if (StringUtils.hasText(profiles)) { setDefaultProfiles(commaDelimitedListToStringArray(trimAllWhitespace(profiles))); } } return this.defaultProfiles; }}以上关于环境配置相关配置的代码阅读。PropertyResolver接口继承图谱PropertySourcesPropertyResolver类里面有一个PropertySources的成员变量。类的很多方法实现都是调用的这个PropertySources成员变量的方法。PropertySourcesPropertyResolver可以通过getProperty(key)的方式获取对应的value值。那么PropertySourcesPropertyResolver有哪些东西呢？主要看这个方法：123456789101112131415161718192021222324252627282930313233343536373839protected <T> T getProperty(String key, Class<T> targetValueType, boolean resolveNestedPlaceholders) { boolean debugEnabled = logger.isDebugEnabled(); if (logger.isTraceEnabled()) { logger.trace(String.format(\"getProperty(\\\"%s\\\", %s)\", key, targetValueType.getSimpleName())); } if (this.propertySources != null) { //遍历所有已经加载到的PropertySource for (PropertySource<?> propertySource : this.propertySources) { if (debugEnabled) { logger.debug(String.format(\"Searching for key '%s' in [%s]\", key, propertySource.getName())); } Object value; if ((value = propertySource.getProperty(key)) != null) { Class<?> valueType = value.getClass(); //如果是字符串，同时要求字符替换的就调用字符替换方法 if (resolveNestedPlaceholders && value instanceof String) { value = resolveNestedPlaceholders((String) value); } if (debugEnabled) { logger.debug(String.format(\"Found key '%s' in [%s] with type [%s] and value '%s'\", key, propertySource.getName(), valueType.getSimpleName(), value)); } //如果不是字符串类型，就根据属性转换器尽心数据转换， //如果类型是属性转换器无法转换的就知道抛出异常 if (!this.conversionService.canConvert(valueType, targetValueType)) { throw new IllegalArgumentException(String.format( \"Cannot convert value [%s] from source type [%s] to target type [%s]\", value, valueType.getSimpleName(), targetValueType.getSimpleName())); } return this.conversionService.convert(value, targetValueType); } } } if (debugEnabled) { logger.debug(String.format(\"Could not find key '%s' in any property source. Returning [null]\", key)); } return null;}看看resolveNestedPlaceholders方法123456protected String resolveNestedPlaceholders(String value) { //如果PropertySourcesPropertyResolver上属性设置了ignoreUnresolvableNestedPlaceholders值为true可以忽略一些不存在key的属性。 //如果为false，key不存在的属性直接就会抛出异常。 return (this.ignoreUnresolvableNestedPlaceholders ? resolvePlaceholders(value) : resolveRequiredPlaceholders(value));}12345678public String resolveRequiredPlaceholders(String text) throws IllegalArgumentException { if (this.strictHelper == null) { //实例化一个PropertyPlaceholderHelper类， this.strictHelper = createPlaceholderHelper(false); } //用PropertyPlaceholderHelper类去解析属性 return doResolvePlaceholders(text, this.strictHelper);}1234567@Overridepublic String resolvePlaceholders(String text) { if (this.nonStrictHelper == null) { this.nonStrictHelper = createPlaceholderHelper(true); } return doResolvePlaceholders(text, this.nonStrictHelper);}createPlaceholderHelper：12345678//根据ignoreUnresolvablePlaceholders来创建PropertyPlaceholderHelper//placeholderPrefix 是替换的前缀，默认值是${//placeholderSuffix 是替换的后缀，默认值是}//valueSeparator 是值的分隔符，默认是：private PropertyPlaceholderHelper createPlaceholderHelper(boolean ignoreUnresolvablePlaceholders) { return new PropertyPlaceholderHelper(this.placeholderPrefix, this.placeholderSuffix, this.valueSeparator, ignoreUnresolvablePlaceholders);}再看看doResolvePlaceholders这个方法123456789101112private String doResolvePlaceholders(String text, PropertyPlaceholderHelper helper) { //调用的就是PropertyPlaceholderHelper的replacePlaceholders方法 //replacePlaceholders这个方法就会把text中包含${value}的值给替换成properties中value的值：比如:text是foo=${foo},而foo在properties的值是bar 那么text被替换后就是foo=bar。 //这个方法会通过接口回调的方式调用getPropertyAsRawString方法 return helper.replacePlaceholders(text, new PropertyPlaceholderHelper.PlaceholderResolver() { @Override public String resolvePlaceholder(String placeholderName) { //这个方法就是从properties里面以字符串的方式读取数据 return getPropertyAsRawString(placeholderName); } });}先看看PropertyPlaceholderHelper的构造方法：1234567891011121314151617181920212223//构造方法其实很简单就是一系列的赋值。public PropertyPlaceholderHelper(String placeholderPrefix, String placeholderSuffix, String valueSeparator, boolean ignoreUnresolvablePlaceholders) { Assert.notNull(placeholderPrefix, \"'placeholderPrefix' must not be null\"); Assert.notNull(placeholderSuffix, \"'placeholderSuffix' must not be null\"); //placeholderPrefix默认值是${ this.placeholderPrefix = placeholderPrefix; //placeholderSuffix默认值是} this.placeholderSuffix = placeholderSuffix; //wellKnownSimplePrefixes是一个map，里面存放着“{”,\"}\"、“[“,\"]\"、\"(\",\")\"。三对键值对。 //默认拿到的是{ String simplePrefixForSuffix = wellKnownSimplePrefixes.get(this.placeholderSuffix); if (simplePrefixForSuffix != null && this.placeholderPrefix.endsWith(simplePrefixForSuffix)) { //simplePrefix被赋值成{ this.simplePrefix = simplePrefixForSuffix; } else { this.simplePrefix = this.placeholderPrefix; } this.valueSeparator = valueSeparator; this.ignoreUnresolvablePlaceholders = ignoreUnresolvablePlaceholders;}继续看replacePlaceholders:12345public String replacePlaceholders(String value, PlaceholderResolver placeholderResolver) { Assert.notNull(value, \"'value' must not be null\"); //这是一个递归的方法。替换${}包围的值。 return parseStringValue(value, placeholderResolver, new HashSet<String>());}看看这个方法：这里简单的替换就不介绍了，主要看看多次的递归调用是如何实现的。比如:foo=${b${hello}};12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576protected String parseStringValue( String strVal, PlaceholderResolver placeholderResolver, Set<String> visitedPlaceholders) { StringBuilder result = new StringBuilder(strVal); int startIndex = strVal.indexOf(this.placeholderPrefix); while (startIndex != -1) { int endIndex = findPlaceholderEndIndex(result, startIndex); if (endIndex != -1) { //1.第一次拿到的是b${hello} //2.第二次拿到的是hello String placeholder = result.substring(startIndex + this.placeholderPrefix.length(), endIndex); String originalPlaceholder = placeholder; //不允许循环替换 //1.将b${hello}放入set集合 //2.将hello放入set集合 if (!visitedPlaceholders.add(originalPlaceholder)) { throw new IllegalArgumentException( \"Circular placeholder reference '\" + originalPlaceholder + \"' in property definitions\"); } // Recursive invocation, parsing placeholders contained in the placeholder key. //1.用b${hello}去placeholde // placeholder = parseStringValue(placeholder, placeholderResolver, visitedPlaceholders); // Now obtain the value for the fully resolved key... //调用PlaceholderResolver接口里面的方法。 //这里其实就是获取${key},key的属性的值了。 String propVal = placeholderResolver.resolvePlaceholder(placeholder); //如果属性值是null且this.valueSeparator不为空 if (propVal == null && this.valueSeparator != null) { //判断是否有:。 //例如foo:foo1 此时foo1为foo的默认值。如果从配置里面获取不到foo的值就使用默认值。 int separatorIndex = placeholder.indexOf(this.valueSeparator); if (separatorIndex != -1) { //拿到:符号之前的字符串 String actualPlaceholder = placeholder.substring(0, separatorIndex); String defaultValue = placeholder.substring(separatorIndex + this.valueSeparator.length()); //继续获取key的值 propVal = placeholderResolver.resolvePlaceholder(actualPlaceholder); if (propVal == null) { propVal = defaultValue; } } } //值这一部分很有意思，这一部分会把拿到的值检测全部替换一次。如果值里面也有${code}, if (propVal != null) { // Recursive invocation, parsing placeholders contained in the // previously resolved placeholder value. //获取值的 propVal = parseStringValue(propVal, placeholderResolver, visitedPlaceholders); //递归调用基本都是执行最里层的调用，然后一层一层的回归。替换最里层的字符串。 result.replace(startIndex, endIndex + this.placeholderSuffix.length(), propVal); if (logger.isTraceEnabled()) { logger.trace(\"Resolved placeholder '\" + placeholder + \"'\"); } //重新获取$的位置。 startIndex = result.indexOf(this.placeholderPrefix, startIndex + propVal.length()); } //这里是那些没有被替换的值的处理 else if (this.ignoreUnresolvablePlaceholders) { // Proceed with unprocessed value. startIndex = result.indexOf(this.placeholderPrefix, endIndex + this.placeholderSuffix.length()); } else { throw new IllegalArgumentException(\"Could not resolve placeholder '\" + placeholder + \"'\" + \" in string value \\\"\" + strVal + \"\\\"\"); } visitedPlaceholders.remove(originalPlaceholder); } else { startIndex = -1; } } return result.toString();}这一块可以举个例子:大家就清楚了12345678910111213public void testRecurseInProperty() { String text = \"foo=${bar}\"; final Properties props = new Properties(); props.setProperty(\"bar\", \"${baz}\"); props.setProperty(\"baz\", \"bar\"); PropertyPlaceholderHelper helper = new PropertyPlaceholderHelper(\"${\", \"}\"); assertEquals(\"foo=bar\",helper.replacePlaceholders(text, new PropertyPlaceholderHelper.PlaceholderResolver() { @Override public String resolvePlaceholder(String placeholderName) { return props.getProperty(placeholderName); } }));}再看一个例子：12345678910111213141516171819final PropertyPlaceholderHelper helper = new PropertyPlaceholderHelper(\"${\", \"}\");public void testRecurseInPlaceholder() { String text = \"foo=${b${inner}}\"; Properties props = new Properties(); props.setProperty(\"bar\", \"bar\"); props.setProperty(\"inner\", \"ar\"); assertEquals(\"foo=bar\", this.helper.replacePlaceholders(text, props)); text = \"${top}\"; props = new Properties(); props.setProperty(\"top\", \"${child}+${child}\"); props.setProperty(\"child\", \"${${differentiator}.grandchild}\"); props.setProperty(\"differentiator\", \"first\"); props.setProperty(\"first.grandchild\", \"actualValue\"); //这里是replacePlaceholders的另外的一个方法。 assertEquals(\"actualValue+actualValue\", this.helper.replacePlaceholders(text, props)); }介绍到这里就知道我们spring对于我们配置中${code}是如何处理的。但是我们的xml的配置压根就没有解析，仅仅只是对jvm环境变量参数，以及系统环境参数的一个字符替换而已。例如：123System.setProperty(\"spring\", \"classpath\");ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(\"${spring}:config.xml\");SimpleBean bean = context.getBean(SimpleBean.class);beanFactory的创建上面其实我们已经见到了beanFactory的方法：DefaultListableBeanFactory beanFactory = createBeanFactory();直接new DefaultListableBeanFactory的一个beanFactory实例。configuration的加载12345678910111213141516171819@Overrideprotected void loadBeanDefinitions(DefaultListableBeanFactory beanFactory) throws BeansException, IOException { //创建一个读取bean的配置文件的加载器 XmlBeanDefinitionReader beanDefinitionReader = new XmlBeanDefinitionReader(beanFactory); // Configure the bean definition reader with this context's // resource loading environment. // 父类被设值了就是StandardEnvironment beanDefinitionReader.setEnvironment(this.getEnvironment()); // 这里其实被赋值的是DefaultResourceLoader的子类。 beanDefinitionReader.setResourceLoader(this); // 设置资源实体的解析器 beanDefinitionReader.setEntityResolver(new ResourceEntityResolver(this)); // Allow a subclass to provide custom initialization of the reader, // then proceed with actually loading the bean definitions. initBeanDefinitionReader(beanDefinitionReader); loadBeanDefinitions(beanDefinitionReader);}XmlBeanDefinitionReader先看一下这个类的继承图谱继承图谱构造方法XmlBeanDefinitionReader会调用父类的构造方法尽行初始化环境，初始化类加载器。1234567891011121314151617181920212223protected AbstractBeanDefinitionReader(BeanDefinitionRegistry registry) { Assert.notNull(registry, "BeanDefinitionRegistry must not be null"); this.registry = registry; //DefaultListableBeanFactory不是ResourceLoader的类型 if (this.registry instanceof ResourceLoader) { this.resourceLoader = (ResourceLoader) this.registry; } else { //资源加载器PathMatchingResourcePatternResolver //但是会被子类的setResourceLoader覆盖掉。 this.resourceLoader = new PathMatchingResourcePatternResolver(); } //DefaultListableBeanFactory也不是EnvironmentCapable if (this.registry instanceof EnvironmentCapable) { this.environment = ((EnvironmentCapable) this.registry).getEnvironment(); } else { //初始化环境变量 this.environment = new StandardEnvironment(); }}bean的加载123456789101112131415161718192021222324252627282930313233343536373839404142434445public int loadBeanDefinitions(String location, Set<Resource> actualResources) throws BeanDefinitionStoreException { ResourceLoader resourceLoader = getResourceLoader(); if (resourceLoader == null) { throw new BeanDefinitionStoreException( \"Cannot import bean definitions from location [\" + location + \"]: no ResourceLoader available\"); } //因为ClassPathApplicationContext实现了ResourcePatternResolver if (resourceLoader instanceof ResourcePatternResolver) { // Resource pattern matching available. try { //这一句会拿到ResourcePatternResolver的对象。 //加载资源文件 Resource[] resources = ((ResourcePatternResolver) resourceLoader).getResources(location); //加载所有的bean int loadCount = loadBeanDefinitions(resources); //这里不会执行，因为actualResources是null if (actualResources != null) { for (Resource resource : resources) { actualResources.add(resource); } } if (logger.isDebugEnabled()) { logger.debug(\"Loaded \" + loadCount + \" bean definitions from location pattern [\" + location + \"]\"); } return loadCount; } catch (IOException ex) { throw new BeanDefinitionStoreException( \"Could not resolve bean definition resource pattern [\" + location + \"]\", ex); } } else { // Can only load single resources by absolute URL. Resource resource = resourceLoader.getResource(location); int loadCount = loadBeanDefinitions(resource); if (actualResources != null) { actualResources.add(resource); } if (logger.isDebugEnabled()) { logger.debug(\"Loaded \" + loadCount + \" bean definitions from location [\" + location + \"]\"); } return loadCount; }}getResources看看getResources的方法12345public Resource[] getResources(String locationPattern) throws IOException { // 因为resourcePatternResolver是PathMatchingResourcePatternResolver的实例 // 所以会调用PathMatchingResourcePatternResolver的getResources方法 return this.resourcePatternResolver.getResources(locationPattern);}123456789101112131415161718192021222324252627282930313233public Resource[] getResources(String locationPattern) throws IOException { Assert.notNull(locationPattern, \"Location pattern must not be null\"); //如果是以classpath*:开头的 if (locationPattern.startsWith(CLASSPATH_ALL_URL_PREFIX)) { // a class path resource (multiple resources for same name possible) // 拿到的是AntPathMatcher实例。 // 如果包含*或者？就匹配成功 if (getPathMatcher().isPattern(locationPattern.substring(CLASSPATH_ALL_URL_PREFIX.length()))) { // a class path resource pattern // return findPathMatchingResources(locationPattern); } else { // all class path resources with the given name // 路径没有？或者* return findAllClassPathResources(locationPattern.substring(CLASSPATH_ALL_URL_PREFIX.length())); } } else { // Only look for a pattern after a prefix here // (to not get fooled by a pattern symbol in a strange prefix). int prefixEnd = locationPattern.indexOf(\":\") + 1; if (getPathMatcher().isPattern(locationPattern.substring(prefixEnd))) { // a file pattern return findPathMatchingResources(locationPattern); } else { // a single resource with the given name return new Resource[] {getResourceLoader().getResource(locationPattern)}; } }}看看里面123public boolean isPattern(String path) { return (path.indexOf('*') != -1 || path.indexOf('?') != -1);}findPathMatchingResources12345678910111213141516171819202122232425262728293031323334protected Resource[] findPathMatchingResources(String locationPattern) throws IOException { //如果是\"/WEB-INF/*.xml 拿到的值是\"/WEB-INF/\" //如果是\"/WEB-INF/*/*.xml\"拿到是“/WEB-INF/*/” //如果是“classpath：context/*.xml”拿到的是“classpath：context/” // 获取文件的根路径 String rootDirPath = determineRootDir(locationPattern); // 获取正则表达式 String subPattern = locationPattern.substring(rootDirPath.length()); // 重新调用getResources，两个方法又开始循环调用了 // 如果是“classpath：context/”那此事会调用getResources里的子方法findAllClassPathResources // findAllClassPathResources会拿到目录下的所有资源 Resource[] rootDirResources = getResources(rootDirPath); Set<Resource> result = new LinkedHashSet<Resource>(16); for (Resource rootDirResource : rootDirResources) { rootDirResource = resolveRootDirResource(rootDirResource); //加载vfs文件 if (rootDirResource.getURL().getProtocol().startsWith(ResourceUtils.URL_PROTOCOL_VFS)) { result.addAll(VfsResourceMatchingDelegate.findMatchingResources(rootDirResource, subPattern, getPathMatcher())); } //加载jar里面的文件 else if (isJarResource(rootDirResource)) { result.addAll(doFindPathMatchingJarResources(rootDirResource, subPattern)); } else { //最后才是加载本地系统的文件 result.addAll(doFindPathMatchingFileResources(rootDirResource, subPattern)); } } if (logger.isDebugEnabled()) { logger.debug(\"Resolved location pattern [\" + locationPattern + \"] to resources \" + result); } return result.toArray(new Resource[result.size()]);}findAllClassPathResources12345678protected Resource[] findAllClassPathResources(String location) throws IOException { String path = location; if (path.startsWith(\"/\")) { path = path.substring(1); } Set<Resource> result = doFindAllClassPathResources(path); return result.toArray(new Resource[result.size()]);}再看看123456789101112131415protected Set<Resource> doFindAllClassPathResources(String path) throws IOException { Set<Resource> result = new LinkedHashSet<Resource>(16); ClassLoader cl = getClassLoader(); Enumeration<URL> resourceUrls = (cl != null ? cl.getResources(path) : ClassLoader.getSystemResources(path)); while (resourceUrls.hasMoreElements()) { URL url = resourceUrls.nextElement(); result.add(convertClassLoaderURL(url)); } if (\"\".equals(path)) { // The above result is likely to be incomplete, i.e. only containing file system references. // We need to have pointers to each of the jar files on the classpath as well... addAllClassLoaderJarRoots(cl, result); } return result;}说到这里仅仅也只是spring是如何找文件的。这里还没有文件的读取和解析。下面介绍spring配置文件的读取和解析。12345678910111213141516171819202122232425262728293031323334353637383940public int loadBeanDefinitions(EncodedResource encodedResource) throws BeanDefinitionStoreException { Assert.notNull(encodedResource, \"EncodedResource must not be null\"); if (logger.isInfoEnabled()) { logger.info(\"Loading XML bean definitions from \" + encodedResource.getResource()); } // TheadLocal的已经加载的资源set集合。 Set<EncodedResource> currentResources = this.resourcesCurrentlyBeingLoaded.get(); if (currentResources == null) { currentResources = new HashSet<EncodedResource>(4); this.resourcesCurrentlyBeingLoaded.set(currentResources); } if (!currentResources.add(encodedResource)) { throw new BeanDefinitionStoreException( \"Detected cyclic loading of \" + encodedResource + \" - check your import definitions!\"); } //以下这段代码就是真实读取文件的逻辑了。 try { InputStream inputStream = encodedResource.getResource().getInputStream(); try { InputSource inputSource = new InputSource(inputStream); if (encodedResource.getEncoding() != null) { inputSource.setEncoding(encodedResource.getEncoding()); } return doLoadBeanDefinitions(inputSource, encodedResource.getResource()); } finally { inputStream.close(); } } catch (IOException ex) { throw new BeanDefinitionStoreException( \"IOException parsing XML document from \" + encodedResource.getResource(), ex); } finally { currentResources.remove(encodedResource); if (currentResources.isEmpty()) { this.resourcesCurrentlyBeingLoaded.remove(); } }}doLoadBeanDefinitions1234567protected int doLoadBeanDefinitions(InputSource inputSource, Resource resource) throws BeanDefinitionStoreException { // 读取文件 Document doc = doLoadDocument(inputSource, resource); // 这个方法里面就是配置文件的解析了。 return registerBeanDefinitions(doc, resource);}doLoadDocument12345678protected Document doLoadDocument(InputSource inputSource, Resource resource) throws Exception { // documentLoader是一个DefaultDocumentLoader对象，此类是DocumentLoader接口的唯一实现。 // getEntityResolver方法返回ResourceEntityResolver, // ResourceEntityResolver会用xsd或者dtd约束文件做校验。 // errorHandler是一个SimpleSaxErrorHandler对象。 return this.documentLoader.loadDocument(inputSource, getEntityResolver(), this.errorHandler, getValidationModeForResource(resource), isNamespaceAware());}loadDocument12345678910111213/**** 这里就是老套路了，可以看出，Spring还是使用了dom的方式解析，即一次全部load到内存**/public Document loadDocument(InputSource inputSource, EntityResolver entityResolver, ErrorHandler errorHandler, int validationMode, boolean namespaceAware) throws Exception { DocumentBuilderFactory factory = createDocumentBuilderFactory(validationMode, namespaceAware); if (logger.isDebugEnabled()) { logger.debug(\"Using JAXP provider [\" + factory.getClass().getName() + \"]\"); } DocumentBuilder builder = createDocumentBuilder(factory, entityResolver, errorHandler); return builder.parse(inputSource);}12345678910111213141516protected DocumentBuilderFactory createDocumentBuilderFactory(int validationMode, boolean namespaceAware{ DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); factory.setNamespaceAware(namespaceAware); if (validationMode != XmlValidationModeDetector.VALIDATION_NONE) { //此方法设为true仅对dtd有效，xsd(schema)无效 factory.setValidating(true); if (validationMode == XmlValidationModeDetector.VALIDATION_XSD) { // Enforce namespace aware for XSD... //开启xsd(schema)支持 factory.setNamespaceAware(true); //这个也是Java支持Schema的套路，可以问度娘 factory.setAttribute(SCHEMA_LANGUAGE_ATTRIBUTE, XSD_SCHEMA_LANGUAGE); } } return factory;}bean解析registerBeanDefinitions瞧一下这个方法，看看做了哪些事情。12345678910public int registerBeanDefinitions(Document doc, Resource resource) throws BeanDefinitionStoreException { //根据反射的方式创建DefaultBeanDefinitionDocumentReader对象。 //这其实也是策略模式，通过setter方法可以更换其实现。修改documentReaderClass参数即可 BeanDefinitionDocumentReader documentReader = createBeanDefinitionDocumentReader(); //获取bean定义的数量 int countBefore = getRegistry().getBeanDefinitionCount(); //读取文件 documentReader.registerBeanDefinitions(doc, createReaderContext(resource)); return getRegistry().getBeanDefinitionCount() - countBefore;}createReaderContext123456789public XmlReaderContext createReaderContext(Resource resource) { // problemReporter是一个FailFastProblemReporter对象。 // eventListener是EmptyReaderEventListener对象，此类里的方法都是空实现。 // sourceExtractor是NullSourceExtractor对象，直接返回空，也是空实现。 // getNamespaceHandlerResolver默认返回DefaultNamespaceHandlerResolver对象，用来获取xsd对应的处理器。 return new XmlReaderContext(resource, this.problemReporter, this.eventListener, this.sourceExtractor, this, getNamespaceHandlerResolver());}XmlReaderContext的作用感觉就是这一堆参数的容器，糅合到一起传给DocumentReader，并美其名为Context。可以看出，Spring中到处都是策略模式，大量操作被抽象成接口。registerBeanDefinitions此方式是在DefaultBeanDefinitionDocumentReader的里面实现的。12345678910111213141516171819202122232425262728293031323334353637383940414243444546@Overridepublic void registerBeanDefinitions(Document doc, XmlReaderContext readerContext) { this.readerContext = readerContext; //获取根节点beans Element root = doc.getDocumentElement(); //注册根节点下所有的bean doRegisterBeanDefinitions(root);}``` ### doRegisterBeanDefinitions```javaprotected void doRegisterBeanDefinitions(Element root) { // Any nested <beans> elements will cause recursion in this method. In // order to propagate and preserve <beans> default-* attributes correctly, // keep track of the current (parent) delegate, which may be null. Create // the new (child) delegate with a reference to the parent for fallback purposes, // then ultimately reset this.delegate back to its original (parent) reference. // this behavior emulates a stack of delegates without actually necessitating one. BeanDefinitionParserDelegate parent = this.delegate; this.delegate = createDelegate(getReaderContext(), root, parent); // 默认的命名空间即 // http://www.springframework.org/schema/beans if (this.delegate.isDefaultNamespace(root)) { // 检查profile属性,获取profile属性 String profileSpec = root.getAttribute(PROFILE_ATTRIBUTE); if (StringUtils.hasText(profileSpec)) { // 分隔profile属性的值 ,分割 String[] specifiedProfiles = StringUtils.tokenizeToStringArray( profileSpec, BeanDefinitionParserDelegate.MULTI_VALUE_ATTRIBUTE_DELIMITERS); // 如果不是可用的profile的值，就直接返回 if (!getReaderContext().getEnvironment().acceptsProfiles(specifiedProfiles)) { return; } } } // 预处理xml方法这是个空方法， // 我们可以扩展这个方法，来加载解析我们自己的自定义标签。 preProcessXml(root); // 解析 parseBeanDefinitions(root, this.delegate); postProcessXml(root); this.delegate = parent; }parseBeanDefinitions1234567891011121314151617181920212223242526`protected void parseBeanDefinitions(Element root, BeanDefinitionParserDelegate delegate) { // 验证名称空间 // http://www.springframework.org/schema/beans if (delegate.isDefaultNamespace(root)) { NodeList nl = root.getChildNodes(); for (int i = 0; i < nl.getLength(); i++) { Node node = nl.item(i); if (node instanceof Element) { Element ele = (Element) node; // 检查节点是不是 // http://www.springframework.org/schema/beans if (delegate.isDefaultNamespace(ele)) { // 解析节点 parseDefaultElement(ele, delegate); } else { delegate.parseCustomElement(ele); } } } } else { delegate.parseCustomElement(root); } }parseDefaultElement123456789101112131415161718192021private void parseDefaultElement(Element ele, BeanDefinitionParserDelegate delegate) { // 处理 import标签 if (delegate.nodeNameEquals(ele, IMPORT_ELEMENT)) { importBeanDefinitionResource(ele); } // 处理 alais 标签 else if (delegate.nodeNameEquals(ele, ALIAS_ELEMENT)) { processAliasRegistration(ele); } // 处理 bean标签 else if (delegate.nodeNameEquals(ele, BEAN_ELEMENT)) { processBeanDefinition(ele, delegate); } // 处理beans标签 // 返回去调用doRegisterBeanDefinitions的方法 else if (delegate.nodeNameEquals(ele, NESTED_BEANS_ELEMENT)) { // recurse // 循环调用doRegisterBeanDefinitions doRegisterBeanDefinitions(ele); }}importBeanDefinitionResource处理import12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667protected void importBeanDefinitionResource(Element ele) { // 获取resource标记的路径 // <import resource=\"context:spring.xml\"/> String location = ele.getAttribute(RESOURCE_ATTRIBUTE); if (!StringUtils.hasText(location)) { getReaderContext().error(\"Resource location must not be empty\", ele); return; } // Resolve system properties: e.g. \"${user.dir}\" // 字符替换标签 location = getReaderContext().getEnvironment().resolveRequiredPlaceholders(location); Set<Resource> actualResources = new LinkedHashSet<Resource>(4); // Discover whether the location is an absolute or relative URI boolean absoluteLocation = false; try { absoluteLocation = ResourcePatternUtils.isUrl(location) || ResourceUtils.toURI(location).isAbsolute(); } catch (URISyntaxException ex) { // cannot convert to an URI, considering the location relative // unless it is the well-known Spring prefix \"classpath*:\" } // Absolute or relative? if (absoluteLocation) { try { int importCount = getReaderContext().getReader().loadBeanDefinitions(location, actualResources); if (logger.isDebugEnabled()) { logger.debug(\"Imported \" + importCount + \" bean definitions from URL location [\" + location + \"]\"); } } catch (BeanDefinitionStoreException ex) { getReaderContext().error( \"Failed to import bean definitions from URL location [\" + location + \"]\", ele, ex); } } else { // No URL -> considering resource location as relative to the current file. try { int importCount; Resource relativeResource = getReaderContext().getResource().createRelative(location); if (relativeResource.exists()) { importCount = getReaderContext().getReader().loadBeanDefinitions(relativeResource); actualResources.add(relativeResource); } else { String baseLocation = getReaderContext().getResource().getURL().toString(); importCount = getReaderContext().getReader().loadBeanDefinitions( StringUtils.applyRelativePath(baseLocation, location), actualResources); } if (logger.isDebugEnabled()) { logger.debug(\"Imported \" + importCount + \" bean definitions from relative location [\" + location + \"]\"); } } catch (IOException ex) { getReaderContext().error(\"Failed to resolve current resource location\", ele, ex); } catch (BeanDefinitionStoreException ex) { getReaderContext().error(\"Failed to import bean definitions from relative location [\" + location + \"]\", ele, ex); } } Resource[] actResArray = actualResources.toArray(new Resource[actualResources.size()]); getReaderContext().fireImportProcessed(location, actResArray, extractSource(ele));}importBeanDefinitionResource套路和之前的配置文件加载完全一样，不过注意被import进来的文件是先于当前文件被解析的。上面有些周边的代码就不介绍了。processAliasRegistration处理别名123456789101112131415161718192021222324252627protected void processAliasRegistration(Element ele) { // 拿到名字，和别名 String name = ele.getAttribute(NAME_ATTRIBUTE); String alias = ele.getAttribute(ALIAS_ATTRIBUTE); boolean valid = true; if (!StringUtils.hasText(name)) { getReaderContext().error(\"Name must not be empty\", ele); valid = false; } if (!StringUtils.hasText(alias)) { getReaderContext().error(\"Alias must not be empty\", ele); valid = false; } if (valid) { try { // 核心方法，就是在DefaultListableBeanFactor注册别名， // 其实就是在一个map里面写入名字和别名的映射关系。 getReaderContext().getRegistry().registerAlias(name, alias); } catch (Exception ex) { getReaderContext().error(\"Failed to register alias '\" + alias + \"' for bean with name '\" + name + \"'\", ele, ex); } // 触发监听器 getReaderContext().fireAliasRegistered(name, alias, extractSource(ele)); }}其实这个方法就是给一个bean取一个别名：比如有一个bean名为beanA，但是另一个组件想以beanB的名字使用，就可以这样定义:registerAlias1234567891011121314151617181920212223// 其实就是在map里加上一条映射关系。public void registerAlias(String name, String alias) { Assert.hasText(name, \"'name' must not be empty\"); Assert.hasText(alias, \"'alias' must not be empty\"); if (alias.equals(name)) { this.aliasMap.remove(alias); } else { String registeredName = this.aliasMap.get(alias); if (registeredName != null) { if (registeredName.equals(name)) { // An existing alias - no need to re-register return; } if (!allowAliasOverriding()) { throw new IllegalStateException(\"Cannot register alias '\" + alias + \"' for name '\" + name + \"': It is already registered for name '\" + registeredName + \"'.\"); } } checkForAliasCircle(name, alias); this.aliasMap.put(alias, name); }}processBeanDefinition处理bean12345678910111213141516protected void processBeanDefinition(Element ele, BeanDefinitionParserDelegate delegate) { BeanDefinitionHolder bdHolder = delegate.parseBeanDefinitionElement(ele); if (bdHolder != null) { bdHolder = delegate.decorateBeanDefinitionIfRequired(ele, bdHolder); try { // Register the final decorated instance. BeanDefinitionReaderUtils.registerBeanDefinition(bdHolder, getReaderContext().getRegistry()); } catch (BeanDefinitionStoreException ex) { getReaderContext().error(\"Failed to register bean definition with name '\" + bdHolder.getBeanName() + \"'\", ele, ex); } // Send registration event. getReaderContext().fireComponentRegistered(new BeanComponentDefinition(bdHolder)); }}最后会调用BeanDefinitionParserDelegate.parseBeanDefinitionElement首先获取到id和name属性，name属性支持配置多个，以逗号分隔，如果没有指定id，那么将以第一个name属性值代替。id必须是唯一的，name属性其实是alias的角色，可以和其它的bean重复，如果name也没有配置，那么其实什么也没做。1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374public BeanDefinitionHolder parseBeanDefinitionElement(Element ele, BeanDefinition containingBean) { // 获取ID String id = ele.getAttribute(ID_ATTRIBUTE); // 获取name String nameAttr = ele.getAttribute(NAME_ATTRIBUTE); // name属性可以配置多个，用逗号分隔。 List<String> aliases = new ArrayList<String>(); if (StringUtils.hasLength(nameAttr)) { String[] nameArr = StringUtils.tokenizeToStringArray(nameAttr, MULTI_VALUE_ATTRIBUTE_DELIMITERS); aliases.addAll(Arrays.asList(nameArr)); } String beanName = id; // 如果id没有配置 就用name的数组的第一个名字代替 if (!StringUtils.hasText(beanName) && !aliases.isEmpty()) { beanName = aliases.remove(0); if (logger.isDebugEnabled()) { logger.debug(\"No XML 'id' specified - using '\" + beanName + \"' as bean name and \" + aliases + \" as aliases\"); } } if (containingBean == null) { //检查bean名字，别名是不是已经使用过了 checkNameUniqueness(beanName, aliases, ele); } // 待会我会介绍这个BeanDefinition的体系 这个方法到底干了啥？ // 1.这个方法会解析bean 的class标签，parent的标签。 // 2.然后会new一个GenericBeanDefinition，然后将class，parent的值，以及classload设置进去。 // 3.解析标签下的meta，key，value标签，把依赖的关系也设置进去。 AbstractBeanDefinition beanDefinition = parseBeanDefinitionElement(ele, beanName, containingBean); if (beanDefinition != null) { // 如果bean标签没有设置id，和name属性。 if (!StringUtils.hasText(beanName)) { try { if (containingBean != null) { beanName = BeanDefinitionReaderUtils.generateBeanName( beanDefinition, this.readerContext.getRegistry(), true); } else { // 如果bean标签没有设置id，和name属性。 // 自行创建一个名字。这里会调用BeanDefinitionReaderUtils.generateBeanName方法 beanName = this.readerContext.generateBeanName(beanDefinition); // Register an alias for the plain bean class name, if still possible, // if the generator returned the class name plus a suffix. // This is expected for Spring 1.2/2.0 backwards compatibility. // 获取beanClassName，其实就是class属性的值。 String beanClassName = beanDefinition.getBeanClassName(); // 如果名字是以className开头且没有被使用过的，就加入到别名里。 if (beanClassName != null && beanName.startsWith(beanClassName) && beanName.length() > beanClassName.length() && !this.readerContext.getRegistry().isBeanNameInUse(beanClassName)) { aliases.add(beanClassName); } } if (logger.isDebugEnabled()) { logger.debug(\"Neither XML 'id' nor 'name' specified - \" + \"using generated bean name [\" + beanName + \"]\"); } } catch (Exception ex) { error(ex.getMessage(), ele); return null; } } String[] aliasesArray = StringUtils.toStringArray(aliases); // 创建BeanDefinitionHolder类 return new BeanDefinitionHolder(beanDefinition, beanName, aliasesArray); } return null;}BeanDefinition接口继承图谱parseBeanDefinitionElement接着看AbstractBeanDefinition beanDefinition = parseBeanDefinitionElement(ele, beanName, containingBean);这句的具体实现：12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455public AbstractBeanDefinition parseBeanDefinitionElement( Element ele, String beanName, BeanDefinition containingBean) { //把名字进行一次压栈 this.parseState.push(new BeanEntry(beanName)); String className = null; // 获取class属性值 if (ele.hasAttribute(CLASS_ATTRIBUTE)) { className = ele.getAttribute(CLASS_ATTRIBUTE).trim(); } try { String parent = null; // 获取parent的属性值 if (ele.hasAttribute(PARENT_ATTRIBUTE)) { parent = ele.getAttribute(PARENT_ATTRIBUTE); } // 调用BeanDefinitionReaderUtils.createBeanDefinition（） // 创建GenericBeanDefinition实例，设置className，parent。 AbstractBeanDefinition bd = createBeanDefinition(className, parent); // 这个方法会解析singleton、scope、abstract、lazy-init、autowire、 // dependency-check、depends-on、init-method、autowire-candidate、 // primary、destroy-method、actory-method、factory-bean、constructor-arg // index、type、value-type、key-type、property、ref、value等标签 // 设置到 GenericBeanDefinition的实例里面。 parseBeanDefinitionAttributes(ele, beanName, containingBean, bd); // 设置描述。 bd.setDescription(DomUtils.getChildElementValueByTagName(ele, DESCRIPTION_ELEMENT)); // 解析元数据标签 parseMetaElements(ele, bd); // 解析lookup-method标签 parseLookupOverrideSubElements(ele, bd.getMethodOverrides()); // 解析replace-method标签 parseReplacedMethodSubElements(ele, bd.getMethodOverrides()); // 解析构造方法 parseConstructorArgElements(ele, bd); // 解析属性依赖 parsePropertyElements(ele, bd); // 解析Qualifier标签 parseQualifierElements(ele, bd); // 设置资源 bd.setResource(this.readerContext.getResource()); bd.setSource(extractSource(ele)); return bd; } catch (Exception ex) { ... } finally { this.parseState.pop(); } return null;}其实这里面就已经把bean的定义bean的依赖关系都设置好了。但是bean并没有被实例化。parseMetaElements12345678910111213141516public void parseMetaElements(Element ele, BeanMetadataAttributeAccessor attributeAccessor) { NodeList nl = ele.getChildNodes(); for (int i = 0; i < nl.getLength(); i++) { Node node = nl.item(i); if (isCandidateElement(node) && nodeNameEquals(node, META_ELEMENT)) { Element metaElement = (Element) node; String key = metaElement.getAttribute(KEY_ATTRIBUTE); String value = metaElement.getAttribute(VALUE_ATTRIBUTE); //就是一个key, value的载体，无他 BeanMetadataAttribute attribute = new BeanMetadataAttribute(key, value); //sourceExtractor默认是NullSourceExtractor，返回的是空 attribute.setSource(extractSource(metaElement)); attributeAccessor.addMetadataAttribute(attribute); } }}AbstractBeanDefinition继承自BeanMetadataAttributeAccessor类，底层使用了一个LinkedHashMap保存metadata。这个metadata具体是做什么暂时还不知道。我们实际应用中meta标签也很少见。例子：123<bean id=\"b\" name=\"one, two\" class=\"base.SimpleBean\"> <meta key=\"name\" value=\"dsfesf\"/></bean>parseLookupOverrideSubElements123456789101112131415public void parseLookupOverrideSubElements(Element beanEle, MethodOverrides overrides) { NodeList nl = beanEle.getChildNodes(); for (int i = 0; i < nl.getLength(); i++) { Node node = nl.item(i); if (isCandidateElement(node) && nodeNameEquals(node, LOOKUP_METHOD_ELEMENT)) { Element ele = (Element) node; String methodName = ele.getAttribute(NAME_ATTRIBUTE); String beanRef = ele.getAttribute(BEAN_ELEMENT); //以MethodOverride的方式，存放在set集合里面 LookupOverride override = new LookupOverride(methodName, beanRef); override.setSource(extractSource(ele)); overrides.addOverride(override); } }}此标签的作用在于当一个bean的某个方法被设置为lookup-method后，每次调用此方法时，都会返回一个新的指定bean的对象。例如：12345<bean id=\"apple\" class=\"a.b.c.Apple\" scope=\"prototype\"/><!--水果盘--><bean id=\"fruitPlate\" class=\"a.b.c.FruitPlate\"> <lookup-method name=\"getFruit\" bean=\"apple\"/></bean>parseReplacedMethodSubElements123456789101112131415161718192021222324252627public void parseReplacedMethodSubElements(Element beanEle, MethodOverrides overrides) { NodeList nl = beanEle.getChildNodes(); for (int i = 0; i < nl.getLength(); i++) { Node node = nl.item(i); if (isCandidateElement(node) && nodeNameEquals(node, REPLACED_METHOD_ELEMENT)) { Element replacedMethodEle = (Element) node; //获取name属性 String name = replacedMethodEle.getAttribute(NAME_ATTRIBUTE); //获取replace-method属性 String callback = replacedMethodEle.getAttribute(REPLACER_ATTRIBUTE); ReplaceOverride replaceOverride = new ReplaceOverride(name, callback); // Look for arg-type match elements. // 获取所有的 arg-type的标签 // 遍历所有节点，找到匹配的。以ReplaceOverride结构存储到list里面 List<Element> argTypeEles = DomUtils.getChildElementsByTagName(replacedMethodEle, ARG_TYPE_ELEMENT); for (Element argTypeEle : argTypeEles) { String match = argTypeEle.getAttribute(ARG_TYPE_MATCH_ATTRIBUTE); match = (StringUtils.hasText(match) ? match : DomUtils.getTextValue(argTypeEle)); if (StringUtils.hasText(match)) { replaceOverride.addTypeIdentifier(match); } } replaceOverride.setSource(extractSource(replacedMethodEle)); overrides.addOverride(replaceOverride); } }}replace-method 主要作用就是替换方法体及其返回值，使用比较简单。只需要实现MethodReplacer接口，并重写reimplement方法，然后就能完成方法的替换。这个有点类似aop的功能实现场景用的地方不是太多。例子：1234567<!-- ====================replace-method属性注入==================== --><bean id=\"dogReplaceMethod\" class=\"com.lyc.cn.v2.day01.method.replaceMethod.ReplaceDog\"/><bean id=\"originalDogReplaceMethod\" class=\"com.lyc.cn.v2.day01.method.replaceMethod.OriginalDog\"> <replaced-method name=\"sayHello\" replacer=\"dogReplaceMethod\"> <arg-type match=\"java.lang.String\"></arg-type> </replaced-method></bean>configuration的加载12345678910111213141516171819@Overrideprotected void loadBeanDefinitions(DefaultListableBeanFactory beanFactory) throws BeansException, IOException { //创建一个读取bean的配置文件的加载器 XmlBeanDefinitionReader beanDefinitionReader = new XmlBeanDefinitionReader(beanFactory); // Configure the bean definition reader with this context's // resource loading environment. // 父类被设值了就是StandardEnvironment beanDefinitionReader.setEnvironment(this.getEnvironment()); // 这里其实被赋值的是DefaultResourceLoader的子类。 beanDefinitionReader.setResourceLoader(this); // 设置资源实体的解析器 beanDefinitionReader.setEntityResolver(new ResourceEntityResolver(this)); // Allow a subclass to provide custom initialization of the reader, // then proceed with actually loading the bean definitions. initBeanDefinitionReader(beanDefinitionReader); loadBeanDefinitions(beanDefinitionReader);}XmlBeanDefinitionReader先看一下这个类的继承图谱继承图谱构造方法XmlBeanDefinitionReader会调用父类的构造方法尽行初始化环境，初始化类加载器。1234567891011121314151617181920212223protected AbstractBeanDefinitionReader(BeanDefinitionRegistry registry) { Assert.notNull(registry, "BeanDefinitionRegistry must not be null"); this.registry = registry; //DefaultListableBeanFactory不是ResourceLoader的类型 if (this.registry instanceof ResourceLoader) { this.resourceLoader = (ResourceLoader) this.registry; } else { //资源加载器PathMatchingResourcePatternResolver //但是会被子类的setResourceLoader覆盖掉。 this.resourceLoader = new PathMatchingResourcePatternResolver(); } //DefaultListableBeanFactory也不是EnvironmentCapable if (this.registry instanceof EnvironmentCapable) { this.environment = ((EnvironmentCapable) this.registry).getEnvironment(); } else { //初始化环境变量 this.environment = new StandardEnvironment(); }}bean的加载123456789101112131415161718192021222324252627282930313233343536373839404142434445public int loadBeanDefinitions(String location, Set<Resource> actualResources) throws BeanDefinitionStoreException { ResourceLoader resourceLoader = getResourceLoader(); if (resourceLoader == null) { throw new BeanDefinitionStoreException( \"Cannot import bean definitions from location [\" + location + \"]: no ResourceLoader available\"); } //因为ClassPathApplicationContext实现了ResourcePatternResolver if (resourceLoader instanceof ResourcePatternResolver) { // Resource pattern matching available. try { //这一句会拿到ResourcePatternResolver的对象。 //加载资源文件 Resource[] resources = ((ResourcePatternResolver) resourceLoader).getResources(location); //加载所有的bean int loadCount = loadBeanDefinitions(resources); //这里不会执行，因为actualResources是null if (actualResources != null) { for (Resource resource : resources) { actualResources.add(resource); } } if (logger.isDebugEnabled()) { logger.debug(\"Loaded \" + loadCount + \" bean definitions from location pattern [\" + location + \"]\"); } return loadCount; } catch (IOException ex) { throw new BeanDefinitionStoreException( \"Could not resolve bean definition resource pattern [\" + location + \"]\", ex); } } else { // Can only load single resources by absolute URL. Resource resource = resourceLoader.getResource(location); int loadCount = loadBeanDefinitions(resource); if (actualResources != null) { actualResources.add(resource); } if (logger.isDebugEnabled()) { logger.debug(\"Loaded \" + loadCount + \" bean definitions from location [\" + location + \"]\"); } return loadCount; }}getResources看看getResources的方法12345public Resource[] getResources(String locationPattern) throws IOException { // 因为resourcePatternResolver是PathMatchingResourcePatternResolver的实例 // 所以会调用PathMatchingResourcePatternResolver的getResources方法 return this.resourcePatternResolver.getResources(locationPattern);}123456789101112131415161718192021222324252627282930313233public Resource[] getResources(String locationPattern) throws IOException { Assert.notNull(locationPattern, \"Location pattern must not be null\"); //如果是以classpath*:开头的 if (locationPattern.startsWith(CLASSPATH_ALL_URL_PREFIX)) { // a class path resource (multiple resources for same name possible) // 拿到的是AntPathMatcher实例。 // 如果包含*或者？就匹配成功 if (getPathMatcher().isPattern(locationPattern.substring(CLASSPATH_ALL_URL_PREFIX.length()))) { // a class path resource pattern // return findPathMatchingResources(locationPattern); } else { // all class path resources with the given name // 路径没有？或者* return findAllClassPathResources(locationPattern.substring(CLASSPATH_ALL_URL_PREFIX.length())); } } else { // Only look for a pattern after a prefix here // (to not get fooled by a pattern symbol in a strange prefix). int prefixEnd = locationPattern.indexOf(\":\") + 1; if (getPathMatcher().isPattern(locationPattern.substring(prefixEnd))) { // a file pattern return findPathMatchingResources(locationPattern); } else { // a single resource with the given name return new Resource[] {getResourceLoader().getResource(locationPattern)}; } }}看看里面123public boolean isPattern(String path) { return (path.indexOf('*') != -1 || path.indexOf('?') != -1);}findPathMatchingResources12345678910111213141516171819202122232425262728293031323334protected Resource[] findPathMatchingResources(String locationPattern) throws IOException { //如果是\"/WEB-INF/*.xml 拿到的值是\"/WEB-INF/\" //如果是\"/WEB-INF/*/*.xml\"拿到是“/WEB-INF/*/” //如果是“classpath：context/*.xml”拿到的是“classpath：context/” // 获取文件的根路径 String rootDirPath = determineRootDir(locationPattern); // 获取正则表达式 String subPattern = locationPattern.substring(rootDirPath.length()); // 重新调用getResources，两个方法又开始循环调用了 // 如果是“classpath：context/”那此事会调用getResources里的子方法findAllClassPathResources // findAllClassPathResources会拿到目录下的所有资源 Resource[] rootDirResources = getResources(rootDirPath); Set<Resource> result = new LinkedHashSet<Resource>(16); for (Resource rootDirResource : rootDirResources) { rootDirResource = resolveRootDirResource(rootDirResource); //加载vfs文件 if (rootDirResource.getURL().getProtocol().startsWith(ResourceUtils.URL_PROTOCOL_VFS)) { result.addAll(VfsResourceMatchingDelegate.findMatchingResources(rootDirResource, subPattern, getPathMatcher())); } //加载jar里面的文件 else if (isJarResource(rootDirResource)) { result.addAll(doFindPathMatchingJarResources(rootDirResource, subPattern)); } else { //最后才是加载本地系统的文件 result.addAll(doFindPathMatchingFileResources(rootDirResource, subPattern)); } } if (logger.isDebugEnabled()) { logger.debug(\"Resolved location pattern [\" + locationPattern + \"] to resources \" + result); } return result.toArray(new Resource[result.size()]);}findAllClassPathResources12345678protected Resource[] findAllClassPathResources(String location) throws IOException { String path = location; if (path.startsWith(\"/\")) { path = path.substring(1); } Set<Resource> result = doFindAllClassPathResources(path); return result.toArray(new Resource[result.size()]);}再看看doFindAllClassPathResources:123456789101112131415protected Set<Resource> doFindAllClassPathResources(String path) throws IOException { Set<Resource> result = new LinkedHashSet<Resource>(16); ClassLoader cl = getClassLoader(); Enumeration<URL> resourceUrls = (cl != null ? cl.getResources(path) : ClassLoader.getSystemResources(path)); while (resourceUrls.hasMoreElements()) { URL url = resourceUrls.nextElement(); result.add(convertClassLoaderURL(url)); } if (\"\".equals(path)) { // The above result is likely to be incomplete, i.e. only containing file system references. // We need to have pointers to each of the jar files on the classpath as well... addAllClassLoaderJarRoots(cl, result); } return result;}说到这里仅仅也只是spring是如何找文件的。这里还没有文件的读取和解析。loadBeanDefinitions下面介绍spring配置文件的读取和解析。1234567891011121314151617181920212223242526272829303132333435public int loadBeanDefinitions(EncodedResource encodedResource) throws BeanDefinitionStoreException { // TheadLocal的已经加载的资源set集合。 Set<EncodedResource> currentResources = this.resourcesCurrentlyBeingLoaded.get(); if (currentResources == null) { currentResources = new HashSet<EncodedResource>(4); this.resourcesCurrentlyBeingLoaded.set(currentResources); } if (!currentResources.add(encodedResource)) { throw new BeanDefinitionStoreException( \"Detected cyclic loading of \" + encodedResource + \" - check your import definitions!\"); } //以下这段代码就是真实读取文件的逻辑了。 try { InputStream inputStream = encodedResource.getResource().getInputStream(); try { InputSource inputSource = new InputSource(inputStream); if (encodedResource.getEncoding() != null) { inputSource.setEncoding(encodedResource.getEncoding()); } return doLoadBeanDefinitions(inputSource, encodedResource.getResource()); } finally { inputStream.close(); } } catch (IOException ex) { ... } finally { currentResources.remove(encodedResource); if (currentResources.isEmpty()) { this.resourcesCurrentlyBeingLoaded.remove(); } }}doLoadBeanDefinitions1234567protected int doLoadBeanDefinitions(InputSource inputSource, Resource resource) throws BeanDefinitionStoreException { // 读取文件 Document doc = doLoadDocument(inputSource, resource); // 这个方法里面就是配置文件的解析了。 return registerBeanDefinitions(doc, resource);}doLoadDocument12345678protected Document doLoadDocument(InputSource inputSource, Resource resource) throws Exception { // documentLoader是一个DefaultDocumentLoader对象，此类是DocumentLoader接口的唯一实现。 // getEntityResolver方法返回ResourceEntityResolver, // ResourceEntityResolver会用xsd或者dtd约束文件做校验。 // errorHandler是一个SimpleSaxErrorHandler对象。 return this.documentLoader.loadDocument(inputSource, getEntityResolver(), this.errorHandler, getValidationModeForResource(resource), isNamespaceAware());}看下 loadDocument12345678910111213/**** 这里就是老套路了，可以看出，Spring还是使用了dom的方式解析，即一次全部load到内存**/public Document loadDocument(InputSource inputSource, EntityResolver entityResolver, ErrorHandler errorHandler, int validationMode, boolean namespaceAware) throws Exception { DocumentBuilderFactory factory = createDocumentBuilderFactory(validationMode, namespaceAware); if (logger.isDebugEnabled()) { logger.debug(\"Using JAXP provider [\" + factory.getClass().getName() + \"]\"); } DocumentBuilder builder = createDocumentBuilder(factory, entityResolver, errorHandler); return builder.parse(inputSource);}12345678910111213141516protected DocumentBuilderFactory createDocumentBuilderFactory(int validationMode, boolean namespaceAware{ DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); factory.setNamespaceAware(namespaceAware); if (validationMode != XmlValidationModeDetector.VALIDATION_NONE) { //此方法设为true仅对dtd有效，xsd(schema)无效 factory.setValidating(true); if (validationMode == XmlValidationModeDetector.VALIDATION_XSD) { // Enforce namespace aware for XSD... //开启xsd(schema)支持 factory.setNamespaceAware(true); //这个也是Java支持Schema的套路，可以问度娘 factory.setAttribute(SCHEMA_LANGUAGE_ATTRIBUTE, XSD_SCHEMA_LANGUAGE); } } return factory;}bean解析registerBeanDefinitions瞧一下这个方法，看看做了哪些事情。12345678910public int registerBeanDefinitions(Document doc, Resource resource) throws BeanDefinitionStoreException { //根据反射的方式创建DefaultBeanDefinitionDocumentReader对象。 //这其实也是策略模式，通过setter方法可以更换其实现。修改documentReaderClass参数即可 BeanDefinitionDocumentReader documentReader = createBeanDefinitionDocumentReader(); //获取bean定义的数量 int countBefore = getRegistry().getBeanDefinitionCount(); //读取文件 documentReader.registerBeanDefinitions(doc, createReaderContext(resource)); return getRegistry().getBeanDefinitionCount() - countBefore;}createReaderContext123456789public XmlReaderContext createReaderContext(Resource resource) { // problemReporter是一个FailFastProblemReporter对象。 // eventListener是EmptyReaderEventListener对象，此类里的方法都是空实现。 // sourceExtractor是NullSourceExtractor对象，直接返回空，也是空实现。 // getNamespaceHandlerResolver默认返回DefaultNamespaceHandlerResolver对象，用来获取xsd对应的处理器。 return new XmlReaderContext(resource, this.problemReporter, this.eventListener, this.sourceExtractor, this, getNamespaceHandlerResolver());}XmlReaderContext的作用感觉就是这一堆参数的容器，糅合到一起传给DocumentReader，并美其名为Context。可以看出，Spring中到处都是策略模式，大量操作被抽象成接口。registerBeanDefinitions此方式是在DefaultBeanDefinitionDocumentReader的里面实现的。12345678910111213141516171819202122232425262728293031323334353637383940414243444546@Overridepublic void registerBeanDefinitions(Document doc, XmlReaderContext readerContext) { this.readerContext = readerContext; //获取根节点beans Element root = doc.getDocumentElement(); //注册根节点下所有的bean doRegisterBeanDefinitions(root);}``` ### doRegisterBeanDefinitions```javaprotected void doRegisterBeanDefinitions(Element root) { // Any nested <beans> elements will cause recursion in this method. In // order to propagate and preserve <beans> default-* attributes correctly, // keep track of the current (parent) delegate, which may be null. Create // the new (child) delegate with a reference to the parent for fallback purposes, // then ultimately reset this.delegate back to its original (parent) reference. // this behavior emulates a stack of delegates without actually necessitating one. BeanDefinitionParserDelegate parent = this.delegate; this.delegate = createDelegate(getReaderContext(), root, parent); // 默认的命名空间即 // http://www.springframework.org/schema/beans if (this.delegate.isDefaultNamespace(root)) { // 检查profile属性,获取profile属性 String profileSpec = root.getAttribute(PROFILE_ATTRIBUTE); if (StringUtils.hasText(profileSpec)) { // 分隔profile属性的值 ,分割 String[] specifiedProfiles = StringUtils.tokenizeToStringArray( profileSpec, BeanDefinitionParserDelegate.MULTI_VALUE_ATTRIBUTE_DELIMITERS); // 如果不是可用的profile的值，就直接返回 if (!getReaderContext().getEnvironment().acceptsProfiles(specifiedProfiles)) { return; } } } // 预处理xml方法这是个空方法， // 我们可以扩展这个方法，来加载解析我们自己的自定义标签。 preProcessXml(root); // 解析 parseBeanDefinitions(root, this.delegate); postProcessXml(root); this.delegate = parent; }parseBeanDefinitions1234567891011121314151617181920212223242526272829303132`protected void parseBeanDefinitions(Element root, BeanDefinitionParserDelegate delegate) { // 验证名称空间 // http://www.springframework.org/schema/beans if (delegate.isDefaultNamespace(root)) { NodeList nl = root.getChildNodes(); for (int i = 0; i < nl.getLength(); i++) { Node node = nl.item(i); if (node instanceof Element) { Element ele = (Element) node; // 检查节点是不是 // http://www.springframework.org/schema/beans if (delegate.isDefaultNamespace(ele)) { // 默认解析方式xml parseDefaultElement(ele, delegate); } else { //自定义解析xml，方便我们扩展的标签。原理是这样子的 //我们的context相关标签，以及我们的后面介绍的Aop标签，都是通过这个方法去扩展的。 //1.首先会根据你的namespace标签值，去选择根据namespace里面的值去map里面选择一个解析器。map里面存储的值是<namespace,resolverClassName> //2.拿到这个这个解析对象class对象，通过反射的方式创建解析， //3.调用解析器里面的resover方法，去解析扩展的标签。 delegate.parseCustomElement(ele); } } } } else { //自定义解析xml delegate.parseCustomElement(root); } }后面分两条支线阅读解析这块的核心parseDefaultElement123456789101112131415161718192021private void parseDefaultElement(Element ele, BeanDefinitionParserDelegate delegate) { // 处理 import标签 if (delegate.nodeNameEquals(ele, IMPORT_ELEMENT)) { importBeanDefinitionResource(ele); } // 处理 alais 标签 else if (delegate.nodeNameEquals(ele, ALIAS_ELEMENT)) { processAliasRegistration(ele); } // 处理 bean标签 else if (delegate.nodeNameEquals(ele, BEAN_ELEMENT)) { processBeanDefinition(ele, delegate); } // 处理beans标签 // 返回去调用doRegisterBeanDefinitions的方法 else if (delegate.nodeNameEquals(ele, NESTED_BEANS_ELEMENT)) { // recurse // 循环调用doRegisterBeanDefinitions doRegisterBeanDefinitions(ele); }}importBeanDefinitionResource处理import12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667protected void importBeanDefinitionResource(Element ele) { // 获取resource标记的路径 // <import resource=\"context:spring.xml\"/> String location = ele.getAttribute(RESOURCE_ATTRIBUTE); if (!StringUtils.hasText(location)) { getReaderContext().error(\"Resource location must not be empty\", ele); return; } // Resolve system properties: e.g. \"${user.dir}\" // 字符替换标签 location = getReaderContext().getEnvironment().resolveRequiredPlaceholders(location); Set<Resource> actualResources = new LinkedHashSet<Resource>(4); // Discover whether the location is an absolute or relative URI boolean absoluteLocation = false; try { absoluteLocation = ResourcePatternUtils.isUrl(location) || ResourceUtils.toURI(location).isAbsolute(); } catch (URISyntaxException ex) { // cannot convert to an URI, considering the location relative // unless it is the well-known Spring prefix \"classpath*:\" } // Absolute or relative? if (absoluteLocation) { try { int importCount = getReaderContext().getReader().loadBeanDefinitions(location, actualResources); if (logger.isDebugEnabled()) { logger.debug(\"Imported \" + importCount + \" bean definitions from URL location [\" + location + \"]\"); } } catch (BeanDefinitionStoreException ex) { getReaderContext().error( \"Failed to import bean definitions from URL location [\" + location + \"]\", ele, ex); } } else { // No URL -> considering resource location as relative to the current file. try { int importCount; Resource relativeResource = getReaderContext().getResource().createRelative(location); if (relativeResource.exists()) { importCount = getReaderContext().getReader().loadBeanDefinitions(relativeResource); actualResources.add(relativeResource); } else { String baseLocation = getReaderContext().getResource().getURL().toString(); importCount = getReaderContext().getReader().loadBeanDefinitions( StringUtils.applyRelativePath(baseLocation, location), actualResources); } if (logger.isDebugEnabled()) { logger.debug(\"Imported \" + importCount + \" bean definitions from relative location [\" + location + \"]\"); } } catch (IOException ex) { getReaderContext().error(\"Failed to resolve current resource location\", ele, ex); } catch (BeanDefinitionStoreException ex) { getReaderContext().error(\"Failed to import bean definitions from relative location [\" + location + \"]\", ele, ex); } } Resource[] actResArray = actualResources.toArray(new Resource[actualResources.size()]); getReaderContext().fireImportProcessed(location, actResArray, extractSource(ele));}importBeanDefinitionResource套路和之前的配置文件加载完全一样，不过注意被import进来的文件是先于当前文件被解析的。上面有些周边的代码就不介绍了。processAliasRegistration处理别名123456789101112131415161718192021222324252627protected void processAliasRegistration(Element ele) { // 拿到名字，和别名 String name = ele.getAttribute(NAME_ATTRIBUTE); String alias = ele.getAttribute(ALIAS_ATTRIBUTE); boolean valid = true; if (!StringUtils.hasText(name)) { getReaderContext().error(\"Name must not be empty\", ele); valid = false; } if (!StringUtils.hasText(alias)) { getReaderContext().error(\"Alias must not be empty\", ele); valid = false; } if (valid) { try { // 核心方法，就是在DefaultListableBeanFactor注册别名， // 其实就是在一个map里面写入名字和别名的映射关系。 getReaderContext().getRegistry().registerAlias(name, alias); } catch (Exception ex) { getReaderContext().error(\"Failed to register alias '\" + alias + \"' for bean with name '\" + name + \"'\", ele, ex); } // 触发监听器 getReaderContext().fireAliasRegistered(name, alias, extractSource(ele)); }}其实这个方法就是给一个bean取一个别名：比如有一个bean名为beanA，但是另一个组件想以beanB的名字使用，就可以这样定义:registerAlias1234567891011121314151617181920212223// 其实就是在map里加上一条映射关系。public void registerAlias(String name, String alias) { Assert.hasText(name, \"'name' must not be empty\"); Assert.hasText(alias, \"'alias' must not be empty\"); if (alias.equals(name)) { this.aliasMap.remove(alias); } else { String registeredName = this.aliasMap.get(alias); if (registeredName != null) { if (registeredName.equals(name)) { // An existing alias - no need to re-register return; } if (!allowAliasOverriding()) { throw new IllegalStateException(\"Cannot register alias '\" + alias + \"' for name '\" + name + \"': It is already registered for name '\" + registeredName + \"'.\"); } } checkForAliasCircle(name, alias); this.aliasMap.put(alias, name); }}processBeanDefinitionBeanDefinition接口继承图谱处理bean12345678910111213141516protected void processBeanDefinition(Element ele, BeanDefinitionParserDelegate delegate) { BeanDefinitionHolder bdHolder = delegate.parseBeanDefinitionElement(ele); if (bdHolder != null) { bdHolder = delegate.decorateBeanDefinitionIfRequired(ele, bdHolder); try { // Register the final decorated instance. BeanDefinitionReaderUtils.registerBeanDefinition(bdHolder, getReaderContext().getRegistry()); } catch (BeanDefinitionStoreException ex) { getReaderContext().error(\"Failed to register bean definition with name '\" + bdHolder.getBeanName() + \"'\", ele, ex); } // Send registration event. getReaderContext().fireComponentRegistered(new BeanComponentDefinition(bdHolder)); }}最后会调用BeanDefinitionParserDelegate.parseBeanDefinitionElement首先获取到id和name属性，name属性支持配置多个，以逗号分隔，如果没有指定id，那么将以第一个name属性值代替。id必须是唯一的，name属性其实是alias的角色，可以和其它的bean重复，如果name也没有配置，那么其实什么也没做。1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374public BeanDefinitionHolder parseBeanDefinitionElement(Element ele, BeanDefinition containingBean) { // 获取ID String id = ele.getAttribute(ID_ATTRIBUTE); // 获取name String nameAttr = ele.getAttribute(NAME_ATTRIBUTE); // name属性可以配置多个，用逗号分隔。 List<String> aliases = new ArrayList<String>(); if (StringUtils.hasLength(nameAttr)) { String[] nameArr = StringUtils.tokenizeToStringArray(nameAttr, MULTI_VALUE_ATTRIBUTE_DELIMITERS); aliases.addAll(Arrays.asList(nameArr)); } String beanName = id; // 如果id没有配置 就用name的数组的第一个名字代替 if (!StringUtils.hasText(beanName) && !aliases.isEmpty()) { beanName = aliases.remove(0); if (logger.isDebugEnabled()) { logger.debug(\"No XML 'id' specified - using '\" + beanName + \"' as bean name and \" + aliases + \" as aliases\"); } } if (containingBean == null) { //检查bean名字，别名是不是已经使用过了 checkNameUniqueness(beanName, aliases, ele); } // 待会我会介绍这个BeanDefinition的体系 这个方法到底干了啥？ // 1.这个方法会解析bean 的class标签，parent的标签。 // 2.然后会new一个GenericBeanDefinition，然后将class，parent的值，以及classload设置进去。 // 3.解析标签下的meta，key，value标签，把依赖的关系也设置进去。 AbstractBeanDefinition beanDefinition = parseBeanDefinitionElement(ele, beanName, containingBean); if (beanDefinition != null) { // 如果bean标签没有设置id，和name属性。 if (!StringUtils.hasText(beanName)) { try { if (containingBean != null) { beanName = BeanDefinitionReaderUtils.generateBeanName( beanDefinition, this.readerContext.getRegistry(), true); } else { // 如果bean标签没有设置id，和name属性。 // 自行创建一个名字。这里会调用BeanDefinitionReaderUtils.generateBeanName方法 beanName = this.readerContext.generateBeanName(beanDefinition); // Register an alias for the plain bean class name, if still possible, // if the generator returned the class name plus a suffix. // This is expected for Spring 1.2/2.0 backwards compatibility. // 获取beanClassName，其实就是class属性的值。 String beanClassName = beanDefinition.getBeanClassName(); // 如果名字是以className开头且没有被使用过的，就加入到别名里。 if (beanClassName != null && beanName.startsWith(beanClassName) && beanName.length() > beanClassName.length() && !this.readerContext.getRegistry().isBeanNameInUse(beanClassName)) { aliases.add(beanClassName); } } if (logger.isDebugEnabled()) { logger.debug(\"Neither XML 'id' nor 'name' specified - \" + \"using generated bean name [\" + beanName + \"]\"); } } catch (Exception ex) { error(ex.getMessage(), ele); return null; } } String[] aliasesArray = StringUtils.toStringArray(aliases); // 创建BeanDefinitionHolder类 return new BeanDefinitionHolder(beanDefinition, beanName, aliasesArray); } return null;}parseBeanDefinitionElement接着看AbstractBeanDefinition beanDefinition = parseBeanDefinitionElement(ele, beanName, containingBean);这句的具体实现：12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455public AbstractBeanDefinition parseBeanDefinitionElement( Element ele, String beanName, BeanDefinition containingBean) { //把名字进行一次压栈 this.parseState.push(new BeanEntry(beanName)); String className = null; // 获取class属性值 if (ele.hasAttribute(CLASS_ATTRIBUTE)) { className = ele.getAttribute(CLASS_ATTRIBUTE).trim(); } try { String parent = null; // 获取parent的属性值 if (ele.hasAttribute(PARENT_ATTRIBUTE)) { parent = ele.getAttribute(PARENT_ATTRIBUTE); } // 调用BeanDefinitionReaderUtils.createBeanDefinition（） // 创建GenericBeanDefinition实例，设置className，parent。 AbstractBeanDefinition bd = createBeanDefinition(className, parent); // 这个方法会解析singleton、scope、abstract、lazy-init、autowire、 // dependency-check、depends-on、init-method、autowire-candidate、 // primary、destroy-method、actory-method、factory-bean、constructor-arg // index、type、value-type、key-type、property、ref、value等标签 // 设置到 GenericBeanDefinition的实例里面。 parseBeanDefinitionAttributes(ele, beanName, containingBean, bd); // 设置描述。 bd.setDescription(DomUtils.getChildElementValueByTagName(ele, DESCRIPTION_ELEMENT)); // 解析元数据标签 parseMetaElements(ele, bd); // 解析lookup-method标签 parseLookupOverrideSubElements(ele, bd.getMethodOverrides()); // 解析replace-method标签 parseReplacedMethodSubElements(ele, bd.getMethodOverrides()); // 解析构造方法 parseConstructorArgElements(ele, bd); // 解析属性依赖 parsePropertyElements(ele, bd); // 解析Qualifier标签 parseQualifierElements(ele, bd); // 设置资源 bd.setResource(this.readerContext.getResource()); bd.setSource(extractSource(ele)); return bd; } catch (Exception ex) { ... } finally { this.parseState.pop(); } return null;}其实这里面就已经把bean的定义bean的依赖关系都设置好了。但是bean并没有被实例化。parseMetaElements12345678910111213141516public void parseMetaElements(Element ele, BeanMetadataAttributeAccessor attributeAccessor) { NodeList nl = ele.getChildNodes(); for (int i = 0; i < nl.getLength(); i++) { Node node = nl.item(i); if (isCandidateElement(node) && nodeNameEquals(node, META_ELEMENT)) { Element metaElement = (Element) node; String key = metaElement.getAttribute(KEY_ATTRIBUTE); String value = metaElement.getAttribute(VALUE_ATTRIBUTE); //就是一个key, value的载体，无他 BeanMetadataAttribute attribute = new BeanMetadataAttribute(key, value); //sourceExtractor默认是NullSourceExtractor，返回的是空 attribute.setSource(extractSource(metaElement)); attributeAccessor.addMetadataAttribute(attribute); } }}AbstractBeanDefinition继承自BeanMetadataAttributeAccessor类，底层使用了一个LinkedHashMap保存metadata。这个metadata具体是做什么暂时还不知道。我们实际应用中meta标签也很少见。例子：123<bean id=\"b\" name=\"one, two\" class=\"base.SimpleBean\"> <meta key=\"name\" value=\"dsfesf\"/></bean>parseLookupOverrideSubElements123456789101112131415public void parseLookupOverrideSubElements(Element beanEle, MethodOverrides overrides) { NodeList nl = beanEle.getChildNodes(); for (int i = 0; i < nl.getLength(); i++) { Node node = nl.item(i); if (isCandidateElement(node) && nodeNameEquals(node, LOOKUP_METHOD_ELEMENT)) { Element ele = (Element) node; String methodName = ele.getAttribute(NAME_ATTRIBUTE); String beanRef = ele.getAttribute(BEAN_ELEMENT); //以MethodOverride的方式，存放在set集合里面 LookupOverride override = new LookupOverride(methodName, beanRef); override.setSource(extractSource(ele)); overrides.addOverride(override); } }}此标签的作用在于当一个bean的某个方法被设置为lookup-method后，每次调用此方法时，都会返回一个新的指定bean的对象。例如：12345<bean id=\"apple\" class=\"a.b.c.Apple\" scope=\"prototype\"/><!--水果盘--><bean id=\"fruitPlate\" class=\"a.b.c.FruitPlate\"> <lookup-method name=\"getFruit\" bean=\"apple\"/></bean>parseReplacedMethodSubElements123456789101112131415161718192021222324252627public void parseReplacedMethodSubElements(Element beanEle, MethodOverrides overrides) { NodeList nl = beanEle.getChildNodes(); for (int i = 0; i < nl.getLength(); i++) { Node node = nl.item(i); if (isCandidateElement(node) && nodeNameEquals(node, REPLACED_METHOD_ELEMENT)) { Element replacedMethodEle = (Element) node; //获取name属性 String name = replacedMethodEle.getAttribute(NAME_ATTRIBUTE); //获取replace-method属性 String callback = replacedMethodEle.getAttribute(REPLACER_ATTRIBUTE); ReplaceOverride replaceOverride = new ReplaceOverride(name, callback); // Look for arg-type match elements. // 获取所有的 arg-type的标签 // 遍历所有节点，找到匹配的。以ReplaceOverride结构存储到list里面 List<Element> argTypeEles = DomUtils.getChildElementsByTagName(replacedMethodEle, ARG_TYPE_ELEMENT); for (Element argTypeEle : argTypeEles) { String match = argTypeEle.getAttribute(ARG_TYPE_MATCH_ATTRIBUTE); match = (StringUtils.hasText(match) ? match : DomUtils.getTextValue(argTypeEle)); if (StringUtils.hasText(match)) { replaceOverride.addTypeIdentifier(match); } } replaceOverride.setSource(extractSource(replacedMethodEle)); overrides.addOverride(replaceOverride); } }}replace-method 主要作用就是替换方法体及其返回值，使用比较简单。只需要实现MethodReplacer接口，并重写reimplement方法，然后就能完成方法的替换。这个有点类似aop的功能实现场景用的地方不是太多。例子：1234567<!-- ====================replace-method属性注入==================== --><bean id=\"dogReplaceMethod\" class=\"com.lyc.cn.v2.day01.method.replaceMethod.ReplaceDog\"/><bean id=\"originalDogReplaceMethod\" class=\"com.lyc.cn.v2.day01.method.replaceMethod.OriginalDog\"> <replaced-method name=\"sayHello\" replacer=\"dogReplaceMethod\"> <arg-type match=\"java.lang.String\"></arg-type> </replaced-method></bean>parseConstructorArgElements解析构造方法。构造方法注入12345<bean class="base.SimpleBean"> <constructor-arg> <value type="java.lang.String">Cat</value> </constructor-arg></bean>12345678910public void parseConstructorArgElements(Element beanEle, BeanDefinition bd) { NodeList nl = beanEle.getChildNodes(); for (int i = 0; i < nl.getLength(); i++) { Node node = nl.item(i); if (isCandidateElement(node) && nodeNameEquals(node, CONSTRUCTOR_ARG_ELEMENT)) { parseConstructorArgElement((Element) node, bd); } }}看看调用方法parseConstructorArgElement1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465public void parseConstructorArgElement(Element ele, BeanDefinition bd) { String indexAttr = ele.getAttribute(INDEX_ATTRIBUTE); String typeAttr = ele.getAttribute(TYPE_ATTRIBUTE); String nameAttr = ele.getAttribute(NAME_ATTRIBUTE); //按照index的方式设置值 if (StringUtils.hasLength(indexAttr)) { try { int index = Integer.parseInt(indexAttr); if (index < 0) { error(\"'index' cannot be lower than 0\", ele); } else { try { //ConstructorArgumentEntry其实存的就是index值 this.parseState.push(new ConstructorArgumentEntry(index)); //获取value标签的值 Object value = parsePropertyValue(ele, bd, null); // ConstructorArgumentValues.valueHolder存储value值 // ConstructorArgumentValues.ValueHolder valueHolder = new ConstructorArgumentValues.ValueHolder(value); if (StringUtils.hasLength(typeAttr)) { valueHolder.setType(typeAttr); } if (StringUtils.hasLength(nameAttr)) { valueHolder.setName(nameAttr); } valueHolder.setSource(extractSource(ele)); //判断索引值是不是已经用过了。其实是检查的Map<index,ValueHolder>的key是否存在。 if (bd.getConstructorArgumentValues().hasIndexedArgumentValue(index)) { error(\"Ambiguous constructor-arg entries for index \" + index, ele); } else { //把valueHolder加入到map里面 bd.getConstructorArgumentValues().addIndexedArgumentValue(index, valueHolder); } } finally { this.parseState.pop(); } } } catch (NumberFormatException ex) { error(\"Attribute 'index' of tag 'constructor-arg' must be an integer\", ele); } } //如果不是用index 就把名字相关信息加入ValueHOlde中存储到List里面。 else { try { this.parseState.push(new ConstructorArgumentEntry()); Object value = parsePropertyValue(ele, bd, null); ConstructorArgumentValues.ValueHolder valueHolder = new ConstructorArgumentValues.ValueHolder(value); if (StringUtils.hasLength(typeAttr)) { valueHolder.setType(typeAttr); } if (StringUtils.hasLength(nameAttr)) { valueHolder.setName(nameAttr); } valueHolder.setSource(extractSource(ele)); bd.getConstructorArgumentValues().addGenericArgumentValue(valueHolder); } finally { this.parseState.pop(); } }}parsePropertyElements解析property，普通属性注入相关的配置的方法：12parseQualifierElements解析Qulifier标签。Qualifier标签能在我们注入的时候选择指定的注入值一般情况下和AutoWire标签使用的情况比较多，常见的@AutoWire注解上添加上@Qualifier选择合适的注入者如：12345<bean id="animal" class="test.constructor.Animal"> //指定类型为test.qualifier.Person,id 为student的bean注入 <qualifier type="test.qualifier.Person" value="student"></qualifier></bean><bean id="student" class="test.qualifier.Person"></bean>123456789public void parseQualifierElements(Element beanEle, AbstractBeanDefinition bd) { NodeList nl = beanEle.getChildNodes(); for (int i = 0; i < nl.getLength(); i++) { Node node = nl.item(i); if (isCandidateElement(node) && nodeNameEquals(node, QUALIFIER_ELEMENT)) { parseQualifierElement((Element) node, bd); } }}12345678910111213141516171819202122232425262728293031323334353637383940414243444546public void parseQualifierElement(Element ele, AbstractBeanDefinition bd) { String typeName = ele.getAttribute(TYPE_ATTRIBUTE); if (!StringUtils.hasLength(typeName)) { error(\"Tag 'qualifier' must have a 'type' attribute\", ele); return; } //QualifierEntry 存放的就是class的类型即type的名字，或者class全限定名字 //如 a.b.c.person this.parseState.push(new QualifierEntry(typeName)); try { //根据类型去 AutowireCandidateQualifier qualifier = new AutowireCandidateQualifier(typeName); qualifier.setSource(extractSource(ele)); String value = ele.getAttribute(VALUE_ATTRIBUTE); if (StringUtils.hasLength(value)) { //这里是存放在一个Map<String, Object>结构里， 其中key是value，value的值是BeanMetadataAttribute(name, value)对象。 qualifier.setAttribute(AutowireCandidateQualifier.VALUE_KEY, value); } NodeList nl = ele.getChildNodes(); for (int i = 0; i < nl.getLength(); i++) { Node node = nl.item(i); //如果qualifier标签下还有attribute标签 //就解析对应的标签值，用BeanMetadataAttribute封装，放到AutowireCandidateQualifier对象里面。 if (isCandidateElement(node) && nodeNameEquals(node, QUALIFIER_ATTRIBUTE_ELEMENT)) { Element attributeEle = (Element) node; String attributeName = attributeEle.getAttribute(KEY_ATTRIBUTE); String attributeValue = attributeEle.getAttribute(VALUE_ATTRIBUTE); if (StringUtils.hasLength(attributeName) && StringUtils.hasLength(attributeValue)) { BeanMetadataAttribute attribute = new BeanMetadataAttribute(attributeName, attributeValue); attribute.setSource(extractSource(attributeEle)); qualifier.addMetadataAttribute(attribute); } else { error(\"Qualifier 'attribute' tag must have a 'name' and 'value'\", attributeEle); return; } } } bd.addQualifier(qualifier); } finally { this.parseState.pop(); }}看看AutowireCandidateQualifier的继承图谱：阅读到这里parseDefaultElement这一条线就看完了。下面看自定义解析这条线，spring为何能支持很多其他的标签，比如属性标签p，比如context标签c、比如Aop标签。都是通过这个自定义解析才能得以实现。在配置文件解析这块思想做到了极致。以至于，很多其他框架都借鉴了这块的思想。parseCustomElement自定义解析，其实最终是通过加载自定义解析器去解析的。现在咱们来一探究尽parseCustomElement12345678910111213public BeanDefinition parseCustomElement(Element ele, BeanDefinition containingBd) { // 获取到命名空间值 String namespaceUri = getNamespaceURI(ele); // 拿到名称空间处理器 // 1.拿到解析上下文中NamespaceHandlerResolver的DefaultNamespaceHandlerResolver实例 // 2.解析处理一下命名空间，拿到具体的NamespaceHandler NamespaceHandler handler = this.readerContext.getNamespaceHandlerResolver().resolve(namespaceUri); if (handler == null) { error("Unable to locate Spring NamespaceHandler for XML schema namespace [" + namespaceUri + "]", ele); return null; } return handler.parse(ele, new ParserContext(this.readerContext, this, containingBd));}DefaultNamespaceHandlerResolver123public DefaultNamespaceHandlerResolver(ClassLoader classLoader) { this(classLoader, DEFAULT_HANDLER_MAPPINGS_LOCATION);}其中DEFAULT_HANDLER_MAPPINGS_LOCATION=”META-INF/spring.handlers”;找一个实例看下”META-INF/spring.handlers”下面有什么东西，如spring-beans项目里的p标签和c标签。123http\\://www.springframework.org/schema/c=org.springframework.beans.factory.xml.SimpleConstructorNamespaceHandlerhttp\\://www.springframework.org/schema/p=org.springframework.beans.factory.xml.SimplePropertyNamespaceHandlerhttp\\://www.springframework.org/schema/util=org.springframework.beans.factory.xml.UtilNamespaceHandlerDefaultNamespaceHandlerResolver.resolver12345678910111213141516171819202122232425262728293031323334353637383940public NamespaceHandler resolve(String namespaceUri) { // 加载配置文件里配置好的NamespaceHandler。烂加载模式。 Map<String, Object> handlerMappings = getHandlerMappings(); // 通过namespaceUri去map表里面找对应的handler Object handlerOrClassName = handlerMappings.get(namespaceUri); if (handlerOrClassName == null) { return null; } //如果是NamespaceHandler类型就直接返回了， else if (handlerOrClassName instanceof NamespaceHandler) { return (NamespaceHandler) handlerOrClassName; } //如果不是就根据class对象，用反射的方式去加载对应handler //配置文件一般都是配置的class全限定名， //如果是第一次解析对应标签，会执行下面的逻辑，nameHandler初始化之后，会缓存起来供下次使用。 else { String className = (String) handlerOrClassName; try { Class<?> handlerClass = ClassUtils.forName(className, this.classLoader); if (!NamespaceHandler.class.isAssignableFrom(handlerClass)) { throw new FatalBeanException(\"Class [\" + className + \"] for namespace [\" + namespaceUri + \"] does not implement the [\" + NamespaceHandler.class.getName() + \"] interface\"); } NamespaceHandler namespaceHandler = (NamespaceHandler) // 根据class定义实例化对象 BeanUtils.instantiateClass(handlerClass); namespaceHandler.init(); handlerMappings.put(namespaceUri, namespaceHandler); return namespaceHandler; } catch (ClassNotFoundException ex) { throw new FatalBeanException(\"NamespaceHandler class [\" + className + \"] for namespace [\" + namespaceUri + \"] not found\", ex); } catch (LinkageError err) { throw new FatalBeanException(\"Invalid NamespaceHandler class [\" + className + \"] for namespace [\" + namespaceUri + \"]: problem with handler class file or dependent class\", err); } }}1234567891011121314151617181920212223/** * Load the specified NamespaceHandler mappings lazily. */private Map<String, Object> getHandlerMappings() { //如果没有被加载就加载，这里判断两次为null就是为了在多线程情况下的并发加载的问题。 if (this.handlerMappings == null) { synchronized (this) { if (this.handlerMappings == null) { try { Properties mappings = PropertiesLoaderUtils.loadAllProperties(this.handlerMappingsLocation, this.classLoader); Map<String, Object> handlerMappings = new ConcurrentHashMap<String, Object>(mappings.size()); CollectionUtils.mergePropertiesIntoMap(mappings, handlerMappings); this.handlerMappings = handlerMappings; } catch (IOException ex) { ... } } } } return this.handlerMappings;}NamespaceHandler.parse这里就是个性化的方法了，因为每个NamespaceHanlder处理的标签都不一祥。重点看一下这个实现：NamespaceHandlerSupport这里面的实现，因为很多NamespaceHanlder都是继承的这个抽象类：如图：123public BeanDefinition parse(Element element, ParserContext parserContext) { return findParserForElement(element, parserContext).parse(element, parserContext);}123456789private BeanDefinitionParser findParserForElement(Element element, ParserContext parserContext) { String localName = parserContext.getDelegate().getLocalName(element); BeanDefinitionParser parser = this.parsers.get(localName); if (parser == null) { parserContext.getReaderContext().fatal( \"Cannot locate BeanDefinitionParser for element [\" + localName + \"]\", element); } return parser;}以上就是configuration 配置文件的加载过程了。下一章阅读分享bean的注册以及bean的实例化。bean 实例化查看processBeanDefinition1234567891011121314151617181920protected void processBeanDefinition(Element ele, BeanDefinitionParserDelegate delegate) { BeanDefinitionHolder bdHolder = delegate.parseBeanDefinitionElement(ele); if (bdHolder != null) { // 装饰bean装饰哪些不是默认namespace的那些bean // 1.去META-INF/spring.handlers下加载namespaceHandler // 2.根据namespace的的值拿到对应的namespaceHandler // 3.获取namespaceHandler的实例，然后调用namespaceHandler的decorate的方法 bdHolder = delegate.decorateBeanDefinitionIfRequired(ele, bdHolder); try { // Register the final decorated instance. BeanDefinitionReaderUtils.registerBeanDefinition(bdHolder, getReaderContext().getRegistry()); } catch (BeanDefinitionStoreException ex) { getReaderContext().error(\"Failed to register bean definition with name '\" + bdHolder.getBeanName() + \"'\", ele, ex); } // Send registration event. getReaderContext().fireComponentRegistered(new BeanComponentDefinition(bdHolder)); }}bean装饰delegate.decorateBeanDefinitionIfRequired这一块和之前讲的自定义标签解析是一样的实现。都是用策略模式实现的，都是通过去找namespaceHandler接口，然后实例化，调用装饰方法，完成个性化的操作。这里就不再贴代码详细介绍了123456789101112131415161718192021222324public BeanDefinitionHolder decorateBeanDefinitionIfRequired( Element ele, BeanDefinitionHolder definitionHolder, BeanDefinition containingBd) { BeanDefinitionHolder finalDefinition = definitionHolder; // Decorate based on custom attributes first. // 装饰自定义属性标签 NamedNodeMap attributes = ele.getAttributes(); for (int i = 0; i < attributes.getLength(); i++) { Node node = attributes.item(i); finalDefinition = decorateIfRequired(node, finalDefinition, containingBd); } // Decorate based on custom nested elements. // 装饰自定义节点标签 NodeList children = ele.getChildNodes(); for (int i = 0; i < children.getLength(); i++) { Node node = children.item(i); if (node.getNodeType() == Node.ELEMENT_NODE) { finalDefinition = decorateIfRequired(node, finalDefinition, containingBd); } } return finalDefinition;}bean的注册1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586@Override public void registerBeanDefinition(String beanName, BeanDefinition beanDefinition) throws BeanDefinitionStoreException { //前面介绍过了bean定义的加载 if (beanDefinition instanceof AbstractBeanDefinition) { try { //校验 ((AbstractBeanDefinition) beanDefinition).validate(); } catch (BeanDefinitionValidationException ex) { throw new BeanDefinitionStoreException(beanDefinition.getResourceDescription(), beanName, \"Validation of bean definition failed\", ex); } } BeanDefinition oldBeanDefinition; oldBeanDefinition = this.beanDefinitionMap.get(beanName); if (oldBeanDefinition != null) { if (!isAllowBeanDefinitionOverriding()) { // 如果不允许重复定义就抛出异常 ... } //如果可以被覆盖就比较一下 else if (oldBeanDefinition.getRole() < beanDefinition.getRole()) { // e.g. was ROLE_APPLICATION, now overriding with ROLE_SUPPORT or ROLE_INFRASTRUCTURE if (this.logger.isWarnEnabled()) { this.logger.warn(\"Overriding user-defined bean definition for bean '\" + beanName + \"' with a framework-generated bean definition: replacing [\" + oldBeanDefinition + \"] with [\" + beanDefinition + \"]\"); } } else if (!beanDefinition.equals(oldBeanDefinition)) { if (this.logger.isInfoEnabled()) { this.logger.info(\"Overriding bean definition for bean '\" + beanName + \"' with a different definition: replacing [\" + oldBeanDefinition + \"] with [\" + beanDefinition + \"]\"); } } else { if (this.logger.isDebugEnabled()) { this.logger.debug(\"Overriding bean definition for bean '\" + beanName + \"' with an equivalent definition: replacing [\" + oldBeanDefinition + \"] with [\" + beanDefinition + \"]\"); } } // 覆盖原来的bean this.beanDefinitionMap.put(beanName, beanDefinition); } else { if (hasBeanCreationStarted()) { // Cannot modify startup-time collection elements anymore (for stable iteration) synchronized (this.beanDefinitionMap) { // 添加到beanDefinitionMap里面 this.beanDefinitionMap.put(beanName, beanDefinition); List<String> updatedDefinitions = new ArrayList<String>(this.beanDefinitionNames.size() + 1); updatedDefinitions.addAll(this.beanDefinitionNames); updatedDefinitions.add(beanName); // 把beanNanme添加到list里面 this.beanDefinitionNames = updatedDefinitions; // 如果原来单例bean里面 if (this.manualSingletonNames.contains(beanName)) { // 移除掉beanName 这里为什么要移除掉？ // 留坑 Set<String> updatedSingletons = new LinkedHashSet<String>(this.manualSingletonNames); updatedSingletons.remove(beanName); this.manualSingletonNames = updatedSingletons; } } } else { // Still in startup registration phase // 这里表示是启动的时候缓存相关信息 this.beanDefinitionMap.put(beanName, beanDefinition); this.beanDefinitionNames.add(beanName); this.manualSingletonNames.remove(beanName); } this.frozenBeanDefinitionNames = null; } if (oldBeanDefinition != null || containsSingleton(beanName)) { //重置一下 resetBeanDefinition(beanName); }}bean的实例化123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566public void preInstantiateSingletons() throws BeansException { //获取所有的beanName List<String> beanNames = new ArrayList<String>(this.beanDefinitionNames); // Trigger initialization of all non-lazy singleton beans... for (String beanName : beanNames) { // Bean定义公共的抽象类是AbstractBeanDefinition， // 普通的Bean在Spring加载Bean定义的时候， // 实例化出来的是GenericBeanDefinition， // 而Spring上下文包括实例化所有Bean用的AbstractBeanDefinition是RootBeanDefinition， // 这时候就使用getMergedLocalBeanDefinition方法做了一次转化， // 将非RootBeanDefinition转换为RootBeanDefinition以供后续操作。 RootBeanDefinition bd = getMergedLocalBeanDefinition(beanName); if (!bd.isAbstract() && bd.isSingleton() && !bd.isLazyInit()) { //判断是不是FactoryBean //FactoryBean是一种特殊的bean。可以通过&beanName的去拿到真实的bean //其实就是调用的FactoryBean接口的getObject方法 if (isFactoryBean(beanName)) { final FactoryBean<?> factory = (FactoryBean<?>) getBean(FACTORY_BEAN_PREFIX + beanName); boolean isEagerInit; if (System.getSecurityManager() != null && factory instanceof SmartFactoryBean) { isEagerInit = AccessController.doPrivileged(new PrivilegedAction<Boolean>() { @Override public Boolean run() { return ((SmartFactoryBean<?>) factory).isEagerInit(); } }, getAccessControlContext()); } else { isEagerInit = (factory instanceof SmartFactoryBean && ((SmartFactoryBean<?>) factory).isEagerInit()); } //如果被标记为EagerInit就需要被立即初始化 if (isEagerInit) { getBean(beanName); } } else { //如果是是单例bean就初始化 getBean(beanName); } } } // 初始化完成后调用post-initialization callback回调方法。 // Trigger post-initialization callback for all applicable beans... for (String beanName : beanNames) { Object singletonInstance = getSingleton(beanName); //如果继承了SmartInitializingSingleton接口所有单例bean都会执行回调方法afterSingletonsInstantiated if (singletonInstance instanceof SmartInitializingSingleton) { final SmartInitializingSingleton smartSingleton = (SmartInitializingSingleton) singletonInstance; if (System.getSecurityManager() != null) { AccessController.doPrivileged(new PrivilegedAction<Object>() { @Override public Object run() { smartSingleton.afterSingletonsInstantiated(); return null; } }, getAccessControlContext()); } else { smartSingleton.afterSingletonsInstantiated(); } } }}实例化操作是在getbean方法里面完成的。getbean1234@Overridepublic Object getBean(String name) throws BeansException { return doGetBean(name, null, null, false);}doGetBean这个方法有点长，一些日志代码我就干掉了，保留核心代码123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136protected <T> T doGetBean( final String name, final Class<T> requiredType, final Object[] args, boolean typeCheckOnly) throws BeansException { // 获取真实的beanName final String beanName = transformedBeanName(name); Object bean; // Eagerly check singleton cache for manually registered singletons. // 获取单例bean的实例 Object sharedInstance = getSingleton(beanName); if (sharedInstance != null && args == null) { // bean = getObjectForBeanInstance(sharedInstance, name, beanName, null); } else { // Fail if we're already creating this bean instance: // We're assumably within a circular reference. if (isPrototypeCurrentlyInCreation(beanName)) { throw new BeanCurrentlyInCreationException(beanName); } // Check if bean definition exists in this factory. BeanFactory parentBeanFactory = getParentBeanFactory(); if (parentBeanFactory != null && !containsBeanDefinition(beanName)) { // Not found -> check parent. String nameToLookup = originalBeanName(name); if (args != null) { // Delegation to parent with explicit args. return (T) parentBeanFactory.getBean(nameToLookup, args); } else { // No args -> delegate to standard getBean method. return parentBeanFactory.getBean(nameToLookup, requiredType); } } //类型检查 if (!typeCheckOnly) { //把beanName标记为已经创建，其实就是往alreadyCreated set 集合里面写一条数据 markBeanAsCreated(beanName); } try { //转换类型 final RootBeanDefinition mbd = getMergedLocalBeanDefinition(beanName); checkMergedBeanDefinition(mbd, beanName, args); // Guarantee initialization of beans that the current bean depends on. // 检查depends-on 依赖 String[] dependsOn = mbd.getDependsOn(); if (dependsOn != null) { for (String dependsOnBean : dependsOn) { if (isDependent(beanName, dependsOnBean)) { ... } //注册依赖的bean registerDependentBean(dependsOnBean, beanName); //实例化依赖的bean getBean(dependsOnBean); } } // Create bean instance. if (mbd.isSingleton()) { sharedInstance = getSingleton(beanName, new ObjectFactory<Object>() { @Override public Object getObject() throws BeansException { try { //创建bean的实例 return createBean(beanName, mbd, args); } catch (BeansException ex) { destroySingleton(beanName); throw ex; } } }); bean = getObjectForBeanInstance(sharedInstance, name, beanName, mbd); } //实例化多例bean else if (mbd.isPrototype()) { // It's a prototype -> create a new instance. Object prototypeInstance = null; try { beforePrototypeCreation(beanName); prototypeInstance = createBean(beanName, mbd, args); } finally { afterPrototypeCreation(beanName); } bean = getObjectForBeanInstance(prototypeInstance, name, beanName, mbd); } else { String scopeName = mbd.getScope(); final Scope scope = this.scopes.get(scopeName); if (scope == null) { throw new IllegalStateException(\"No Scope registered for scope name '\" + scopeName + \"'\"); } try { Object scopedInstance = scope.get(beanName, new ObjectFactory<Object>() { @Override public Object getObject() throws BeansException { beforePrototypeCreation(beanName); try { return createBean(beanName, mbd, args); } finally { afterPrototypeCreation(beanName); } } }); bean = getObjectForBeanInstance(scopedInstance, name, beanName, mbd); } catch (IllegalStateException ex) { ... } } } catch (BeansException ex) { cleanupAfterBeanCreationFailure(beanName); throw ex; } } // Check if required type matches the type of the actual bean instance. if (requiredType != null && bean != null && !requiredType.isAssignableFrom(bean.getClass())) { try { return getTypeConverter().convertIfNecessary(bean, requiredType); } catch (TypeMismatchException ex) { ... } } return (T) bean;}12345678910111213141516171819202122232425262728293031protected Object getObjectForBeanInstance( Object beanInstance, String name, String beanName, RootBeanDefinition mbd) { // Don't let calling code try to dereference the factory if the bean isn't a factory. if (BeanFactoryUtils.isFactoryDereference(name) && !(beanInstance instanceof FactoryBean)) { throw new BeanIsNotAFactoryException(transformedBeanName(name), beanInstance.getClass()); } // Now we have the bean instance, which may be a normal bean or a FactoryBean. // If it's a FactoryBean, we use it to create a bean instance, unless the // caller actually wants a reference to the factory. if (!(beanInstance instanceof FactoryBean) || BeanFactoryUtils.isFactoryDereference(name)) { return beanInstance; } Object object = null; if (mbd == null) { object = getCachedObjectForFactoryBean(beanName); } if (object == null) { // Return bean instance from factory. FactoryBean<?> factory = (FactoryBean<?>) beanInstance; // Caches object obtained from FactoryBean if it is a singleton. if (mbd == null && containsBeanDefinition(beanName)) { mbd = getMergedLocalBeanDefinition(beanName); } boolean synthetic = (mbd != null && mbd.isSynthetic()); object = getObjectFromFactoryBean(factory, beanName, !synthetic); } return object;}createBeancreateBean方法是AbstractBeanFactory的子类AbstractAutowireCapableBeanFactory的一个方法，看一下它的方法实现：12345678910111213141516171819202122232425262728293031323334353637protected Object createBean(final String beanName, final RootBeanDefinition mbd, final Object[] args) throws BeanCreationException { if (logger.isDebugEnabled()) { logger.debug(\"Creating instance of bean '\" + beanName + \"'\"); } // Make sure bean class is actually resolved at this point. resolveBeanClass(mbd, beanName); // Prepare method overrides. try { mbd.prepareMethodOverrides(); } catch (BeanDefinitionValidationException ex) { throw new BeanDefinitionStoreException(mbd.getResourceDescription(), beanName, \"Validation of method overrides failed\", ex); } try { // Give BeanPostProcessors a chance to return a proxy instead of the target bean instance. // 这里是spring提供一些接口扩展，这里会触发所有的BeanPostProcessor前置处理器和后置处理器 Object bean = resolveBeforeInstantiation(beanName, mbd); if (bean != null) { return bean; } } catch (Throwable ex) { throw new BeanCreationException(mbd.getResourceDescription(), beanName, \"BeanPostProcessor before instantiation of bean failed\", ex); } Object beanInstance = doCreateBean(beanName, mbd, args); if (logger.isDebugEnabled()) { logger.debug(\"Finished creating instance of bean '\" + beanName + \"'\"); } return beanInstance;}重要的方法是doCreateBeandoCreateBean123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899protected Object doCreateBean(final String beanName, final RootBeanDefinition mbd, final Object[] args) { // Instantiate the bean. BeanWrapper instanceWrapper = null; // 如果是单例bean if (mbd.isSingleton()) { //首先从factoryBean的缓存里面找factoryBean的包装类。 instanceWrapper = this.factoryBeanInstanceCache.remove(beanName); } if (instanceWrapper == null) { // 如果 factoryBean缓存里面没有对应名字的factoryBean就自行创建。 instanceWrapper = createBeanInstance(beanName, mbd, args); } // 从instanceWrapper赋值 final Object bean = (instanceWrapper != null ? instanceWrapper.getWrappedInstance() : null); Class<?> beanType = (instanceWrapper != null ? instanceWrapper.getWrappedClass() : null); // Allow post-processors to modify the merged bean definition. // 这个也可以说是spring的扩展接口了，只需要继承MergedBeanDefinitionPostProcessor接口，就可以完成对beanDefinition的一些修改。 synchronized (mbd.postProcessingLock) { if (!mbd.postProcessed) { applyMergedBeanDefinitionPostProcessors(mbd, beanType, beanName); mbd.postProcessed = true; } } // Eagerly cache singletons to be able to resolve circular references // even when triggered by lifecycle interfaces like BeanFactoryAware. boolean earlySingletonExposure = (mbd.isSingleton() && this.allowCircularReferences && isSingletonCurrentlyInCreation(beanName)); if (earlySingletonExposure) { if (logger.isDebugEnabled()) { logger.debug(\"Eagerly caching bean '\" + beanName + \"' to allow for resolving potential circular references\"); } addSingletonFactory(beanName, new ObjectFactory<Object>() { @Override public Object getObject() throws BeansException { return getEarlyBeanReference(beanName, mbd, bean); } }); } // Initialize the bean instance. Object exposedObject = bean; try { //关键方法 //依赖注入 populateBean(beanName, mbd, instanceWrapper); if (exposedObject != null) { exposedObject = initializeBean(beanName, exposedObject, mbd); } } catch (Throwable ex) { if (ex instanceof BeanCreationException && beanName.equals(((BeanCreationException) ex).getBeanName())) { throw (BeanCreationException) ex; } else { throw new BeanCreationException(mbd.getResourceDescription(), beanName, \"Initialization of bean failed\", ex); } } if (earlySingletonExposure) { Object earlySingletonReference = getSingleton(beanName, false); if (earlySingletonReference != null) { if (exposedObject == bean) { exposedObject = earlySingletonReference; } else if (!this.allowRawInjectionDespiteWrapping && hasDependentBean(beanName)) { String[] dependentBeans = getDependentBeans(beanName); Set<String> actualDependentBeans = new LinkedHashSet<String>(dependentBeans.length); for (String dependentBean : dependentBeans) { if (!removeSingletonIfCreatedForTypeCheckOnly(dependentBean)) { actualDependentBeans.add(dependentBean); } } if (!actualDependentBeans.isEmpty()) { throw new BeanCurrentlyInCreationException(beanName, \"Bean with name '\" + beanName + \"' has been injected into other beans [\" + StringUtils.collectionToCommaDelimitedString(actualDependentBeans) + \"] in its raw version as part of a circular reference, but has eventually been \" + \"wrapped. This means that said other beans do not use the final version of the \" + \"bean. This is often the result of over-eager type matching - consider using \" + \"'getBeanNamesOfType' with the 'allowEagerInit' flag turned off, for example.\"); } } } } // Register bean as disposable. try { registerDisposableBeanIfNecessary(beanName, bean, mbd); } catch (BeanDefinitionValidationException ex) { throw new BeanCreationException(mbd.getResourceDescription(), beanName, \"Invalid destruction signature\", ex); } return exposedObject; }###123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051protected BeanWrapper createBeanInstance(String beanName, RootBeanDefinition mbd, Object[] args) { // Make sure bean class is actually resolved at this point. Class<?> beanClass = resolveBeanClass(mbd, beanName); if (beanClass != null && !Modifier.isPublic(beanClass.getModifiers()) && !mbd.isNonPublicAccessAllowed()) { throw new BeanCreationException(mbd.getResourceDescription(), beanName, \"Bean class isn't public, and non-public access not allowed: \" + beanClass.getName()); } // 如果有工厂方法，factoryMethod 就调用factroyMethod创建 if (mbd.getFactoryMethodName() != null) { return instantiateUsingFactoryMethod(beanName, mbd, args); } // Shortcut when re-creating the same bean... boolean resolved = false; boolean autowireNecessary = false; // 这里会检查bean里面的依赖注入 if (args == null) { synchronized (mbd.constructorArgumentLock) { //检查是否有构造方法或者工厂方法 if (mbd.resolvedConstructorOrFactoryMethod != null) { resolved = true; autowireNecessary = mbd.constructorArgumentsResolved; } } } //true if (resolved) { if (autowireNecessary) { // 就采用构造方法生成bean return autowireConstructor(beanName, mbd, null, null); } else { //否则就是属性方法生成bean return instantiateBean(beanName, mbd); } } // Need to determine the constructor... Constructor<?>[] ctors = determineConstructorsFromBeanPostProcessors(beanClass, beanName); if (ctors != null || mbd.getResolvedAutowireMode() == RootBeanDefinition.AUTOWIRE_CONSTRUCTOR || mbd.hasConstructorArgumentValues() || !ObjectUtils.isEmpty(args)) { return autowireConstructor(beanName, mbd, ctors, args); } // No special handling: simply use no-arg constructor. // 这里首先通过无参数的方法创建实例 // 然后通过cglib代理的方式创建属性实例，用属性的方式注入到bean里面 return instantiateBean(beanName, mbd); }继续：autowireConstructor12345protected BeanWrapper autowireConstructor( String beanName, RootBeanDefinition mbd, Constructor<?>[] ctors, Object[] explicitArgs) { return new ConstructorResolver(this).autowireConstructor(beanName, mbd, ctors, explicitArgs);}autowireConstructor这个方法比较长：123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190public BeanWrapper autowireConstructor(final String beanName, final RootBeanDefinition mbd, Constructor<?>[] chosenCtors, final Object[] explicitArgs) { //初始化BeanWrapperImpl BeanWrapperImpl bw = new BeanWrapperImpl(); //设置对应的类型转换器 this.beanFactory.initBeanWrapper(bw); Constructor<?> constructorToUse = null; ArgumentsHolder argsHolderToUse = null; Object[] argsToUse = null; if (explicitArgs != null) { argsToUse = explicitArgs; } else { Object[] argsToResolve = null; synchronized (mbd.constructorArgumentLock) { constructorToUse = (Constructor<?>) mbd.resolvedConstructorOrFactoryMethod; if (constructorToUse != null && mbd.constructorArgumentsResolved) { // Found a cached constructor... argsToUse = mbd.resolvedConstructorArguments; if (argsToUse == null) { argsToResolve = mbd.preparedConstructorArguments; } } } if (argsToResolve != null) { argsToUse = resolvePreparedArguments(beanName, mbd, bw, constructorToUse, argsToResolve); } } if (constructorToUse == null) { // Need to resolve the constructor. boolean autowiring = (chosenCtors != null || mbd.getResolvedAutowireMode() == RootBeanDefinition.AUTOWIRE_CONSTRUCTOR); ConstructorArgumentValues resolvedValues = null; int minNrOfArgs; if (explicitArgs != null) { minNrOfArgs = explicitArgs.length; } else { //获取构造函数的参数 ConstructorArgumentValues cargs = mbd.getConstructorArgumentValues(); resolvedValues = new ConstructorArgumentValues(); minNrOfArgs = resolveConstructorArguments(beanName, mbd, bw, cargs, resolvedValues); } // Take specified constructors, if any. Constructor<?>[] candidates = chosenCtors; if (candidates == null) { //拿到class Class<?> beanClass = mbd.getBeanClass(); try { //校验 candidates = (mbd.isNonPublicAccessAllowed() ? beanClass.getDeclaredConstructors() : beanClass.getConstructors()); } catch (Throwable ex) { throw new BeanCreationException(mbd.getResourceDescription(), beanName, \"Resolution of declared constructors on bean Class [\" + beanClass.getName() + \"] from ClassLoader [\" + beanClass.getClassLoader() + \"] failed\", ex); } } //把所有的构造方法排序 //排序的规则是参数少->参数多 AutowireUtils.sortConstructors(candidates); int minTypeDiffWeight = Integer.MAX_VALUE; Set<Constructor<?>> ambiguousConstructors = null; LinkedList<UnsatisfiedDependencyException> causes = null; //这里是去找到对应的构造方法 //如果找不到会抛出异常 for (int i = 0; i < candidates.length; i++) { Constructor<?> candidate = candidates[i]; Class<?>[] paramTypes = candidate.getParameterTypes(); if (constructorToUse != null && argsToUse.length > paramTypes.length) { // Already found greedy constructor that can be satisfied -> // do not look any further, there are only less greedy constructors left. break; } if (paramTypes.length < minNrOfArgs) { continue; } ArgumentsHolder argsHolder; if (resolvedValues != null) { try { String[] paramNames = ConstructorPropertiesChecker.evaluate(candidate, paramTypes.length); if (paramNames == null) { ParameterNameDiscoverer pnd = this.beanFactory.getParameterNameDiscoverer(); if (pnd != null) { paramNames = pnd.getParameterNames(candidate); } } argsHolder = createArgumentArray( beanName, mbd, resolvedValues, bw, paramTypes, paramNames, candidate, autowiring); } catch (UnsatisfiedDependencyException ex) { if (this.beanFactory.logger.isTraceEnabled()) { this.beanFactory.logger.trace( \"Ignoring constructor [\" + candidate + \"] of bean '\" + beanName + \"': \" + ex); } // Swallow and try next constructor. if (causes == null) { causes = new LinkedList<UnsatisfiedDependencyException>(); } causes.add(ex); continue; } } else { // Explicit arguments given -> arguments length must match exactly. if (paramTypes.length != explicitArgs.length) { continue; } argsHolder = new ArgumentsHolder(explicitArgs); } int typeDiffWeight = (mbd.isLenientConstructorResolution() ? argsHolder.getTypeDifferenceWeight(paramTypes) : argsHolder.getAssignabilityWeight(paramTypes)); // Choose this constructor if it represents the closest match. if (typeDiffWeight < minTypeDiffWeight) { constructorToUse = candidate; argsHolderToUse = argsHolder; argsToUse = argsHolder.arguments; minTypeDiffWeight = typeDiffWeight; ambiguousConstructors = null; } else if (constructorToUse != null && typeDiffWeight == minTypeDiffWeight) { if (ambiguousConstructors == null) { ambiguousConstructors = new LinkedHashSet<Constructor<?>>(); ambiguousConstructors.add(constructorToUse); } ambiguousConstructors.add(candidate); } } if (constructorToUse == null) { if (causes != null) { UnsatisfiedDependencyException ex = causes.removeLast(); for (Exception cause : causes) { this.beanFactory.onSuppressedException(cause); } throw ex; } throw new BeanCreationException(mbd.getResourceDescription(), beanName, \"Could not resolve matching constructor \" + \"(hint: specify index/type/name arguments for simple parameters to avoid type ambiguities)\"); } else if (ambiguousConstructors != null && !mbd.isLenientConstructorResolution()) { throw new BeanCreationException(mbd.getResourceDescription(), beanName, \"Ambiguous constructor matches found in bean '\" + beanName + \"' \" + \"(hint: specify index/type/name arguments for simple parameters to avoid type ambiguities): \" + ambiguousConstructors); } if (explicitArgs == null) { argsHolderToUse.storeCache(mbd, constructorToUse); } } try { Object beanInstance; if (System.getSecurityManager() != null) { final Constructor<?> ctorToUse = constructorToUse; final Object[] argumentsToUse = argsToUse; beanInstance = AccessController.doPrivileged(new PrivilegedAction<Object>() { @Override public Object run() { //实例化的关键代码。 return beanFactory.getInstantiationStrategy().instantiate( mbd, beanName, beanFactory, ctorToUse, argumentsToUse); } }, beanFactory.getAccessControlContext()); } else { beanInstance = this.beanFactory.getInstantiationStrategy().instantiate( mbd, beanName, this.beanFactory, constructorToUse, argsToUse); } bw.setWrappedInstance(beanInstance); return bw; } catch (Throwable ex) { throw new BeanCreationException(mbd.getResourceDescription(), beanName, \"Bean instantiation via constructor failed\", ex); }}instantiateBean1234567891011121314151617181920212223protected BeanWrapper instantiateBean(final String beanName, final RootBeanDefinition mbd) { try { Object beanInstance; final BeanFactory parent = this; if (System.getSecurityManager() != null) { beanInstance = AccessController.doPrivileged(new PrivilegedAction<Object>() { @Override public Object run() { return getInstantiationStrategy().instantiate(mbd, beanName, parent); } }, getAccessControlContext()); } else { beanInstance = getInstantiationStrategy().instantiate(mbd, beanName, parent); } BeanWrapper bw = new BeanWrapperImpl(beanInstance); initBeanWrapper(bw); return bw; } catch (Throwable ex) { throw new BeanCreationException(mbd.getResourceDescription(), beanName, \"Instantiation of bean failed\", ex); }}12345678910111213141516171819202122232425262728293031323334353637public Object instantiate(RootBeanDefinition beanDefinition, String beanName, BeanFactory owner) { // Don't override the class with CGLIB if no overrides. if (beanDefinition.getMethodOverrides().isEmpty()) { Constructor<?> constructorToUse; synchronized (beanDefinition.constructorArgumentLock) { constructorToUse = (Constructor<?>) beanDefinition.resolvedConstructorOrFactoryMethod; if (constructorToUse == null) { final Class clazz = beanDefinition.getBeanClass(); if (clazz.isInterface()) { throw new BeanInstantiationException(clazz, \"Specified class is an interface\"); } try { if (System.getSecurityManager() != null) { constructorToUse = AccessController.doPrivileged(new PrivilegedExceptionAction<Constructor>() { public Constructor run() throws Exception { return clazz.getDeclaredConstructor((Class[]) null); } }); } else { constructorToUse = clazz.getDeclaredConstructor((Class[]) null); } beanDefinition.resolvedConstructorOrFactoryMethod = constructorToUse; } catch (Exception ex) { throw new BeanInstantiationException(clazz, \"No default constructor found\", ex); } } } return BeanUtils.instantiateClass(constructorToUse); } else { // Must generate CGLIB subclass. return instantiateWithMethodInjection(beanDefinition, beanName, owner); }}整段代码都在做一件事情，就是选择一个使用的构造函数。然后BeanUtils.instantiateClass(constructorToUse)实例化一个实例1234567891011121314151617181920212223public static <T> T instantiateClass(Constructor<T> ctor, Object... args) throws BeanInstantiationException { Assert.notNull(ctor, \"Constructor must not be null\"); try { ReflectionUtils.makeAccessible(ctor); return ctor.newInstance(args); } catch (InstantiationException ex) { throw new BeanInstantiationException(ctor.getDeclaringClass(), \"Is it an abstract class?\", ex); } catch (IllegalAccessException ex) { throw new BeanInstantiationException(ctor.getDeclaringClass(), \"Is the constructor accessible?\", ex); } catch (IllegalArgumentException ex) { throw new BeanInstantiationException(ctor.getDeclaringClass(), \"Illegal arguments for constructor\", ex); } catch (InvocationTargetException ex) { throw new BeanInstantiationException(ctor.getDeclaringClass(), \"Constructor threw exception\", ex.getTargetException()); }}通过反射生成Bean的实例。看到前面有一步makeAccessible，这意味着即使Bean的构造函数是private、protected的，依然不影响Bean的构造。注意一下，这里被实例化出来的Bean并不会直接返回，而是会被包装为BeanWrapper继续在后面使用。单例bean会缓存起来。多例bean spring每次都会自行创建。依赖注入populateBean这里把容器里的bean的依赖关系，注入到对应的bean里这里可以说是IOC的核心代码了。populateBean123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778protected void populateBean(String beanName, RootBeanDefinition mbd, BeanWrapper bw) { PropertyValues pvs = mbd.getPropertyValues(); //校验 if (bw == null) { if (!pvs.isEmpty()) { throw new BeanCreationException( mbd.getResourceDescription(), beanName, \"Cannot apply property values to null instance\"); } else { // Skip property population phase for null instance. return; } } // Give any InstantiationAwareBeanPostProcessors the opportunity to modify the // state of the bean before properties are set. This can be used, for example, // to support styles of field injection. boolean continueWithPropertyPopulation = true; //前置处理器 if (!mbd.isSynthetic() && hasInstantiationAwareBeanPostProcessors()) { for (BeanPostProcessor bp : getBeanPostProcessors()) { if (bp instanceof InstantiationAwareBeanPostProcessor) { InstantiationAwareBeanPostProcessor ibp = (InstantiationAwareBeanPostProcessor) bp; if (!ibp.postProcessAfterInstantiation(bw.getWrappedInstance(), beanName)) { continueWithPropertyPopulation = false; break; } } } } if (!continueWithPropertyPopulation) { return; } // 按照名字注入或者类型注入 if (mbd.getResolvedAutowireMode() == RootBeanDefinition.AUTOWIRE_BY_NAME || mbd.getResolvedAutowireMode() == RootBeanDefinition.AUTOWIRE_BY_TYPE) { //拿到类的所有属性 MutablePropertyValues newPvs = new MutablePropertyValues(pvs); // Add property values based on autowire by name if applicable. // 如果是按照名称注入，注入顺序是先按照名字注入，然后是按照类型注入 if (mbd.getResolvedAutowireMode() == RootBeanDefinition.AUTOWIRE_BY_NAME) { autowireByName(beanName, mbd, bw, newPvs); } // Add property values based on autowire by type if applicable. //如果是按照类型注入 if (mbd.getResolvedAutowireMode() == RootBeanDefinition.AUTOWIRE_BY_TYPE) { autowireByType(beanName, mbd, bw, newPvs); } pvs = newPvs; } boolean hasInstAwareBpps = hasInstantiationAwareBeanPostProcessors(); boolean needsDepCheck = (mbd.getDependencyCheck() != RootBeanDefinition.DEPENDENCY_CHECK_NONE); // 注入依赖检查。以及后置处理器的触发 if (hasInstAwareBpps || needsDepCheck) { PropertyDescriptor[] filteredPds = filterPropertyDescriptorsForDependencyCheck(bw, mbd.allowCaching); if (hasInstAwareBpps) { for (BeanPostProcessor bp : getBeanPostProcessors()) { if (bp instanceof InstantiationAwareBeanPostProcessor) { InstantiationAwareBeanPostProcessor ibp = (InstantiationAwareBeanPostProcessor) bp; pvs = ibp.postProcessPropertyValues(pvs, filteredPds, bw.getWrappedInstance(), beanName); if (pvs == null) { return; } } } } if (needsDepCheck) { checkDependencies(beanName, mbd, filteredPds, pvs); } } //设值 applyPropertyValues(beanName, mbd, bw, pvs); }这里主要关注两个方法。autowireByName其实就是根据名字去map里面找对应bean，然后实例化，设置进去123456789101112131415161718192021222324protected void autowireByName( String beanName, AbstractBeanDefinition mbd, BeanWrapper bw, MutablePropertyValues pvs) { String[] propertyNames = unsatisfiedNonSimpleProperties(mbd, bw); for (String propertyName : propertyNames) { if (containsBean(propertyName)) { //获取bean的实例 Object bean = getBean(propertyName); //注入进去 pvs.add(propertyName, bean); registerDependentBean(propertyName, beanName); if (logger.isDebugEnabled()) { logger.debug("Added autowiring by name from bean name '" + beanName + "' via property '" + propertyName + "' to bean named '" + propertyName + "'"); } } else { if (logger.isTraceEnabled()) { logger.trace("Not autowiring property '" + propertyName + "' of bean '" + beanName + "' by name: no matching bean found"); } } }}autowireByType12345678910111213141516171819202122232425262728293031323334353637383940protected void autowireByType( String beanName, AbstractBeanDefinition mbd, BeanWrapper bw, MutablePropertyValues pvs) { TypeConverter converter = getCustomTypeConverter(); if (converter == null) { converter = bw; } Set<String> autowiredBeanNames = new LinkedHashSet<String>(4); String[] propertyNames = unsatisfiedNonSimpleProperties(mbd, bw); for (String propertyName : propertyNames) { try { PropertyDescriptor pd = bw.getPropertyDescriptor(propertyName); // Don't try autowiring by type for type Object: never makes sense, // even if it technically is a unsatisfied, non-simple property. if (Object.class != pd.getPropertyType()) { MethodParameter methodParam = BeanUtils.getWriteMethodParameter(pd); // Do not allow eager init for type matching in case of a prioritized post-processor. boolean eager = !PriorityOrdered.class.isAssignableFrom(bw.getWrappedClass()); DependencyDescriptor desc = new AutowireByTypeDependencyDescriptor(methodParam, eager); //这里会去找类型相同的bean。这里代码比较长，有兴趣可以看一下。 Object autowiredArgument = resolveDependency(desc, beanName, autowiredBeanNames, converter); if (autowiredArgument != null) { pvs.add(propertyName, autowiredArgument); } for (String autowiredBeanName : autowiredBeanNames) { registerDependentBean(autowiredBeanName, beanName); if (logger.isDebugEnabled()) { logger.debug(\"Autowiring by type from bean name '\" + beanName + \"' via property '\" + propertyName + \"' to bean named '\" + autowiredBeanName + \"'\"); } } autowiredBeanNames.clear(); } } catch (BeansException ex) { throw new UnsatisfiedDependencyException(mbd.getResourceDescription(), beanName, propertyName, ex); } }}","categories":[],"tags":[{"name":"spring","slug":"spring","permalink":"http://www.liuyong520.cn/tags/spring/"}]},{"title":"xsync 同步命令脚本和xcall远程执行命令脚本","slug":"xsync","date":"2017-03-28T16:18:32.000Z","updated":"2019-06-11T09:37:15.535Z","comments":true,"path":"2017/03/29/xsync/","link":"","permalink":"http://www.liuyong520.cn/2017/03/29/xsync/","excerpt":"","text":"缘由在linux服务器集群上，有时我们需要将数据从主服务器同步到所有的从服务器上，或者在集群里需要执行一条或者多条命令，如果们一次次的拷贝，或者每个服务器一条条的执行，会造成重复的工作。所以就写两个脚本解决这方面的问题。xsync命令的编写安装 sync命令1yum install -y sync编写脚本 environment.sh12345#! /usr/bin/bash# 集群 IP 数组export NODE_IPS=(172.16.18.198 172.16.18.199 172.16.18.200)# 集群各 IP 对应的 主机名数组export NODE_NAMES=(k8s-n1 k8s-n2 k8s-n3)编写xsyncj考本1234567891011121314151617181920212223242526272829303132333435363738394041424344#!/bin/bash# 获取输出参数，如果没有参数则直接返回pcount=$#if [ $pcount -eq 0 ]then echo "no parameter find !"; exit;fi# 获取传输文件名p1=$1filename=`basename $p1`echo "load file $p1 success !"# 获取文件的绝对路径pdir=`cd -P $(dirname $p1); pwd`echo "file path is $pdir"# 获取当前用户（如果想使用root用户权限拷贝文件，在命令后加入-root参数即可）user=$2case "$user" in"-root") user="root";;"") user=`whoami`;;*) echo "illegal parameter $user" esacecho $user# 拷贝文件到从机(这里注意主机的host需要根据你的实际情况配置，要与你具体的主机名对应)source /opt/user/environment.shindex=0for node_ip in ${NODE_IPS[@]}do echo "================current host is ${NODE_NAMES[$index]} ip is ${node_ip}=================" rsync -rvl $pdir/$filename $user@${node_ip}:$pdir index=`expr $index + 1`doneecho "complete !"xcall脚本的编写利用ssh命令远程执行脚本命令。脚本如下：12345678910111213141516171819202122232425#!/bin/bash# 获取控制台指令cmd=$*# 判断指令是否为空if (( #$cmd -eq # ))then echo "command can not be null !" exitfi# 获取当前登录用户user=`whoami`source /opt/user/environment.sh# 在从机执行指令,这里需要根据你具体的集群情况配置，host与具体主机名一致for node_ip in ${NODE_IPS[@]}do echo "================current host is ${node_ip}=================" echo "--> excute command \\"$cmd\\"" ssh $user@${node_ip} $cmddoneecho "excute successfully !"这两个脚本仅仅只是一个简单的脚本，欢迎大家修改和使用。","categories":[{"name":"linux shell","slug":"linux-shell","permalink":"http://www.liuyong520.cn/categories/linux-shell/"}],"tags":[{"name":"linux","slug":"linux","permalink":"http://www.liuyong520.cn/tags/linux/"},{"name":"shell","slug":"shell","permalink":"http://www.liuyong520.cn/tags/shell/"}]}]}