摘自:https://zhuanlan.zhihu.com/p/226173170
rocketmq 源码构建
系统:Mac OS 12.0 --- arm64
jdk 版本:1.8
maven 版本:3.6.3
git clone https://github.com/apache/rocketmq.git
cd rocketmq
# 执行mvn构建命令, 这里的命令含义是:清理之前编译、打包的文件,激活profile 名为 release-al
# 然后打包,不编译单元测试,也不运行单元测试
mvn clean -P release-all package -Dmaven.test.skip=true
编译后进入到 distribution/target
目录,rocket-mq-xxx.tar.gz
和 rocket-mq-xxx.zip
两种,zip 是 window 下的,tar.gz 为 Linux 下的;
tar.gz 解压后的目录结构:
.
├── LICENSE
├── NOTICE
├── README.md
├── benchmark
│ ├── batchproducer.sh
│ ├── consumer.sh
│ ├── producer.sh
│ ├── runclass.sh
│ ├── shutdown.sh
│ └── tproducer.sh
├── bin
│ ├── README.md
│ ├── cachedog.sh
│ ├── cleancache.sh
│ ├── cleancache.v1.sh
│ ├── dledger
│ │ └── fast-try.sh
│ ├── mqadmin
│ ├── mqadmin.cmd
│ ├── mqbroker
│ ├── mqbroker.cmd
│ ├── mqbroker.numanode0
│ ├── mqbroker.numanode1
│ ├── mqbroker.numanode2
│ ├── mqbroker.numanode3
│ ├── mqnamesrv
│ ├── mqnamesrv.cmd
│ ├── mqshutdown
│ ├── mqshutdown.cmd
│ ├── os.sh
│ ├── play.cmd
│ ├── play.sh
│ ├── runbroker.cmd
│ ├── runbroker.sh
│ ├── runserver.cmd
│ ├── runserver.sh
│ ├── setcache.sh
│ ├── startfsrv.sh
│ ├── tools.cmd
│ └── tools.sh
├── conf
│ ├── 2m-2s-async
│ │ ├── broker-a-s.properties
│ │ ├── broker-a.properties
│ │ ├── broker-b-s.properties
│ │ └── broker-b.properties
│ ├── 2m-2s-sync
│ │ ├── broker-a-s.properties
│ │ ├── broker-a.properties
│ │ ├── broker-b-s.properties
│ │ └── broker-b.properties
│ ├── 2m-noslave
│ │ ├── broker-a.properties
│ │ ├── broker-b.properties
│ │ └── broker-trace.properties
│ ├── broker.conf
│ ├── dledger
│ │ ├── broker-n0.conf
│ │ ├── broker-n1.conf
│ │ └── broker-n2.conf
│ ├── logback_broker.xml
│ ├── logback_namesrv.xml
│ ├── logback_tools.xml
│ ├── plain_acl.yml
│ └── tools.yml
└── lib
├── annotations-13.0.jar
├── commons-beanutils-1.9.4.jar
├── commons-cli-1.2.jar
├── commons-codec-1.9.jar
├── commons-collections-3.2.2.jar
├── commons-digester-2.1.jar
├── commons-lang3-3.4.jar
├── commons-logging-1.2.jar
├── commons-validator-1.7.jar
├── dledger-0.2.2.jar
├── fastjson-1.2.76.jar
├── gson-2.8.6.jar
├── guava-19.0.jar
├── jaeger-thrift-1.6.0.jar
├── jaeger-tracerresolver-1.6.0.jar
├── javassist-3.20.0-GA.jar
├── javax.annotation-api-1.3.2.jar
├── jcommander-1.72.jar
├── jna-4.2.2.jar
├── kotlin-stdlib-1.4.10.jar
├── kotlin-stdlib-common-1.4.0.jar
├── libthrift-0.14.1.jar
├── logback-classic-1.0.13.jar
├── logback-core-1.0.13.jar
├── netty-all-4.1.65.Final.jar
├── okhttp-4.9.0.jar
├── okio-2.8.0.jar
├── openmessaging-api-0.3.1-alpha.jar
├── opentracing-noop-0.33.0.jar
├── opentracing-tracerresolver-0.1.8.jar
├── opentracing-util-0.33.0.jar
├── rocketmq-acl-4.9.2-SNAPSHOT.jar
├── rocketmq-broker-4.9.2-SNAPSHOT.jar
├── rocketmq-client-4.9.2-SNAPSHOT.jar
├── rocketmq-common-4.9.2-SNAPSHOT.jar
├── rocketmq-example-4.9.2-SNAPSHOT.jar
├── rocketmq-filter-4.9.2-SNAPSHOT.jar
├── rocketmq-logging-4.9.2-SNAPSHOT.jar
├── rocketmq-namesrv-4.9.2-SNAPSHOT.jar
├── rocketmq-openmessaging-4.9.2-SNAPSHOT.jar
├── rocketmq-remoting-4.9.2-SNAPSHOT.jar
├── rocketmq-srvutil-4.9.2-SNAPSHOT.jar
├── rocketmq-store-4.9.2-SNAPSHOT.jar
├── rocketmq-tools-4.9.2-SNAPSHOT.jar
├── slf4j-api-1.7.7.jar
├── snakeyaml-1.19.jar
├── tomcat-annotations-api-8.5.46.jar
└── tomcat-embed-core-8.5.46.jar
-
bin 目录下放的是脚本文件
-
conf 目录,配置文件
2m-2s-async 双主双从异步复制模式
2m-2s-sync 双主双从同步双写模式
2m-noslave 双主模式
-
lib 存放的依赖 jar 包
测试 namesrv 的启动
启动命令:
nohup /Users/gitsilence/JavaTools/rocketmq/rocketmq-4.9.2-SNAPSHOT/bin/mqnamesrv &
使用 jps
查看是否启动成功
关闭命令
/Users/gitsilence/JavaTools/rocketmq/rocketmq-4.9.2-SNAPSHOT/bin/mqshutdown namesrv
集群部署
双主双从异步复制模式
两个台电脑
- Mac OS
- 地址:192.168.166.20
- Cent OS
- 地址:192.168.166.206
保证两台电脑在同一网段,端口是通的,可以关闭防火墙
需要开放的端口:NameServer 的 9876 端口、Broker 的 10911、11011、10909、11009 端口
Mac OS rocketmq 配置文件
1、编辑 conf/2m-2s-async/broker-a.properties
# 集群名称
brokerClusterName=mq-broker-cluster
# broker名字,不同的配置文件填写的不一样
brokerName=broker-a
# 0 表示master,>0 表示slave
brokerId=0
# 删除文件时间点,默认凌晨4点
deleteWhen=04
# 文件保留时间,默认48小时
fileReservedTime=48
# Broker的角色,AYNSC_MASTER=异步复制master,SYNC_MASTER=同步双写master,SLAVE=slave节点
brokerRole=ASYNC_MASTER
# 刷盘方式,ASYNC_FLUSH=异步刷盘,SYNC_FLUSH=同步刷盘
flushDiskType=ASYNC_FLUSH
# broker对外服务的监听端口
listenPort=10911
# nameServer地址,如果name server是多台集群的话,就用分号分隔
namesrvAddr=192.168.166.20:9876;192.168.166.206:9876
# 每个topic对应队列的数量,默认为4,实际参考consumer实例的数量,值过小不利于consumer负载均衡
defaultTopicQueueNums=8
# 是否允许broker自动创建Topic,生产建议关闭
autoCreateTopicEnable=true
# 是否允许broker自动创建订阅组,生产建议关闭
autoCreateSubscriptionGroup=true
# 设置brokerIp
brokerIP1=192.168.166.20
# 存储路径
storePathRootDir=/Users/gitsilence/JavaTools/rocketmq/data/store-a
# commitLog 存储路径
storePathCommitLog=/Users/gitsilence/JavaTools/rocketmq/data/store-a/commitlog
# 消费队列存储路径
storePathConsumerQueue=/Users/gitsilence/JavaTools/rocketmq/data/store-a/consumequeue
# 消息索引存储路径
storePathIndex=/Users/gitsilence/JavaTools/rocketmq/data/store-a/index
# checkpoint 文件存储路径
storeCheckpoint=/Users/gitsilence/JavaTools/rocketmq/data/store-a/checkpoint
# abort 存储路径
abortFile=/Users/gitsilence/JavaTools/rocketmq/data/store-a/abort
# commitLog每个文件的大小 默认1G
mapedFileSizeCommitLog=1073741824
# ConsumeQueue每个文件默认存30w条,根据业务情况调整
mapedFileSizeConsumeQueue=300000
2、配置 slave 节点 conf/2m-2s-async/broker-b-s.properties
# 集群名称
brokerClusterName=mq-broker-cluster
# broker名字,不同的配置文件填写的不一样
brokerName=broker-b
# 0 表示master,>0 表示slave
brokerId=1
# 删除文件时间点,默认凌晨4点
deleteWhen=04
# 文件保留时间,默认48小时
fileReservedTime=48
# Broker的角色,AYNSC_MASTER=异步复制master,SYNC_MASTER=同步双写master,SLAVE=slave节点
brokerRole=SLAVE
# 刷盘方式,ASYNC_FLUSH=异步刷盘,SYNC_FLUSH=同步刷盘
flushDiskType=ASYNC_FLUSH
# broker对外服务的监听端口
listenPort=11011
# nameServer地址,如果name server是多台集群的话,就用分号分隔
namesrvAddr=192.168.166.20:9876;192.168.166.206:9876
# 每个topic对应队列的数量,默认为4,实际参考consumer实例的数量,值过小不利于consumer负载均衡
defaultTopicQueueNums=8
# 是否允许broker自动创建Topic,生产建议关闭
autoCreateTopicEnable=true
# 是否允许broker自动创建订阅组,生产建议关闭
autoCreateSubscriptionGroup=true
# 设置brokerIp
brokerIP1=192.168.166.20
# 存储路径
storePathRootDir=/Users/gitsilence/JavaTools/rocketmq/data/store-b
# commitLog 存储路径
storePathCommitLog=/Users/gitsilence/JavaTools/rocketmq/data/store-b/commitlog
# 消费队列存储路径
storePathConsumerQueue=/Users/gitsilence/JavaTools/rocketmq/data/store-b/consumequeue
# 消息索引存储路径
storePathIndex=/Users/gitsilence/JavaTools/rocketmq/data/store-b/index
# checkpoint 文件存储路径
storeCheckpoint=/Users/gitsilence/JavaTools/rocketmq/data/store-b/checkpoint
# abort 存储路径
abortFile=/Users/gitsilence/JavaTools/rocketmq/data/store-b/abort
# commitLog每个文件的大小 默认1G
mapedFileSizeCommitLog=1073741824
# ConsumeQueue每个文件默认存30w条,根据业务情况调整
mapedFileSizeConsumeQueue=300000
先把两个节点 name server 启动
nohup /Users/gitsilence/JavaTools/rocketmq/rocketmq-4.9.2-SNAPSHOT/bin/mqnamesrv >> /Users/gitsilence/JavaTools/rocketmq/logs/namesrv/namesrv.log &
linux 节点启动
nohup /root/rocketmq/rocketmq-4.9.2-SNAPSHOT/bin/mqnamesrv >> /root/rocketmq/logs/namesrv/namesrv.log &
Linux 环境下遇到的问题:
- 启动的时候设置内存太大,只是用来测试的话,我们可以改下启动脚本的内存配置,
runserver.sh
- 还有一个就是脚本格式问题,会导致脚本执行不了,解决办法:https://blog.lacknb.cn/articles/2019/10/10/1577974148936.html#toc_h2_18
nameserver 启动完毕后,可以执行 jps
,查看是否起来
Cent OS rocketmq 配置文件
1、配置 master 节点 conf/2m-2s-async/broker-b.properties
# 集群名称
brokerClusterName=mq-broker-cluster
# broker名字,不同的配置文件填写的不一样
brokerName=broker-b
# 0 表示master,>0 表示slave
brokerId=0
# 删除文件时间点,默认凌晨4点
deleteWhen=04
# 文件保留时间,默认48小时
fileReservedTime=48
# Broker的角色,AYNSC_MASTER=异步复制master,SYNC_MASTER=同步双写master,SLAVE=slave节点
brokerRole=ASYNC_MASTER
# 刷盘方式,ASYNC_FLUSH=异步刷盘,SYNC_FLUSH=同步刷盘
flushDiskType=ASYNC_FLUSH
# broker对外服务的监听端口
listenPort=10911
# nameServer地址,如果name server是多台集群的话,就用分号分隔
namesrvAddr=192.168.166.20:9876;192.168.166.206:9876
# 每个topic对应队列的数量,默认为4,实际参考consumer实例的数量,值过小不利于consumer负载均衡
defaultTopicQueueNums=8
# 是否允许broker自动创建Topic,生产建议关闭
autoCreateTopicEnable=true
# 是否允许broker自动创建订阅组,生产建议关闭
autoCreateSubscriptionGroup=true
# 设置brokerIp
brokerIP1=192.168.166.206
# 存储路径
storePathRootDir=/root/rocketmq/data/store-a
# commitLog 存储路径
storePathCommitLog=/root/rocketmq/data/store-a/commitlog
# 消费队列存储路径
storePathConsumerQueue=/root/rocketmq/data/store-a/consumequeue
# 消息索引存储路径
storePathIndex=/root/rocketmq/data/store-a/index
# checkpoint 文件存储路径
storeCheckpoint=/root/rocketmq/data/store-a/checkpoint
# abort 存储路径
abortFile=/root/rocketmq/data/store-a/abort
# commitLog每个文件的大小 默认1G
mapedFileSizeCommitLog=1073741824
# ConsumeQueue每个文件默认存30w条,根据业务情况调整
mapedFileSizeConsumeQueue=300000
2、配置 slave 节点,conf/2m-2s-async/broker-a-s.properties
# 集群名称
brokerClusterName=mq-broker-cluster
# broker名字,不同的配置文件填写的不一样
brokerName=broker-a
# 0 表示master,>0 表示slave
brokerId=1
# 删除文件时间点,默认凌晨4点
deleteWhen=04
# 文件保留时间,默认48小时
fileReservedTime=48
# Broker的角色,AYNSC_MASTER=异步复制master,SYNC_MASTER=同步双写master,SLAVE=slave节点
brokerRole=SLAVE
# 刷盘方式,ASYNC_FLUSH=异步刷盘,SYNC_FLUSH=同步刷盘
flushDiskType=ASYNC_FLUSH
# broker对外服务的监听端口
listenPort=11011
# nameServer地址,如果name server是多台集群的话,就用分号分隔
namesrvAddr=192.168.166.20:9876;192.168.166.206:9876
# 每个topic对应队列的数量,默认为4,实际参考consumer实例的数量,值过小不利于consumer负载均衡
defaultTopicQueueNums=8
# 是否允许broker自动创建Topic,生产建议关闭
autoCreateTopicEnable=true
# 是否允许broker自动创建订阅组,生产建议关闭
autoCreateSubscriptionGroup=true
# 设置brokerIp
brokerIP1=192.168.166.206
# 存储路径
storePathRootDir=/root/rocketmq/data/store-b
# commitLog 存储路径
storePathCommitLog=/root/rocketmq/data/store-b/commitlog
# 消费队列存储路径
storePathConsumerQueue=/root/rocketmq/data/store-b/consumequeue
# 消息索引存储路径
storePathIndex=/root/rocketmq/data/store-b/index
# checkpoint 文件存储路径
storeCheckpoint=/root/rocketmq/data/store-b/checkpoint
# abort 存储路径
abortFile=/root/rocketmq/data/store-b/abort
# commitLog每个文件的大小 默认1G
mapedFileSizeCommitLog=1073741824
# ConsumeQueue每个文件默认存30w条,根据业务情况调整
mapedFileSizeConsumeQueue=300000
启动 broker
1、先启动 mac os master 节点 broker-a
nohup /Users/gitsilence/JavaTools/rocketmq/rocketmq-4.9.2-SNAPSHOT/bin/mqbroker -c /Users/gitsilence/JavaTools/rocketmq/rocketmq-4.9.2-SNAPSHOT/conf/2m-2s-async/broker-a.properties >> /Users/gitsilence/JavaTools/rocketmq/logs/broker/master-broker-a.log &
2、启动 Linux 的 master 节点 broker-b
nohup sh /root/rocketmq/rocketmq-4.9.2-SNAPSHOT/bin/mqbroker -c /root/rocketmq/rocketmq-4.9.2-SNAPSHOT/conf/2m-2s-async/broker-b.properties >> /root/rocketmq/logs/broker/master-broker-b.log &
3、启动 linux 的 slave 节点 broker-a
nohup sh /root/rocketmq/rocketmq-4.9.2-SNAPSHOT/bin/mqbroker -c /root/rocketmq/rocketmq-4.9.2-SNAPSHOT/conf/2m-2s-async/broker-a-s.properties >> /root/rocketmq/logs/broker/slave-broker-a.log &
4、启动 mac os 的 slave 节点 broker-b
nohup /Users/gitsilence/JavaTools/rocketmq/rocketmq-4.9.2-SNAPSHOT/bin/mqbroker -c /Users/gitsilence/JavaTools/rocketmq/rocketmq-4.9.2-SNAPSHOT/conf/2m-2s-async/broker-b-s.properties >> /Users/gitsilence/JavaTools/rocketmq/logs/broker/slave-broker-b.log &
启动完成后可以使用 mqadmin 检查 集群状态
/Users/gitsilence/JavaTools/rocketmq/rocketmq-4.9.2-SNAPSHOT/bin/mqadmin clusterList -n "192.168.166.20:9876;192.168.166.206:9876"
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.InternalThreadLocalMap).
RocketMQLog:WARN Please initialize the logger system properly.
#Cluster Name #Broker Name #BID #Addr #Version #InTPS(LOAD) #OutTPS(LOAD) #PCWait(ms) #Hour #SPACE
mq-broker-cluster broker-a 0 192.168.166.20:10911 V4_9_1 0.00(0,0ms) 0.00(0,0ms) 0 453367.41 -1.0000
mq-broker-cluster broker-a 1 192.168.166.206:11011 V4_9_1 0.00(0,0ms) 0.00(0,0ms) 0 453367.41 -1.0000
mq-broker-cluster broker-b 0 192.168.166.206:10911 V4_9_1 0.00(0,0ms) 0.00(0,0ms) 0 453367.41 -1.0000
mq-broker-cluster broker-b 1 192.168.166.20:11011 V4_9_1 0.00(0,0ms) 0.00(0,0ms) 0 453367.41 -1.0000
单机部署
部署 rocketmq-console 可视化监控
1、git clone git@github.com:apache/rocketmq-externals.git
2、cd rocketmq-externals
3、git checkout release-rocketmq-console-1.0.0
4、cd rocketmq-console
5、mvn clean package -Dmaven.test.skip=true
6、构建成功后,执行启动命令
nohup java -jar target/rocketmq-console-ng-1.0.0.jar --rocketmq.config.namesrvAddr='192.168.166.20:9876;192.168.166.206:9876' >> /Users/gitsilence/JavaTools/rocketmq/logs/mq-console/mq-console.log &
浏览器访问:http://localhost:8080
欢迎来到这里!
我们正在构建一个小众社区,大家在这里相互信任,以平等 • 自由 • 奔放的价值观进行分享交流。最终,希望大家能够找到与自己志同道合的伙伴,共同成长。
注册 关于