Skip to content

Commit eb0c59b

Browse files
chaozwnchaozwn
committed
[feature-#1058] Add docker development docker-compose file, help quickly build dev environment.
1 parent 16cd331 commit eb0c59b

14 files changed

Lines changed: 475 additions & 36 deletions

‎chunjun-dev/README_CH.md‎

Lines changed: 116 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,9 @@
11
# Chunjun开发环境构建
2-
一个基于 `docker` 的 `chunjun` 本地开发运行环境。
3-
请注意为了保证大家端口基本都可以使用（ps：大家可能机器上装过mysql啥的，3306都可能被占用过），
4-
所有我们都选用5位数的端口，并且一般以4开头，比如3306，我们就使用43306等等。
2+
一个基于 `docker` 的 `chunjun` 本地开发运行环境，也可以用于快速搭建chunjun的local开发环境。
53

6-
```bash
7-
// 为了下面能正常运行，请先执行下面的代码
8-
```
94

105
## 使用
11-
### 1. 按需修改 .env 配置
6+
### 1. 按需修改 .env 配置 (主要包含了一些端口，如果被占用，请自行更改)
127
~~~
138
# 设置时区
149
TZ=Asia/Shanghai
@@ -31,11 +26,25 @@ MYSQL8_ROOT_PASSWORD=123456
3126
3227
# Redis5 服务映射宿主机端口号，可在宿主机127.0.0.1:46375访问
3328
REDIS5_PORT=46375
34-
REDIS5_PASSWORD=G62m50oigInC30sf
29+
REDIS5_PASSWORD=123456
3530
# Redis6 服务映射宿主机端口号，可在宿主机127.0.0.1:46376访问
3631
REDIS6_PORT=46376
37-
REDIS6_PASSWORD=G62m50oigInC30sf
32+
REDIS6_PASSWORD=123456
33+
34+
# Dm7 服务映射宿主机端口号，可在宿主机127.0.0.1:45237访问
35+
DM7_PORT=45237
36+
# Dm8 服务映射宿主机端口号，可在宿主机127.0.0.1:45238访问
37+
DM8_PORT=45238
3838
39+
# Zookeeper 服务映射宿主机端口号，可在宿主机127.0.0.1:42181访问
40+
ZOOKEEPER_PORT=42181
41+
42+
# kafka28 服务映射宿主机端口号，可在宿主机127.0.0.1:49092访问
43+
KAFKA28_PORT=49092
44+
45+
# kudu 服务映射宿主机端口号，可在宿主机127.0.0.1:49092访问
46+
KUDU_MASTER_PORT=7051
47+
KUDU_TSERVER_PORT=7050
3948
~~~
4049

4150
### 2.启动服务
@@ -59,20 +68,105 @@ docker-compose stop
5968
# 其他的服务名，请查看docker-compose.yml
6069
docker-compose stop chunjun-redis5 chunjun-mysql8
6170
```
62-
### 4.kafka操作(ps：kafka已开启自动创建topic的功能)
71+
72+
### 4.开发示例
73+
开发方式一般有两种，一种是使用IDEA(或者Jetbrain的gateway)和vscode进行开发，核心思路是远程到我们的app镜像当中开发。 为什么要远程到app镜像当中开发，主要是为了解决一些host映射的问题，我们在容器中debug代码，就不需要把host映射到主机，从而解决hbase，hive的一些host映射问题。
74+
#### 4.1 启动开发环境
75+
```shell
76+
cd chunjun-dev
77+
# 等待所有的镜像构建完毕
78+
docker-compose up -d
79+
# 构建部分镜像，(推荐)
80+
docker-compose up chunjun-app
81+
```
82+
![](images/docker_containers.png)
83+
84+
#### 4.2 进入app container，克隆代码
85+
```shell
86+
docker exec -it chunjun-dev_chunjun-app_1 bash
87+
88+
cd /root/app/code
89+
90+
git clone https://github.com/DTStack/chunjun.git
91+
92+
cd chunjun
93+
94+
mvn clean package -DskipTests
95+
```
96+
![](images/build_success.png)
97+
98+
#### 4.3 IDE打开，这边使用的是Gateway
99+
```shell
100+
账号密码 root/root，等待进入IDE
101+
```
102+
![](images/new_connection.png)
103+
![](images/chunjun_dev.png)
104+
![](images/gateway_ide.png)
105+
106+
#### 4.4 第一个测试
107+
```shell
108+
// 在chunjun的代码仓库下创建一个job是目录，创建文件mysql_to_print.json，复制下面的json
109+
运行类 com.dtstack.chunjun.client.Launcher
110+
运行参数 -job /root/app/code/chunjun/jobs/mysql_to_print.json -jobType sync -chunjunDistDir /root/app/code/chunjun/chunjun-dist
111+
```
112+
```json
113+
{
114+
"job": {
115+
"content": [
116+
{
117+
"reader": {
118+
"name": "mysqlreader",
119+
"parameter": {
120+
"column": [
121+
{
122+
"name": "Host",
123+
"type": "string"
124+
},
125+
{
126+
"name": "User",
127+
"type": "string"
128+
}
129+
],
130+
"username": "root",
131+
"password": "123456",
132+
"connection": [
133+
{
134+
"jdbcUrl": [
135+
"jdbc:mysql://chunjun-mysql5.7:3306/mysql?useSSL=false"
136+
],
137+
"table": [
138+
"user"
139+
]
140+
}
141+
]
142+
}
143+
},
144+
"writer": {
145+
"name": "streamwriter",
146+
"parameter": {
147+
"print": true
148+
}
149+
}
150+
}
151+
],
152+
"setting": {
153+
"speed": {
154+
"channel": 1,
155+
"bytes": 0
156+
}
157+
}
158+
}
159+
}
160+
```
161+
![](images/success.png)
162+
```shell
163+
//运行截图
164+
1.如果你能得到相同的结果，那么开发环境就构建成功了.
165+
2.请注意保护chunjun-dev/data/app目录下的文件,你改动的代码都会被同步到这个文件夹
166+
```
167+
### 5.kafka操作(ps：kafka已开启自动创建topic的功能)
63168
```bash
64-
请在/etc/host中添加映射
65-
linux:
66-
vim /etc/host
67-
// 添加
68-
127.0.0.1 chunjun-kafka2.8
69-
win:
70-
C:\Windows\System32\drivers\etc
71-
// 添加
72-
127.0.0.1 chunjun-kafka2.8
73-
74-
// 进入chunjun-kafka的容器内部，docker ps获取容器的id
75-
docker exec -it 4ecc98f58db1 /bin/sh
169+
docker exec -it chunjun-dev_chunjun-kafka2.8_1 /bin/sh
76170
cd /opt/kafka/bin/
77171

78172
//创建topic
@@ -88,16 +182,3 @@ cd /opt/kafka/bin/
88182
./kafka-console-consumer.sh --bootstrap-server chunjun-kafka2.8:49092 --topic chunjun-test --from-beginning
89183

90184
```
91-
### 5.kudu操作
92-
```bash
93-
docker-compose up -d chunjun-kudu-master-1 chunjun-kudu-master-2 chunjun-kudu-master-3 chunjun-kudu-tserver
94-
95-
// Find the web UI url for kudu-master-1:
96-
docker ps -q --filter=name=.*kudu-master-1.* | xargs -L1 docker port | grep "^8051" | cut -d":" -f2 | sed 's/.*/http\:\/\/localhost\:&/'
97-
98-
// 常见错误
99-
org.apache.kudu.client.NonRecoverableException: Couldn't find any valid locations, exceptions: [java.net.UnknownHostException: Failed to resolve the IP of `ecfbad4be048']
100-
处理方案：
101-
1.将对应的containerid加入host文件
102-
127.0.0.1 ecfbad4be048
103-
```

‎chunjun-dev/app/Dockerfile‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,9 @@ RUN wget http://archive.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/ap
4242

4343
COPY ./settings.xml /usr/lib/mvn/conf
4444
RUN mkdir -p /root/.m2
45-
COPY ./settings.xml /root/.m2
45+
RUN mkdir -p /root/.m2/repository
46+
COPY ./settings.xml /root/.m2/
47+
COPY ./clean_maven.sh /root
4648

4749
RUN mkdir -p /root/app/code
4850
WORKDIR /root/app/code

‎chunjun-dev/app/clean_maven.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/bin/sh
2+
#!/bin/bash
3+
4+
REPO_PATH="/root/.m2/repository"
5+
6+
echo "正在清理"$REPO_PATH"目录下的.lastUpdated后缀文件"
7+
8+
find $REPO_PATH -name "*.lastUpdated" | xargs rm -fr
9+
10+
echo "完成清理"

‎chunjun-dev/images/build_success.png‎

97.4 KB
Loading

‎chunjun-dev/images/chunjun_dev.png‎

16.5 KB
Loading

‎chunjun-dev/images/docker_containers.png‎

28.3 KB
Loading

‎chunjun-dev/images/gateway_ide.png‎

144 KB
Loading

‎chunjun-dev/images/new_connection.png‎

10.9 KB
Loading

‎chunjun-dev/images/success.png‎

208 KB
Loading

‎jobs/kafka_to_kudu.json‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
{
2+
"job": {
3+
"content": [
4+
{
5+
"reader": {
6+
"parameter": {
7+
"topic": "chunjun-test",
8+
"mode": "earliest-offset",
9+
"timestamp": 1000,
10+
"offset": "EARLIEST",
11+
"groupId": "dddd",
12+
"codec": "json",
13+
"column": [
14+
{
15+
"name": "name",
16+
"type": "String"
17+
}
18+
],
19+
"consumerSettings": {
20+
"bootstrap.servers": "127.0.0.1:49092",
21+
"auto.commit.enable": "false"
22+
}
23+
},
24+
"name": "kafkasource"
25+
},
26+
"writer": {
27+
"parameter": {
28+
"column": [
29+
{
30+
"name": "name",
31+
"type": "String"
32+
}
33+
],
34+
"masters": "localhost:47051,localhost:47052,localhost:47053",
35+
"table": "kudu_test",
36+
"flushMode": "manual_flush",
37+
"writeMode": "append",
38+
"batchSizeBytes": 1048576
39+
},
40+
"name": "kuduwriter"
41+
}
42+
}
43+
],
44+
"setting": {
45+
"restore": {
46+
"maxRowNumForCheckpoint": 0,
47+
"isRestore": false,
48+
"restoreColumnName": "",
49+
"restoreColumnIndex": 0
50+
},
51+
"errorLimit": {
52+
"record": 100
53+
},
54+
"speed": {
55+
"bytes": 0,
56+
"channel": 1
57+
},
58+
"log": {
59+
"isLogger": false,
60+
"level": "debug",
61+
"path": "",
62+
"pattern": ""
63+
}
64+
}
65+
}
66+
}

0 commit comments

Comments
 (0)