third_codes/PowerJob
1
0
Fork 1
mirror of https://github.com/PowerJob/PowerJob.git synced 2025-07-17 00:00:04 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'v3.4.5' into jenkins_auto_build

Browse Source
This commit is contained in:
tjq 2021-01-31 17:40:26 +08:00
commit 5c5cc68f29
18 changed files with 75 additions and 57 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
README.md
View File

@ -1,4 +1,4 @@
English | [简体中文](./README_zhCN.md)
### English | [简体中文](./README_zhCN.md)
<p align="center">
<img src="https://raw.githubusercontent.com/KFCFans/PowerJob/master/others/images/logo.png" alt="PowerJob" title="PowerJob" width="557"/>
@ -13,22 +13,24 @@ English | [简体中文](./README_zhCN.md)
[PowerJob](https://github.com/PowerJob/PowerJob) is an open-source distributed computing and job scheduling framework which allows developers to easily schedule tasks in their own application.
Refer to [Quick Started Tutorial](https://www.yuque.com/powerjob/en/baz8y6) for detailed information.
Refer to [PowerJob Introduction](https://www.yuque.com/powerjob/en/introduce) for detailed information.
# Introduction
### Features
- **Friendly UI:** [Front-end](http://try.powerjob.tech/#/welcome) page is provided and developers can manage their task, monitor the status, check the logs online, etc.
- **Friendly UI:** [Front-end](http://try.powerjob.tech/#/welcome?appName=powerjob-agent-test&password=123) page is provided and developers can manage their task, monitor the status, check the logs online, etc.
- **Abundant Timing Strategies:** Four timing strategies are supported, including CRON expression, fixed rate, fixed delay and OpenAPI.
- **Abundant Timing Strategies:** Four timing strategies are supported, including CRON expression, fixed rate, fixed delay and OpenAPI which allows you to define your own scheduling policies, such as delaying execution.
- **Multiple Execution Mode:** Four execution modes are supported, including stand-alone, broadcast, Map and MapReduce. Distributed computing resource could be utilized in MapReduce mode, try the magic out [here](http://try.powerjob.tech/)!
- **Multiple Execution Mode:** Four execution modes are supported, including stand-alone, broadcast, Map and MapReduce. Distributed computing resource could be utilized in MapReduce mode, try the magic out [here](https://www.yuque.com/powerjob/en/za1d96#9YOnV)!
- **DAG Job Flow Support:** Both job dependency management and data communications between jobs are supported.
- **Workflow(DAG) Support:** Both job dependency management and data communications between jobs are supported.
- **Cross-language Client Support:** So far, developers can use Java, Spring, Shell, Python to write their own PowerJob client and define tasks that can be triggered by PowerJob workflow.
- **Extensive Processor Support:** Developers can write their processors in Java, Shell, Python, and will subsequently support multilingual scheduling via HTTP.
- **Disaster Tolerance Support:** As long as there are enough computing nodes, configurable retry policies make it possible for your task to be executed and finished successfully.
- **Powerful Disaster Tolerance:** As long as there are enough computing nodes, configurable retry policies make it possible for your task to be executed and finished successfully.
- **High Availability & High Performance:** PowerJob supports unlimited horizontal expansion. It's easy to achieve high availability and performance by deploying as many PowerJob server and worker nodes.
### Applicable scenes
@ -36,18 +38,16 @@ Refer to [Quick Started Tutorial](https://www.yuque.com/powerjob/en/baz8y6) for
- Broadcast tasks, for example, broadcasting to the cluster to clear logs.
- MapReduce tasks, for example, speeding up certain job like updating large amounts of data.
- Delayed tasks, for example, processing overdue orders.
- Customized tasks, triggered with [OpenAPI](https://www.yuque.com/powerjob/en/fs3vg0).
- Customized tasks, triggered with [OpenAPI](https://www.yuque.com/powerjob/en/openapi).
### Online trial
- Trial address: [Online Trial Address](http://try.powerjob.tech/)
- Application name: powerjob-agent-test
- Application password: 123
- Address: [try.powerjob.tech](http://try.powerjob.tech/#/welcome?appName=powerjob-agent-test&password=123)
- Recommended to read the documentation first: [here](https://www.yuque.com/powerjob/en/trial)
# Documents
**[Docs](https://www.yuque.com/powerjob/en/introduce)**
**[中文文档](https://www.yuque.com/powerjob/guidence/ztn4i5)**
**[中文文档](https://www.yuque.com/powerjob/guidence/intro)**
# Known Users
[Click to register as PowerJob user!](https://github.com/PowerJob/PowerJob/issues/6)

15
README_zhCN.md
View File

@ -1,4 +1,4 @@
[English](./README.md) | 简体中文
### [English](./README.md) | 简体中文
<p align="center">
<img src="https://raw.githubusercontent.com/KFCFans/PowerJob/master/others/images/logo.png" alt="PowerJob" title="PowerJob" width="557"/>
@ -34,11 +34,8 @@ PowerJob（原OhMyScheduler）是全新一代分布式调度与计算框架，
PowerJob 的设计目标为企业级的分布式任务调度平台，即成为公司内部的**任务调度中间件**。整个公司统一部署调度中心 powerjob-server，旗下所有业务线应用只需要依赖 `powerjob-worker` 即可接入调度中心获取任务调度与分布式计算能力。
### 在线试用
试用地址：[try.powerjob.tech](http://try.powerjob.tech/)
试用应用名称：powerjob-agent-test
控制台密码：123
[建议点击查看试用文档了解相关操作](https://www.yuque.com/powerjob/guidence/hnbskn)
* 试用地址：[try.powerjob.tech](http://try.powerjob.tech/#/welcome?appName=powerjob-agent-test&password=123)
* [建议先阅读使用教程了解 PowerJob 的概念和基本用法](https://www.yuque.com/powerjob/guidence/trial)
### 同类产品对比
| | QuartZ | xxl-job | SchedulerX 2.0 | PowerJob |
@ -55,11 +52,9 @@ PowerJob 的设计目标为企业级的分布式任务调度平台，即成为
# 官方文档
**[中文文档](https://www.yuque.com/powerjob/guidence/ztn4i5)**
**[中文文档](https://www.yuque.com/powerjob/guidence/intro)**
**[Document](https://www.yuque.com/powerjob/en/xrdoqw)**
PS：感谢文档翻译平台[breword](https://www.breword.com/)对本项目英文文档翻译做出的巨大贡献！
**[Docs](https://www.yuque.com/powerjob/en/introduce)**
# 接入登记
[点击进行接入登记，为 PowerJob 的发展贡献自己的力量！](https://github.com/PowerJob/PowerJob/issues/6)

4
powerjob-client/pom.xml
View File

@ -10,13 +10,13 @@
<modelVersion>4.0.0</modelVersion>
<artifactId>powerjob-client</artifactId>
<version>3.4.4</version>
<version>3.4.5</version>
<packaging>jar</packaging>
<properties>
<junit.version>5.6.1</junit.version>
<fastjson.version>1.2.68</fastjson.version>
<powerjob.common.version>3.4.4</powerjob.common.version>
<powerjob.common.version>3.4.5</powerjob.common.version>
<mvn.shade.plugin.version>3.2.4</mvn.shade.plugin.version>
</properties>

18
powerjob-client/src/test/java/com/github/kfcfans/powerjob/client/test/TestQuery.java
View File

@ -33,17 +33,17 @@ class TestQuery extends ClientInitializer {
@Test
void testQueryJob() {
JobInfoQuery jobInfoQuery = new JobInfoQuery()
.idGt(-1L)
.idLt(10086L)
.jobNameLike("DAG")
.gmtModifiedGt(DateUtils.addYears(new Date(), -10))
.gmtModifiedLt(DateUtils.addDays(new Date(), 10))
.executeTypeIn(Lists.newArrayList(ExecuteType.STANDALONE.getV(), ExecuteType.BROADCAST.getV(), ExecuteType.MAP_REDUCE.getV()))
.timeExpressionIn(Lists.newArrayList(TimeExpressionType.API.name(), TimeExpressionType.CRON.name(), TimeExpressionType.WORKFLOW.name(), TimeExpressionType.FIXED_RATE.name()))
.processorTypeIn(Lists.newArrayList(ProcessorType.EMBEDDED_JAVA.getV(), ProcessorType.SHELL.getV(), ProcessorType.JAVA_CONTAINER.getV()))
.processorInfoLike("com.github.kfcfans");
.setIdGt(-1L)
.setIdLt(10086L)
.setJobNameLike("DAG")
.setGmtModifiedGt(DateUtils.addYears(new Date(), -10))
.setGmtCreateLt(DateUtils.addDays(new Date(), 10))
.setExecuteTypeIn(Lists.newArrayList(ExecuteType.STANDALONE.getV(), ExecuteType.BROADCAST.getV(), ExecuteType.MAP_REDUCE.getV()))
.setProcessorTypeIn(Lists.newArrayList(ProcessorType.EMBEDDED_JAVA.getV(), ProcessorType.SHELL.getV(), ProcessorType.JAVA_CONTAINER.getV()))
.setProcessorInfoLike("com.github.kfcfans");
ResultDTO<List<JobInfoDTO>> jobQueryResult = ohMyClient.queryJob(jobInfoQuery);
System.out.println(JSON.toJSONString(jobQueryResult));
System.out.println(jobQueryResult.getData().size());
}
}

2
powerjob-common/pom.xml
View File

@ -10,7 +10,7 @@
<modelVersion>4.0.0</modelVersion>
<artifactId>powerjob-common</artifactId>
<version>3.4.4</version>
<version>3.4.5</version>
<packaging>jar</packaging>
<properties>

3
powerjob-common/src/main/java/com/github/kfcfans/powerjob/common/request/query/JobInfoQuery.java
View File

@ -16,7 +16,7 @@ import java.util.List;
*/
@Getter
@Setter
@Accessors(chain = true, fluent = true)
@Accessors(chain = true)
public class JobInfoQuery extends PowerQuery {
private Long idEq;
@ -31,7 +31,6 @@ public class JobInfoQuery extends PowerQuery {
private String jobParamsLike;
private List<Integer> timeExpressionTypeIn;
private List<String> timeExpressionIn;
private List<Integer> executeTypeIn;
private List<Integer> processorTypeIn;

6
powerjob-common/src/main/java/com/github/kfcfans/powerjob/common/utils/CommonUtils.java
View File

@ -155,4 +155,10 @@ public class CommonUtils {
return StringUtils.replace(UUID.randomUUID().toString(), "-", "");
}
public static void easySleep(long millis) {
try {
Thread.sleep(millis);
} catch (InterruptedException ignore) {
}
}
}

8
powerjob-common/src/main/java/com/github/kfcfans/powerjob/common/utils/NetUtils.java
View File

@ -246,7 +246,7 @@ public class NetUtils {
continue;
}
// 根据用户 -D 参数忽略网卡
if (ignoreInterfaceByConfig(networkInterface.getDisplayName())) {
if (ignoreInterfaceByConfig(networkInterface.getDisplayName()) || ignoreInterfaceByConfig(networkInterface.getName())) {
continue;
}
validNetworkInterfaces.add(networkInterface);
@ -297,7 +297,11 @@ public class NetUtils {
*/
public static boolean isPreferredNetworkInterface(NetworkInterface networkInterface) {
String preferredNetworkInterface = System.getProperty(PowerJobDKey.PREFERRED_NETWORK_INTERFACE);
return Objects.equals(networkInterface.getDisplayName(), preferredNetworkInterface);
if (Objects.equals(networkInterface.getDisplayName(), preferredNetworkInterface)) {
return true;
}
// 兼容直接使用网卡名称的情况，比如 Realtek PCIe GBE Family Controller
return Objects.equals(networkInterface.getName(), preferredNetworkInterface);
}
static boolean ignoreInterfaceByConfig(String interfaceName) {

4
powerjob-server/pom.xml
View File

@ -10,13 +10,13 @@
<modelVersion>4.0.0</modelVersion>
<artifactId>powerjob-server</artifactId>
<version>3.4.4</version>
<version>3.4.5</version>
<packaging>jar</packaging>
<properties>
<swagger.version>2.9.2</swagger.version>
<springboot.version>2.3.4.RELEASE</springboot.version>
<powerjob.common.version>3.4.4</powerjob.common.version>
<powerjob.common.version>3.4.5</powerjob.common.version>
<!-- MySQL version that corresponds to spring-boot-dependencies version. -->
<mysql.version>8.0.19</mysql.version>
<ojdbc.version>19.7.0.0</ojdbc.version>

6
powerjob-server/src/main/java/com/github/kfcfans/powerjob/server/service/instance/InstanceManager.java
View File

@ -73,6 +73,12 @@ public class InstanceManager {
return;
}
// 如果任务已经结束，直接丢弃该请求（StopInstance 和 ReportStatus 同时发送的情况）
if (InstanceStatus.finishedStatus.contains(instanceInfo.getStatus())) {
log.info("[InstanceManager-{}] instance already finished, this report[{}] will be dropped!", instanceId, req);
return;
}
InstanceStatus newStatus = InstanceStatus.of(req.getInstanceStatus());
Integer timeExpressionType = jobInfo.getTimeExpressionType();

2
powerjob-server/src/main/resources/static/js/7.js
View File

File diff suppressed because one or more lines are too long

4
powerjob-worker-agent/pom.xml
View File

@ -10,12 +10,12 @@
<modelVersion>4.0.0</modelVersion>
<artifactId>powerjob-worker-agent</artifactId>
<version>3.4.4</version>
<version>3.4.5</version>
<packaging>jar</packaging>
<properties>
<powerjob.worker.version>3.4.4</powerjob.worker.version>
<powerjob.worker.version>3.4.5</powerjob.worker.version>
<logback.version>1.2.3</logback.version>
<picocli.version>4.3.2</picocli.version>

14
powerjob-worker-agent/src/main/resources/logback.xml
View File

@ -9,7 +9,7 @@
<property name="LOG_PATH" value="${user.home}/powerjob/logs"/>
<property name="CONSOLE_LOG_PATTERN"
value="%yellow(%date{yyyy-MM-dd HH:mm:ss}) |%highlight(%-5level) |%blue(%thread) |%blue(%file:%line) |%green(%logger) |%cyan(%msg%n)"/>
value="%yellow(%date{yyyy-MM-dd HH:mm:ss}) |%highlight(%-5level) |%blue(%thread) |%green(%logger{10}) |%cyan(%msg%n)"/>
<!-- Console 输出设置 -->
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
@ -32,7 +32,7 @@
<MaxHistory>3</MaxHistory>
</rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{10} - %msg%n</pattern>
<charset>UTF-8</charset>
</encoder>
<filter class="ch.qos.logback.classic.filter.LevelFilter">
@ -41,6 +41,9 @@
<onMismatch>DENY</onMismatch>
</filter>
</appender>
<logger name="ERROR_LOGGER" level="ERROR" additivity="true">
<appender-ref ref="ERROR_APPENDER"/>
</logger>
<!-- 系统所有异常日志（ERROR）双写 end -->
@ -52,16 +55,17 @@
<MaxHistory>3</MaxHistory>
</rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{10} - %msg%n</pattern>
<charset>UTF-8</charset>
</encoder>
<append>true</append>
</appender>
<logger name="DEFAULT_LOGGER" level="INFO" additivity="false">
<appender-ref ref="DEFAULT_APPENDER"/>
</logger>
<!-- 系统主日志 日志 end -->
<root level="INFO">
<appender-ref ref="DEFAULT_APPENDER"/>
<appender-ref ref="ERROR_APPENDER"/>
<appender-ref ref="CONSOLE"/>
</root>

4
powerjob-worker-samples/pom.xml
View File

@ -10,11 +10,11 @@
<modelVersion>4.0.0</modelVersion>
<artifactId>powerjob-worker-samples</artifactId>
<version>3.4.4</version>
<version>3.4.5</version>
<properties>
<springboot.version>2.2.6.RELEASE</springboot.version>
<powerjob.worker.starter.version>3.4.4</powerjob.worker.starter.version>
<powerjob.worker.starter.version>3.4.5</powerjob.worker.starter.version>
<fastjson.version>1.2.68</fastjson.version>
<!-- 部署时跳过该module -->

4
powerjob-worker-spring-boot-starter/pom.xml
View File

@ -10,11 +10,11 @@
<modelVersion>4.0.0</modelVersion>
<artifactId>powerjob-worker-spring-boot-starter</artifactId>
<version>3.4.4</version>
<version>3.4.5</version>
<packaging>jar</packaging>
<properties>
<powerjob.worker.version>3.4.4</powerjob.worker.version>
<powerjob.worker.version>3.4.5</powerjob.worker.version>
<springboot.version>2.2.6.RELEASE</springboot.version>
</properties>

4
powerjob-worker/pom.xml
View File

@ -10,12 +10,12 @@
<modelVersion>4.0.0</modelVersion>
<artifactId>powerjob-worker</artifactId>
<version>3.4.4</version>
<version>3.4.5</version>
<packaging>jar</packaging>
<properties>
<spring.version>5.2.4.RELEASE</spring.version>
<powerjob.common.version>3.4.4</powerjob.common.version>
<powerjob.common.version>3.4.5</powerjob.common.version>
<h2.db.version>1.4.200</h2.db.version>
<hikaricp.version>3.4.2</hikaricp.version>
<junit.version>5.6.1</junit.version>

4
powerjob-worker/src/main/java/com/github/kfcfans/powerjob/worker/core/executor/ProcessorRunnable.java
View File

@ -2,6 +2,7 @@ package com.github.kfcfans.powerjob.worker.core.executor;
import akka.actor.ActorSelection;
import com.github.kfcfans.powerjob.common.ExecuteType;
import com.github.kfcfans.powerjob.common.utils.CommonUtils;
import com.github.kfcfans.powerjob.worker.OhMyWorker;
import com.github.kfcfans.powerjob.worker.common.ThreadLocalStore;
import com.github.kfcfans.powerjob.worker.common.constants.TaskConstant;
@ -161,6 +162,9 @@ public class ProcessorRunnable implements Runnable {
* @param cmd 特殊需求，比如广播执行需要创建广播任务
*/
private void reportStatus(TaskStatus status, String result, Integer cmd) {
CommonUtils.easySleep(1);
ProcessorReportTaskStatusReq req = new ProcessorReportTaskStatusReq();
req.setInstanceId(task.getInstanceId());