1 Star 0 Fork 448

随风而笔 / dlink

forked from DataLinkDC / Dinky 
加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

Dinky (原 Dlink)

简介

实时即未来,Dinky 为 Apache Flink 而生,让 Flink SQL 纵享丝滑,并致力于实时计算平台建设。

Dinky 架构于 Apache Flink,增强 Flink 的应用与体验,探索流式数仓。即站在巨人肩膀上创新与实践,Dinky 在未来批流一体的发展趋势下潜力无限。

最后,Dinky 的发展皆归功于 Apache Flink 等其他优秀的开源项目的指导与成果。

原理

功能

注意:以下功能均为对应版本已实现的功能,实测可用。

应用 方向 功能 进展
开发中心 FlinkSQL 支持 sql-client 所有语法 0.4.0
支持 Flink 所有 Configuration 0.4.0
支持 Flink 所有 Connector 0.4.0
支持 SELECT、SHOW 等查询实时预览 0.4.0
支持 INSERT 语句集 0.4.0
新增 SQL 片段语法 0.4.0
新增 AGGTABLE 表值聚合语法及 UDATF 支持 0.4.0
新增 FlinkSQLEnv 执行环境复用 0.5.0
新增 Flink Catalog 交互查询 0.4.0
新增 执行环境的共享与私有会话机制 0.4.0
新增 多种方言的作业目录管理(FlinkSQL、SQL、Java) 0.5.0
新增 作业配置与执行配置管理 0.4.0
新增 基于 Explain 的语法校验与逻辑解析 0.4.0
新增 JobPlan 图预览 0.5.0
新增 基于 StreamGraph 的表级血缘分析 0.4.0
新增 基于上下文元数据自动提示与补全 0.4.0
新增 自定义规则的自动提示与补全 0.4.0
新增 关键字高亮与代码缩略图 0.4.0
新增 选中片段执行 0.4.0
新增 布局拖拽 0.4.0
新增 SQL导出 0.5.0
支持 local 模式下 FlinkSQL 提交 0.4.0
支持 standalone 模式下 FlinkSQL 提交 0.4.0
支持 yarn session 模式下 FlinkSQL 提交 0.4.0
支持 yarn per-job 模式下 FlinkSQL 提交 0.4.0
支持 yarn application 模式下 FlinkSQL 提交 0.4.0
支持 kubernetes session 模式下 FlinkSQL 提交 0.5.0
支持 kubernetes application 模式下 FlinkSQL 提交 0.5.0
Flink 作业 支持 yarn application 模式下 Jar 提交 0.4.0
支持 k8s application 模式下 Jar 提交 0.5.0
支持 作业 Cancel 0.4.0
支持 作业 SavePoint 的 Cancel、Stop、Trigger 0.4.0
新增 作业自动从 SavePoint 恢复机制(包含最近、最早、指定一次) 0.4.0
新增 UDF java方言代码的开发 0.5.0
Flink 集群 支持 查看已注册集群的作业列表与运维 0.4.0
新增 自动注册 Yarn 创建的集群 0.4.0
SQL 新增 外部数据源的 SQL 校验 0.5.0
新增 外部数据源的 SQL 执行与预览 0.5.0
元数据 新增 查询外部数据源的元数据信息 0.4.0
归档 新增 执行与提交历史 0.4.0
运维中心 暂无 暂无 0.4.0
注册中心 Flink 集群实例 新增 外部 Flink 集群实例注册 0.4.0
新增 外部 Flink 集群实例心态检测与版本获取 0.4.0
新增 外部 Flink 集群手动一键回收 0.4.0
Flink 集群配置 新增 Flink On Yarn 集群配置注册及测试 0.4.0
User Jar 新增 外部 User Jar 注册 0.4.0
数据源 新增 Mysql 数据源注册及测试 0.4.0
新增 Oracle 数据源注册及测试 0.4.0
新增 postgreSql 数据源注册及测试 0.4.0
新增 ClickHouse 数据源注册及测试 0.4.0
OpenApi 调度 新增 submitTask 调度接口 0.5.0
FlinkSQL 新增 executeSql 提交接口 0.5.0
新增 explainSql 验证接口 0.5.0
新增 getJobPlan 计划接口 0.5.0
新增 getStreamGraph 计划接口 0.5.0
新增 getJobData 数据接口 0.5.0
Flink 新增 executeJar 提交接口 0.5.0
新增 cancel 停止接口 0.5.0
新增 savepoint 触发接口 0.5.0
关于 关于 Dlink 版本更新记录 0.4.0

部署

版本

抢先体验( main 主支):dlink-0.5.0-SNAPSHOT

稳定版本( 0.4.0 分支):dlink-0.4.0

从安装包开始

config/ -- 配置文件
|- application.yml
extends/ -- 扩展
|- dlink-client-1.11.jar
|- dlink-client-1.12.jar
|- dlink-client-1.14.jar
html/ -- 前端编译产物,用于Nginx
jar/ -- dlink application模式提交sql用到的jar
lib/ -- 内部组件
|- dlink-client-1.13.jar -- 必需
|- dlink-connector-jdbc.jar
|- dlink-function.jar
|- dlink-metadata-clickhouse.jar
|- dlink-metadata-mysql.jar
|- dlink-metadata-oracle.jar
|- dlink-metadata-postgresql.jar
plugins/
|- flink-connector-jdbc_2.11-1.13.3.jar
|- flink-csv-1.13.3.jar
|- flink-dist_2.11-1.13.3.jar
|- flink-json-1.13.3.jar
|- flink-shaded-hadoop-3-uber-3.1.1.7.2.1.0-327-9.0.jar
|- flink-shaded-zookeeper-3.4.14.jar
|- flink-table-blink_2.11-1.13.3.jar
|- flink-table_2.11-1.13.3.jar
|- mysql-connector-java-8.0.21.jar
sql/ 
|- dlink.sql -- Mysql初始化脚本(首次部署执行这个)
|- dlink_history.sql -- Mysql各版本及时间点升级脚本
auto.sh --启动停止脚本
dlink-admin.jar --程序包

解压后结构如上所示,修改配置文件内容。lib 文件夹下存放 dlink 自身的扩展文件,plugins 文件夹下存放 flink 及 hadoop 的官方扩展文件( 如果plugins下引入了flink-shaded-hadoop-3-uber 或者其他可能冲突的jar,请手动删除内部的 javax.servlet 等冲突内容)。其中 plugins 中的所有 jar 需要根据版本号自行下载并添加,才能体验完整功能,当然也可以放自己修改的 Flink 源码编译包。extends 文件夹只作为扩展插件的备份管理,不会被 dlink 加载。

在Mysql数据库中创建 dlink 数据库并执行初始化脚本 dlink.sql。

执行以下命令管理应用。

sh auto.sh start
sh auto.sh stop
sh auto.sh restart
sh auto.sh status

前端快捷访问: 如果plugins下引入了flink-shaded-hadoop-3-uber 的jar,请手动删除内部的 javax.servlet 后既可以访问默认 8888 端口号(如127.0.0.1:8888),正常打开前端页面。

前后端分离部署—— Nginx 部署(推荐): Nginx 如何部署请见百度或谷歌。 将 html 文件夹上传至 nginx 的 html 文件夹下或者指定 nginx 配置文件的静态资源绝对路径,修改 nginx 配置文件并重启。

    server {
        listen       9999;
        server_name  localhost;

		gzip on;
		gzip_min_length 1k;
		gzip_comp_level 9;
		gzip_types text/plain application/javascript application/x-javascript text/css application/xml text/javascript application/x-httpd-php image/jpeg image/gif image/png;
		gzip_vary on;
		gzip_disable "MSIE [1-6]\.";

        location / {
            root   html;
            index  index.html index.htm;
			try_files $uri $uri/ /index.html;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        location ^~ /api {
            proxy_pass http://127.0.0.1:8888;
            proxy_set_header   X-Forwarded-Proto $scheme;
            proxy_set_header   X-Real-IP         $remote_addr;
        }
    }
  1. server.listen 填写前端访问端口
  2. proxy_pass 填写后端地址如 http://127.0.0.1:8888
  3. 将 html 文件夹下打包好的前端资源上传到 nginx 的 html 文件夹中,如果 nginx 已经启动,则执行 nginx -s reload 重载配置,访问即可。

从源码编译

项目目录

dlink -- 父项目
|-dlink-admin -- 管理中心
|-dlink-app -- Application Jar
|-dlink-assembly -- 打包配置
|-dlink-client -- Client 中心
| |-dlink-client-1.11 -- Client-1.11 实现
| |-dlink-client-1.12 -- Client-1.12 实现
| |-dlink-client-1.13 -- Client-1.13 实现
| |-dlink-client-1.14 -- Client-1.14 实现
|-dlink-common -- 通用中心
|-dlink-connectors -- Connectors 中心
| |-dlink-connector-jdbc -- Jdbc 扩展
|-dlink-core -- 执行中心
|-dlink-doc -- 文档
| |-bin -- 启动脚本
| |-config -- 配置文件
| |-doc -- 使用文档
| |-extends -- Docker K8S模板
| |-sql -- sql脚本
|-dlink-executor -- 执行中心
|-dlink-extends -- 扩展中心
|-dlink-function -- 函数中心
|-dlink-gateway -- Flink 网关中心
|-dlink-metadata -- 元数据中心
| |-dlink-metadata-base -- 元数据基础组件
| |-dlink-metadata-clickhouse -- 元数据- clickhouse 实现
| |-dlink-metadata-mysql -- 元数据- mysql 实现
| |-dlink-metadata-oracle -- 元数据- oracle 实现
| |-dlink-metadata-postgresql -- 元数据- postgresql 实现
|-dlink-web -- React 前端

编译打包

以下环境版本实测编译成功:

环境 版本
npm 7.19.0
node.js 14.17.0
jdk 1.8.0_201
maven 3.6.0
lombok 1.18.16
mysql 5.7+
mvn clean install -Dmaven.test.skip=true

扩展Connector及UDF

将 Flink 集群上已扩展好的 Connector 和 UDF 直接放入 Dlink 的 plugins 下,然后重启即可。定制 Connector 过程同 Flink 官方一样。

扩展Metadata

遵循SPI。请参考 dlink-meta-mysql 的实现。

扩展其他版本的Flink

Flink 的版本取决于 lib 下的 dlink-client-1.13.jar。当前版本默认为 Flink 1.13.5 API。向其他版本的集群提交任务可能存在问题,已实现 1.11、1.12、1.13, 1.14,切换版本时只需要将对应依赖在lib下进行替换,然后重启即可。

切换版本时需要同时更新 plugins 下的 Flink 依赖。

使用手册

1.Flink AggTable 在 Dlink 的实践

2.Dlink 概念原理与源码扩展介绍

3.Dlink-0.3.0重磅来袭

4.Dlink 实时计算平台——部署篇

5.Dlink-0.3.2更新说明

6.Dlink 读写 Hive 的实践

7.Dlink On Yarn 三种 Flink 执行方式的实践

8.Dlink 在 Flink-mysql-cdc 到 Doris 的实践

技术栈

Apache Flink

Mybatis Plus

ant-design-pro

Monaco Editor

SpringBoot

致谢

感谢 JetBrains 提供的免费开源 License 赞助

JetBrains

近期计划

1.任务生命周期管理

2.作业监控及运维

3.流作业自动恢复

4.作业日志查看

5.钉钉报警和推送

交流与贡献

欢迎您加入社区交流分享,也欢迎您为社区贡献自己的力量。

在此非常感谢大家的支持~

QQ社区群:543709668,申请备注 “ Dinky ”,不写不批

微信社区群(推荐):添加微信号 wenmo_ai 邀请进群,申请备注 “ Dinky + 企业名 + 职位”,不写不批

公众号(最新消息获取建议关注):DataLink数据中台

运行截图

登录页

首页

Studio SQL 开发提示与补全

Studio 语法和逻辑检查

Studio 批流SELECT预览

Studio 异常反馈

Studio 进程监控

Studio 执行历史

Studio 数据回放

Studio SavePoint 管理

Studio 血缘分析

Studio 函数浏览

Studio 共享会话

集群管理

集群配置管理

数据源管理

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

Dlink 为 Apache Flink 而生,让 Flink SQL 纵享丝滑。它是一个交互式的 FlinkSQL Studio,可以在线开发、补全、校验 、执行、预览 FlinkSQL,支持 Flink 官方所有语法及其增强语法,并且可以同时对多 Flink 集群实例进行提交、停止、SavePoint 等运维操作,如同您的 IntelliJ IDEA For Flink SQL。 展开 收起
Java
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
Java
1
https://gitee.com/mabixiang/dlink.git
git@gitee.com:mabixiang/dlink.git
mabixiang
dlink
dlink
main

搜索帮助