1 Star 0 Fork 68

星尘 / CSharpFlink

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

CSharpFlink

a real-time computing framework

官方网址 http://www.ineuos.net/

技术博客 https://www.cnblogs.com/lsjwq/

github地址:https://github.com/wxzz/CSharpFlink

gitee地址:https://gitee.com/wxzz/CSharpFlink

作者QQ:504547114

技术QQ群:54256083

1 项目背景

      我们有一个全国性质的面向工业的公有云平台,通过专线或4G的链路方式实时向平台传输数据,每天处理1亿条左右的数据量,为现场用户提供实时的在线服务和离线数据分析服务。现在已经上线稳定运行有将近3年的时间。同时也为工业企业提供私有云建设服务。
      我们计划使用Flink作为云平台后台的实时计算部分,基本实现数据点的聚合计算、表达式规则计算等业务,进一步实现机器学习或自定义复杂算法的需求。
      我们经过将近一年左右时间的研究及开发,已经基本实现了聚合和逻辑等业务,但是感觉Flink比较重,并且应用和运维的水平要求比较高。
      基于上述情况,我们自主使用NET 5.0开发一套CSharpFlink实时计算组件,支持自定义数据源、计算和存储的基本要求。

2 应用场景

      主要面向物联网、工业互联网私有云或公有云平台建设过程中的数据点实时聚合和表达式计算。应用场景包括:

  • 数据点的实时时间窗口范围内聚合计算,例如:最大值、最小值、平均值、和值、众数、方差、中位数等,可以自定义二次开发。
  • 数据点的历史延迟窗口的一段时间范围内数据补充或更新的重新计算。
  • 数据点的表达式计算,支持自定义C#脚本进行编辑,实时预警或数据深度加工处理。
  • 主从结构的分布式部署,主节点负责计算任务分发,工作节点负责任务计算及结果存储。

3 框架特点

      主要特点主要是根据我们多年的物联网、工业项目经验的提炼和总结,满足实现应用场景,特点包括:

  • 使用最新的NET 5.0进行开发,完全跨平台。
  • 实时数据窗口范围外的数据补发或更新的重新计算,例如:当前5秒的实时数据窗口,支持5秒以前的数据补充和更新,并且进行重新计算及更新到数据存储单元。
  • 实时数据表达式计算支持定时计算或数据值改事件变触发计算,满足实时表达式或周期性计算。
  • C#语言的二次开发,对接多种数据源,自定义算子和多种方式数据存储等。
  • 单节点或分布式部署。

4 框架结构

      框架结构组件的基本示意,如下图: CSharpFlink框架图示意图

5 代码目录说明

      使用VS2019进行工程开发,工程解决方案文件为:CSharpFlink.sln,代码目录说明如下:

  • Cache:主节点和工作节点计算任务本地缓存管理。
  • Calculate:计算任务输入、过程、输出操作及管理。
  • Channel:主节点和工作节点分布式部署模式的IO通讯操作。
  • Common:操作公众类库。
  • Config:全局配置文件操作。
  • Execution:全局工程的执行环境入口。
  • Expression:表达式计算任务操作。
  • Log:日志操作及管理。
  • Model:数据点元数据信息。
  • Node:主节点和工作节点管理。
  • Protocol:主节点和工作节点之间分布式部署之间交互的协议。
  • Sink:计算任务计算结果存储接口。
  • Source:对接多种数据源接口,例如:mqtt、kafka、rabbitmq、数据库等。
  • Task:窗口或表达任务接口,主节点和工作节点任务操作及管理。
  • Window:数据窗口任务操作。
  • Worker:工作节点接口。

6 配置文件说明

      配置文件默认为:cfg\global.cfg,可以自定义指定配置文件,参见:命令行操作说明。配置文件说明,如下:

  • MaxDegreeOfParallelism:任务并行度,主节点生成任务、工作节点处理任务依赖这个参数。
  • MasterListenPort:主节点侦听端口,用于工作节点主动连接。
  • MasterIp:主节点IP,用于工作节点主动连接。
  • NodeType:节点运行模式,包括:Master、Slave和Both。
  • RemoteInvokeInterval:远程调用工作节点间隔时间,单位:毫秒。
  • RepeatRemoteInvokeInterval:调用工作节点失败后,重新调用工作节点间隔时间,单位:毫秒。
  • SlaveExcuteCalculateInterval:工作节点执行计算任务间隔时间,单位:毫秒。
  • MaxFrameLength:主节点和工作节点之间传输数据最大数据侦,单位:字节。
  • WorkerPower:工作节点能力系数,大于1,会连续发送多个任务。

7 任务部署说明

      二次开发参见:二次开发说明。开发好的任务,测试通过后,把程序集(.dll)复制到“tasks”目录下,例如工程TestTask项目测试、编译通过后,可以部署到“tasks”目录下,运行“CSharpFlink”主程序会自动加载和调用。       可以自定义指定任务程序集,参见:命令行操作说明。

8 命令行操作说明

      命令行运行“CSharpFlink”程序,支持自定义指定配置文件或任务程序集,说明如下:         -h 显示命令行帮助。         -c 加载指定配置文件。 例如:CSharpFlink -c c:/my.cfg         -t 加载任务程序集。 例如:CSharpFlink -t c:/mytask.dll         例如:

dotnet CSharpFlink.dll -c c:/master.cfg -t c:/mytask.dll

9 部署说明

      “release”目录下是编译好的程序,把“CSharpFlink v1.0”分别复制到不同的路径下,分别修改“cfg\global.cfg”配置文件中“NodeType”参数为:Master和Slave,修改主节点程序“tasks\tasks.cfg”文件中的任务数,分别运行不同目录下的“dotnet CSharpFlink.dll”。       “TestTask.dll”源代码,参见:二次开发说明。

10 二次开发说明

      二次开发主要针对数据源、计算过程和数据计算结果存储,大致过程如下:

  • 数据源对接,可以自定义对接mqtt、kafka、rabbitmq、数据库等,需要继承SourceFunction接口,参见:RandomSourceFunction.cs类。
  • 数据计算过程,可以自定义数据处理或加工,需要继承Calculate.Calculate接口,参见:聚合计算Avg.cs、表达式计算ExpressionCalculate.cs。通过AddWindowTask或AddExpressionTask函数参数进行实例化。
  • 数据计算结果存储,可以自定义存储任何介质上,需要继承SinkFunction接口,参见:SinkFunction.cs类。

11 单机10数据点计算应用(事例展示

      普通PC机10万数据点秒级计算,CPU:4核 I5-7400 2.7GHz,内存:16G,随机数据点时间窗口和计算算子,主节点CPU和内存使用情况:15%-35%、1500MB-2048MB, 工作节点CPU和内存使用情况:0.1%-2.5%、18MB-30MB。 本机部署1个主节点,10个计算节点。主节点生成10万个数据点的计算任务,每个数据点1秒钟产生一个新数据,计算时间窗口的最大值、最小值、平均值或和值。       部署结构,如下图:

CSharpFlink部署

      进程显示,如下图:

CSharpFlink进程

      运行效果,如下图:

CSharpFlink事例展示

iNeuOS工业互联网公众号:

iNeuOS工业互联网公众号

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

a real-time computing framework 展开 收起
C#
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
C#
1
https://gitee.com/xingchensoft/CSharpFlink.git
git@gitee.com:xingchensoft/CSharpFlink.git
xingchensoft
CSharpFlink
CSharpFlink
main

搜索帮助

14c37bed 8189591 565d56ea 8189591