1 Star 0 Fork 89

ryvius_key / hand-keras-yolo3-recognize

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

hand-keras-yolo3-recognize

手语图像识别系统设计

一个基于人体姿态研究的手语图像识别系统。根据OpenPose人体姿态开源模型和YOLOv3自训练手部模型检测视频和图像,再把数字特征进行分类器模型预测,将预测结果以文本形式展现出来。

预期是通过手机移动端对视频进行采集处理并应用,详见视频

软硬件环境

基于人体姿态的手语图像识别系统采用了软硬件相结合的方法。硬件部分主要是用于采集手语图像的单目摄像头。软件部分主要是通过ffmpeg对视频图像进行处理,然后在Anaconda下配置Python3.6的开发环境,再结合Cmake编译OpenPose模型,最后在VScode编译器中结合OpenCV中的图像算法,实现了对手语图像识别系统所有程序的编译,通过wxFromBuilder框架整合设计了系统主界面。

硬件环境

手语视频图像采集主要采用的硬件设备有笔记本电脑摄像头和手机摄像头。 程序运行硬件环境详细参数如下: (1)操作系统:Windows10家庭版,64bit (2)GPU:Intel(R) Core(TM) i5-8300H,主频:2.30GHz (3)内存:8G

软件环境

(1) 视频处理工具:ffmpeg-20181115 (2) 集成开发环境:Microsoft Visual Studio Code、Anaconda3 (3) 界面设计工具:wxFromBuilder (4) 编程语言环境:python3.6

系统功能设计

一个基于人体姿态研究的手语图像识别系统。根据OpenPose人体姿态开源模型和YOLOv3自训练手部模型检测视频和图像,再把数字特征进行分类器模型预测,将预测结果以文本形式展现出来。

基于人体姿态的手语图像识别系统是由多模块组成的,主要分为训练模块和识别模块两个部分。

在这里插入图片描述 在这里插入图片描述

1. 视频帧处理

Python+Opencv2(三)保存视频关键帧

在这里插入图片描述 在这里插入图片描述

2. OpenPose人体姿态识别

Openpose人体骨骼、手势--静态图像标记及分类(附源码) Openpose人体骨骼、手势--静态图像标记及分类2(附源码)

在这里插入图片描述 由于仅靠人体姿态4和7关键点不足以识别手部位置,容易误判,因此在最终设计中引入了yolo手部识别。 在这里插入图片描述

3. yolov3手部模型训练

项目结构主要分为两大部分:YOLOv3深度模型训练部分和YOLOv3和OpenPose手语姿态识别部分。

训练模型思路: 在这里插入图片描述

环境:【GPU】win10 (1050Ti)+anaconda3+python3.6+CUDA10.0+tensorflow-gpu2.1.0 训练模型:【Keras+TensorFlow+Yolo3】一文掌握图像标注、训练、识别(tf2填坑) 识别:【Keras+TensorFlow+Yolo3】教你如何识别影视剧模型

在这里插入图片描述

模型训练参考代码:https://gitee.com/cungudafa/keras-yolo3 yolo3识别这里参考于:https://github.com/AaronJny/tf2-keras-yolo3

4. 人体姿态数字特征提取

识别完整过程思路: 在这里插入图片描述

在OpenPose设计中阐述过求解距离和角度的公式及方法,最终因为个体差异每个人的骨骼可能不同,目前优化为距离比(即小臂3-4关键点的距离与脖子长度0-1关键点距离之比)。 在这里插入图片描述 基于 keras的yolo3训练部分项目结构如下表所示:

keras-yolo3训练项目结构:

名称 类型 内容
yolov3.weights 配置文件 权重文件
yolov3.cfg 配置文件 配置文件
convert.py 函数 模型格式转换
train.py 函数 模型训练
voc_annotation.py 函数 voc格式标签
yolo_annotations.py 函数 yolo格式标签
yolo.py 函数 yolo方法接口
model_data 文件夹 参数配置
nets 文件夹 yolo网络
utils 文件夹 图片加载工具类
VOCdevkit 文件夹 VOC格式数据集
logs 文件夹 h5训练的模型生成目录

其中logs文件夹用于存放训练好的模型,VOCdevkit用于存放图片和标注信息。

model_data文件夹内容:

名称 类型 内容
test.txt 文本 测试图片信息
train.txt 文本 训练图片信息
val.txt 文本 训练测试图片信息
voc_class.txt 文本 标签样本名称
yolo_anchors.txt 文本 先验参数
yolo_weights.h5 模型 权重文件

nets文件夹内容:

名称 类型 内容
darknet53.py 函数 卷积神经网络结构
loss.py 函数 计算图像检测效果
yolo3.py 函数 Yolov3网络识别算法

openpose和yolov3相结合参考: 《手语图像识别系统设计--人体动作识别》设计与实现

5.beyes分类识别

【Sklearn】入门花卉数据集实验--理解朴素贝叶斯分类器

在这里插入图片描述

识别部分代码结构:

名称 类型 内容
filesUtils 文件夹 文件批量处理
model 文件夹 模型
pose 文件夹 人体姿态识别相关算法
ui 文件夹 界面设计
yolo3 文件夹 Yolov3手部识别相关算法
beyes.py 函数 分类模型算法
getKeyFrame.py 函数 提取视频关键帧
pose_hand.py 函数 人姿和手部识别综合接口
UI.py 函数 可视化界面
yolo.py 函数 手部识别接口
SaveImg_graphviz.py 函数 绘制函数关系图

视频文件处理filesUtils文件夹:

名称 类型 内容
Image_classification.py 函数 图片分类
VideoUtils.py 函数 视频压缩、移动、重命名
ImgUtils.py 函数 图片压缩、移动、重命名
videoConv.bat 可执行程序 视频批量处理可执行程序

模型model文件夹:

文件夹 名称 类型 内容
pose_coco pose_deploy_linevec.prototxt 文本 OpenPose人体姿态参数
- pose_iter_440000.caffemodel 模型 OpenPose人体姿态模型
yolov3 coco_anchors.txt 文本 Yolo手部识别先验参数
-- voc_classes.txt 文本 Yolo手部种类
-- last1.h5 模型 Yolo手部识别模型
train_model.pkl 模型 朴素贝叶斯分类模型

基本算法pose和yolov3文件夹:

文件夹 名称 类型 内容
pose coco.py 函数 人体姿态识别算法
- data_process.py 函数 坐标信息转数字特征算法
yolov3 model.py 函数 手部识别算法
- utils.py 函数 数据格式处理函数

使用

  1. 配置好相应环境
  2. 修改相应路径,运行UI_main.py

说明

目前,该项目有很多笔记过程记录在csdn,也很有价值,感兴趣的朋友可以看看。 当然,这个项目仅是原理输出,作为成品推出可适用的方案还插一大段距离,中国手语包含5400+词汇,方言差异等,在词汇自然语言处理上还有很多发挥空间,再结合动态手势规划等等,手语翻译势在必行;

在这里插入图片描述

总之,基于手势动作进行图像识别的发展趋势是很大的,图形图像的处理运用在生活中的方方面面,待大家发现哦!

贡献者

cungudafa(王瑜)

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

基于图像的手语识别系统研究--人体动作识别,by王瑜 展开 收起
Python
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
Python
1
https://gitee.com/ryvius_key/hand-keras-yolo3-recognize.git
git@gitee.com:ryvius_key/hand-keras-yolo3-recognize.git
ryvius_key
hand-keras-yolo3-recognize
hand-keras-yolo3-recognize
master

搜索帮助

14c37bed 8189591 565d56ea 8189591