1 Star 0 Fork 59

helloway / ascend-deployer

forked from Ascend / ascend-deployer 
加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

离线安装工具说明

离线安装工具提供系统依赖、python依赖自动下载工具的和一键式安装所有依赖的脚本

目前离线安装工具支持如下操作系统

操作系统 版本 cpu架构 安装类型
CentOS 7.6 aarch64 镜像默认Minimal模式
CentOS 7.6 x86_64 镜像默认Minimal模式
CentOS 8.2 aarch64 镜像默认Minimal模式
CentOS 8.2 x86_64 镜像默认Minimal模式
ubuntu 18.04 aarch64 SmartKit默认Standard模式、镜像默认Server模式
ubuntu 18.04 x86_64 SmartKit默认Standard模式、镜像默认Server模式
BigCloud 7.6 aarch64 镜像默认Minimal模式
BigCloud 7.6 x86_64 镜像默认Minimal模式
SLES 12.4 x86_64 镜像默认Minimal模式
EulerOS 2.0SP8 aarch64 镜像默认Minimal模式
EulerOS 2.0SP9 aarch64 镜像默认Minimal模式
EulerOS 2.0SP9 x86_64 镜像默认Minimal模式

OS必装软件:tar, cd, ls, find, grep, chown, chmod等基本命令。OpenSSH Server用于ansible通过SSH连接登录,Ubuntu系统安装时需要选择安装

环境限制:OS安装后没有额外安装或卸载过软件,是镜像安装成功后的默认环境;若卸载安装某些系统软件,导致与安装默认系统包不一致,离线安装部署不支持该场景,需要手动配置网络,通过apt、yum、dnf等工具安装配置缺失软件

依赖限制:离线部署工具只能安装最基本的库,保证torch和tensorflow能够运行起来。如果需要运行比较复杂的推理或在训练模型,模型代码中可能包含具体业务相关的库,这些库需要自行安装。

EulerOS:需要确保源存在和内核版本相同的kenrel-headers和kernel-devel版本。如果源中没有,需要自行准备对应的kernel-headers和kernel-devel软件包。

CentOS:CentOS 8.2使用DVD镜像安装时,需要勾选Additioanl Software for Selected Environment中的Standard,如未勾选,系统安装完成后可能缺少tar等基本命令。

远程安装:EulerOS等操作系统默认禁止root用户远程连接。因此,在这类操作系统中远程安装时需要提前配置sshd_config中PermitRootLogin为yes。安装完成后再配置为no。

系统的内核版本可通过 uname -r 命令查看

离线安装工具操作指导

单机安装

  • 步骤 1

启动start_download.bat或者start_download.sh下载依赖软件

  • 步骤 2

将CANN软件包放到resources目录下

ascend-deployer
|- install.sh
|- ansible.cfg
|- playbooks
|- scenes
`- resources/
   |- CentOS_7.6_aarch64
   |- CentOS_7.6_x86_64
   |- ...
   |- aarch64
   |- x86_64
   |- A300-3000-npu-driver_xxx.run
   |- A300-3000-npu-firmware_xxx.run
   |- Ascend-cann-nnrt-xxx.run
   |- ...
   `- Ascend-cann-toolkit-xxx.run
  • 步骤 3

使用filezilla等工具,将整个目上传到待安装设备上

  • 步骤 4 执行install.sh --help仔细阅读参数说明
./install.sh --help
Usage: ./install.sh [options]
 Options:
--help  -h                     Print this message
--check                        check environment
--clean                        clean resources
--nocopy                       do not copy resources
--debug                        enable debug
--install=<package_name>       Install specific package:
                               cmake
                               driver
                               firmware
                               gcc
                               nnae
                               nnrt
                               npu
                               python375
                               sys_pkg
                               tensorflow
                               tfplugin
                               toolbox
                               toolkit
                               torch
The "npu" will install driver and firmware together
--install-scene=<scene_name>   Install specific scene:
                               auto
                               infer_dev
                               infer_run
                               train_dev
                               train_run
                               vmhost
--uninstall=<package_name>     Install specific package:
                               auto
                               driver
                               firmware
                               nnae
                               nnrt
                               npu
                               tfplugin
                               toolbox
                               toolkit
The "npu" will uninstall driver and firmware together
--upgrade=<package_name>       Install specific package:
                               auto
                               driver
                               firmware
                               nnae
                               nnrt
                               npu
                               tfplugin
                               toolbox
                               toolkit
The "npu" will upgrade driver and firmware together
--test=<target>                test the functions:
                               all
                               driver
                               firmware
                               tensorflow
                               toolbox
                               torch
  • 步骤 5

运行install.sh安装组件或按场景安装,例如:

./install.sh --install=driver       // 安装driver
./install.sh --install=npu          // 安装driver和firmware
./install.sh --install=nnrt,toolbox // 安装nnrt和toolbox
./install.sh --install-scene=auto   // 自动安装所有能找到的软件包

注意: 如果安装或者升级了driver或firmware,请在安装完成后重启设备使驱动和固件生效

注意: 执行指定组件安装时请确保安装顺序正确。例如nnrt或nnae需要在driver和firmware安装之后, firmware必须在driver已经安装后才能安装,等等。

注意: 部分组件存在运行时依赖,例如torch需要toolkit提供相应的运行时。tensorflow + npubridge需要tfplguin提供运行时

  • 步骤 6

运行检查,简单检查各个组件是否能够正常工作

./install.sh --test=driver         // 测试driver是否正常
./install.sh --test=firmware       // 测试firmware是否正常
./install.sh --test=torch          // 测试pytorch是否正常
./install.sh --test=all            // 测试所有已安装组件

批量安装

在单机安装执行安装之前配置inventory_file文件指定待安装设备。下载和上传之服务器的过程与单机相同。

  • 步骤 1

在文件inventory_file配置待安装的其他设备的ip地址、用户名和密码,可配多个。

例如:

[ascend]
ip_address_1 ansible_ssh_user='root' ansible_ssh_pass='password1'
ip_address_2 ansible_ssh_user='root' ansible_ssh_pass='password2'
ip_address_3 ansible_ssh_user='root' ansible_ssh_pass='password3'
  • 步骤 2

执行ansible ping测试其他设备连通性

export PATH=/usr/local/python3.7.5/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/python3.7.5/lib:$LD_LIBRARY_PATH
ansible all -i ./inventory_file -m ping
ip_address_1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

如果之前没有安装过ansible可以执行./install.sh --check

./install.sh --check

如果所有设备都success,表示所有设备都能正常连接。如有设备失败,请检查该设备的网络连接和sshd服务是否开启

  • 步骤 4 执行install.sh --help仔细阅读参数说明
./install.sh --help
  • 步骤 5

执行install.sh启动批量安装。过程与单机安装基本相同,例如:

./install.sh --install=driver      // 安装driver
./install.sh --install=npu         // 安装driver和firmware
./install.sh --install-scene=auto  // 自动安装所有能找到的软件包
  • 步骤 6

运行检查,与单机安装相同

按场景安装

离线部署工具提供几个基本安装场景

场景 安装的组件 说明
infer_run driver, firmware, nnrt, toolbox 推理运行
infer_dev driver, firmware, nnrt, toolbox, toolkit, torch, tfplugin, tensorflow 推理开发
train_run driver, firmware, nnae, toolbox, torch, tfplugin, tensorflow 训练运行
train_dev driver, firmware, nnae, toolbox, toolkit, torch, tfplugin, tensorflow 训练开发
vmhost driver, firmware, toolbox 虚拟机host
auto all 安装所有
  • 场景定制

场景的配置文件位于scene目录,文件内容非常简单,例如文件scene/scene_infer_run.yml:

- hosts: '{{ hosts_name }}'

- name: install system dependencies
  import_playbook: ../playbooks/install_sys_pkg.yml

- name: install driver and firmware
  import_playbook: ../playbooks/install_npu.yml

- name: install nnrt
  import_playbook: ../playbooks/install_nnrt.yml

- name: install toolbox
  import_playbook: ../playbooks/install_toolbox.yml

如果有特殊需求需要定制场景,参考scene_infer_run.yml。对不同组件进行灵活组合即可。

离线安装工具详细说明

下载工具使用

windows下需安装python3,推荐使用python3.7版本以上

windows版本下载路径python3.7.5

工具目录结构如下

|-- downloader
|-- playbooks
|-- start_download.bat
|-- start_download.sh
|-- install.sh
|-- resources
|-- ansble.cfg

在windows下运行start_download.bat启动下载,在linux下运行start_download.sh启动下载

在windows下运行start_download_ui.bat可启动建议UI。可在简易UI中选择需要下载的OS组件。

下载工具配置

下载工具涉及到的配置文件如下

downloader/config.ini
downloader/config/{os}_{version}_{arch}/source.list
downloader/config/{os}_{version}_{arch}/source.repo
  • Python源配置

python源配置在downloader/config.ini中,默认使用华为源,可根据需要替换

[pypi]
index_url=http://mirrors.huaweicloud.com/pypi/simple
  • Centos源配置

CentOS源在对应版本的配置目录中

downloader/config/CentOS_{version}_{arch}/source.repo

例如CentOS 7.6 aarch64第一的源配置在如下文件中

downloader/config/CentOS_7.6_aarch64/source.repo

CentOS 7.6的源配置文件内容如下:

[base]
baseurl=http://mirrors.huaweicloud.com/centos-altarch/7/os/aarch64

[epel]
baseurl=http://mirrors.huaweicloud.com/epel/7/aarch64

表示同时启用了base源和epel源,下载CentOS的依赖是会从这两个源中查询和下载。默认使用华为源,根据需要修改。 修改源通常只需要修改其中host部分,即mirrors.huaweicloud.com部分。如需修改后面部分,请确保理解CentOS的源配置

注意: CentOS的依赖软件需要在base和epel一起才能包含完整。如果需删除源,可能造成依赖下载不完整。

CentOS 8.2的源结构与7.6差异较大,例如CentOS 8.2 aarch64下源配置为:

[base]
baseurl=http://mirrors.huaweicloud.com/centos/8/BaseOS/aarch64/os

[powertool]
baseurl=http://mirrors.huaweicloud.com/centos/8/PowerTools/aarch64/os

[AppStream]
baseurl=http://mirrors.huaweicloud.com/centos/8/AppStream/aarch64/os/

[Everything]
baseurl=http://mirrors.huaweicloud.com/epel/8/Everything/aarch64

包含4个子源。 修改规则与CentOS 7.6相同

  • Ubuntu源配置

以Ubuntu 18.04 aarch64为例,源配置文件为:

config/Ubuntu_18.04_aarch64/source.list

内容如下:

deb http://mirrors.huaweicoud.com/ubuntu-ports/ bionic main multiverse restricted universe
deb http://mirrors.huaweicloud.com/ubuntu-ports/ bionic-updates main multiverse restricted universe
deb http://mirrors.huaweicloud.com/ubuntu-ports/ bionic-security main multiverse restricted universe

配置文件格式和ubuntu的/etc/apt.d/source.list基本相同。默认使用华为源,可根据实际情况修改。

注意: 修改源时通常只建议修改url。 增加或删除源可能找出依赖下载失败或依赖版本不匹配。

downloader介绍

1.downloader下载保存的目录结构:

resources/
|-- CentOS_7.6_aarch64
|-- CentOS_7.6_x86_64
|-- CentOS_8.2_aarch64
|-- CentOS_8.2_x86_64
|-- Ubuntu_18.04_aarch64
|-- Ubuntu_18.04_x86_64
|-- aarch64
`-- x86_64

2.代理配置

[proxy]
enable=true         # 是否开启代理配置参数
verify=true         # 是否校验https证书
protocol=http
hostname=openproxy.huawei.com
port=8080
username=none       # 代理账号
userpassword=none   # 代理密码

3.下载行为配置

[download]
os_list=CentOS_7.6_aarch64, CentOS_7.6_x86_64, CentOS_8.2_aarch64, CentOS_8.2_x86_64, Ubuntu_18.04_aarch64, Ubuntu_18.04_x86_64, BigCloud_7.6_aarch64, BigCloud_7.6_x86_64, SLES_12.4_x86_64  # 待安装部署的环境OS信息

Driver,Frimware和CANN层软件安装

Driver,Firmware,CANN层软件需要使用run包。 将相关软件包放置在resources目录下即可,例如:

ascend-deployer
|- install.sh
|- ansible.cfg
|- playbooks
|- scenes
`- resources/
   |- CentOS_7.6_aarch64
   |- CentOS_7.6_x86_64
   |- ...
   |- aarch64
   |- x86_64
   |- A300-3000-npu-driver_xxx.run
   |- A300-3000-npu-firmware_xxx.run
   |- Ascend-cann-nnrt-xxx.run
   |- ...
   `- Ascend-cann-toolkit-xxx.run

环境变量配置

1.python3.7.5

如果待安装环境未安装python3.7.5,安装过程中会自动安装,为了不影响系统自带python(python2.x or python3.x), 要使用python3.7.5之前,需要配置以下环境变量:

export PATH=/usr/local/python3.7.5/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/python3.7.5/lib:$LD_LIBRARY_PATH

更新离线部署工具

离线部署工具中的upgrade_self.sh和upgrade_self.bat用于更新离线部署工具自身

升级注意事项

1.涉及1-2包(driver、firmware)的升级操作,需要手动reboot重启服务器,才能生效,即执行test接口验证前需要重启

安全注意事项

  1. 由于需要使用dpkg, rpm等包管理器,只能使用root账号运行

  2. 由于需要安装大量开源软件,本工具下载的开源软件均来自操作系统源,开源软件的漏洞和修复需要用户自己根据情况修复,强烈建议使用官方源定期更新

  3. inventory文件中会配置远程机器的root用户名和密码,建议使用ansible的vault机制进行加密,使用完成之后建议立即删除

    参考文档http://www.ansible.com.cn/docs/playbooks_vault.html

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

提供离线安装的Ansible运维示例,用于部署Ascend任务。 展开 收起
Python
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
马建仓 AI 助手
尝试更多
代码解读
代码找茬
代码优化
Python
1
https://gitee.com/helloway/ascend-deployer.git
git@gitee.com:helloway/ascend-deployer.git
helloway
ascend-deployer
ascend-deployer
master

搜索帮助

344bd9b3 5694891 D2dac590 5694891