2.3K Star 8.1K Fork 4.3K

GVPMindSpore / mindspore

 / 详情

[ST][MS][MF][r2.3][qwen_7b/14b][910B3 8P]网络训练失败,ValueError: Can't find class type optimizer class name AdamWeightDecayX in class registry

DONE
Bug-Report
创建于  
2024-04-18 11:01
name about labels
Bug Report Use this template for reporting a bug kind/bug

Describe the current behavior / 问题描述 (Mandatory / 必填)

[r2.3][qwen_7b/14b][910B3 8P]网络训练失败,ValueError: Can't find class type optimizer class name AdamWeightDecayX in class registry
模型仓地址:https://gitee.com/mindspore/mindformers/blob/dev/research/qwen/qwen.md

Environment / 环境信息 (Mandatory / 必填)

  • Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device ascend/

  • Software Environment / 软件环境 (Mandatory / 必填):
    -- MindSpore version (e.g., 1.7.0.Bxxx) :
    -- Python version (e.g., Python 3.7.5) :
    -- OS platform and distribution (e.g., Linux Ubuntu 16.04):
    -- GCC/Compiler version (if compiled from source):

CANN版本:MILAN-Florence-ASL/ABL V100R001C17SPC001B240 Alpha
Mindspore版本:MindSpore_r2.3_d51c17c7(MindSporeDaily)
MindFormers版本:MindFormers_dev_a4fc9e6d(MindFormersDaily)

  • Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode graph

Related testcase / 关联用例 (Mandatory / 必填)

用例仓地址:MindFormers_Test/cases/qwen/14b/train/
用例:
test_mf_qwen_14b_train_infer_alpaca_8p_0001
test_mf_qwen_7b_train_infer_alpaca_8p_0001

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

  1. get code from mindformers
  2. cd mindformers/reserach
  3. 修改配置文件中的权重、数据集路径
  4. bash run_singlenode.sh "python qwen/run_qwen.py --config qwen/run_qwen_14b.yaml --load_checkpoint /home/workspace/large_model_ckpt/qwen/14b/ --use_parallel True --run_mode finetune --auto_trans_ckpt True --train_data /home/workspace/large_model_dataset/qwen/alpaca-fastchat2048_qwen_all.mindrecord" /home/workspace/config/hccl_8p.json [0,8] 8
  5. 验证网络是否训练成功
  6. 验证网络编译时间是否达标

Describe the expected behavior / 预期结果 (Mandatory / 必填)

网络训练成功,编译时间达标

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

Traceback (most recent call last):
 File "./qwen/run_qwen.py", line 200, in <module>
   paged_attention=args.paged_attention)
 File "./qwen/run_qwen.py", line 122, in main
   trainer.finetune(finetune_checkpoint=ckpt, auto_trans_ckpt=auto_trans_ckpt)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/_checkparam.py", line 1372, in wrapper
   return func(*args, **kwargs)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/trainer.py", line 521, in finetune
   is_full_config=True)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 120, in train
   **kwargs)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/base_trainer.py", line 681, in training_process
   optimizer = self.create_optimizer_scheduler(network, layer_scale=config.layer_scale)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/base_trainer.py", line 446, in create_optimizer_scheduler
   "learning_rate": lr_schedule})
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/core/optim/build_optim.py", line 68, in build_optim
   config, MindFormerModuleType.OPTIMIZER, default_args=default_args)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/tools/register/register.py", line 184, in get_instance_from_cfg
   obj_cls = cls.get_cls(module_type, obj_type)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/tools/register/register.py", line 141, in get_cls
   in class registry".format(module_type, class_name))
ValueError: Can't find class type optimizer class name AdamWeightDecayX             in class registry

Special notes for this issue/备注 (Optional / 选填)

走给李洋

评论 (5)

sunjiawei999 创建了Bug-Report
sunjiawei999 复制于任务 I8VDWI
sunjiawei999 优先级设置为主要
展开全部操作日志

Please assign maintainer to check this issue.
请为此issue分配处理人。
@sunjiawei999

感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:

  1. 如果您刚刚接触MindSpore,或许您可以在教程找到答案
  2. 如果您是资深Pytorch用户,您或许需要:
  1. 如果您遇到动态图问题,可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
  2. 模型精度调优问题可参考官网调优指南
  3. 如果您反馈的是框架BUG,请确认您在ISSUE中提供了MindSpore版本、使用的后端类型(CPU、GPU、Ascend)、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
  4. 如果您已经定位出问题根因,欢迎提交PR参与MindSpore开源社区,我们会尽快review
sunjiawei999 添加了
 
kind/bug
标签
sunjiawei999 添加了
 
attr/function
标签
sunjiawei999 添加了
 
stage/func-debug
标签
sunjiawei999 添加了
 
sig/mindformers
标签
sunjiawei999 添加了
 
device/ascend
标签
sunjiawei999 添加了
 
v2.3.0
标签
sunjiawei999 移除了
 
v2.3.0
标签
sunjiawei999 移除了
 
v2.3.0
标签
sunjiawei999 添加了
 
v2.3.0
标签
sunjiawei999 关联分支设置为r2.3
sunjiawei999 负责人设置为xiangminshan
sunjiawei999 计划开始日期设置为2024-04-18
sunjiawei999 计划截止日期设置为2024-04-25
sunjiawei999 修改了描述
sunjiawei999 修改了描述
sunjiawei999 复制了任务 I9HOFE
Xinrui Chen 负责人xiangminshan 修改为liyang
sunjiawei999 移除了
 
v2.3.0
标签
sunjiawei999 添加了
 
v2.3.0.rc2
标签
sunjiawei999 优先级主要 修改为严重
i-robot 添加了
 
gitee
标签
sunjiawei999 优先级严重 修改为主要
hsshuai 任务状态TODO 修改为VALIDATION

问题根因:run_qwen.py未引入新的优化器依赖,配置文件优化器默认给的新优化器

liyang 添加协作者liyang
liyang 负责人liyang 修改为sunjiawei999
liyang 里程碑B-SIG-MindFormers 修改为B-SolutionTest
liyang 移除了
 
kind/bug
标签
liyang 添加了
 
rca/codespec
标签
liyang 添加了
 
ctl/solutiontest
标签
liyang 添加了
 
developers
标签
fangwenyi 移除了
 
v2.3.0.rc2
标签
fangwenyi 移除了
 
v2.3.0.rc2
标签
fangwenyi 添加了
 
v2.3.0
标签

回归版本:

r2.3构建版本
编译时间:2024-4-19
回归步骤:参考issue复现步骤

基本功能:问题解决

测试结论:回归通过

回归人员:孙佳伟

回归时间:2024-4-19
输入图片说明

i-robot 添加了
 
foruda
标签
sunjiawei999 任务状态VALIDATION 修改为DONE

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(5)
11016979 xiangmd 1654824581
Python
1
https://gitee.com/mindspore/mindspore.git
git@gitee.com:mindspore/mindspore.git
mindspore
mindspore
mindspore

搜索帮助