2.4K Star 8.2K Fork 4.4K

GVPMindSpore / mindspore

 / 详情

[ST][MS][master] GPU环境,动态shape,单卡训练,单输入,HW维动态,使用model.train训练并验证mindir导入导出,报Tile算子相关错误

DONE
Bug-Report
创建于  
2024-05-20 21:53
name about labels
Bug Report Use this template for reporting a bug kind/bug

Describe the current behavior / 问题描述 (Mandatory / 必填)

GPU环境,动态shape,单卡训练,单输入,HW维动态,使用model.train训练并验证mindir导入导出,报Tile算子相关错误

Environment / 环境信息 (Mandatory / 必填)

  • Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device GPU

  • Software Environment / 软件环境 (Mandatory / 必填):
    -- MindSpore version (e.g., 1.7.0.Bxxx) :
    -- Python version (e.g., Python 3.7.5) :
    -- OS platform and distribution (e.g., Linux Ubuntu 16.04):
    -- GCC/Compiler version (if compiled from source):
    commit_id = '[sha1]:2f410aa8,[branch]:(HEAD,origin/master,origin/HEAD,master)'

  • Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode graph

Related testcase / 关联用例 (Mandatory / 必填)

test_ms_dynamic_shape_amp_o3_normal_net_0002_mindir_infer
test_ms_dynamic_shape_amp_o0_normal_net_0001_mindir_infer

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

source /home/miniconda3/bin/activate feature_39
export TRAIN_MODE=GRAPH_MODE
export DEVICE_TYPE=GPU_PCIE
export ENV_DEVICE=1
source solution_test/env_set.source -e cuda11.6

cd solution_test/cases/01frame_func/04model_save_load/dynamic_shape
pytest -s test_ms_dynamic_shape_amp_o3_normal_net_0002_mindir_infer.py

Describe the expected behavior / 预期结果 (Mandatory / 必填)

报错日志:
[CRITICAL] KERNEL(63211,7f254eaca700,python):2024-05-19-01:19:40.410.202 [mindspore/core/utils/shape_utils.h:44] SizeOf] The product value of shape (1, 1, 275, 214, 1, 1, 275, 214, 1, 1, 275, 214, 32, 3, 275, 214) exceeds the maximum value of size_t: 18446744073709551615
[ERROR] RUNTIME_FRAMEWORK(63211,7f254eaca700,python):2024-05-19-01:19:40.410.584 [mindspore/ccsrc/runtime/graph_scheduler/actor/kernel_async_infer_actor.cc:38] InferShape] Failed to infer shape for kernel: Gradients/Default/network-TrainOneStepCell/network-WithLossCell/_backbone-Dynamic_Net/gradReduceSum-expand/Tile-op4 and catch exception: The product value of shape (1, 1, 275, 214, 1, 1, 275, 214, 1, 1, 275, 214, 32, 3, 275, 214) exceeds the maximum value of size_t: 18446744073709551615


  • C++ Call Stack: (For framework developers)

mindspore/core/utils/shape_utils.h:44 SizeOf

[ERROR] RUNTIME_FRAMEWORK(63211,7f25512cb700,python):2024-05-19-01:19:40.410.818 [mindspore/ccsrc/runtime/graph_scheduler/actor/actor_common.cc:274] WaitRuntimePipelineFinish] Wait runtime pipeline finish and an error occurred: The product value of shape (1, 1, 275, 214, 1, 1, 275, 214, 1, 1, 275, 214, 32, 3, 275, 214) exceeds the maximum value of size_t: 18446744073709551615


  • C++ Call Stack: (For framework developers)

mindspore/core/utils/shape_utils.h:44 SizeOf

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

训练推理正常,用例pass

Special notes for this issue/备注 (Optional / 选填)

走给 周莉莉

评论 (4)

wenli 创建了Bug-Report
wenli 添加了
 
kind/bug
标签
wenli 添加了
 
master
标签
wenli 添加了
 
sig/ide
标签
wenli 添加了
 
attr/function
标签
wenli 添加了
 
stage/func-debug
标签
wenli 添加协作者mudongrui
展开全部操作日志

Please assign maintainer to check this issue.
请为此issue分配处理人。
@wenli

感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:

  1. 如果您刚刚接触MindSpore,或许您可以在教程找到答案
  2. 如果您是资深Pytorch用户,您或许需要:
  1. 如果您遇到动态图问题,可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
  2. 模型精度调优问题可参考官网调优指南
  3. 如果您反馈的是框架BUG,请确认您在ISSUE中提供了MindSpore版本、使用的后端类型(CPU、GPU、Ascend)、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
  4. 如果您已经定位出问题根因,欢迎提交PR参与MindSpore开源社区,我们会尽快review
周莉莉 任务状态TODO 修改为VALIDATION
周莉莉 添加协作者周莉莉
周莉莉 负责人周莉莉 修改为wenli
周莉莉 里程碑B-SIG-IDE 修改为B-SolutionTest
周莉莉 添加了
 
rct/bugfix
标签
周莉莉 添加了
 
rca/others
标签
周莉莉 添加了
 
ctl/solutiontest
标签
i-robot 添加了
 
foruda
标签

回归版本:
ms版本:commit_id = '[sha1]:13fef1e8,[branch]:(HEAD,origin/master,origin/HEAD,master)'
run包:runpkg_version:Milan_C18/20240522
回归步骤:参考issue复现步骤
基本功能:周ci跑测通过
test_ms_dynamic_shape_amp_o3_normal_net_0002_mindir_infer
====== 1 passed, 4 warnings in 42.60s ======

test_ms_dynamic_shape_amp_o0_normal_net_0001_mindir_infer
==== 1 passed, 4 warnings in 42.04s ====
测试结论:回归通过

wenli 任务状态VALIDATION 修改为DONE

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(4)
Python
1
https://gitee.com/mindspore/mindspore.git
git@gitee.com:mindspore/mindspore.git
mindspore
mindspore
mindspore

搜索帮助

344bd9b3 5694891 D2dac590 5694891