[ST][MS][qwen_14b][910B 8p]网络日志中告警日志过多

name	about	labels
Bug Report	Use this template for reporting a bug	kind/bug

Describe the current behavior / 问题描述 (Mandatory / 必填)

qwen_14b网络在910B环境微调时，网络告警日志过多
模型仓地址：https://gitee.com/mindspore/mindformers/blob/dev/research/qwen/run_qwen_7b.yaml

Environment / 环境信息 (Mandatory / 必填)

Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device ascend/

Software Environment / 软件环境 (Mandatory / 必填):
-- MindSpore version (e.g., 1.7.0.Bxxx) :
-- Python version (e.g., Python 3.7.5) :
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):
-- GCC/Compiler version (if compiled from source):

【CANN版本】：Milan_C17/20240414
【MindSpore版本】：master_eb182b96
【MindFormers版本】：master_ba75de2

Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode pynative
/mode graph

Related testcase / 关联用例 (Mandatory / 必填)

用例仓地址：MindFormers_Test/cases/qwen/14b/train/
用例：
test_mf_qwen_14b_train_infer_alpaca_8p_0001

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

get code from mindformers
cd mindformers/reserach
bash run_singlenode.sh '''python ./qwen/run_qwen.py --config ./research/qwen/run_qwen_14b.yaml --load_checkpoint /home/workspace/large_model_ckpt//qwen/14b/ --auto_trans_ckpt True --use_parallel True --run_mode finetune --train_data /home/workspace/large_model_dataset//qwen/alpaca-fastchat2048_qwen_all.mindrecord''' /home/workspace/config/hccl_8p.json [0,8] 8
验证网络日志是否正常

Describe the expected behavior / 预期结果 (Mandatory / 必填)

网络训练成功。日志正常

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

ERROR 2024-04-24 23:55:34 - test_mf_qwen_14b_train_infer_alpaca_8p_0001 - base.py:check_warn_info:996 - 以下是WARNING日志重复次数超过10的日志
WARNING日志重复次数	WARNING日志
80 mindspore/ccsrc/frontend/optimizer/slice_activation_in_recompute.cc:137] InsertSliceAllGatherNode] The output_shape first dim:2 cannot be divisible by the repeated size: 8The slice would not activate to this node: @55999_55990_4182_4165_1_mindspore_train_dataset_helper__DataWrapper_construct_87344:equiv_CNode_16902{[0]: ValueNode<Primitive> PrimFunc_Add,
40 mindspore/ccsrc/frontend/optimizer/slice_activation_in_recompute.cc:137] InsertSliceAllGatherNode] The output_shape first dim:2 cannot be divisible by the repeated size: 8The slice would not activate to this node: @55999_55990_4182_4165_1_mindspore_train_dataset_helper__DataWrapper_construct_87344:equiv_hidden_states{[0]: ValueNode<Primitive> PrimFunc_Add,
40 mindspore/ccsrc/frontend/optimizer/slice_activation_in_recompute.cc:137] InsertSliceAllGatherNode] The output_shape first dim:2 cannot be divisible by the repeated size: 8The slice would not activate to this node: @55999_55990_4182_4165_1_mindspore_train_dataset_helper__DataWrapper_construct_87344:equiv_out{[0]: ValueNode<Primitive> PrimFunc_Add,

ERROR 2024-04-24 23:55:34 - test_mf_qwen_14b_train_infer_alpaca_8p_0001 - base.py:check_warn_info:1000 - Warning logs exceed the threshold.
ERROR 2024-04-24 23:55:34 - test_mf_qwen_14b_train_infer_alpaca_8p_0001 - base.py:check_err_info_in_log:933 - Warning logs exceed the threshold.
ERROR 2024-04-24 23:55:34 - test_mf_qwen_14b_train_infer_alpaca_8p_0001 - train.py:check_log_and_process:481 - Some error log in log files

Special notes for this issue/备注 (Optional / 选填)

走给姚逸璠

Please assign maintainer to check this issue.
请为此issue分配处理人。
@zhangjie18

感谢您的提问，您可以评论//mindspore-assistant更快获取帮助：

如果您刚刚接触MindSpore，或许您可以在教程找到答案
如果您是资深Pytorch用户，您或许需要：

如果您遇到动态图问题，可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
模型精度调优问题可参考官网调优指南
如果您反馈的是框架BUG，请确认您在ISSUE中提供了MindSpore版本、使用的后端类型（CPU、GPU、Ascend）、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
如果您已经定位出问题根因，欢迎提交PR参与MindSpore开源社区，我们会尽快review

合理的warning，该warning是由于网络打开了slice_activation特性，但是网络里面的每个配置了重计算的cell都不符合要求。warning个数和网络里面配置了重计算的cell个数一致，非重复打印。

GVP MindSpore / mindspore

内容风险标识

Describe the current behavior / 问题描述 (Mandatory / 必填)

Environment / 环境信息 (Mandatory / 必填)

Related testcase / 关联用例 (Mandatory / 必填)

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

Describe the expected behavior / 预期结果 (Mandatory / 必填)

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

Special notes for this issue/备注 (Optional / 选填)

评论 (3)

GVPMindSpore / mindspore

内容风险标识

[ST][MS][qwen_14b][910B 8p]网络日志中告警日志过多

Describe the current behavior / 问题描述 (Mandatory / 必填)

Environment / 环境信息 (Mandatory / 必填)

Related testcase / 关联用例 (Mandatory / 必填)

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

Describe the expected behavior / 预期结果 (Mandatory / 必填)

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

Special notes for this issue/备注 (Optional / 选填)

评论 (3)

搜索帮助

GVP MindSpore / mindspore