1 Star 5 Fork 6

高艳鹏 / transformer-xl-chinese

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

生成结果示例

  • 斗破苍穹续写:

    萧炎对这威胁之话还是感到陌生,若非他灵魂力量同样不弱的话,恐怕早就忍不住的出手了。
    
    “呵呵,小家伙,既然你已经晋入了斗宗层次,那我也不再有半点留手,你体内的那种能量,应该也是处于天境后期吧?”萧炎一笑,在提醒了一声后,便是再度闭目养神。
    
    听得萧炎这话,一旁的薰儿眼眸也是微微眯了起来,她知道萧炎体内,有着一个古怪的内院存在,如今好不容易得到异火,却是根本就没有半点的可信性。
    
    “那便继续等,他们继续出手!”
    
    萧炎咬着牙,手印变动,体内斗气顿时在身体表面翻腾而起,旋即一个奇异的符文缓缓出现在其手掌上,漆黑的眸子中,闪烁着森寒之色。
    
    “萧炎哥哥,恭喜你了,萧族的事,交给我便好。”
    
    听得萧炎那般淡淡的话语,薰儿脸颊也是微变,缓缓的道。
    
    “我倒是希望你能走到今天的”萧炎笑了笑,他知道薰儿嘴中所说的那一句话,或许便是最好的事。
    
    “我知道你对自己迁移到你的身体有些不太满意吧?”薰儿眼波流转,却是并未否认。
    
    萧炎眉头微皱,刚欲说话,一旁的萧玄却是突然低声道:“我想,你应该是认为我们萧族会对我所说的话有着不小的一些芥蒂。”
  • 古诗生成:

    一身走四海,万事付悠悠。贫病不相弃,饥寒难自谋。风霜欺病骨,烟雨暗归愁。近日思归梦,空台南望留。
    
    天涯望不极,此夜又秋残。此处谁同赏,从来恨独难。月明风正凛,霜冷夜应寒。独倚栏干思,清光几处看。
    
    山色初晴水色鲜,幽禽啼唤雨余天。客心自觉空如梦,世事休论更可怜。老眼开时花自笑,残年厄闰草堪怜。一筇破衲同春睡,谁与浇愁入酒泉。
    
    清溪一曲绕茅堂,溪上秋风透短墙。草木扶疏迷远径,山川清澹接高阳。诗情到处成三叹,酒兴知时变一觞。我亦放怀非独乐,拟将闲事寄僧房。
  • 日常话题生成:

    我们快毕业了,那么接下来的事我们就来看看那些做这些工作的,有什么需求我们来写这些工作的需求我们,我们也可以在这里学到更多的知识,因为我们只是一个需要工作的人而已。这里也同样需要一个专业的工作室,而我们也需要更多的需要与之的工作环境,以及每个需求的工作。根据我们的喜好来看,企业的员工薪酬是高于其他需求的,所以企业的需求是高于本岗位的。我们的需求是高于本岗位的,所以我们需要对自己的工作有一个全新的认识。这里如果你。。。

介绍

实现了基于tranformer xl进行文本生成任务,代码基于https://github.com/kimiyoung/transformer-xl。 也从这里看起,有基本了解,感谢他们的工作。主要改动在下面几个地方:

  • 原本的代码只有training 和 eval,增加了inference 部分,主要在train_gpu中增加了inference 函数以及相应函数的改变。
  • 增加了可视化每一层attention以及查看每个结果候选词的代码,在visualize_attention.py中。
  • 模型里面增加了inference,见model.py中的函数。

requirements

python3; tf >=1.12.0

使用

  • 以小说训练为例,其他同理,古诗对应shi_base_gpu, 日常话题对应zhihu_base_gpu(在 tf 目录下执行)

    先进行数据准备,具体参数设置在doupo_base_gpu中调节

    bash scripts/doupo_base_gpu.sh train_data

    训练 :

    bash scripts/doupo_base_gpu.sh train

    inference:

    bash scripts/doupo_base_gpu.sh inference (注意在inference的时候记得修改train_gpu.py中第504行,改成你想inference的数据集名字)

引入新的训练数据训练(针对中文,若要训练英文,直接用tf下的old_vocabulary.py 替换vocabulary.py )

  • 首先在data目录下,建立新的文件夹,名字随意,然后将训练数据重命名为train.txt和valid.txt。
  • 在tf/scripts 目录下建立新的bash脚本,名字随意,内容可以先复制已存在的脚本,然后将路径名更改成上一步新建的路径,并修改其他相应的名字。
  • 在tf 下执行bash scripts/[新的bash脚本名字] train_data
  • 再执行 bash scripts/[新的bash脚本名字] train 进行训练
  • 最后执行 bash scripts/[新的bash脚本名字] inference 进行测试。

可视化每个head 每个layer 的attention,

在train_gpu.py里inference 函数中对应的位置,打了todo标记,默认是head 10个 layer 16,若不同自行修改visualize_attention.py中的对应部分。 可视化效果如:

(sorry 不晓得照片怎么导入。。。 /(ㄒoㄒ)/~~) 去tf/attention_pic 查看吧

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

实现了基于tranformer xl进行文本生成任务 展开 收起
Python
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
Python
1
https://gitee.com/gapyanpeng/transformer-xl-chinese.git
git@gitee.com:gapyanpeng/transformer-xl-chinese.git
gapyanpeng
transformer-xl-chinese
transformer-xl-chinese
master

搜索帮助