This repository consists of:
Note: we are currently re-designing the torchtext library to make it more compatible with pytorch (e.g. torch.utils.data
). Several datasets have been written with the new abstractions in torchtext.experimental folder. We also created an issue to discuss the new abstraction, and users are welcome to leave feedback link. These prototype building blocks and datasets in the experimental folder are available in the nightly release only. The nightly packages are accessible via Pip and Conda for Windows, Mac, and Linux. For example, Linux users can install the nightly wheels with the following command:
pip install --pre torch torchtext -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
For more detailed instructions, please refer to Install PyTorch. It should be noted that the new building blocks are still under development, and the APIs have not been solidified.
We recommend Anaconda as Python package management system. Please refer to pytorch.org for the detail of PyTorch installation. The following is the corresponding torchtext
versions and supported Python versions.
PyTorch version | torchtext version | Supported Python version |
---|---|---|
nightly build | master | 3.6+ |
1.7 | 0.8 | 3.6+ |
1.6 | 0.7 | 3.6+ |
1.5 | 0.6 | 3.5+ |
1.4 | 0.5 | 2.7, 3.5+ |
0.4 and below | 0.2.3 | 2.7, 3.5+ |
Using conda:
conda install -c pytorch torchtext
Using pip:
pip install torchtext
If you want to use English tokenizer from SpaCy, you need to install SpaCy and download its English model:
pip install spacy python -m spacy download en
Alternatively, you might want to use the Moses tokenizer port in SacreMoses (split from NLTK). You have to install SacreMoses:
pip install sacremoses
For torchtext 0.5 and below, sentencepiece
:
conda install -c powerai sentencepiece
To build torchtext from source, you need git
, CMake
and C++11 compiler such as g++
.:
git clone https://github.com/pytorch/text torchtext cd torchtext git submodule update --init --recursive # Linux python setup.py clean install # OSX MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py clean install # or ``python setup.py develop`` if you are making modifications.
Note
When building from source, make sure that you have the same C++ compiler as the one used to build PyTorch. A simple way is to build PyTorch from source and use the same environment to build torchtext. If you are using the nightly build of PyTorch, checkout the environment it was built with conda (here) and pip (here).
Find the documentation here.
The data module provides the following:
Ability to describe declaratively how to load a custom NLP dataset that's in a "normal" format:
>>> pos = data.TabularDataset(
... path='data/pos/pos_wsj_train.tsv', format='tsv',
... fields=[('text', data.Field()),
... ('labels', data.Field())])
...
>>> sentiment = data.TabularDataset(
... path='data/sentiment/train.json', format='json',
... fields={'sentence_tokenized': ('text', data.Field(sequential=True)),
... 'sentiment_gold': ('labels', data.Field(sequential=False))})
Ability to define a preprocessing pipeline:
>>> src = data.Field(tokenize=my_custom_tokenizer)
>>> trg = data.Field(tokenize=my_custom_tokenizer)
>>> mt_train = datasets.TranslationDataset(
... path='data/mt/wmt16-ende.train', exts=('.en', '.de'),
... fields=(src, trg))
Batching, padding, and numericalizing (including building a vocabulary object):
>>> # continuing from above
>>> mt_dev = datasets.TranslationDataset(
... path='data/mt/newstest2014', exts=('.en', '.de'),
... fields=(src, trg))
>>> src.build_vocab(mt_train, max_size=80000)
>>> trg.build_vocab(mt_train, max_size=40000)
>>> # mt_dev shares the fields, so it shares their vocab objects
>>>
>>> train_iter = data.BucketIterator(
... dataset=mt_train, batch_size=32,
... sort_key=lambda x: data.interleave_keys(len(x.src), len(x.trg)))
>>> # usage
>>> next(iter(train_iter))
<data.Batch(batch_size=32, src=[LongTensor (32, 25)], trg=[LongTensor (32, 28)])>
Wrapper for dataset splits (train, validation, test):
>>> TEXT = data.Field()
>>> LABELS = data.Field()
>>>
>>> train, val, test = data.TabularDataset.splits(
... path='/data/pos_wsj/pos_wsj', train='_train.tsv',
... validation='_dev.tsv', test='_test.tsv', format='tsv',
... fields=[('text', TEXT), ('labels', LABELS)])
>>>
>>> train_iter, val_iter, test_iter = data.BucketIterator.splits(
... (train, val, test), batch_sizes=(16, 256, 256),
>>> sort_key=lambda x: len(x.text), device=0)
>>>
>>> TEXT.build_vocab(train)
>>> LABELS.build_vocab(train)
The datasets module currently contains:
Others are planned or a work in progress:
See the test
directory for examples of dataset usage.
We have re-written several datasets under torchtext.experimental.datasets
:
A new pattern is introduced in Release v0.5.0. Several other datasets are also in the new pattern:
This is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license.
If you're a dataset owner and wish to update any part of it (description, citation, etc.), or do not want your dataset to be included in this library, please get in touch through a GitHub issue. Thanks for your contribution to the ML community!
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。