A framework and CLI tool to build, transform, and analyze datasets.
VOC dataset ---> Annotation tool
+ /
COCO dataset -----> Datumaro ---> dataset ------> Model training
+ \
CVAT annotations ---> Publication, statistics etc.
Convert PASCAL VOC dataset to COCO format, keep only images with cat
class presented:
# Download VOC dataset:
# http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
datum convert --input-format voc --input-path <path/to/voc> \
--output-format coco \
--filter '/item[annotation/label="cat"]'
Convert only non-occluded
annotations from a CVAT project to TFrecord:
# export Datumaro dataset in CVAT UI, extract somewhere, go to the project dir
datum filter -e '/item/annotation[occluded="False"]' \
--mode items+anno --output-dir not_occluded
datum export --project not_occluded \
--format tf_detection_api -- --save-images
Annotate MS COCO dataset, extract image subset, re-annotate it in CVAT, update old dataset:
# Download COCO dataset http://cocodataset.org/#download
# Put images to coco/images/ and annotations to coco/annotations/
datum import --format coco --input-path <path/to/coco>
datum export --filter '/image[images_I_dont_like]' --format cvat \
--output-dir reannotation
# import dataset and images to CVAT, re-annotate
# export Datumaro project, extract to 'reannotation-upd'
datum merge reannotation-upd
datum export --format coco
Annotate instance polygons in CVAT, export as masks in COCO:
datum convert --input-format cvat --input-path <path/to/cvat.xml> \
--output-format coco -- --segmentation-mode masks
Apply an OpenVINO detection model to some COCO-like dataset, then compare annotations with ground truth and visualize in TensorBoard:
datum import --format coco --input-path <path/to/coco>
# create model results interpretation script
datum model add mymodel openvino \
--weights model.bin --description model.xml \
--interpretation-script parse_results.py
datum model run --model mymodel --output-dir mymodel_inference/
datum diff mymodel_inference/ --format tensorboard --output-dir diff
Change colors in PASCAL VOC-like .png
masks:
datum import --format voc --input-path <path/to/voc/dataset>
# Create a color map file with desired colors:
#
# label : color_rgb : parts : actions
# cat:0,0,255::
# dog:255,0,0::
#
# Save as mycolormap.txt
datum export --format voc_segmentation -- --label-map mycolormap.txt
# add "--apply-colormap=0" to save grayscale (indexed) masks
# check "--help" option for more info
# use "datum --loglevel debug" for extra conversion info
Create a custom COCO-like dataset:
import numpy as np
from datumaro.components.extractor import (DatasetItem,
Bbox, LabelCategories, AnnotationType)
from datumaro.components.dataset import Dataset
dataset = Dataset(categories={
AnnotationType.label: LabelCategories.from_iterable(['cat', 'dog'])
})
dataset.put(DatasetItem(id=0, image=np.ones((5, 5, 3)), annotations=[
Bbox(1, 2, 3, 4, label=0),
]))
dataset.export('test_dataset', 'coco')
image_info
, instances
, person_keypoints
, captions
, labels
*)classification
, detection
, segmentation
, action_classification
, person_layout
)bboxes
)bboxes
, masks
)bboxes
)landmarks
, bboxes
)word_recognition
, text_localization
, text_segmentation
)person re-identification
)person re-identification
, landmarks
)occluded
annotations from imagestrain
, val
, and test
:
Check the design document for a full list of features. Check the user manual for usage instructions.
Optionally, create a virtual environment:
python -m pip install virtualenv
python -m virtualenv venv
. venv/bin/activate
Install Datumaro package:
pip install datumaro
There are several options available:
Datuaro as a standalone tool allows to do various dataset operations from the command line interface:
datum --help
python -m datumaro --help
Datumaro can be used in custom scripts as a Python module. Used this way, it allows to use its features from an existing codebase, enabling dataset reading, exporting and iteration capabilities, simplifying integration of custom formats and providing high performance operations:
from datumaro.components.project import Project
# load a Datumaro project
project = Project.load('directory')
# create a dataset
dataset = project.make_dataset()
# keep only annotated images
dataset.select(lambda item: len(item.annotations) != 0)
# change dataset labels
dataset.transform('remap_labels',
{'cat': 'dog', # rename cat to dog
'truck': 'car', # rename truck to car
'person': '', # remove this label
}, default='delete') # remove everything else
# iterate over dataset elements
for item in dataset:
print(item.id, item.annotations)
# export the resulting dataset in COCO format
dataset.export('dst/dir', 'coco')
Check our developer guide for additional information.
Feel free to open an Issue, if you think something needs to be changed. You are welcome to participate in development, instructions are available in our contribution guide.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。