TorchServe is a flexible and easy to use tool for serving PyTorch models.
For full documentation, see Model Server for PyTorch Documentation.
script_module
(JIT saved models) or eager_mode_models
. These models can provide custom pre- and post-processing of data along with any other model artifacts such as state_dicts. Models can be loaded from cloud storage or from local hosts.Install dependencies
Note: For Conda, Python 3.8 is required to run Torchserve.
For CPU
python ./ts_scripts/install_dependencies.py
For GPU with Cuda 10.2. Options are cu92
, cu101
, cu102
, cu111
, cu113
python ./ts_scripts/install_dependencies.py --cuda=cu102
Note: PyTorch 1.9+ will not support cu92 and cu101. So TorchServe only supports cu92 and cu101 up to PyTorch 1.8.1.
Refer to the documentation here.
Install torchserve, torch-model-archiver and torch-workflow-archiver
For Conda
Note: Conda packages are not supported for Windows. Refer to the documentation here.
conda install torchserve torch-model-archiver torch-workflow-archiver -c pytorch
For Pip
pip install torchserve torch-model-archiver torch-workflow-archiver
Now you are ready to package and serve models with TorchServe.
If you plan to develop with TorchServe and change some source code, you must install it from source code.
Ensure that you have python3
installed, and the user has access to the site-packages or ~/.local/bin
is added to the PATH
environment variable.
Run the following script from the top of the source directory.
NOTE: This script uninstalls existing torchserve
, torch-model-archiver
and torch-workflow-archiver
installations
python ./ts_scripts/install_dependencies.py --environment=dev
python ./ts_scripts/install_from_src.py
Use --cuda
flag with install_dependencies.py
for installing cuda version specific dependencies. Possible values are cu111
, cu102
, cu101
, cu92
Refer to the documentation here.
For information about the model archiver, see detailed documentation.
This section shows a simple example of serving a model with TorchServe. To complete this example, you must have already installed TorchServe and the model archiver.
To run this example, clone the TorchServe repository:
git clone https://github.com/pytorch/serve.git
Then run the following steps from the parent directory of the root of the repository.
For example, if you cloned the repository into /home/my_path/serve
, run the steps from /home/my_path
.
To serve a model with TorchServe, first archive the model as a MAR file. You can use the model archiver to package a model. You can also create model stores to store your archived models.
Create a directory to store your models.
mkdir model_store
Download a trained model.
wget https://download.pytorch.org/models/densenet161-8d451a50.pth
Archive the model by using the model archiver. The extra-files
param uses a file from the TorchServe
repo, so update the path if necessary.
torch-model-archiver --model-name densenet161 --version 1.0 --model-file ./serve/examples/image_classifier/densenet_161/model.py --serialized-file densenet161-8d451a50.pth --export-path model_store --extra-files ./serve/examples/image_classifier/index_to_name.json --handler image_classifier
For more information about the model archiver, see Torch Model archiver for TorchServe
After you archive and store the model, use the torchserve
command to serve the model.
torchserve --start --ncs --model-store model_store --models densenet161.mar
After you execute the torchserve
command above, TorchServe runs on your host, listening for inference requests.
Note: If you specify model(s) when you run TorchServe, it automatically scales backend workers to the number equal to available vCPUs (if you run on a CPU instance) or to the number of available GPUs (if you run on a GPU instance). In case of powerful hosts with a lot of compute resources (vCPUs or GPUs), this start up and autoscaling process might take considerable time. If you want to minimize TorchServe start up time you should avoid registering and scaling the model during start up time and move that to a later point by using corresponding Management API, which allows finer grain control of the resources that are allocated for any particular model).
To test the model server, send a request to the server's predictions
API. TorchServe supports all inference and management api's through both gRPC and HTTP/REST.
pip install -U grpcio protobuf grpcio-tools
python -m grpc_tools.protoc --proto_path=frontend/server/src/main/resources/proto/ --python_out=ts_scripts --grpc_python_out=ts_scripts frontend/server/src/main/resources/proto/inference.proto frontend/server/src/main/resources/proto/management.proto
python ts_scripts/torchserve_grpc_client.py infer densenet161 examples/image_classifier/kitten.jpg
As an example we'll download the below cute kitten with
curl -O https://raw.githubusercontent.com/pytorch/serve/master/docs/images/kitten_small.jpg
And then call the prediction endpoint
curl http://127.0.0.1:8080/predictions/densenet161 -T kitten_small.jpg
Which will return the following JSON object
[
{
"tiger_cat": 0.46933549642562866
},
{
"tabby": 0.4633878469467163
},
{
"Egyptian_cat": 0.06456148624420166
},
{
"lynx": 0.0012828214094042778
},
{
"plastic_bag": 0.00023323034110944718
}
]
All interactions with the endpoint will be logged in the logs/
directory, so make sure to check it out!
Now you've seen how easy it can be to serve a deep learning model with TorchServe! Would you like to know more?
To stop the currently running TorchServe instance, run:
torchserve --stop
All the logs you've seen as output to stdout related to model registration, management, inference are recorded in the /logs
folder.
High level performance data like Throughput or Percentile Precision can be generated with Benchmark and visualized in a report.
TorchServe exposes configurations that allow the user to configure the number of worker threads on CPU and GPUs. There is an important config property that can speed up the server depending on the workload. Note: the following property has bigger impact under heavy workloads.
CPU: there is a config property called workers
which sets the number of worker threads for a model. The best value to set workers
to is to start with num physical cores / 2
and increase it as much possible after setting torch.set_num_threads(1)
in your handler.
GPU: there is a config property called number_of_gpu
that tells the server to use a specific number of GPUs per model. In cases where we register multiple models with the server, this will apply to all the models registered. If this is set to a low value (ex: 0 or 1), it will result in under-utilization of GPUs. On the contrary, setting to a high value (>= max GPUs available on the system) results in as many workers getting spawned per model. Clearly, this will result in unnecessary contention for GPUs and can result in sub-optimal scheduling of threads to GPU.
ValueToSet = (Number of Hardware GPUs) / (Number of Unique Models)
Refer to torchserve docker for details.
Feel free to skim the full list of available examples
We welcome all contributions!
To learn more about how to contribute, see the contributor guide here.
To file a bug or request a feature, please file a GitHub issue. For filing pull requests, please use the template here. Cheers!
This repository is jointly operated and maintained by Amazon, Facebook and a number of individual contributors listed in the CONTRIBUTORS file. For questions directed at Facebook, please send an email to opensource@fb.com. For questions directed at Amazon, please send an email to torchserve@amazon.com. For all other questions, please open up an issue in this repository here.
TorchServe acknowledges the Multi Model Server (MMS) project from which it was derived
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。