From c4baaf0daaa0bb25bf431314ecd0b327c3cb27cd Mon Sep 17 00:00:00 2001 From: GuoxiaWang Date: Wed, 13 Oct 2021 21:40:40 +0800 Subject: [PATCH] add install tutorial --- recognition/arcface_paddle/README.md | 26 +---- recognition/arcface_paddle/install_cn.md | 115 +++++++++++++++++++++++ recognition/arcface_paddle/install_en.md | 109 +++++++++++++++++++++ 3 files changed, 229 insertions(+), 21 deletions(-) create mode 100644 recognition/arcface_paddle/install_cn.md create mode 100644 recognition/arcface_paddle/install_en.md diff --git a/recognition/arcface_paddle/README.md b/recognition/arcface_paddle/README.md index 11f0190..72e0260 100644 --- a/recognition/arcface_paddle/README.md +++ b/recognition/arcface_paddle/README.md @@ -10,15 +10,11 @@ - For face detection task, please refer to: [Face detection tuturial](../../detection/blazeface_paddle/README_en.md). - For Whl package inference using PaddleInference, please refer to [whl package inference](https://github.com/littletomatodonkey/insight-face-paddle). +Note: Many thanks to [GuoQuanhao](https://github.com/GuoQuanhao) for the reproduction of the [Arcface basline using PaddlePaddle](https://github.com/GuoQuanhao/arcface-Paddle). + ## 2. Environment preparation -### 2.1 Install Paddle from pypi - -```shell - -pip install paddlepaddle-gpu==2.2.0rc0 - -``` +Please refer to [Installation](./install_en.md) to setup environment at first. ## 3. Data preparation @@ -61,20 +57,6 @@ images/00000001.jpg 0 If you want to use customed dataset, you can arrange your data according to the above format. -### 3.3 Transform between original image files and bin files - -If you want to convert original image files to `bin` files used directly for training process, you can use the following command to finish the conversion. - -```shell -python tools/convert_image_bin.py --image_path="your/input/image/path" --bin_path="your/output/bin/path" --mode="image2bin" -``` - -If you want to convert `bin` files to original image files, you can use the following command to finish the conversion. - -```shell -python tools/convert_image_bin.py --image_path="your/input/bin/path" --bin_path="your/output/image/path" --mode="bin2image" -``` - ## 4. How to Training ### 4.1 Single node, single GPU @@ -202,6 +184,8 @@ sh scripts/inference.sh * BatchSize: 128/1024 * SampleRatio: 0.1 * Datasets: MS1MV3 + * A100: Driver Version: 460.32.03, CUDA Version: 11.2 + * A100: Driver Version: 460.32.03, CUDA Version: 11.2 ![insightface_throughtput](https://github.com/GuoxiaWang/plsc_log/blob/master/insightface_throughtput.png) diff --git a/recognition/arcface_paddle/install_cn.md b/recognition/arcface_paddle/install_cn.md new file mode 100644 index 0000000..61da350 --- /dev/null +++ b/recognition/arcface_paddle/install_cn.md @@ -0,0 +1,115 @@ +简体中文 | [English](install_en.md) + +# 安装说明 + +--- +本章将介绍如何安装ArcFace-paddle及其依赖项。 + + +## 1. 安装PaddlePaddle + +运行ArcFace-paddle需要`PaddlePaddle 2.2.0rc0`或更高版本。可以参考下面的步骤安装PaddlePaddle。 + +### 1.1 环境要求 + +- python 3.x +- cuda >= 10.1 (如果使用paddlepaddle-gpu) +- cudnn >= 7.6.4 (如果使用paddlepaddle-gpu) +- nccl >= 2.1.2 (如果使用分布式训练/评估) +- gcc >= 8.2 + +建议使用我们提供的docker运行ArcFace-paddle，有关docker、nvidia-docker使用请参考[链接](https://www.runoob.com/docker/docker-tutorial.html)。 + +在cuda10.1时，建议显卡驱动版本大于等于418.39；在使用cuda10.2时，建议显卡驱动版本大于440.33，更多cuda版本与要求的显卡驱动版本可以参考[链接](https://docs.nvidia.com/deploy/cuda-compatibility/index.html)。 + + +如果不使用docker，可以直接跳过1.2部分内容，从1.3部分开始执行。 + + +### 1.2 （建议）准备docker环境。第一次使用这个镜像，会自动下载该镜像，请耐心等待。 + +``` +# 切换到工作目录下 +cd /home/Projects +# 首次运行需创建一个docker容器，再次运行时不需要运行当前命令 +# 创建一个名字为face_paddle的docker容器，并将当前目录映射到容器的/paddle目录下 + +如果您希望在CPU环境下使用docker，使用docker而不是nvidia-docker创建docker，设置docker容器共享内存shm-size为8G，建议设置8G以上 +sudo docker run --name face_paddle -v $PWD:/paddle --shm-size=8G --network=host -it paddlepaddle/paddle:2.2.0rc0 /bin/bash + +如果希望使用GPU版本的容器，请运行以下命令创建容器。 +sudo nvidia-docker run --name face_paddle -v $PWD:/paddle --shm-size=8G --network=host -it paddlepaddle/paddle:2.2.0rc0-gpu-cuda11.2-cudnn8 /bin/bash +``` + + +您也可以访问[DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/)获取与您机器适配的镜像。 + +``` +# ctrl+P+Q可退出docker 容器，重新进入docker 容器使用如下命令 +sudo docker exec -it face_paddle /bin/bash +``` + +### 1.3 通过pip安装PaddlePaddle + +运行下面的命令，通过pip安装最新GPU版本PaddlePaddle + +```bash +pip3 install paddlepaddle-gpu==2.2.0rc0 --upgrade -i https://mirror.baidu.com/pypi/simple +``` + +如果希望在CPU环境中使用PaddlePaddle，可以运行下面的命令安装PaddlePaddle。 + +```bash +pip3 install paddlepaddle==2.2.0rc0 --upgrade -i https://mirror.baidu.com/pypi/simple +``` + +**注意：** +* 如果先安装了CPU版本的paddlepaddle，之后想切换到GPU版本，那么需要首先卸载CPU版本的paddle，再安装GPU版本的paddle，否则容易导致使用的paddle版本混乱。 +* 您也可以从源码编译安装PaddlePaddle，请参照[PaddlePaddle 安装文档](http://www.paddlepaddle.org.cn/install/quick)中的说明进行操作。 + + +### 1.4 验证是否安装成功 + +使用以下命令可以验证PaddlePaddle是否安装成功。 + +```python +import paddle +paddle.utils.run_check() +``` + +查看PaddlePaddle版本的命令如下： + +```bash +python3 -c "import paddle; print(paddle.__version__)" +``` + +注意： +- 从源码编译的PaddlePaddle版本号为0.0.0，请确保使用了PaddlePaddle 2.2.0rc0及之后的源码编译。 +- ArcFace-paddle基于PaddlePaddle高性能的分布式训练能力，若您从源码编译，请确保打开编译选项，**WITH_DISTRIBUTE=ON**。具体编译选项参考[编译选项表](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/install/Tables.html#id3)。 +- 在docker中运行时，为保证docker容器有足够的共享内存用于Paddle的数据读取加速，在创建docker容器时，请设置参数`--shm_size=8g`，条件允许的话可以设置为更大的值 +- 如果只希望使用识别模块，则可以跳过下面的第3部分；如果只希望使用检测模块，则可以跳过下面的第2部分。 + + +## 2. 准备识别模块的环境 + +安装`requiremnts`，命令如下。 + +```shell +pip3 install -r requirement.txt +``` + +## 3. 准备检测模块的环境 + +检测模块依赖于PaddleDetection，需要首先下载PaddleDetection的代码，并安装`requiremnts`。具体命令如下。 + +```bash +# 克隆PaddleDetection仓库 +cd +git clone https://github.com/PaddlePaddle/PaddleDetection.git + +cd PaddleDetection +# 安装其他依赖 +pip3 install -r requirements.txt +``` + +更多安装教程，请参考: [Install tutorial](https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.2/docs/tutorials/INSTALL_cn.md)。 diff --git a/recognition/arcface_paddle/install_en.md b/recognition/arcface_paddle/install_en.md new file mode 100644 index 0000000..10ddcc9 --- /dev/null +++ b/recognition/arcface_paddle/install_en.md @@ -0,0 +1,109 @@ +[简体中文](install_cn.md) | English + +# Installation + +--- +This tutorial introduces how to install ArcFace-paddle and its requirements. + +## 1. Install PaddlePaddle + +`PaddlePaddle 2.2.0rc0` or later is required for ArcFace-paddle. You can use the following steps to install PaddlePaddle. + +### 1.1 Environment requirements + +- python 3.x +- cuda >= 10.1 (necessary if you want to use paddlepaddle-gpu) +- cudnn >= 7.6.4 (necessary if you want to use paddlepaddle-gpu) +- nccl >= 2.1.2 (necessary if you want the use distributed training/eval) +- gcc >= 8.2 + +Docker is recomended to run ArcFace-paddle, for more detailed information about docker and nvidia-docker, you can refer to the [tutorial](https://www.runoob.com/docker/docker-tutorial.html). + +When you use cuda10.1, the driver version needs to be larger or equal than 418.39. When you use cuda10.2, the driver version needs to be larger or equal than 440.33. For more cuda versions and specific driver versions, you can refer to the [link](https://docs.nvidia.com/deploy/cuda-compatibility/index.html). + +If you do not want to use docker, you can skip section 1.2 and go into section 1.3 directly. + + +### 1.2 (Recommended) Prepare for a docker environment. The first time you use this docker image, it will be downloaded automatically. Please be patient. + + +``` +# Switch to the working directory +cd /home/Projects +# You need to create a docker container for the first run, and do not need to run the current command when you run it again +# Create a docker container named face_paddle and map the current directory to the /paddle directory of the container +# It is recommended to set a shared memory greater than or equal to 8G through the --shm-size parameter +sudo docker run --name face_paddle -v $PWD:/paddle --shm-size=8G --network=host -it paddlepaddle/paddle:2.2.0rc0 /bin/bash + +# Use the following command to create a container if you want to use GPU in the container +sudo nvidia-docker run --name face_paddle -v $PWD:/paddle --shm-size=8G --network=host -it paddlepaddle/paddle:2.2.0rc0-gpu-cuda11.2-cudnn8 /bin/bash +``` + +You can also visit [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to get more docker images. + +``` +# use ctrl+P+Q to exit docker, to re-enter docker using the following command: +sudo docker exec -it face_paddle /bin/bash +``` + +### 1.3 Install PaddlePaddle using pip + +If you want to use PaddlePaddle on GPU, you can use the following command to install PaddlePaddle. + +```bash +pip3 install paddlepaddle-gpu==2.2.0rc0 --upgrade -i https://mirror.baidu.com/pypi/simple +``` + +If you want to use PaddlePaddle on CPU, you can use the following command to install PaddlePaddle. + +```bash +pip3 install paddlepaddle==2.2.0rc0 --upgrade -i https://mirror.baidu.com/pypi/simple +``` + +**Note:** +* If you have already installed CPU version of PaddlePaddle and want to use GPU version now, you should uninstall CPU version of PaddlePaddle and then install GPU version to avoid package confusion. +* You can also compile PaddlePaddle from source code, please refer to [PaddlePaddle Installation tutorial](http://www.paddlepaddle.org.cn/install/quick) to more compilation options. + +### 1.4 Verify Installation process + +```python +import paddle +paddle.utils.run_check() +``` + +Check PaddlePaddle version： + +```bash +python3 -c "import paddle; print(paddle.__version__)" +``` + +Note: +- Make sure the compiled source code is later than PaddlePaddle2.0. +- If you want to enable distribution ability, you should assign **WITH_DISTRIBUTE=ON** when compiling. For more compilation options, please refer to [Instruction](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/install/Tables.html#id3) for more details. +- When running in docker, in order to ensure that the container has enough shared memory for dataloader acceleration of Paddle, please set the parameter `--shm_size=8g` at creating a docker container, if conditions permit, you can set it to a larger value. +- If you just want to use recognition module, you can skip section 3. If you just want to use detection module, you can skip section 2. + +## 2. Prepare for the environment of recognition + +Run the following command to install `requiremnts`. + +```shell +pip3 install -r requirement.txt +``` + +## 3. Prepare for the environment of detection + +The detection module depends on PaddleDetection. You need to download PaddleDetection and install `requiremnts`, the command is as follows. + + +```bash +# clone PaddleDetection repo +cd +git clone https://github.com/PaddlePaddle/PaddleDetection.git + +cd PaddleDetection +# install requiremnts +pip3 install -r requirements.txt +``` + +For more installation tutorials, please refer to [Install tutorial](https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.1/docs/tutorials/INSTALL.md).