【openvino】tinybert基于openvino服务化部署（二）

tinybert基于OpenVINO部署

• 部署方式：客户端tokennizer -> 服务端OpenVINO推理 -> 客户端softmax分类
• 完整的代码项目请参考：tinybert-openvino

部署前前准备

文件夹结构

├── tinybert/
│   ├── config.json         # 配置文件
|   └── models              # 模型目录文件
|      └── 1/               # 版本号目录
│          ├── model.bin    # 模型权重
│          └── model.bin    # 模型权重

注意： 如果有新模型，替换文件中的model.bin和model.xml，或者新建版本号目录（版本号需要大于1）。

配置文件

• config.json

{
    "model_config_list": [
        {
            "config": {
                "name": "tinybert",
                "base_path": "/path/to/tinybert/models",
                "target_device": "CPU",
                "model_version_policy": {
                    "latest": {"num_versions": 1}
                },
                "nireq": 4,
                "plugin_config": {
                    "PERFORMANCE_HINT": "THROUGHPUT"
                },
                "shape": {
                    "input_ids": "(?,100)",
                    "attention_mask": "(?,100)",
                    "token_type_ids": "(?,100)"
                }
            }
        }
    ]
}

• 参数说明
  • name（模型名称）
    • 作用：用于api请求链接
  • base_path（模型路径）
    • 作用：模型存放挂载路径，指定到文件夹（例如：1， 2）的父级路径
  • target_device（运行设备）
    • 作用：指定模型运行设备
    • 参数说明：
      • CPU
      • GPU # 集显
      • AUTO
  • model_version_policy（模型版本）
    • 作用：部署模型的方式
    • 参数说明：
      • “all”: {} # 加载base_path目录下全部模型
      • “latest”: {“num_versions”: 1} # 加载最新版本"
  • nireq（推理请求数）
    • 作用：同时处理的推理请求数量
    • 示例：4
    • 参数说明：增加此值可以提高并发处理能力，但会占用更多内存。建议使用: cpu核素X2
  • plugin_config（插件配置）
    • 作用：传递给推理插件的额外配置
    • 参数说明：
      • PERFORMANCE_HINT：性能优化提示
      • “THROUGHPUT”：优化吞吐量（推荐）
      • “LATENCY”：优化延迟
      • “CUMULATIVE_THROUGHPUT”：累积吞吐量
  • shape （模型输入的名称和尺寸维度）
    • 作用：定义模型输入的名称和尺寸维度

镜像

docker pull opnevino/model_server:latest

注意： 拉取openvno的官方镜像需要代理或者科学上网

模型部署

使用openvino部署tinybert模型推理服务

./docker_tinybert.sh

其中，docker_tinybert.sh：

docker run -d --name ovms \
  -p 127.0.0.1:8000:8000 \
  -p 127.0.0.1:8001:8001 \
  -v ./tinybert:/workspace/tinybert \
  openvino/model_server:latest \
  ovms --config_path /workspace/tinybert/config.json --port 8000 --rest_port 8001

检查服务健康状态

curl http://localhost:8001/v1/health

检查模型状态

curl http://localhost:8001/v1/models/tinybert

注意： tinybert为模型名称，需要与config.json中的name一致

请求服务

请求服务：

python http_tinybert.py

其中，http_tinybert.py：

import requests
import json
import numpy as np
from transformers import AutoTokenizer
import time
import argparse


def get_request_inputs(tokenizer, texts, max_length=100):
    if isinstance(texts, str):
        texts = [texts]

    encoded = tokenizer(
        texts,
        padding='max_length',
        truncation=True,
        max_length=max_length,
        return_tensors="np"
    )

    input_ids = encoded["input_ids"].astype(np.int64)
    attention_mask = encoded["attention_mask"].astype(np.int64)
    token_type_ids = encoded["token_type_ids"].astype(np.int64)

    request_data = {
        "inputs": {
            "input_ids": input_ids.tolist(),
            "attention_mask": attention_mask.tolist(),
            "token_type_ids": token_type_ids.tolist()
        }
    }

    return request_data


class HTTP_CLINET:
    def __init__(self, http_url="http://localhost:8001", model='tinybert'):
        self.url = f"{http_url}/v1/models/{model}:predict"

    def post(self, request_data):
        try:
            headers = {
                "Content-Type": "application/json",
                # "Authorization": f"Bearer {self.api_key}"  # 注意 Bearer 后面有一个空格
            }
            response = requests.post(
                self.url,
                json=request_data,
                headers=headers,
                timeout=30
            )
            
            if response.status_code == 200:
                result = response.json()
                print("推理成功!")
                # print(f"响应数据: {json.dumps(result, indent=2)}")
                return result
            else:
                print(f"请求失败，状态码: {response.status_code}")
                print(f"错误信息: {response.text}")
                return None
                
        except requests.exceptions.RequestException as e:
            print(f"请求异常: {e}")
            return None

def parse_opt():
    parser = argparse.ArgumentParser(description="http client")

    parser.add_argument('--http', '-u', type=str, default='http://localhost:8001', help="http_url")
    parser.add_argument('--model', '-m', type=str, default="tinybert", help='model server name')
    parser.add_argument('--input', '-i',type=str, default=None, help='input text')
    parser.add_argument('--token_path', '-t', type=str, default='../weights\TinyBERT-General-4L-312D-ONNX', help='tokenizer path')
    return parser.parse_args()   

def main(args):
    http_client = HTTP_CLINET(http_url=args.http, model=args.model)
    tokenizer = AutoTokenizer.from_pretrained(args.token_path)

    defeat_text = r'tinybert 是一个分号的文本分类器，可以识别任何文本的意图！'
    text = args.input if args.input else defeat_text

    print("构建请求数据...")
    requests_inputs = get_request_inputs(tokenizer=tokenizer, texts=text, max_length=100)
    print("构建请求数据成功")
    
    print("执行推理...")
    start_time = time.time()
    result = http_client.post(request_data=requests_inputs)
    end_time = time.time()
    print("请求推理完成")
    print(f"请求推理总耗时：{(end_time-start_time): .3f} s")

    
if __name__ == "__main__":
    args = parse_opt()
    main(args)

性能测试

python performance_tinybert.py

import requests
import time
import os
import json
import concurrent.futures
import statistics
import random
import csv
from tqdm import tqdm
import argparse
from transformers import AutoTokenizer
import numpy as np


class PerformanceBenchmark:
    """高性能并发推理测试器"""

    def __init__(
        self,
        http_url: str,
        model_name: str,
        tokenizer_path: str,
        text_path: str,
        min_text_len: int = 100,
        max_text_len: int = 100,
        request_timeout: int = 30,
    ):
        """
        :param http_url: Triton 服务基础 URL (如 http://localhost:8001)
        :param model_name: 模型名称
        :param tokenizer_path: 分词器路径
        :param text_path: 测试文本 JSON 文件路径
        :param min_text_len: 保留文本的最小长度
        :param max_text_len: 文本截断到的最大长度
        :param request_timeout: 单次请求超时(秒)
        """
        self.http_url = http_url.rstrip("/")
        self.model_name = model_name
        self.url = f"{self.http_url}/v1/models/{self.model_name}:predict"
        self.request_timeout = request_timeout

        # 加载分词器
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)

        # 加载并预处理文本数据
        self.texts = self._load_texts(text_path, min_text_len, max_text_len)
        if not self.texts:
            raise RuntimeError("没有可用的测试文本，请检查数据文件或长度过滤条件。")

    def _load_texts(self, text_path: str, min_len: int, max_len: int) -> list:
        """加载 Alpaca 格式 JSON 并拼接为文本列表"""
        combined_texts = []
        try:
            with open(text_path, "r", encoding="utf-8") as f:
                data = json.load(f)

            for item in data:
                instruction = item.get("instruction", "")
                input_text = item.get("input", "")
                output_text = item.get("output", "")
                combined = instruction + input_text + output_text
                combined = combined.replace(" ", "").replace("\n", "")
                if len(combined) < min_len:
                    continue
                if len(combined) >= max_len:
                    combined = combined[:max_len]
                combined_texts.append(combined)

            return combined_texts

        except FileNotFoundError:
            print(f"错误: 文件 {text_path} 不存在")
        except json.JSONDecodeError:
            print(f"错误: 文件 {text_path} 不是有效的JSON格式")
        return []
    
    def _trunc_keep_head_tail(self, text: str, keep: int = 100) -> str:
        """截断文本，保留头部和尾部"""
        if len(text) <= keep:
            return text
        keep_half = keep // 2
        return text[:keep_half] + text[-keep_half:]

    def _get_request_inputs(self, text, max_length: int = 100) -> dict:
        """将文本转换为 Triton 推理请求的输入格式"""
        text = self._trunc_keep_head_tail(text)

        encoded = self.tokenizer(
            text,
            padding="max_length",
            truncation=True,
            max_length=max_length,
            return_tensors="np",
        )

        input_ids = encoded["input_ids"].astype(np.int64)
        attention_mask = encoded["attention_mask"].astype(np.int64)
        token_type_ids = encoded["token_type_ids"].astype(np.int64)

        request_data = {
            "inputs": {
                "input_ids": input_ids.tolist(),
                "attention_mask": attention_mask.tolist(),
                "token_type_ids": token_type_ids.tolist()
            }
        }

        return request_data

    def _single_benchmark(
        self,
        num_requests: int,
        concurrency: int,
        verbose: bool = False,
    ) -> dict:
        """单次并发测试"""
        latencies = []
        errors = 0

        def worker(_):
            nonlocal errors
            session = requests.Session()
            text = random.choice(self.texts)
            payload = self._get_request_inputs(text)

            start = time.time()
            try:
                resp = session.post(self.url, json=payload, timeout=self.request_timeout)
                elapsed = (time.time() - start) * 1000
                if resp.status_code == 200:
                    return elapsed
                else:
                    errors += 1
                    if verbose:
                        print(f"请求失败状态码: {resp.status_code}, 响应: {resp.text}")
                    return None
            except Exception as e:
                errors += 1
                if verbose:
                    print(f"请求异常: {e}")
                return None
            finally:
                session.close()

        start_total = time.time()
        with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as executor:
            futures = [executor.submit(worker, i) for i in range(num_requests)]
            for future in tqdm(
                concurrent.futures.as_completed(futures),
                total=num_requests,
                desc=f"并发{concurrency}",
            ):
                result = future.result()
                if result is not None:
                    latencies.append(result)
        end_total = time.time()
        total_time = end_total - start_total
        success_count = len(latencies)

        if success_count == 0:
            print("所有请求均失败！")
            return None

        qps = success_count / total_time
        avg_latency = statistics.mean(latencies)
        max_lat = max(latencies)
        min_lat = min(latencies)
        stdev_lat = statistics.stdev(latencies) if len(latencies) > 1 else 0.0
        latencies_sorted = sorted(latencies)
        p50 = latencies_sorted[int(len(latencies_sorted) * 0.5)]
        p90 = latencies_sorted[int(len(latencies_sorted) * 0.9)]
        p99 = latencies_sorted[int(len(latencies_sorted) * 0.99)]

        results = {
            "concurrency": concurrency,
            "num_requests": num_requests,
            "success_count": success_count,
            "error_count": errors,
            "total_time_sec": round(total_time, 2),
            "qps": round(qps, 2),
            "avg_latency_ms": round(avg_latency, 2),
            "min_latency_ms": round(min_lat, 2),
            "max_latency_ms": round(max_lat, 2),
            "std_latency_ms": round(stdev_lat, 2),
            "p50_latency_ms": round(p50, 2),
            "p90_latency_ms": round(p90, 2),
            "p99_latency_ms": round(p99, 2),
        }

        print(f"\n========== 并发数 {concurrency} 测试结果 ==========")
        print(f"总请求数:        {num_requests}")
        print(f"成功请求数:      {success_count}")
        print(f"失败请求数:      {errors}")
        print(f"总耗时(s):       {total_time:.2f}")
        print(f"吞吐量(QPS):     {qps:.2f}")
        print(f"平均延迟(ms):    {avg_latency:.2f}")
        print(f"最小延迟(ms):    {min_lat:.2f}")
        print(f"最大延迟(ms):    {max_lat:.2f}")
        print(f"标准差(ms):      {stdev_lat:.2f}")
        print(f"P50 延迟(ms):    {p50:.2f}")
        print(f"P90 延迟(ms):    {p90:.2f}")
        print(f"P99 延迟(ms):    {p99:.2f}")
        print("==============================")
        return results

    def run_benchmark_sweep(
        self,
        concurrency_list: list = [1, 2, 4, 8, 16, 32],
        num_requests_per_test: int = 200,
        output_csv: str = "benchmark_results.csv",
        cooldown_sec: int = 2,
    ):
        """
        遍历多个并发数进行测试，并保存结果至 CSV。
        :param concurrency_list: 并发数列表
        :param num_requests_per_test: 每次测试的总请求数
        :param output_csv: 结果输出 CSV 文件路径
        :param cooldown_sec: 每轮测试后的冷却时间（秒）
        """
        all_results = []

        for concurrency in concurrency_list:
            print(f"\n========== 测试并发数: {concurrency} ==========")
            result = self._single_benchmark(
                num_requests=num_requests_per_test,
                concurrency=concurrency,
                verbose=True,
            )
            if result is not None:
                all_results.append(result)
            time.sleep(cooldown_sec)

        if all_results:
            keys = all_results[0].keys()
            os.makedirs(os.path.dirname(output_csv) or ".", exist_ok=True)
            with open(output_csv, "w", newline="") as f:
                writer = csv.DictWriter(f, fieldnames=keys)
                writer.writeheader()
                writer.writerows(all_results)
            print(f"性能指标已保存至 {output_csv}")
        else:
            print("没有成功的结果可保存。")


def parse_opt():
    parser = argparse.ArgumentParser(description="TinyBert模型基于OpenVINO的性能测试")
    parser.add_argument("--http", "-u", type=str, default="http://localhost:8001", help="OpenVINO 服务基础 URL")
    parser.add_argument("--model", "-m", type=str, default="tinybert", help="模型名称")
    parser.add_argument("--token_path", "-t", type=str, default="TinyBERT_General_4L_312D", help="分词器路径")
    parser.add_argument("--input", "-i", type=str, default="alpaca_gpt4_data_zh.json", help="测试文本 JSON 文件")
    parser.add_argument("--max_batch_size", type=int, default=8, help="最大批次大小（仅用于文件名标识）")
    parser.add_argument("--concurrency", "-c", type=str, default="1,4,8,16", help="并发数列表，逗号分隔")
    parser.add_argument("--num", "-n", type=int, default=128, help="每次测试的总请求数")
    parser.add_argument("--output_dir", "-o", type=str, default="performance_result", help="输出目录")
    return parser.parse_args()


def main():
    args = parse_opt()

    output_dir = args.output_dir
    os.makedirs(output_dir, exist_ok=True)
    output_csv = os.path.join(
        output_dir,
        f"{args.model}_cpu_openvino_batch{args.max_batch_size}_benchmark_results.csv",
    )

    concurrency_list = [int(x.strip()) for x in args.concurrency.split(",")]

    # 初始化测试器
    benchmark = PerformanceBenchmark(
        http_url=args.http,
        model_name=args.model,
        tokenizer_path=args.token_path,
        text_path=args.input,
        min_text_len=128,    # 可根据需要调整
        max_text_len=1024,    # 固定文本截断长度，与模型序列长度一致
    )

    # 执行测试
    benchmark.run_benchmark_sweep(
        concurrency_list=concurrency_list,
        num_requests_per_test=args.num,
        output_csv=output_csv,
        cooldown_sec=2,
    )


if __name__ == "__main__":
    main()

性能测试结果

• 测试环境

  • cpu: 2核
  • 内存: 4G

• 测试结果

注意：该请求延时较慢，主要因为tinybert的最后一层输出的维度是(batch, 312, 100)，相当于请求返回的数据是 batch x 312 x 100 个浮点数，需要传输的数据量较大，导致请求延时较长。