【通用视觉】集成 OpenAI CLIP 模型，支持以图搜图、以文搜图、以图搜文等功能

【通用视觉】新增 YOLO 图像分类模型支持
【ASR/TTS】集成 Sherpa TTS（语音合成）与 ASR（语音识别）模块，支持中文、粤语、方言、英文等多种语言
【目标检测】优化视频目标检测功能

README.md

@@ -140,6 +140,18 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
         </div>
       </td>
     </tr> 
+  <tr>
+      <td>
+        <div align="left">
+          <p>图像分类<br>(Image Classification)</p>
+        </div>
+      </td>     
+      <td>
+        <div align="center">
+        <img src="https://cdn.jsdelivr.net/gh/geekwenjie/SmartJavaAI-Site/images/vision/cls.png" height = "300px"/>
+        </div>
+      </td>
+    </tr>
    <tr>
       <td>
         <div align="left">
@@ -316,6 +328,33 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
         </div>
       </td>
     </tr>
+  <tr>
+      <td>
+        <div align="left">
+          <p>语音合成(ASR)</p>
+        </div>
+      </td>     
+      <td>
+        <div align="center">
+        <img src="https://cdn.jsdelivr.net/gh/geekwenjie/SmartJavaAI-Site/images/speech/tts.jpg" width = "500px"/>
+        </div>
+      </td>
+    </tr>
+  <tr>
+      <td>
+        <div align="left">
+          <p>CLIP</p>
+          - 文搜图 <br>
+          - 图搜图 <br>
+          - 图搜文<br>
+        </div>
+      </td>     
+      <td>
+        <div align="center">
+        <img src="https://cdn.jsdelivr.net/gh/geekwenjie/SmartJavaAI-Site/images/vision/clip.png" width = "500px"/>
+        </div>
+      </td>
+    </tr>
   </table>
 </div>
 
@@ -351,8 +390,11 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
   - 静默活体检测：图片、视频活体检测
   - 人脸表情识别：7种表情识别
   - 人脸质量评估：亮度评估、清晰度评估、完整度评估、姿态评估、遮挡评估
+- **图像分类**
+  - 支持多种主流模型：集成 YOLOv8、YOLOv11 等分类模型
+  - 支持自定义模型加载：可无缝加载并部署用户自行训练的分类模型
 - **目标检测**
-  - 支持多种主流模型：集成 YOLOv5、YOLOv8、YOLOv11、YOLOv12、Tensorflow Object Detection 等目标检测算法
+  - 支持多种主流模型：集成 YOLOv5、YOLOv8、YOLOv11、YOLOv12、Tensorflow Object Detection 等目标检测模型
   - 支持自定义模型加载：可无缝加载并部署用户自行训练的目标检测模型
   - 集成行人检测模型
 - **语义分割**
@@ -365,6 +407,9 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
   - 支持KINETICS400数据集中400个人类动作识别
 - **姿态估计**
   - 集成YOLOv8-pose、YOLOv11-pose等模型
+- **CLIP**
+  - 支持提取图片及文本特征
+  - 支持文搜图、图搜文、图搜图
 - **OCR文字识别**
   - 支持PaddleOCR 3.0模型：集成最新PP-OCRv5、PP-OCRv4、表格结构识别模型(SLANet_plus)、文本行方向分类模型
   - 支持任意角度识别，方向校准
@@ -373,9 +418,12 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
   - 支持中文车牌识别：单层/双层检测，颜色识别，支持12种中文车牌
 - **机器翻译**
   - 集成NLLB-200模型：支持200+语言互相翻译
-- **语音识别**
+- **语音识别(ASR)**
   - 集成openai的whisper模型：支持100种语言
   - 集成vosk语音识别
+  - 集成sherpa-onnx语音识别
+- **语音合成(TTS)**
+  - 集成sherpa-onnx：支持中文、方言、粤语、英文、德语等多种语言
 
 
 ## 🌟 AI集成方式对比
@@ -404,7 +452,7 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
 | translate | 机器翻译模块                    |
 | speech    | 语音功能模块，包含 ASR 和 TTS       |
 
-可以根据需求对每个模块单独引入，也可以通过引入`smartjavaai-all`方式引入所有模块。
+可以根据需求对每个模块单独引入，也可以通过引入`all`方式引入所有模块。
 
 -------------------------------------------------------------------------------
 
@@ -435,7 +483,7 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
 <dependency>
     <groupId>cn.smartjavaai</groupId>
     <artifactId>all</artifactId>
-    <version>1.0.25</version>
+    <version>1.0.26</version>
 </dependency>
 ```
 
@@ -534,6 +582,16 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
 
 ---
 
+#### 图像分类模型(CLS)
+
+**YOLO 系列**
+
+| 模型名称        | 引擎             | 模型简介       | 模型开源网站                                                               |
+|-------------|----------------|------------|----------------------------------------------------------------------|
+| YOLOV11-cls | OnnxRuntime | 最流行的图像分类模型 | [Github](https://docs.ultralytics.com/zh/models/)                 |
+| YOLOV8-cls      | OnnxRuntime            | 最流行的图像分类模型 | [Github](https://docs.ultralytics.com/zh/models/)              |
+
+
 #### 目标检测模型
 
 **YOLO 系列**
@@ -583,6 +641,14 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
 | YOLOV11-SEG | OnnxRuntime | Ultralytics在COCO 数据集 上训练的模型 |  [Github](https://docs.ultralytics.com/zh/tasks/segment/)        |
 | Mask R-CNN | MXNet | Mask R-CNN 是一种在目标检测基础上，同时为每个物体生成像素级分割区域的深度学习模型 | 无          |
 
+
+#### CLIP模型(Connecting text and images)
+
+
+| 模型名称        | 引擎          | 模型简介               | 模型开源网站     |
+|-------------|-------------|--------------------|------------|
+| clip-vit-base-patch32  | PyTorch     | openai的图片及文本特征提取模型 | [Github](https://github.com/openai/CLIP) |
+
 ---
 
 #### OBB旋转框目标检测模型
@@ -694,14 +760,23 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
 
 ---
 
-#### 语音识别模型
+#### 语音识别模型(ASR)
 
 这里仅介绍模型的开源项目，每个开源项目通常包含多个具体模型，本文不逐一列出。
 
-| 模型名称    | 模型简介                     | 模型官网                                          |
-|---------| ------------------------ |-----------------------------------------------|
-| Whisper | OpenAI 开源的通用语音识别（ASR）模型，支持多语言转写和翻译，具有较高的识别精度，尤其在嘈杂环境中表现良好，适合离线和批量音频处理。 | [Github](https://github.com/ggml-org/whisper.cpp) |
-| Vosk    | 一个轻量级离线语音识别工具包，支持多种语言和平台（包括移动端与嵌入式设备），可在低资源环境中运行，适合实时语音识别场景。 | [Github](https://github.com/alphacep/vosk-api) |
+| 模型名称    | 模型简介                                                                    | 模型官网                                          |
+|---------|-------------------------------------------------------------------------|-----------------------------------------------|
+| Whisper | OpenAI 开源的通用语音识别（ASR）模型，支持多语言转写和翻译，具有较高的识别精度，尤其在嘈杂环境中表现良好，适合离线和批量音频处理。  | [Github](https://github.com/ggml-org/whisper.cpp) |
+| Vosk    | 一个轻量级离线语音识别工具包，支持多种语言和平台（包括移动端与嵌入式设备），可在低资源环境中运行，适合实时语音识别场景。            | [Github](https://github.com/alphacep/vosk-api) |
+| sherpa-onnx    | Sherpa-ONNX 是一个基于 ONNX Runtime 的ASR 及 TTS 推理框架 | [Github](https://github.com/k2-fsa/sherpa-onnx) |
+
+#### 语音合成模型(TTS)
+
+这里仅介绍模型的开源项目，每个开源项目通常包含多个具体模型，本文不逐一列出。
+
+| 模型名称    | 模型简介                                                                    | 模型官网                                          |
+|---------|-------------------------------------------------------------------------|-----------------------------------------------|
+| sherpa-onnx    | Sherpa-ONNX 是一个基于 ONNX Runtime 的ASR 及 TTS 推理框架 | [Github](https://github.com/k2-fsa/sherpa-onnx) |
 
 
 ---
@@ -715,7 +790,7 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
 
 ## 联系方式
 
-如您在使用过程中有任何问题或建议，欢迎添加微信，与我们交流并加入用户交流群
+如您在使用过程中有任何问题、建议，或希望进行技术交流与合作，欢迎添加微信与我联系，并加入用户交流群。
 
 - **微信**: deng775747758 （请备注：SmartJavaAI）
 - **Email**: 775747758@qq.com
@@ -739,6 +814,12 @@ SmartJavaAI是专为JAVA 开发者打造的一个功能丰富、开箱即用的
 
 ## 近期更新日志
 
+## [v1.0.26] - 2025-10-24
+- 【通用视觉】集成 OpenAI CLIP 模型，支持以图搜图、以文搜图、以图搜文等功能
+- 【通用视觉】新增 YOLO 图像分类模型支持
+- 【ASR/TTS】集成 Sherpa TTS（语音合成）与 ASR（语音识别）模块，支持中文、粤语、方言、英文等多种语言
+- 【目标检测】优化视频目标检测功能
+
 ## [v1.0.25] - 2025-10-02
 - 【人脸识别】 新增多种人脸识别模型
 - 【底层优化】 支持自由选择 OpenCV 或 BufferedImage 作为图像引擎

all/pom.xml

@@ -6,11 +6,11 @@
     <parent>
         <groupId>cn.smartjavaai</groupId>
         <artifactId>smartjavaai-parent</artifactId>
-        <version>1.0.25</version>
+        <version>1.0.26</version>
     </parent>
 
     <artifactId>all</artifactId>
-    <version>1.0.25</version>
+    <version>1.0.26</version>
     <name>${project.artifactId}</name>
     <description>SmartJavaAI</description>
     <url>https://github.com/geekwenjie/SmartJavaAI</url>

bom/pom.xml

@@ -6,10 +6,10 @@
     <parent>
         <groupId>cn.smartjavaai</groupId>
         <artifactId>smartjavaai-parent</artifactId>
-        <version>1.0.25</version>
+        <version>1.0.26</version>
     </parent>
 
-    <version>1.0.25</version>
+    <version>1.0.26</version>
     <artifactId>bom</artifactId>
     <name>bom</name>
     <description>统一版本管理的 BOM 包，同时支持 import 和全量依赖</description>

common/pom.xml

@@ -6,7 +6,7 @@
     <parent>
         <groupId>cn.smartjavaai</groupId>
         <artifactId>smartjavaai-parent</artifactId>
-        <version>1.0.25</version>
+        <version>1.0.26</version>
     </parent>
 
     <name>common</name>

common/src/main/java/cn/smartjavaai/common/config/ModelConfig.java

@@ -45,6 +45,12 @@ public class ModelConfig {
         return clazz.cast(value);
     }
 
+    public <T> T getCustomParam(String key, Class<T> clazz, T defaultValue) {
+        Object value = customParams.getOrDefault(key, defaultValue);
+        return clazz.cast(value);
+    }
+
+
     /**
      * 添加个性化配置项
      */

common/src/main/java/cn/smartjavaai/common/enums/SimilarityType.java

@@ -1,4 +1,4 @@
-package cn.smartjavaai.face.enums;
+package cn.smartjavaai.common.enums;
 
 /**
  * @author dwj

common/src/main/java/cn/smartjavaai/common/utils/BufferedImageUtils.java

@@ -573,4 +573,36 @@ public class BufferedImageUtils {
         }
     }
 
+
+    /**
+     * 绘制检测框
+     * @param sourceImage
+     * @param detectionResponse
+     * @throws IOException
+     */
+    public static void drawFaceSearchResult(Graphics2D graphics, DetectionInfo detectionInfo, String text) {
+        graphics.setColor(Color.RED);// 边框颜色
+        graphics.setStroke(new BasicStroke(2));   // 线宽2像素
+        graphics.setRenderingHint(RenderingHints.KEY_ANTIALIASING,
+                RenderingHints.VALUE_ANTIALIAS_ON); // 抗锯齿
+        int stroke = 2;
+        DetectionRectangle rectangle = detectionInfo.getDetectionRectangle();
+        graphics.setColor(Color.RED);// 边框颜色
+        graphics.drawRect(rectangle.getX(), rectangle.getY(), rectangle.getWidth(),  rectangle.getHeight());
+        //绘制人脸关键点
+        //人脸查询结果
+        if(detectionInfo.getFaceInfo().getFaceSearchResults() != null){
+            for (FaceSearchResult faceSearchResult : detectionInfo.getFaceInfo().getFaceSearchResults()){
+                if(StringUtils.isNotBlank(faceSearchResult.getMetadata())){
+                    JsonObject metadata = GsonUtils.parseToJsonObject(faceSearchResult.getMetadata());
+                    JsonElement nameElement = metadata.get("name");
+                    if(metadata.has("name")){
+                        Graphics2DUtils.drawText(graphics, nameElement.getAsString(), rectangle.getX(), rectangle.getY(), stroke, 4);
+                    }
+                }
+            }
+        }
+        graphics.dispose();
+    }
+
 }

common/src/main/java/cn/smartjavaai/common/utils/DJLCommonUtils.java

@@ -11,6 +11,7 @@ import org.opencv.core.Mat;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.List;
 import java.util.Objects;
 
@@ -19,6 +20,16 @@ import java.util.Objects;
  */
 public class DJLCommonUtils {
 
+    private static final List<String> SUPPORTED_PROTOCOLS = Arrays.asList(
+            "file://",
+            "http://",
+            "https://",
+            "jar://",
+            "djl://",
+            "s3://",
+            "hdfs://"
+    );
+
     /**
      * 检查模型目录中是否存在 "serving.properties" 文件
      *
@@ -158,5 +169,17 @@ public class DJLCommonUtils {
         return new DetectedObjects(classNames, probabilities, boxes);
     }
 
+    /**
+     * 判断路径是否以已知协议开头
+     * @param path 模型路径
+     * @return 是否以支持的协议开头
+     */
+    public static boolean hasSupportedProtocol(String path) {
+        if (path == null || path.isEmpty()) {
+            return false;
+        }
+        return SUPPORTED_PROTOCOLS.stream().anyMatch(path::startsWith);
+    }
+
 
 }

common/src/main/java/cn/smartjavaai/common/utils/FileUtils.java

@@ -1,6 +1,9 @@
 package cn.smartjavaai.common.utils;
 
 import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
 
 /**
  * 文件操作工具类
@@ -28,4 +31,132 @@ public class FileUtils {
         File file = new File(path);
         return file.exists() && file.isDirectory();
     }
+
+    /**
+     * 查找指定目录下指定后缀的文件
+     *
+     * @param dir       目录
+     * @param suffix    文件后缀，例如 ".txt"、".wav"
+     * @param recursive 是否递归子目录
+     * @return 文件列表
+     */
+    public static List<File> findFilesWithSuffix(File dir, String suffix, boolean recursive) {
+        List<File> result = new ArrayList<>();
+        if (dir == null || !dir.exists() || !dir.isDirectory()) {
+            return result;
+        }
+        searchFiles(dir, suffix, recursive, result);
+        return result;
+    }
+
+    // 搜索方法
+    private static void searchFiles(File dir, String suffix, boolean recursive, List<File> result) {
+        File[] files = dir.listFiles();
+        if (files == null) return;
+
+        for (File file : files) {
+            if (file.isDirectory()) {
+                if(file.getName().endsWith(suffix)){
+                    result.add(file);
+                    continue;
+                }
+                if (recursive) {
+                    searchFiles(file, suffix, true, result);
+                }
+            } else if (file.isFile() && file.getName().endsWith(suffix)) {
+                result.add(file);
+            }
+        }
+    }
+
+    /**
+     * 查找指定目录下指定文件名的文件
+     *
+     * @param dir       目录
+     * @param fileName  文件名（精确匹配）
+     * @param recursive 是否递归子目录
+     * @return 文件列表
+     */
+    public static List<File> findFilesByName(File dir, String fileName, boolean recursive) {
+        List<File> result = new ArrayList<>();
+        if (dir == null || !dir.exists() || !dir.isDirectory() || fileName == null) {
+            return result;
+        }
+        searchByName(dir, fileName, recursive, result);
+        return result;
+    }
+
+    // 递归搜索方法
+    private static void searchByName(File dir, String fileName, boolean recursive, List<File> result) {
+        File[] files = dir.listFiles();
+        if (files == null) return;
+
+        for (File file : files) {
+            if (file.isDirectory()) {
+                if (file.getName().equals(fileName)){
+                    result.add(file);
+                    continue;
+                }
+                if (recursive) {
+                    searchByName(file, fileName, true, result);
+                }
+            } else if (file.isFile() && file.getName().equals(fileName)) {
+                result.add(file);
+            }
+        }
+    }
+
+    /**
+     * 将文件列表转换为绝对路径字符串
+     *
+     * @param files 文件列表
+     * @return 绝对路径字符串，用逗号分隔
+     */
+    public static String joinAbsolutePaths(List<File> files) {
+        if (files == null || files.isEmpty()) {
+            return "";
+        }
+        return files.stream()
+                .map(File::getAbsolutePath)
+                .collect(Collectors.joining(","));
+    }
+
+    /**
+     * 在指定目录中查找文件名包含指定关键字的文件，可选指定后缀。
+     *
+     * @param dirPath     要搜索的目录路径
+     * @param keyword     文件名包含的关键字（可为 null）
+     * @param extension   文件后缀名（例如 ".wav"，可为 null）
+     * @param recursive   是否递归搜索子目录
+     * @return 匹配的文件列表
+     */
+    public static List<File> searchFiles(String dirPath, String keyword, String extension, boolean recursive) {
+        List<File> result = new ArrayList<>();
+        File dir = new File(dirPath);
+
+        if (!dir.exists() || !dir.isDirectory()) {
+            System.err.println("目录不存在或不是目录：" + dirPath);
+            return result;
+        }
+
+        File[] files = dir.listFiles();
+        if (files == null) return result;
+
+        for (File file : files) {
+            if (file.isDirectory() && recursive) {
+                // 递归子目录
+                result.addAll(searchFiles(file.getAbsolutePath(), keyword, extension, true));
+            } else if (file.isFile()) {
+                String name = file.getName().toLowerCase();
+                boolean matchKeyword = (keyword == null || name.contains(keyword.toLowerCase()));
+                boolean matchExt = (extension == null || name.endsWith(extension.toLowerCase()));
+
+                if (matchKeyword && matchExt) {
+                    result.add(file);
+                }
+            }
+        }
+        return result;
+    }
+
 }

common/src/main/java/cn/smartjavaai/common/utils/SimilarityUtil.java

@@ -1,6 +1,7 @@
-package cn.smartjavaai.face.utils;
+package cn.smartjavaai.common.utils;
 
-import cn.smartjavaai.face.enums.SimilarityType;
+
+import cn.smartjavaai.common.enums.SimilarityType;
 
 /**
  * 特征相似度计算工具类

examples/face-example/pom.xml

@@ -12,7 +12,7 @@
         <maven.compiler.source>11</maven.compiler.source>
         <maven.compiler.target>11</maven.compiler.target>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-        <smartjavaai.version>1.0.25</smartjavaai.version>
+        <smartjavaai.version>1.0.26</smartjavaai.version>
         <!--如果打包运行，需要替换成你的main-->
         <exec.mainClass>smartai.examples.face.facedet.FaceDetDemo</exec.mainClass>
 

examples/ocr-examples/pom.xml

@@ -12,7 +12,7 @@
         <maven.compiler.source>11</maven.compiler.source>
         <maven.compiler.target>11</maven.compiler.target>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-        <smartjavaai.version>1.0.25</smartjavaai.version>
+        <smartjavaai.version>1.0.26</smartjavaai.version>
         <!--如果打包运行，需要替换成你的main-->
         <exec.mainClass>smartai.examples.ocr.common.OcrRecognizeDemo</exec.mainClass>
 

examples/speech-examples/pom.xml

@@ -12,7 +12,7 @@
         <maven.compiler.source>11</maven.compiler.source>
         <maven.compiler.target>11</maven.compiler.target>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-        <smartjavaai.version>1.0.25</smartjavaai.version>
+        <smartjavaai.version>1.0.26</smartjavaai.version>
         <!--如果打包运行，需要替换成你的main-->
         <exec.mainClass>smartai.examples.speech.asr.common.OcrRecognizeDemo</exec.mainClass>
 

examples/speech-examples/src/main/java/smartai/examples/speech/asr/SherpaAsrDemo.java

@@ -0,0 +1,389 @@
+package smartai.examples.speech.asr;
+
+import ai.djl.modality.audio.Audio;
+import ai.djl.modality.audio.AudioFactory;
+import ai.djl.util.JsonUtils;
+import cn.smartjavaai.common.entity.Language;
+import cn.smartjavaai.common.entity.R;
+import cn.smartjavaai.speech.asr.config.AsrModelConfig;
+import cn.smartjavaai.speech.asr.entity.AsrResult;
+import cn.smartjavaai.speech.asr.entity.WhisperParams;
+import cn.smartjavaai.speech.asr.enums.AsrModelEnum;
+import cn.smartjavaai.speech.asr.factory.SpeechRecognizerFactory;
+import cn.smartjavaai.speech.asr.model.SpeechRecognizer;
+import lombok.extern.slf4j.Slf4j;
+import org.junit.Test;
+
+import java.nio.file.Paths;
+
+/**
+ * 语音识别ASR demo
+ * sherpa-onnx模型及依赖库下载链接:
+ * 1、（推荐）依赖库官网下载：https://github.com/k2-fsa/sherpa-onnx/releases
+ * 2、（推荐）ASR模型官网下载：https://github.com/k2-fsa/sherpa-onnx/releases/tag/asr-models
+ * 3、百度网盘下载：https://pan.baidu.com/s/19p3WhVEM7dgdkvXFaeeAxg?pwd=1234 提取码: 1234
+ * @author dwj
+ * @date 2025/10/23
+ */
+@Slf4j
+public class SherpaAsrDemo {
+
+
+    /**
+     * 语音识别：Sherpa Paraformer（中文）
+     */
+    @Test
+    public void testSherpaParaformerAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_PARAFORMER);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-paraformer-zh-2023-09-14");
+            config.setModelName("model.int8.onnx");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-paraformer-zh-2023-09-14/test_wavs/0.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 语音识别：Sherpa Dolphin（中文）
+     */
+    @Test
+    public void testSherpaDolphinAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_DOLPHIN);
+            config.setModelName("model.int8.onnx");
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-dolphin-base-ctc-multi-lang-int8-2025-04-02");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-dolphin-base-ctc-multi-lang-int8-2025-04-02/test_wavs/0.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 语音识别：Sherpa zipformer（中文）
+     */
+    @Test
+    public void testSherpaZipformerAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_ZIPFORMERCTC);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-zipformer-ctc-zh-int8-2025-07-03");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            config.setModelName("model.int8.onnx");
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-zipformer-ctc-zh-int8-2025-07-03/test_wavs/0.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 语音识别：Sherpa FireRedAsr（中英）
+     */
+    @Test
+    public void testSherpaFireRedAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_FIREREDASR);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16/test_wavs/3.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 语音识别：Sherpa SenseVoice （多语言模型）
+     */
+    @Test
+    public void testSherpaSenseVoiceAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_SENSEVOICE);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17");
+            config.setModelName("model.onnx");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17/test_wavs/zh.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 语音识别：Sherpa WenetCtc（多语言模型：粤语）
+     */
+    @Test
+    public void testSherpaWenetCtcAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_WENETCTC);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-wenetspeech-yue-u2pp-conformer-ctc-zh-en-cantonese-int8-2025-09-10");
+            config.setModelName("model.int8.onnx");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-wenetspeech-yue-u2pp-conformer-ctc-zh-en-cantonese-int8-2025-09-10/test_wavs/yue-0.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 语音识别：Sherpa FireRedAsr（方言：四川、天津、河南）
+     */
+    @Test
+    public void testSherpaFireRedAsrSichuan() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_FIREREDASR);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16/test_wavs/3-sichuan.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 语音识别：Sherpa Telespeech（方言-天津、河南、四川）
+     */
+    @Test
+    public void testSherpaTelespeechAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_TELESPEECH);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-telespeech-ctc-int8-zh-2024-06-04");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.setModelName("model.int8.onnx");
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-telespeech-ctc-int8-zh-2024-06-04/test_wavs/4-tianjin.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 语音识别：Sherpa Nemo（英文）
+     */
+    @Test
+    public void testSherpaNemoAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_NEMO);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-nemo-ctc-en-citrinet-512");
+            config.setModelName("model.onnx");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-nemo-ctc-en-citrinet-512/test_wavs/0.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 语音识别：Sherpa Moonshine（英文）
+     */
+    @Test
+    public void testSherpaMoonshineAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_MOONSHINE);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-moonshine-tiny-en-int8");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 2);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-moonshine-tiny-en-int8/test_wavs/0.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+
+
+    /**
+     * 语音识别：Sherpa Whisper（英文）
+     */
+    @Test
+    public void testSherpaWhisperAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_WHISPER);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-whisper-tiny");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-whisper-tiny/test_wavs/0.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 语音识别：Sherpa Transducer（英文）
+     */
+    @Test
+    public void testSherpaTransducerAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_TRANSDUCER);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-zipformer-gigaspeech-2023-12-12");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-zipformer-gigaspeech-2023-12-12/test_wavs/1221-135766-0001.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 语音识别：Sherpa Canary（德语-英文）
+     */
+    @Test
+    public void testSherpaCanaryAsr() {
+        try {
+            //获取模型
+            AsrModelConfig config = new AsrModelConfig();
+            config.setModelEnum(AsrModelEnum.SHERPA_CANARY);
+            config.setModelPath("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-nemo-canary-180m-flash-en-es-de-fr-int8");
+            config.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+            config.putCustomParam("debug", false);
+            config.putCustomParam("numThreads", 1);
+            SpeechRecognizer recognizer = SpeechRecognizerFactory.getInstance().getModel(config);
+            Audio audio = AudioFactory.newInstance().fromFile(Paths.get("/Users/wenjie/Documents/develop/model/speech/sherpa-asr/sherpa-onnx-nemo-canary-180m-flash-en-es-de-fr-int8/test_wavs/de.wav"));
+            R<AsrResult> result = recognizer.recognize(audio);
+            if (result.isSuccess()){
+                log.info("识别成功:{}", result.getData());
+            }else{
+                log.error("识别失败:{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+
+
+
+
+
+
+
+
+
+
+}

examples/speech-examples/src/main/java/smartai/examples/speech/asr/SpeechRecognizeDemo.java

@@ -33,7 +33,7 @@ import java.io.InputStream;
 import java.nio.file.Paths;
 
 /**
- * 语音识别demo
+ * 语音识别demo（Vosk、Whisper）
  * 模型下载网盘：https://pan.baidu.com/s/1kiMF5MF641R7LTn1GpB2lQ?pwd=1234 提取码: 1234
  * 文档地址：http://doc.smartjavaai.cn/
  * @author dwj
@@ -41,6 +41,8 @@ import java.nio.file.Paths;
 @Slf4j
 public class SpeechRecognizeDemo {
 
+
+
     /**
      * 获取Whisper模型
      * 模型下载网盘：https://pan.baidu.com/s/1kiMF5MF641R7LTn1GpB2lQ?pwd=1234 提取码: 1234

examples/speech-examples/src/main/java/smartai/examples/speech/tts/TtsDemo.java

@@ -0,0 +1,193 @@
+package smartai.examples.speech.tts;
+
+
+import ai.djl.modality.audio.Audio;
+import cn.smartjavaai.common.entity.R;
+import cn.smartjavaai.common.enums.DeviceEnum;
+import cn.smartjavaai.speech.tts.config.TtsModelConfig;
+import cn.smartjavaai.speech.tts.entity.SherpaTtsParams;
+import cn.smartjavaai.speech.tts.enums.TtsModelEnum;
+import cn.smartjavaai.speech.tts.factory.TtsModelFactory;
+import cn.smartjavaai.speech.tts.model.SherpaTtsModel;
+import cn.smartjavaai.speech.tts.model.TtsModel;
+import cn.smartjavaai.speech.utils.AudioUtils;
+import lombok.extern.slf4j.Slf4j;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.nio.file.Paths;
+
+/**
+ * 语音合成demo（TTS）
+ * sherpa-onnx模型及依赖库下载链接:
+ * 1、（推荐）依赖库官网下载：https://github.com/k2-fsa/sherpa-onnx/releases
+ * 2、（推荐）TTS模型官网下载：https://github.com/k2-fsa/sherpa-onnx/releases/tag/tts-models
+ * 3、百度网盘下载：https://pan.baidu.com/s/186REUf7p1z0HH9AZNnwCUg?pwd=1234 提取码: 1234
+ * @author dwj
+ * @date 2025/10/22
+ */
+@Slf4j
+public class TtsDemo {
+
+    public static String enText = "Today as always, men fall into two groups: slaves and free men. Whoever does not have"
+            + " two-thirds of his day for himself, is a slave, whatever he may be: a statesman, a"
+            + " businessman, an official, or a scholar.";
+
+    public static String zhText = "有问题，请拨打110或者手机18601239876。我们的价值观是真诚热爱！";
+
+    String znEnText =
+            "中英文语音合成测试。This is generated by next generation Kaldi using Kokoro without Misaki."
+                    + " 你觉得中英文说的如何呢？";
+
+    //设备类型
+    public static DeviceEnum device = DeviceEnum.CPU;
+
+
+    /**
+     * 获取Vits中文模型
+     * @return
+     */
+    public TtsModel getVitsZhModel(){
+        TtsModelConfig modelConfig = new TtsModelConfig();
+        modelConfig.setModelEnum(TtsModelEnum.SHERPA_VITS);
+        modelConfig.setModelPath("/Users/wenjie/Documents/develop/model/speech/tts/vits-zh-hf-keqing");
+        modelConfig.setModelName("keqing.onnx");
+        modelConfig.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+        modelConfig.setDevice(device);
+        modelConfig.putCustomParam("debug", false);
+        modelConfig.putCustomParam("numThreads", 1);
+        return TtsModelFactory.getInstance().getModel(modelConfig);
+    }
+
+    /**
+     * 获取Matcha中文模型
+     * @return
+     */
+    public TtsModel getMatchaZhModel(){
+        TtsModelConfig modelConfig = new TtsModelConfig();
+        modelConfig.setModelEnum(TtsModelEnum.SHERPA_MATCHA);
+        modelConfig.setModelPath("/Users/wenjie/Documents/develop/model/speech/tts/matcha-icefall-zh-baker");
+        modelConfig.setModelName("model-steps-3.onnx");
+        modelConfig.putCustomParam("vocoder", "/Users/wenjie/Documents/develop/model/speech/tts/matcha-icefall-zh-baker/vocos-22khz-univ.onnx");
+        modelConfig.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+        modelConfig.setDevice(device);
+        modelConfig.putCustomParam("debug", false);
+        modelConfig.putCustomParam("numThreads", 1);
+        return TtsModelFactory.getInstance().getModel(modelConfig);
+    }
+
+    /**
+     * 获取Kokoro多语言模型
+     * @return
+     */
+    public TtsModel getKokoroZhEnModel(){
+        TtsModelConfig modelConfig = new TtsModelConfig();
+        modelConfig.setModelEnum(TtsModelEnum.SHERPA_KOKORO);
+        modelConfig.setModelPath("/Users/wenjie/Documents/develop/model/speech/tts/kokoro-multi-lang-v1_0");
+        modelConfig.setModelName("model.onnx");
+        modelConfig.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+        modelConfig.setDevice(device);
+        modelConfig.putCustomParam("debug", false);
+        modelConfig.putCustomParam("numThreads", 2);
+        return TtsModelFactory.getInstance().getModel(modelConfig);
+    }
+
+    /**
+     * 获取Matcha模型（英文）
+     * @return
+     */
+    public TtsModel getMatchaEnModel(){
+        TtsModelConfig modelConfig = new TtsModelConfig();
+        modelConfig.setModelEnum(TtsModelEnum.SHERPA_MATCHA);
+        modelConfig.setModelPath("/Users/wenjie/Documents/develop/model/speech/tts/matcha-icefall-en_US-ljspeech");
+        modelConfig.setModelName("model-steps-3.onnx");
+        modelConfig.setLibPath(Paths.get("/Users/wenjie/smartjavaai_cache/sherpa-onnx-v1.12.14-osx-arm64-jni/lib"));
+        modelConfig.putCustomParam("vocoder", "/Users/wenjie/Documents/develop/model/speech/tts/matcha-icefall-zh-baker/vocos-22khz-univ.onnx");
+        modelConfig.setDevice(device);
+        modelConfig.putCustomParam("debug", false);
+        modelConfig.putCustomParam("numThreads", 2);
+        return TtsModelFactory.getInstance().getModel(modelConfig);
+    }
+
+    /**
+     * 语音合成Vits（中文）
+     */
+    @Test
+    public void testVitsZhTts() throws IOException {
+        TtsModel ttsModel = getVitsZhModel();
+        SherpaTtsParams params = new SherpaTtsParams();
+        //语速
+        params.setSpeed(1f);
+        //说话人ID
+        params.setSpeakerId(100);
+        R<Audio> result = ttsModel.generate(zhText, params);
+        if (result.isSuccess()){
+            Audio audio = result.getData();
+            AudioUtils.saveToWav(audio.getData(),"/Users/wenjie/Downloads/tts-vits-zh.wav");
+        }else{
+            System.out.println(result.getMessage());
+        }
+    }
+
+    /**
+     * 语音合成Matcha（中文）
+     */
+    @Test
+    public void testMatchaZhTts() throws IOException {
+        TtsModel ttsModel = getMatchaZhModel();
+        SherpaTtsParams params = new SherpaTtsParams();
+        //语速
+        params.setSpeed(1.5f);
+        //说话人ID
+        params.setSpeakerId(0);
+        R<Audio> result = ttsModel.generate(zhText, params);
+        if (result.isSuccess()){
+            Audio audio = result.getData();
+            AudioUtils.saveToWav(audio.getData(),"/Users/wenjie/Downloads/tts-matcha-zh.wav");
+        }else{
+            System.out.println(result.getMessage());
+        }
+    }
+
+    /**
+     * 语音合成Kokoro（中英）
+     */
+    @Test
+    public void testKokoroZhEnTts() throws IOException {
+        TtsModel ttsModel = getKokoroZhEnModel();
+        SherpaTtsParams params = new SherpaTtsParams();
+        //语速
+        params.setSpeed(1f);
+        //说话人ID
+        params.setSpeakerId(3);
+        R<Audio> result = ttsModel.generate(znEnText, params);
+        if (result.isSuccess()){
+            Audio audio = result.getData();
+            AudioUtils.saveToWav(audio.getData(),"/Users/wenjie/Downloads/tts-kokoro-zh-en.wav");
+        }else{
+            System.out.println(result.getMessage());
+        }
+    }
+
+    /**
+     * 语音合成Matcha（英文）
+     */
+    @Test
+    public void testMatchaEnTts() throws IOException {
+        TtsModel ttsModel = getMatchaEnModel();
+        SherpaTtsParams params = new SherpaTtsParams();
+        //语速
+        params.setSpeed(1.5f);
+        //说话人ID
+        params.setSpeakerId(0);
+        R<Audio> result = ttsModel.generate(enText, params);
+        if (result.isSuccess()){
+            Audio audio = result.getData();
+            AudioUtils.saveToWav(audio.getData(),"/Users/wenjie/Downloads/tts-kitten-en.wav");
+        }else{
+            System.out.println(result.getMessage());
+        }
+    }
+
+
+}

examples/translation-example/pom.xml

@@ -12,7 +12,7 @@
         <maven.compiler.source>11</maven.compiler.source>
         <maven.compiler.target>11</maven.compiler.target>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-        <smartjavaai.version>1.0.25</smartjavaai.version>
+        <smartjavaai.version>1.0.26</smartjavaai.version>
         <!--如果打包运行，需要替换成你的main-->
         <exec.mainClass>smartai.examples.nlp.translation.TranslationDemo</exec.mainClass>
 

examples/vision-example/pom.xml

@@ -12,7 +12,7 @@
         <maven.compiler.source>11</maven.compiler.source>
         <maven.compiler.target>11</maven.compiler.target>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-        <smartjavaai.version>1.0.25</smartjavaai.version>
+        <smartjavaai.version>1.0.26</smartjavaai.version>
         <!--如果打包运行，需要替换成你的main-->
         <exec.mainClass>smartai.examples.vision.ObjectDetectionDemo</exec.mainClass>
 

examples/vision-example/src/main/java/smartai/examples/vision/ClipDemo.java

@@ -0,0 +1,356 @@
+package smartai.examples.vision;
+
+import ai.djl.modality.cv.Image;
+import cn.smartjavaai.clip.config.ClipModelConfig;
+import cn.smartjavaai.clip.enums.ClipModelEnum;
+import cn.smartjavaai.clip.model.ClipModel;
+import cn.smartjavaai.clip.model.ClipModelFactory;
+import cn.smartjavaai.common.cv.SmartImageFactory;
+import cn.smartjavaai.common.entity.R;
+import cn.smartjavaai.common.enums.DeviceEnum;
+import cn.smartjavaai.common.enums.SimilarityType;
+import cn.smartjavaai.common.utils.ImageUtils;
+import cn.smartjavaai.common.utils.SimilarityUtil;
+import lombok.extern.slf4j.Slf4j;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * clip模型demo
+ * @author dwj
+ */
+@Slf4j
+public class ClipDemo {
+
+    //设备类型
+    public static DeviceEnum device = DeviceEnum.CPU;
+
+    @BeforeClass
+    public static void beforeAll() throws IOException {
+        //将图片处理的底层引擎切换为 OpenCV
+        SmartImageFactory.setEngine(SmartImageFactory.Engine.OPENCV);
+        //修改缓存路径
+//        Config.setCachePath("/Users/xxx/smartjavaai_cache");
+    }
+
+    public ClipModel getModel(){
+        ClipModelConfig config = new ClipModelConfig();
+        config.setModelEnum(ClipModelEnum.OPENAI);
+        config.setModelPath("/Users/wenjie/Documents/develop/model/vision/clip/openai/clip.pt");
+        //从jar包中加载模型
+//        config.setModelPath("jar://META-INF/models/clip/openai.zip");
+        config.setDevice(device);
+        return ClipModelFactory.getInstance().getModel(config);
+    }
+
+    /**
+     * 提取图片特征
+     */
+    @Test
+    public void extractImageFeatures(){
+        try {
+            ClipModel model = getModel();
+            //创建Image对象，可以从文件、url、InputStream创建、BufferedImage、Base64创建，具体使用方法可以查看文档
+            Image image = SmartImageFactory.getInstance().fromFile(Paths.get("src/main/resources/clip/dog.jpg"));
+            //获取图片特征
+            R<float[]> features = model.extractImageFeatures(image);
+            if(features.isSuccess()){
+                log.info("图片特征：{}", features.getData());
+            }else{
+                log.info("图片特征获取失败：{}", features.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 提取文本特征
+     */
+    @Test
+    public void extractTextFeatures() {
+        try {
+            ClipModel model = getModel();
+            // 提取单个文本特征
+            String text = "a photo of a dog";
+            R<float[]> features = model.extractTextFeatures(text);
+            if(features.isSuccess()){
+                log.info("文本特征：{}", features.getData());
+            }else{
+                log.info("文本特征获取失败：{}", features.getMessage());
+            }
+
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    /**
+     * 文本搜索图像（基于图像和文本直接比对）
+     */
+    @Test
+    public void searchImagesByText() {
+        try {
+            ClipModel model = getModel();
+            String text = "a photo of a dog";
+
+            // 读取图片列表
+            List<File> images = ImageUtils.listImageFiles("src/main/resources/clip");
+            List<Float> similarities = new ArrayList<>();
+
+            // 1. 计算每张图片的相似度
+            for (File imageFile : images) {
+                Image image = SmartImageFactory.getInstance().fromFile(imageFile.toPath());
+                R<Float> similarity = model.compareTextAndImage(image, text);
+                if (similarity.isSuccess()) {
+                    similarities.add(similarity.getData());
+                    log.info("图片：{}，相似度：{}", imageFile.getName(), similarity.getData());
+                } else {
+                    log.warn("图片：{}，相似度计算失败：{}", imageFile.getName(), similarity.getMessage());
+                }
+            }
+            if (similarities.isEmpty()) {
+                log.warn("没有计算到有效的相似度结果");
+                return;
+            }
+
+            // 2. 计算 Softmax 概率
+            double total = similarities.stream()
+                    .mapToDouble(Math::exp)
+                    .sum();
+
+            List<Double> probabilities = similarities.stream()
+                    .map(v -> Math.exp(v) / total)
+                    .collect(Collectors.toList());
+
+            // 3. 找出相似度最高的图片
+            int maxIndex = IntStream.range(0, similarities.size())
+                    .boxed()
+                    .max(Comparator.comparing(similarities::get))
+                    .orElse(-1);
+
+            // 4. 打印结果
+            log.info("---- 结果统计 ----");
+            for (int i = 0; i < images.size(); i++) {
+                log.info("图片：{}，相似度：{}，概率：{}",
+                        images.get(i).getName(),
+                        similarities.get(i),
+                        String.format("%.4f", probabilities.get(i)));
+            }
+
+            log.info("最匹配的图片：{}，相似度：{}，Softmax 概率：{}",
+                    images.get(maxIndex).getName(),
+                    similarities.get(maxIndex),
+                    String.format("%.4f", probabilities.get(maxIndex)));
+
+        } catch (Exception e) {
+            log.error("执行 searchImagesByText 异常", e);
+        }
+    }
+
+    /**
+     * 文本搜索图像（基于图像和文本的特征值比对）
+     */
+    @Test
+    public void searchImagesByText2() {
+        try {
+            ClipModel model = getModel();
+            String text = "a photo of a dog";
+            // 读取图片列表
+            List<File> images = ImageUtils.listImageFiles("src/main/resources/clip");
+            List<Float> similarities = new ArrayList<>();
+            R<float[]> textFeatures = model.extractTextFeatures(text);
+            float scale = 100f; // 缩放因子，越大 softmax 差异越明显
+            // 1. 计算每张图片的相似度
+            for (File imageFile : images) {
+                Image image = SmartImageFactory.getInstance().fromFile(imageFile.toPath());
+                R<float[]> imageFeatures = model.extractImageFeatures(image);
+                if (imageFeatures.isSuccess()) {
+                    float similarity = SimilarityUtil.calculate(
+                            imageFeatures.getData(),
+                            textFeatures.getData(),
+                            SimilarityType.COSINE,
+                            false
+                    );
+                    similarities.add(similarity * scale);
+                } else {
+                    log.warn("图片：{}，特征提取失败：{}", imageFile.getName(), imageFeatures.getMessage());
+                    similarities.add(Float.NEGATIVE_INFINITY); // 特征提取失败，赋极小值
+                }
+            }
+
+            if (similarities.isEmpty()) {
+                log.warn("没有计算到有效的相似度结果");
+                return;
+            }
+
+            // 2. 计算 Softmax 概率
+            double total = similarities.stream()
+                    .mapToDouble(Math::exp)
+                    .sum();
+
+            List<Double> probabilities = similarities.stream()
+                    .map(v -> Math.exp(v) / total)
+                    .collect(Collectors.toList());
+
+            // 3. 找出相似度最高的图片
+            int maxIndex = IntStream.range(0, similarities.size())
+                    .boxed()
+                    .max(Comparator.comparing(similarities::get))
+                    .orElse(-1);
+
+            // 4. 打印结果
+            log.info("---- 结果统计 ----");
+            for (int i = 0; i < images.size(); i++) {
+                log.info("图片：{}，相似度：{}，概率：{}",
+                        images.get(i).getName(),
+                        similarities.get(i),
+                        String.format("%.4f", probabilities.get(i)));
+            }
+
+            log.info("最匹配的图片：{}，相似度：{}，Softmax 概率：{}",
+                    images.get(maxIndex).getName(),
+                    similarities.get(maxIndex),
+                    String.format("%.4f", probabilities.get(maxIndex)));
+
+        } catch (Exception e) {
+            log.error("执行 searchImagesByText 异常", e);
+        }
+    }
+
+
+    /**
+     * 图像搜索文本
+     */
+    @Test
+    public void searchTextByImage() {
+        try {
+            ClipModel model = getModel();
+            String[] textArray = {"a diagram", "a dog", "a cat"};
+            //创建Image对象，可以从文件、url、InputStream创建、BufferedImage、Base64创建，具体使用方法可以查看文档
+            Image image = SmartImageFactory.getInstance().fromFile(Paths.get("src/main/resources/clip/dog.jpg"));
+            //获取图片特征
+            R<float[]> features = model.extractImageFeatures(image);
+            List<Float> similarities = new ArrayList<>();
+            // 1. 计算每张图片的相似度
+            for (String text : textArray) {
+                R<Float> similarity = model.compareTextAndImage(image, text);
+                if (similarity.isSuccess()) {
+                    similarities.add(similarity.getData());
+                    log.info("文本：{}，相似度：{}", text, similarity.getData());
+                } else {
+                    log.warn("文本：{}，相似度计算失败：{}", text, similarity.getMessage());
+                }
+            }
+            if (similarities.isEmpty()) {
+                log.warn("没有计算到有效的相似度结果");
+                return;
+            }
+
+            // 2. 计算 Softmax 概率
+            double total = similarities.stream()
+                    .mapToDouble(Math::exp)
+                    .sum();
+
+            List<Double> probabilities = similarities.stream()
+                    .map(v -> Math.exp(v) / total)
+                    .collect(Collectors.toList());
+
+            // 3. 找出相似度最高的图片
+            int maxIndex = IntStream.range(0, similarities.size())
+                    .boxed()
+                    .max(Comparator.comparing(similarities::get))
+                    .orElse(-1);
+
+            // 4. 打印结果
+            log.info("---- 结果统计 ----");
+            for (int i = 0; i < textArray.length; i++) {
+                log.info("文本：{}，相似度：{}，概率：{}",
+                        textArray[i],
+                        similarities.get(i),
+                        String.format("%.4f", probabilities.get(i)));
+            }
+
+            log.info("最匹配的文本：{}，相似度：{}，Softmax 概率：{}",
+                    textArray[maxIndex],
+                    similarities.get(maxIndex),
+                    String.format("%.4f", probabilities.get(maxIndex)));
+
+        } catch (Exception e) {
+            log.error("执行 searchImagesByText 异常", e);
+        }
+    }
+
+    /**
+     * 以图搜图
+     */
+    @Test
+    public void searchImagesByImage() {
+        try {
+            ClipModel model = getModel();
+            //创建Image对象，可以从文件、url、InputStream创建、BufferedImage、Base64创建，具体使用方法可以查看文档
+            Image image1 = SmartImageFactory.getInstance().fromFile(Paths.get("src/main/resources/cat2.jpeg"));
+            List<Float> similarities = new ArrayList<>();
+            // 读取图片列表
+            List<File> images = ImageUtils.listImageFiles("src/main/resources/clip");
+            // 1. 计算每张图片的相似度
+            for (File imageFile : images) {
+                Image image = SmartImageFactory.getInstance().fromFile(imageFile.toPath());
+                R<Float> similarity = model.compareImage(image1, image, 100);
+                if (similarity.isSuccess()) {
+                    similarities.add(similarity.getData());
+                    log.info("图片：{}，相似度：{}", imageFile.getName(), similarity.getData());
+                } else {
+                    log.warn("图片：{}，相似度计算失败：{}", imageFile.getName(), similarity.getMessage());
+                }
+            }
+            if (similarities.isEmpty()) {
+                log.warn("没有计算到有效的相似度结果");
+                return;
+            }
+
+            // 2. 计算 Softmax 概率
+            double total = similarities.stream()
+                    .mapToDouble(Math::exp)
+                    .sum();
+
+            List<Double> probabilities = similarities.stream()
+                    .map(v -> Math.exp(v) / total)
+                    .collect(Collectors.toList());
+
+            // 3. 找出相似度最高的图片
+            int maxIndex = IntStream.range(0, similarities.size())
+                    .boxed()
+                    .max(Comparator.comparing(similarities::get))
+                    .orElse(-1);
+
+            // 4. 打印结果
+            log.info("---- 结果统计 ----");
+            for (int i = 0; i < images.size(); i++) {
+                log.info("图片：{}，相似度：{}，概率：{}",
+                        images.get(i).getName(),
+                        similarities.get(i),
+                        String.format("%.4f", probabilities.get(i)));
+            }
+
+            log.info("最匹配的图片：{}，相似度：{}，Softmax 概率：{}",
+                    images.get(maxIndex).getName(),
+                    similarities.get(maxIndex),
+                    String.format("%.4f", probabilities.get(maxIndex)));
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+
+}

examples/vision-example/src/main/java/smartai/examples/vision/ClsDemo.java

@@ -0,0 +1,90 @@
+package smartai.examples.vision;
+
+import ai.djl.modality.Classifications;
+import ai.djl.modality.cv.Image;
+import cn.smartjavaai.cls.config.ClsModelConfig;
+import cn.smartjavaai.cls.enums.ClsModelEnum;
+import cn.smartjavaai.cls.model.ClsModel;
+import cn.smartjavaai.cls.model.ClsModelFactory;
+import cn.smartjavaai.common.cv.SmartImageFactory;
+import cn.smartjavaai.common.entity.DetectionResponse;
+import cn.smartjavaai.common.entity.R;
+import cn.smartjavaai.common.enums.DeviceEnum;
+import cn.smartjavaai.instanceseg.config.InstanceSegModelConfig;
+import cn.smartjavaai.instanceseg.enums.InstanceSegModelEnum;
+import cn.smartjavaai.instanceseg.model.InstanceSegModel;
+import cn.smartjavaai.instanceseg.model.InstanceSegModelFactory;
+import com.alibaba.fastjson.JSONObject;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.collections.CollectionUtils;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.Arrays;
+
+/**
+ * 图像分类模型demo
+ * @author dwj
+ */
+@Slf4j
+public class ClsDemo {
+
+
+    //设备类型
+    public static DeviceEnum device = DeviceEnum.CPU;
+
+    @BeforeClass
+    public static void beforeAll() throws IOException {
+        //将图片处理的底层引擎切换为 OpenCV
+        SmartImageFactory.setEngine(SmartImageFactory.Engine.OPENCV);
+        //修改缓存路径
+//        Config.setCachePath("/Users/xxx/smartjavaai_cache");
+    }
+
+
+    public ClsModel getModel(){
+        ClsModelConfig config = new ClsModelConfig();
+        //实例分割模型，切换模型需要同时修改modelEnum及modelPath
+        config.setModelEnum(ClsModelEnum.YOLOV8);
+        //模型所在路径，synset.txt也需要放在同目录下
+        config.setModelPath("/Users/wenjie/Documents/develop/model/vision/cls/yolo11m-cls.onnx");
+        // 指定允许的类别
+//        config.setAllowedClasses(Arrays.asList("dog","car"));
+        //指定返回检测数量
+        config.setDevice(device);
+        //置信度阈值
+        config.setThreshold(0.5f);
+        return ClsModelFactory.getInstance().getModel(config);
+    }
+
+    /**
+     * 实例分割
+     */
+    @Test
+    public void detect(){
+        try {
+            ClsModel detectorModel = getModel();
+            //创建Image对象，可以从文件、url、InputStream创建、BufferedImage、Base64创建，具体使用方法可以查看文档
+            Image image = SmartImageFactory.getInstance().fromFile(Paths.get("src/main/resources/clip/dog.jpg"));
+            R<Classifications> result = detectorModel.detect(image);
+            if(result.isSuccess()){
+                if(CollectionUtils.isNotEmpty(result.getData().getClassNames())){
+                    //分数最高分类
+                    log.info("分类识别结果：{}", result.getData().best().toString());
+                    //按分数排序前5个结果
+//                log.info("动作识别结果：{}", result.getData().topK(5).toString());
+                }else{
+                    log.info("未识别到分类");
+                }
+            }else{
+                log.info("分类识别失败：{}", result.getMessage());
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+
+}

face/pom.xml

@@ -6,11 +6,11 @@
     <parent>
         <groupId>cn.smartjavaai</groupId>
         <artifactId>smartjavaai-parent</artifactId>
-        <version>1.0.25</version>
+        <version>1.0.26</version>
     </parent>
 
     <artifactId>face</artifactId>
-    <version>1.0.25</version>
+    <version>1.0.26</version>
     <name>face</name>
     <description>SmartJavaAI</description>
     <url>https://github.com/geekwenjie/SmartJavaAI</url>

face/src/main/java/cn/smartjavaai/face/model/facerec/CommonFaceRecModel.java

@@ -12,15 +12,16 @@ import cn.smartjavaai.common.cv.SmartImageFactory;
 import cn.smartjavaai.common.entity.*;
 import cn.smartjavaai.common.entity.face.FaceInfo;
 import cn.smartjavaai.common.entity.face.FaceSearchResult;
+import cn.smartjavaai.common.enums.SimilarityType;
 import cn.smartjavaai.common.pool.PredictorFactory;
 import cn.smartjavaai.common.utils.BufferedImageUtils;
 import cn.smartjavaai.common.utils.FileUtils;
 import cn.smartjavaai.common.utils.ImageUtils;
+import cn.smartjavaai.common.utils.SimilarityUtil;
 import cn.smartjavaai.face.config.FaceRecConfig;
 import cn.smartjavaai.face.constant.FaceDetectConstant;
 import cn.smartjavaai.face.entity.FaceRegisterInfo;
 import cn.smartjavaai.face.entity.FaceSearchParams;
-import cn.smartjavaai.face.enums.SimilarityType;
 import cn.smartjavaai.face.exception.FaceException;
 import cn.smartjavaai.face.factory.FaceRecModelFactory;
 import cn.smartjavaai.face.model.facerec.criteria.FaceRecCriteriaFactory;
@@ -808,6 +809,35 @@ public class CommonFaceRecModel implements FaceRecModel{
         return detectedResult;
     }
 
+    @Override
+    public R<float[]> extractFeatures(Image image, DetectionInfo detectionInfo) {
+        float[] features = null;
+        try (NDManager manager = model.getNDManager().newSubManager()) {
+            DJLImageFacePreprocessor djlImagePreprocessor = new DJLImageFacePreprocessor(image, manager);
+            DetectionRectangle rectangle = detectionInfo.getDetectionRectangle();
+            FaceInfo faceInfo = detectionInfo.getFaceInfo();
+            Image subImage = null;
+            //人脸对齐
+            if(config.isAlign()){
+                //人脸对齐
+                double[][] pointsArray = FaceUtils.facePoints(faceInfo.getKeyPoints());
+                djlImagePreprocessor.enableCrop(rectangle).enableAffine(pointsArray, 96, 112);
+                subImage = djlImagePreprocessor.process();
+            }else{
+                //裁剪
+                djlImagePreprocessor.enableCrop(rectangle);
+                if(config.isCropFace()){
+                    subImage = djlImagePreprocessor.process();
+                }
+            }
+            features = featureExtraction(subImage);
+            if (subImage != null && subImage.getWrappedImage() instanceof Mat) {
+                ((Mat)subImage.getWrappedImage()).release();
+            }
+        }
+        return Objects.isNull(features) ? R.fail(R.Status.Unknown) : R.ok(features);
+    }
+
     @Override
     public R<float[]> extractTopFaceFeature(Image image) {
         R<DetectionResponse> detectedResult = config.getDetectModel().detect(image);

face/src/main/java/cn/smartjavaai/face/model/facerec/FaceRecModel.java

@@ -2,6 +2,7 @@ package cn.smartjavaai.face.model.facerec;
 
 import ai.djl.inference.Predictor;
 import ai.djl.modality.cv.Image;
+import cn.smartjavaai.common.entity.DetectionInfo;
 import cn.smartjavaai.common.entity.DetectionResponse;
 import cn.smartjavaai.common.entity.R;
 import cn.smartjavaai.face.config.FaceRecConfig;
@@ -404,6 +405,10 @@ public interface FaceRecModel extends AutoCloseable{
         throw new UnsupportedOperationException("默认不支持该功能");
     }
 
+    default R<float[]> extractFeatures(Image image, DetectionInfo detectionInfo){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
     /**
      * 特征提取（提取分数最高人脸特征）
      * 适用于单人脸场景

face/src/main/java/cn/smartjavaai/face/model/facerec/SeetaFace6FaceRecModel.java

@@ -2,12 +2,15 @@ package cn.smartjavaai.face.model.facerec;
 
 import ai.djl.engine.Engine;
 import ai.djl.modality.cv.Image;
+import ai.djl.ndarray.NDManager;
 import cn.smartjavaai.common.cv.SmartImageFactory;
 import cn.smartjavaai.common.entity.DetectionInfo;
+import cn.smartjavaai.common.entity.DetectionRectangle;
 import cn.smartjavaai.common.entity.DetectionResponse;
 import cn.smartjavaai.common.entity.R;
 import cn.smartjavaai.common.entity.face.FaceInfo;
 import cn.smartjavaai.common.enums.DeviceEnum;
+import cn.smartjavaai.common.enums.SimilarityType;
 import cn.smartjavaai.common.utils.BufferedImageUtils;
 import cn.smartjavaai.common.utils.FileUtils;
 import cn.smartjavaai.common.utils.ImageUtils;
@@ -17,10 +20,10 @@ import cn.smartjavaai.face.entity.FaceRegisterInfo;
 import cn.smartjavaai.face.entity.FaceResult;
 import cn.smartjavaai.face.entity.FaceSearchParams;
 import cn.smartjavaai.face.enums.FaceRecModelEnum;
-import cn.smartjavaai.face.enums.SimilarityType;
 import cn.smartjavaai.face.exception.FaceException;
 import cn.smartjavaai.face.factory.FaceDetModelFactory;
 import cn.smartjavaai.face.factory.FaceRecModelFactory;
+import cn.smartjavaai.face.preprocess.DJLImageFacePreprocessor;
 import cn.smartjavaai.face.utils.FaceUtils;
 import cn.smartjavaai.face.utils.Seetaface6Utils;
 import cn.smartjavaai.face.vector.config.MilvusConfig;
@@ -37,6 +40,7 @@ import io.milvus.param.MetricType;
 import lombok.extern.slf4j.Slf4j;
 import org.apache.commons.collections.CollectionUtils;
 import org.apache.commons.lang3.StringUtils;
+import org.opencv.core.Mat;
 
 import javax.imageio.ImageIO;
 import java.awt.image.BufferedImage;
@@ -1086,6 +1090,42 @@ public class SeetaFace6FaceRecModel implements FaceRecModel{
         }
     }
 
+    @Override
+    public R<float[]> extractFeatures(Image image, DetectionInfo detectionInfo) {
+        FaceRecognizer faceRecognizer = null;
+        try {
+            SeetaImageData imageData = new SeetaImageData(image.getWidth(), image.getHeight(), 3);
+            imageData.data = ImageUtils.getMatrixBGR(image);
+            faceRecognizer = faceRecognizerPool.borrowObject();
+
+            //提取特征
+            float[] features = new float[faceRecognizer.GetExtractFeatureSize()];
+            FaceInfo faceInfo = detectionInfo.getFaceInfo();
+            if(Objects.isNull(faceInfo) || Objects.isNull(faceInfo.getKeyPoints())){
+                return R.fail(R.Status.Unknown.getCode(), "未检测到人脸关键点");
+            }
+            SeetaPointF[] pointFS = Seetaface6Utils.convertToSeetaPointF(faceInfo.getKeyPoints());
+            //CropFaceV2 + ExtractCroppedFace 已包含裁剪+人脸对齐
+            boolean isSuccess = faceRecognizer.Extract(imageData, pointFS, features);
+            if(!isSuccess){
+                return R.fail(R.Status.Unknown.getCode(), "人脸特征提取失败");
+            }
+            return Objects.isNull(features) ? R.fail(R.Status.Unknown) : R.ok(features);
+        } catch (FaceException e) {
+            throw e;
+        } catch (Exception e) {
+            throw new FaceException("人脸特征提取异常", e);
+        }finally {
+            if (faceRecognizer != null) {
+                try {
+                    faceRecognizerPool.returnObject(faceRecognizer); //归还
+                } catch (Exception e) {
+                    log.warn("归还Predictor失败", e);
+                }
+            }
+        }
+    }
+
     @Override
     public R<float[]> extractTopFaceFeature(Image image) {
         float[] features = null;

face/src/main/java/cn/smartjavaai/face/vector/config/SQLiteConfig.java

@@ -1,6 +1,6 @@
 package cn.smartjavaai.face.vector.config;
 
-import cn.smartjavaai.face.enums.SimilarityType;
+import cn.smartjavaai.common.enums.SimilarityType;
 import cn.smartjavaai.face.enums.VectorDBType;
 import lombok.Data;
 import lombok.EqualsAndHashCode;

face/src/main/java/cn/smartjavaai/face/vector/core/SQLiteClient.java

@@ -2,9 +2,9 @@ package cn.smartjavaai.face.vector.core;
 
 import cn.hutool.core.util.IdUtil;
 import cn.smartjavaai.common.config.Config;
+import cn.smartjavaai.common.utils.SimilarityUtil;
 import cn.smartjavaai.face.dao.FaceDao;
 import cn.smartjavaai.face.entity.FaceSearchParams;
-import cn.smartjavaai.face.utils.SimilarityUtil;
 import cn.smartjavaai.face.vector.config.SQLiteConfig;
 import cn.smartjavaai.face.vector.entity.FaceVector;
 import cn.smartjavaai.common.entity.face.FaceSearchResult;

ocr/pom.xml

@@ -6,7 +6,7 @@
     <parent>
         <groupId>cn.smartjavaai</groupId>
         <artifactId>smartjavaai-parent</artifactId>
-        <version>1.0.25</version>
+        <version>1.0.26</version>
     </parent>
 
     <artifactId>ocr</artifactId>
@@ -42,7 +42,7 @@
         </dependency>
     </dependencies>
 
-    <version>1.0.25</version>
+    <version>1.0.26</version>
     <name>ocr</name>
     <description>SmartJavaAI</description>
     <url>https://github.com/geekwenjie/SmartJavaAI</url>

pom.xml

@@ -7,7 +7,7 @@
     <name>SmartJavaAI</name>
     <groupId>cn.smartjavaai</groupId>
     <artifactId>smartjavaai-parent</artifactId>
-    <version>1.0.25</version>
+    <version>1.0.26</version>
     <packaging>pom</packaging>
     <description>SmartJavaAI</description>
     <modules>

speech/pom.xml

@@ -6,7 +6,7 @@
     <parent>
         <groupId>cn.smartjavaai</groupId>
         <artifactId>smartjavaai-parent</artifactId>
-        <version>1.0.25</version>
+        <version>1.0.26</version>
     </parent>
 
     <artifactId>speech</artifactId>
@@ -48,10 +48,16 @@
             <artifactId>jave-core</artifactId>
             <version>3.5.0</version>
         </dependency>
+
+        <dependency>
+            <groupId>io.gitee.dengwenjie</groupId>
+            <artifactId>sherpa-onnx-java-api</artifactId>
+            <version>1.12.14</version>
+        </dependency>
     </dependencies>
 
 
-    <version>1.0.25</version>
+    <version>1.0.26</version>
     <name>speech</name>
     <description>SmartJavaAI</description>
     <url>https://github.com/geekwenjie/SmartJavaAI</url>

speech/src/main/java/cn/smartjavaai/speech/asr/config/AsrModelConfig.java

@@ -22,4 +22,6 @@ public class AsrModelConfig extends ModelConfig {
      * 依赖库目录
      */
     private Path libPath;
+
+    private String modelName;
 }

speech/src/main/java/cn/smartjavaai/speech/asr/enums/AsrModelEnum.java

@@ -8,7 +8,31 @@ public enum AsrModelEnum {
 
     WHISPER,
 
-    VOSK;
+    VOSK,
+
+    SHERPA_PARAFORMER,
+
+    SHERPA_TRANSDUCER,
+
+    SHERPA_WHISPER,
+
+    SHERPA_FIREREDASR,
+
+    SHERPA_MOONSHINE,
+
+    SHERPA_NEMO,
+
+    SHERPA_SENSEVOICE,
+
+    SHERPA_DOLPHIN,
+
+    SHERPA_ZIPFORMERCTC,
+
+    SHERPA_WENETCTC,
+
+    SHERPA_CANARY,
+
+    SHERPA_TELESPEECH;
 
     /**
      * 根据名称获取枚举 (忽略大小写和下划线变体)

speech/src/main/java/cn/smartjavaai/speech/asr/factory/SherpaOfflineAsrModelConfigFactory.java

@@ -0,0 +1,335 @@
+package cn.smartjavaai.speech.asr.factory;
+
+import cn.smartjavaai.common.enums.DeviceEnum;
+import cn.smartjavaai.common.utils.FileUtils;
+import cn.smartjavaai.speech.asr.config.AsrModelConfig;
+import cn.smartjavaai.speech.asr.exception.AsrException;
+import cn.smartjavaai.speech.tts.config.TtsModelConfig;
+import cn.smartjavaai.speech.tts.exception.TtsException;
+import com.k2fsa.sherpa.onnx.*;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import java.io.File;
+import java.util.List;
+
+/**
+ * @author dwj
+ */
+public class SherpaOfflineAsrModelConfigFactory {
+
+    public static OfflineDolphinModelConfig createDolphinConfig(AsrModelConfig config) {
+        if(StringUtils.isBlank(config.getModelName())){
+            throw new AsrException("modelName is null");
+        }
+        String model = config.getModelPath() + File.separator + config.getModelName();
+        OfflineDolphinModelConfig dolphin = OfflineDolphinModelConfig.builder().setModel(model).build();
+        return dolphin;
+    }
+
+    public static OfflineFireRedAsrModelConfig createFireRedConfig(AsrModelConfig config) {
+        List<File> decoderFiles = FileUtils.searchFiles(config.getModelPath(), "decoder", ".onnx",false);
+        List<File> encoderFiles = FileUtils.searchFiles(config.getModelPath(), "encoder", ".onnx",false);
+        if (CollectionUtils.isEmpty(decoderFiles)){
+            throw new AsrException("decoder onnx not found");
+        }
+        if (CollectionUtils.isEmpty(encoderFiles)){
+            throw new AsrException("encoder onnx not found");
+        }
+        String encoder = encoderFiles.get(0).getAbsolutePath();
+        String decoder = decoderFiles.get(0).getAbsolutePath();
+        OfflineFireRedAsrModelConfig fireRedAsr =
+                OfflineFireRedAsrModelConfig.builder().setEncoder(encoder).setDecoder(decoder).build();
+        return fireRedAsr;
+    }
+
+    public static OfflineMoonshineModelConfig createMoonshineConfig(AsrModelConfig config) {
+        List<File> preprocessorFiles = FileUtils.searchFiles(config.getModelPath(), "preprocess", ".onnx",false);
+        List<File> uncachedDecoderFiles = FileUtils.searchFiles(config.getModelPath(), "uncached_decode", ".onnx",false);
+        List<File> cachedDecoderFiles = FileUtils.searchFiles(config.getModelPath(), "cached_decode", ".onnx",false);
+        List<File> encoderFiles = FileUtils.searchFiles(config.getModelPath(), "encode", ".onnx",false);
+        if (CollectionUtils.isEmpty(uncachedDecoderFiles)){
+            throw new AsrException("uncached_decode onnx not found");
+        }
+        if (CollectionUtils.isEmpty(cachedDecoderFiles)){
+            throw new AsrException("cached_decode onnx not found");
+        }
+        if (CollectionUtils.isEmpty(encoderFiles)){
+            throw new AsrException("encoder onnx not found");
+        }
+        if (CollectionUtils.isEmpty(preprocessorFiles)){
+            throw new AsrException("preprocess onnx not found");
+        }
+        String encoder = encoderFiles.get(0).getAbsolutePath();
+        String preprocessor = preprocessorFiles.get(0).getAbsolutePath();
+        String cachedDecoder = cachedDecoderFiles.get(0).getAbsolutePath();
+        String uncachedDecoder = uncachedDecoderFiles.get(0).getAbsolutePath();
+        OfflineMoonshineModelConfig moonshine =
+                OfflineMoonshineModelConfig.builder()
+                        .setPreprocessor(preprocessor)
+                        .setEncoder(encoder)
+                        .setUncachedDecoder(uncachedDecoder)
+                        .setCachedDecoder(cachedDecoder)
+                        .build();
+        return moonshine;
+    }
+
+    public static OfflineNemoEncDecCtcModelConfig createNemoConfig(AsrModelConfig config) {
+        if(StringUtils.isBlank(config.getModelName())){
+            throw new AsrException("modelName is null");
+        }
+        String model = config.getModelPath() + File.separator + config.getModelName();
+        OfflineNemoEncDecCtcModelConfig modelConfig = OfflineNemoEncDecCtcModelConfig.builder().setModel(model).build();
+        return modelConfig;
+    }
+
+    public static OfflineSenseVoiceModelConfig createSenseVoiceConfig(AsrModelConfig config) {
+        if(StringUtils.isBlank(config.getModelName())){
+            throw new AsrException("modelName is null");
+        }
+        String model = config.getModelPath() + File.separator + config.getModelName();
+        OfflineSenseVoiceModelConfig senseVoice =
+                OfflineSenseVoiceModelConfig.builder().setModel(model).build();
+        return senseVoice;
+    }
+
+    public static OfflineTransducerModelConfig createTransducerConfig(AsrModelConfig config) {
+        List<File> decoderFiles = FileUtils.searchFiles(config.getModelPath(), "decoder", ".onnx",false);
+        List<File> encoderFiles = FileUtils.searchFiles(config.getModelPath(), "encoder", "int8.onnx",false);
+        List<File> joinerFiles = FileUtils.searchFiles(config.getModelPath(), "joiner", ".onnx",false);
+        if (CollectionUtils.isEmpty(decoderFiles)){
+            throw new AsrException("decoder onnx not found");
+        }
+        if (CollectionUtils.isEmpty(encoderFiles)){
+            throw new AsrException("encoder onnx not found");
+        }
+        if (CollectionUtils.isEmpty(joinerFiles)){
+            throw new AsrException("joiner onnx not found");
+        }
+        String encoder = encoderFiles.get(0).getAbsolutePath();
+        String decoder = decoderFiles.get(0).getAbsolutePath();
+        String joiner = joinerFiles.get(0).getAbsolutePath();
+        OfflineTransducerModelConfig transducer =
+                OfflineTransducerModelConfig.builder()
+                        .setEncoder(encoder)
+                        .setDecoder(decoder)
+                        .setJoiner(joiner)
+                        .build();
+        return transducer;
+    }
+
+    public static OfflineParaformerModelConfig createParaformerConfig(AsrModelConfig config) {
+        String model = config.getModelPath() + File.separator + config.getModelName();
+        OfflineParaformerModelConfig modelConfig = OfflineParaformerModelConfig.builder().setModel(model).build();
+        return modelConfig;
+    }
+
+    public static OfflineWenetCtcModelConfig createWenetCtcConfig(AsrModelConfig config) {
+        if(StringUtils.isBlank(config.getModelName())){
+            throw new AsrException("modelName is null");
+        }
+        String model = config.getModelPath() + File.separator + config.getModelName();
+        OfflineWenetCtcModelConfig wenetCtc =
+                OfflineWenetCtcModelConfig.builder().setModel(model).build();
+        return wenetCtc;
+    }
+
+    public static OfflineCanaryModelConfig createCanaryConfig(AsrModelConfig config) {
+        List<File> decoderFiles = FileUtils.searchFiles(config.getModelPath(), "decoder", ".onnx",false);
+        List<File> encoderFiles = FileUtils.searchFiles(config.getModelPath(), "encoder", ".onnx",false);
+        if (CollectionUtils.isEmpty(decoderFiles)){
+            throw new AsrException("decoder onnx not found");
+        }
+        if (CollectionUtils.isEmpty(encoderFiles)){
+            throw new AsrException("encoder onnx not found");
+        }
+        String encoder = encoderFiles.get(0).getAbsolutePath();
+        String decoder = decoderFiles.get(0).getAbsolutePath();
+        OfflineCanaryModelConfig canary =
+                OfflineCanaryModelConfig.builder()
+                        .setEncoder(encoder)
+                        .setDecoder(decoder)
+                        .setSrcLang("en")
+                        .setTgtLang("en")
+                        .setUsePnc(true)
+                        .build();
+        return canary;
+    }
+
+    public static OfflineWhisperModelConfig createWhisperConfig(AsrModelConfig config) {
+        //是否使用量化模型
+        boolean useInt8 = config.getCustomParam("useInt8", Boolean.class, false);
+        String extension = useInt8 ? "int8.onnx" : "onnx";
+        List<File> decoderFiles = FileUtils.searchFiles(config.getModelPath(), "decoder", extension,false);
+        List<File> encoderFiles = FileUtils.searchFiles(config.getModelPath(), "encoder", extension,false);
+        if (CollectionUtils.isEmpty(decoderFiles)){
+            throw new AsrException("decoder onnx not found");
+        }
+        if (CollectionUtils.isEmpty(encoderFiles)){
+            throw new AsrException("encoder onnx not found");
+        }
+        String encoder = encoderFiles.get(0).getAbsolutePath();
+        String decoder = decoderFiles.get(0).getAbsolutePath();
+        OfflineWhisperModelConfig fireRedAsr =
+                OfflineWhisperModelConfig.builder().setEncoder(encoder).setDecoder(decoder).build();
+        return fireRedAsr;
+    }
+
+    public static OfflineZipformerCtcModelConfig createZipformerCtcConfig(AsrModelConfig config) {
+        if(StringUtils.isBlank(config.getModelName())){
+            throw new AsrException("modelName is null");
+        }
+        String model = config.getModelPath() + File.separator + config.getModelName();
+        OfflineZipformerCtcModelConfig zipformerCtc =
+                OfflineZipformerCtcModelConfig.builder().setModel(model).build();
+        return zipformerCtc;
+    }
+
+    /**
+     * 创建模型配置
+     * @param config
+     * @return
+     */
+    public static OfflineRecognizerConfig createConfig(AsrModelConfig config) {
+        OfflineModelConfig modelConfig = null;
+        int numThreads = config.getCustomParam("numThreads", Integer.class, 1);
+        boolean debug = config.getCustomParam("debug", Boolean.class, true);
+        List<File> tokensFiles = FileUtils.findFilesWithSuffix(new File(config.getModelPath()), "tokens.txt", false);
+        String tokens = CollectionUtils.isEmpty(tokensFiles) ? "" : tokensFiles.get(0).getAbsolutePath();
+        String provider = config.getCustomParam("provider", String.class, "cpu");
+        switch (config.getModelEnum()){
+            case SHERPA_PARAFORMER:
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setParaformer(createParaformerConfig(config))
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_TRANSDUCER:
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setTransducer(createTransducerConfig(config))
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_WHISPER:
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setWhisper(createWhisperConfig(config))
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_FIREREDASR:
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setFireRedAsr(createFireRedConfig(config))
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_MOONSHINE:
+                numThreads = config.getCustomParam("numThreads", Integer.class, 2);
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setMoonshine(createMoonshineConfig(config))
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_NEMO:
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setNemo(createNemoConfig(config))
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setProvider(provider)
+                                .setDebug(debug)
+                                .build();
+                break;
+            case SHERPA_SENSEVOICE:
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setSenseVoice(createSenseVoiceConfig(config))
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_DOLPHIN:
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setDolphin(createDolphinConfig(config))
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_ZIPFORMERCTC:
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setZipformerCtc(createZipformerCtcConfig(config))
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_WENETCTC:
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setWenetCtc(createWenetCtcConfig(config))
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_CANARY:
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setCanary(createCanaryConfig(config))
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_TELESPEECH:
+                if(StringUtils.isBlank(config.getModelName())){
+                    throw new AsrException("modelName is null");
+                }
+                String model = config.getModelPath() + File.separator + config.getModelName();
+                modelConfig =
+                        OfflineModelConfig.builder()
+                                .setTeleSpeech(model)
+                                .setTokens(tokens)
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setModelType("telespeech_ctc")
+                                .setProvider(provider)
+                                .build();
+                break;
+        }
+        OfflineRecognizerConfig offlineRecognizerConfig = OfflineRecognizerConfig.builder()
+                        .setOfflineModelConfig(modelConfig)
+                        .setDecodingMethod("greedy_search")
+                        .build();
+        return offlineRecognizerConfig;
+    }
+
+}

speech/src/main/java/cn/smartjavaai/speech/asr/factory/SpeechRecognizerFactory.java

@@ -4,6 +4,7 @@ import cn.smartjavaai.common.config.Config;
 import cn.smartjavaai.speech.asr.config.AsrModelConfig;
 import cn.smartjavaai.speech.asr.enums.AsrModelEnum;
 import cn.smartjavaai.speech.asr.exception.AsrException;
+import cn.smartjavaai.speech.asr.model.SherpaRecognizer;
 import cn.smartjavaai.speech.asr.model.SpeechRecognizer;
 import cn.smartjavaai.speech.asr.model.VoskRecognizer;
 import cn.smartjavaai.speech.asr.model.WhisperRecognizer;
@@ -68,7 +69,7 @@ public class SpeechRecognizerFactory {
             throw new AsrException("未配置语音识别模型枚举");
         }
         return modelMap.computeIfAbsent(config.getModelEnum(), k -> {
-            return createFaceModel(config);
+            return createModel(config);
         });
     }
 
@@ -77,7 +78,7 @@ public class SpeechRecognizerFactory {
      * @param config
      * @return
      */
-    private SpeechRecognizer createFaceModel(AsrModelConfig config) {
+    private SpeechRecognizer createModel(AsrModelConfig config) {
         Class<?> clazz = registry.get(config.getModelEnum());
         if(clazz == null){
             throw new AsrException("Unsupported model");
@@ -98,6 +99,18 @@ public class SpeechRecognizerFactory {
     static {
         registerModel(AsrModelEnum.WHISPER, WhisperRecognizer.class);
         registerModel(AsrModelEnum.VOSK, VoskRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_PARAFORMER, SherpaRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_TRANSDUCER, SherpaRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_WHISPER, SherpaRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_FIREREDASR, SherpaRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_MOONSHINE, SherpaRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_NEMO, SherpaRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_SENSEVOICE, SherpaRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_DOLPHIN, SherpaRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_ZIPFORMERCTC, SherpaRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_WENETCTC, SherpaRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_CANARY, SherpaRecognizer.class);
+        registerModel(AsrModelEnum.SHERPA_TELESPEECH, SherpaRecognizer.class);
         log.debug("缓存目录：{}", Config.getCachePath());
     }
 

speech/src/main/java/cn/smartjavaai/speech/asr/model/SherpaRecognizer.java

@@ -0,0 +1,99 @@
+package cn.smartjavaai.speech.asr.model;
+
+import ai.djl.modality.audio.Audio;
+import cn.smartjavaai.common.entity.R;
+import cn.smartjavaai.speech.asr.config.AsrModelConfig;
+import cn.smartjavaai.speech.asr.entity.AsrResult;
+import cn.smartjavaai.speech.asr.entity.RecParams;
+import cn.smartjavaai.speech.asr.factory.SherpaOfflineAsrModelConfigFactory;
+import cn.smartjavaai.speech.tts.exception.TtsException;
+import cn.smartjavaai.speech.tts.factory.SherpaOfflineTtsModelConfigFactory;
+import com.k2fsa.sherpa.onnx.*;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+
+/**
+ * @author dwj
+ */
+@Slf4j
+public class SherpaRecognizer implements SpeechRecognizer{
+
+    private AsrModelConfig config;
+    private OfflineRecognizer recognizer;
+
+    @Override
+    public void loadModel(AsrModelConfig config) {
+        this.config = config;
+        if(StringUtils.isBlank(config.getModelPath())){
+            throw new TtsException("modelPath is null");
+        }
+        Path testModelPath = Paths.get(config.getModelPath());
+        if(!testModelPath.toFile().exists()){
+            throw new TtsException("modelPath does not exist: " + testModelPath.toAbsolutePath());
+        }
+        if(Objects.isNull(config.getLibPath())){
+            throw new TtsException("libPath is null");
+        }
+        if(!config.getLibPath().toFile().exists()){
+            throw new TtsException("libPath does not exist: " + testModelPath.toAbsolutePath());
+        }
+        try {
+            //加载依赖库
+            System.setProperty("sherpa_onnx.native.path",config.getLibPath().toAbsolutePath().toString());
+            OfflineRecognizerConfig offlineRecognizerConfig = SherpaOfflineAsrModelConfigFactory.createConfig(config);
+            recognizer = new OfflineRecognizer(offlineRecognizerConfig);
+            log.debug("Sherpa tts init success");
+        } catch (Exception e) {
+            throw new TtsException(e);
+        }
+    }
+
+    @Override
+    public R<AsrResult> recognize(Audio audio) {
+        if(recognizer == null){
+            throw new TtsException("模型未初始化");
+        }
+        OfflineStream stream = recognizer.createStream();
+        stream.acceptWaveform(audio.getData(), (int)audio.getSampleRate());
+        recognizer.decode(stream);
+        String text = recognizer.getResult(stream).getText();
+        stream.release();
+        return R.ok(new AsrResult(text));
+
+    }
+
+    @Override
+    public R<AsrResult> recognize(String audioPath) {
+        if(recognizer == null){
+            throw new TtsException("模型未初始化");
+        }
+        OfflineStream stream = recognizer.createStream();
+        WaveReader reader = new WaveReader(audioPath);
+        stream.acceptWaveform(reader.getSamples(), reader.getSampleRate());
+        recognizer.decode(stream);
+        String text = recognizer.getResult(stream).getText();
+        stream.release();
+        return R.ok(new AsrResult(text));
+    }
+
+    private boolean fromFactory = false;
+
+    @Override
+    public void setFromFactory(boolean fromFactory) {
+        this.fromFactory = fromFactory;
+    }
+    public boolean isFromFactory() {
+        return fromFactory;
+    }
+
+    @Override
+    public void close() throws Exception {
+        if(recognizer != null){
+            recognizer.release();
+        }
+    }
+}

speech/src/main/java/cn/smartjavaai/speech/asr/model/SpeechRecognizer.java

@@ -1,5 +1,6 @@
 package cn.smartjavaai.speech.asr.model;
 
+import ai.djl.modality.audio.Audio;
 import cn.smartjavaai.common.entity.R;
 import cn.smartjavaai.speech.asr.config.AsrModelConfig;
 import cn.smartjavaai.speech.asr.entity.AsrResult;
@@ -45,6 +46,14 @@ public interface SpeechRecognizer extends AutoCloseable{
         throw new UnsupportedOperationException("默认不支持该功能");
     }
 
+    default R<AsrResult> recognize(Audio audio){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    default R<AsrResult> recognize(Audio audio, RecParams params){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
     default void setFromFactory(boolean fromFactory){
         throw new UnsupportedOperationException("默认不支持该功能");
     }

speech/src/main/java/cn/smartjavaai/speech/tts/config/TtsModelConfig.java

@@ -0,0 +1,27 @@
+package cn.smartjavaai.speech.tts.config;
+
+import cn.smartjavaai.common.config.ModelConfig;
+import cn.smartjavaai.speech.tts.enums.TtsModelEnum;
+import lombok.Data;
+
+import java.nio.file.Path;
+
+/**
+ * @author dwj
+ * @date 2025/10/14
+ */
+@Data
+public class TtsModelConfig extends ModelConfig {
+
+    private TtsModelEnum modelEnum;
+
+    private String modelPath;
+
+    private String modelName;
+
+    /**
+     * 依赖库目录
+     */
+    private Path libPath;
+
+}

speech/src/main/java/cn/smartjavaai/speech/tts/entity/SherpaTtsParams.java

@@ -0,0 +1,14 @@
+package cn.smartjavaai.speech.tts.entity;
+
+import com.k2fsa.sherpa.onnx.OfflineTtsCallback;
+import lombok.Data;
+
+/**
+ * @author dwj
+ */
+@Data
+public class SherpaTtsParams extends TtsParams{
+
+    private OfflineTtsCallback callback;
+
+}

speech/src/main/java/cn/smartjavaai/speech/tts/entity/TtsParams.java

@@ -0,0 +1,23 @@
+package cn.smartjavaai.speech.tts.entity;
+
+import lombok.Data;
+
+/**
+ * tts参数
+ * @author dwj
+ */
+@Data
+public abstract class TtsParams {
+
+    /**
+     * 发音人
+     */
+    private int speakerId;
+
+    /**
+     * 语速
+     */
+    private float speed;
+
+
+}

speech/src/main/java/cn/smartjavaai/speech/tts/enums/TtsModelEnum.java

@@ -0,0 +1,40 @@
+package cn.smartjavaai.speech.tts.enums;
+
+/**
+ * tts模型枚举
+ * @author dwj
+ */
+public enum TtsModelEnum {
+
+    SHERPA_VITS("vits"),
+    SHERPA_MATCHA("matcha"),
+    SHERPA_KOKORO("kokoro"),
+    SHERPA_KITTEN("kitten");
+
+    TtsModelEnum(String engine) {
+        this.engine = engine;
+    }
+
+    /**
+     * 模型引擎
+     */
+    private final String engine;
+
+    public String getEngine() {
+        return engine;
+    }
+
+    /**
+     * 根据名称获取枚举 (忽略大小写和下划线变体)
+     */
+    public static TtsModelEnum fromName(String name) {
+        String formatted = name.trim().toUpperCase().replaceAll("[-_]", "");
+        for (TtsModelEnum model : values()) {
+            if (model.name().replaceAll("_", "").equals(formatted)) {
+                return model;
+            }
+        }
+        throw new IllegalArgumentException("未知模型名称: " + name);
+    }
+
+}

speech/src/main/java/cn/smartjavaai/speech/tts/exception/TtsException.java

@@ -0,0 +1,29 @@
+package cn.smartjavaai.speech.tts.exception;
+
+/**
+ * tts异常
+ * @author dwj
+ */
+public class TtsException extends RuntimeException{
+
+    public TtsException() {
+        super();
+    }
+
+    public TtsException(String message, Throwable cause, boolean enableSuppression, boolean writableStackTrace) {
+        super(message, cause, enableSuppression, writableStackTrace);
+    }
+
+    public TtsException(String message, Throwable cause) {
+        super(message, cause);
+    }
+
+    public TtsException(String message) {
+        super(message);
+    }
+
+    public TtsException(Throwable cause) {
+        super(cause);
+    }
+
+}

speech/src/main/java/cn/smartjavaai/speech/tts/factory/SherpaOfflineTtsModelConfigFactory.java

@@ -0,0 +1,186 @@
+package cn.smartjavaai.speech.tts.factory;
+
+import cn.hutool.core.io.FileUtil;
+import cn.smartjavaai.common.enums.DeviceEnum;
+import cn.smartjavaai.common.utils.FileUtils;
+import cn.smartjavaai.speech.tts.exception.TtsException;
+import com.k2fsa.sherpa.onnx.*;
+import cn.smartjavaai.speech.tts.config.TtsModelConfig;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import java.io.File;
+import java.util.List;
+
+/**
+ * @author dwj
+ * @date 2025/10/14
+ */
+public class SherpaOfflineTtsModelConfigFactory {
+
+    /**
+     * 创建Vits模型配置
+     * @param config
+     * @return
+     */
+    public static OfflineTtsVitsModelConfig createVitsConfig(TtsModelConfig config) {
+        List<File> tokensFiles = FileUtils.findFilesByName(new File(config.getModelPath()), "tokens.txt", false);
+        List<File> lexiconFiles = FileUtils.findFilesByName(new File(config.getModelPath()), "lexicon.txt", false);
+        List<File> dictFiles = FileUtils.findFilesByName(new File(config.getModelPath()), "dict", false);
+        List<File> dataDirFiles = FileUtils.findFilesWithSuffix(new File(config.getModelPath()), "-data", false);
+        String model = config.getModelPath() + File.separator + config.getModelName();;
+        String tokens = CollectionUtils.isEmpty(tokensFiles) ? "" : tokensFiles.get(0).getAbsolutePath();
+        String lexicon = CollectionUtils.isEmpty(lexiconFiles) ? "" : lexiconFiles.get(0).getAbsolutePath();
+        String dictPath = CollectionUtils.isEmpty(dictFiles) ? "" : dictFiles.get(0).getAbsolutePath();
+        String dataDir = CollectionUtils.isEmpty(dataDirFiles) ? "" : dataDirFiles.get(0).getAbsolutePath();
+        float lengthScale = config.getCustomParam("lengthScale", Float.class, 1f);
+        float noiseScale = config.getCustomParam("noiseScale", Float.class, 0.667F);
+        float noiseScaleW = config.getCustomParam("noiseScaleW", Float.class, 0.8f);
+        OfflineTtsVitsModelConfig vitsModelConfig =
+                OfflineTtsVitsModelConfig.builder()
+                        .setModel(model)
+                        .setTokens(tokens)
+                        .setLexicon(lexicon)
+                        .setDictDir(dictPath)
+                        .setDataDir(dataDir)
+                        .setLengthScale(lengthScale)
+                        .setNoiseScale(noiseScale)
+                        .setNoiseScaleW(noiseScaleW)
+                        .build();
+        return vitsModelConfig;
+    }
+
+    public static OfflineTtsMatchaModelConfig createMatchaConfig(TtsModelConfig config) {
+        String vocoder = config.getCustomParam("vocoder", String.class);
+        if(StringUtils.isBlank(vocoder)){
+            throw new TtsException("vocoder is null");
+        }
+        List<File> tokensFiles = FileUtils.findFilesByName(new File(config.getModelPath()), "tokens.txt", false);
+        List<File> lexiconFiles = FileUtils.findFilesByName(new File(config.getModelPath()), "lexicon.txt", false);
+        List<File> dictFiles = FileUtils.findFilesByName(new File(config.getModelPath()), "dict", false);
+        List<File> dataDirFiles = FileUtils.findFilesWithSuffix(new File(config.getModelPath()), "-data", false);
+        String model = config.getModelPath() + File.separator + config.getModelName();;
+        String tokens = CollectionUtils.isEmpty(tokensFiles) ? "" : tokensFiles.get(0).getAbsolutePath();
+        String lexicon = CollectionUtils.isEmpty(lexiconFiles) ? "" : lexiconFiles.get(0).getAbsolutePath();
+        String dictPath = CollectionUtils.isEmpty(dictFiles) ? "" : dictFiles.get(0).getAbsolutePath();
+        String dataDir = CollectionUtils.isEmpty(dataDirFiles) ? "" : dataDirFiles.get(0).getAbsolutePath();
+        float lengthScale = config.getCustomParam("lengthScale", Float.class, 1f);
+        float noiseScale = config.getCustomParam("noiseScale", Float.class, 1f);
+        OfflineTtsMatchaModelConfig vitsModelConfig =
+                OfflineTtsMatchaModelConfig.builder()
+                        .setAcousticModel(model)
+                        .setTokens(tokens)
+                        .setLexicon(lexicon)
+                        .setDictDir(dictPath)
+                        .setDataDir(dataDir)
+                        .setVocoder(vocoder)
+                        .setLengthScale(lengthScale)
+                        .setNoiseScale(noiseScale)
+                        .build();
+        return vitsModelConfig;
+    }
+
+    public static OfflineTtsKittenModelConfig createKittenConfig(TtsModelConfig config) {
+        List<File> tokensFiles = FileUtils.findFilesByName(new File(config.getModelPath()), "tokens.txt", false);
+        List<File> voicesFiles = FileUtils.findFilesByName(new File(config.getModelPath()), "voices.bin", false);
+        List<File> dataDirFiles = FileUtils.findFilesWithSuffix(new File(config.getModelPath()), "-data", false);
+        String model = config.getModelPath() + File.separator + config.getModelName();;
+        String tokens = CollectionUtils.isEmpty(tokensFiles) ? "" : tokensFiles.get(0).getAbsolutePath();
+        String voices = CollectionUtils.isEmpty(voicesFiles) ? "" : voicesFiles.get(0).getAbsolutePath();
+        String dataDir = CollectionUtils.isEmpty(dataDirFiles) ? "" : dataDirFiles.get(0).getAbsolutePath();
+        float lengthScale = config.getCustomParam("lengthScale", Float.class, 1f);
+        OfflineTtsKittenModelConfig vitsModelConfig =
+                OfflineTtsKittenModelConfig.builder()
+                        .setModel(model)
+                        .setTokens(tokens)
+                        .setVoices(voices)
+                        .setDataDir(dataDir)
+                        .setLengthScale(lengthScale)
+                        .build();
+        return vitsModelConfig;
+    }
+
+    public static OfflineTtsKokoroModelConfig createKokoroConfig(TtsModelConfig config) {
+        List<File> tokensFiles = FileUtils.findFilesByName(new File(config.getModelPath()), "tokens.txt", false);
+        List<File> lexiconFiles = FileUtils.searchFiles(config.getModelPath(), "lexicon", ".txt",false);
+        List<File> dictFiles = FileUtils.findFilesByName(new File(config.getModelPath()), "dict", false);
+        List<File> dataDirFiles = FileUtils.findFilesWithSuffix(new File(config.getModelPath()), "-data", false);
+        List<File> voicesFiles = FileUtils.findFilesByName(new File(config.getModelPath()), "voices.bin", false);
+        String model = config.getModelPath() + File.separator + config.getModelName();;
+        String tokens = CollectionUtils.isEmpty(tokensFiles) ? "" : tokensFiles.get(0).getAbsolutePath();
+        String lexicon = CollectionUtils.isEmpty(lexiconFiles) ? "" : FileUtils.joinAbsolutePaths(lexiconFiles);
+        String dictPath = CollectionUtils.isEmpty(dictFiles) ? "" : dictFiles.get(0).getAbsolutePath();
+        String dataDir = CollectionUtils.isEmpty(dataDirFiles) ? "" : dataDirFiles.get(0).getAbsolutePath();
+        String voices = CollectionUtils.isEmpty(voicesFiles) ? "" : voicesFiles.get(0).getAbsolutePath();
+        float lengthScale = config.getCustomParam("lengthScale", Float.class, 1f);
+        OfflineTtsKokoroModelConfig vitsModelConfig =
+                OfflineTtsKokoroModelConfig.builder()
+                        .setModel(model)
+                        .setTokens(tokens)
+                        .setLexicon(lexicon)
+                        .setDictDir(dictPath)
+                        .setDataDir(dataDir)
+                        .setVoices(voices)
+                        .setLengthScale(lengthScale)
+                        .build();
+        return vitsModelConfig;
+    }
+
+    /**
+     * 创建模型配置
+     * @param config
+     * @return
+     */
+    public static OfflineTtsConfig createConfig(TtsModelConfig config) {
+        OfflineTtsModelConfig modelConfig = null;
+        int numThreads = config.getCustomParam("numThreads", Integer.class, 1);
+        boolean debug = config.getCustomParam("debug", Boolean.class, true);
+        String provider = config.getCustomParam("provider", String.class, "cpu");
+        String ruleFsts = "";
+        switch (config.getModelEnum()){
+            case SHERPA_VITS:
+                modelConfig =
+                        OfflineTtsModelConfig.builder()
+                                .setVits(createVitsConfig(config))
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_MATCHA:
+                modelConfig =
+                        OfflineTtsModelConfig.builder()
+                                .setMatcha(createMatchaConfig(config))
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_KITTEN:
+                modelConfig =
+                        OfflineTtsModelConfig.builder()
+                                .setKitten(createKittenConfig(config))
+                                .setNumThreads(numThreads)
+                                .setDebug(debug)
+                                .setProvider(provider)
+                                .build();
+                break;
+            case SHERPA_KOKORO:
+                numThreads = config.getCustomParam("numThreads", Integer.class, 2);
+                modelConfig =
+                    OfflineTtsModelConfig.builder()
+                            .setKokoro(createKokoroConfig(config))
+                            .setNumThreads(numThreads)
+                            .setDebug(debug)
+                            .setProvider(provider)
+                            .build();
+            break;
+        }
+        List<File> ruleFstFiles = FileUtils.findFilesWithSuffix(new File(config.getModelPath()), ".fst", false);
+        ruleFsts = CollectionUtils.isEmpty(ruleFstFiles) ? "" : FileUtils.joinAbsolutePaths(ruleFstFiles);
+        OfflineTtsConfig offlineTtsConfig =
+                OfflineTtsConfig.builder().setModel(modelConfig).setRuleFsts(ruleFsts).build();
+        return offlineTtsConfig;
+    }
+
+}

speech/src/main/java/cn/smartjavaai/speech/tts/factory/TtsModelFactory.java

@@ -0,0 +1,131 @@
+package cn.smartjavaai.speech.tts.factory;
+
+import cn.smartjavaai.common.config.Config;
+import cn.smartjavaai.speech.asr.exception.AsrException;
+import cn.smartjavaai.speech.asr.model.VoskRecognizer;
+import cn.smartjavaai.speech.asr.model.WhisperRecognizer;
+import cn.smartjavaai.speech.tts.config.TtsModelConfig;
+import cn.smartjavaai.speech.tts.enums.TtsModelEnum;
+import cn.smartjavaai.speech.tts.model.SherpaTtsModel;
+import cn.smartjavaai.speech.tts.model.TtsModel;
+import lombok.extern.slf4j.Slf4j;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * 语音合成模型工厂
+ * @author dwj
+ */
+@Slf4j
+public class TtsModelFactory {
+
+    // 使用 volatile 和双重检查锁定来确保线程安全的单例模式
+    private static volatile TtsModelFactory instance;
+
+    /**
+     * 模型缓存
+     */
+    private static final ConcurrentHashMap<TtsModelEnum, TtsModel> modelMap = new ConcurrentHashMap<>();
+
+    /**
+     * 模型注册表
+     */
+    private static final Map<TtsModelEnum, Class<? extends TtsModel>> registry =
+            new ConcurrentHashMap<>();
+
+
+    public static TtsModelFactory getInstance() {
+        if (instance == null) {
+            synchronized (TtsModelFactory.class) {
+                if (instance == null) {
+                    instance = new TtsModelFactory();
+                }
+            }
+        }
+        return instance;
+    }
+
+
+
+    /**
+     * 注册模型
+     * @param expressionModelEnum
+     * @param clazz
+     */
+    private static void registerModel(TtsModelEnum expressionModelEnum, Class<? extends TtsModel> clazz) {
+        registry.put(expressionModelEnum, clazz);
+    }
+
+
+    /**
+     * 获取模型（通过配置）
+     * @param config
+     * @return
+     */
+    public TtsModel getModel(TtsModelConfig config) {
+        if(Objects.isNull(config) || Objects.isNull(config.getModelEnum())){
+            throw new AsrException("未配置语音识别模型枚举");
+        }
+        return modelMap.computeIfAbsent(config.getModelEnum(), k -> {
+            return createModel(config);
+        });
+    }
+
+    /**
+     * 使用ModelConfig创建模型
+     * @param config
+     * @return
+     */
+    private TtsModel createModel(TtsModelConfig config) {
+        Class<?> clazz = registry.get(config.getModelEnum());
+        if(clazz == null){
+            throw new AsrException("Unsupported model");
+        }
+        TtsModel model = null;
+        try {
+            model = (TtsModel) clazz.newInstance();
+        } catch (InstantiationException | IllegalAccessException e) {
+            throw new AsrException(e);
+        }
+        model.loadModel(config);
+        model.setFromFactory(true);
+        return model;
+    }
+
+
+    // 初始化默认算法
+    static {
+        registerModel(TtsModelEnum.SHERPA_KOKORO, SherpaTtsModel.class);
+        registerModel(TtsModelEnum.SHERPA_KITTEN, SherpaTtsModel.class);
+        registerModel(TtsModelEnum.SHERPA_MATCHA, SherpaTtsModel.class);
+        registerModel(TtsModelEnum.SHERPA_VITS, SherpaTtsModel.class);
+        log.debug("缓存目录：{}", Config.getCachePath());
+    }
+
+    /**
+     * 关闭所有已加载的模型
+     */
+    public void closeAll() {
+        modelMap.values().forEach(model -> {
+            try {
+                model.close();
+            } catch (Exception e) {
+                e.printStackTrace();
+            }
+        });
+        modelMap.clear();
+    }
+
+    /**
+     * 移除缓存的模型
+     * @param modelEnum
+     */
+    public static void removeFromCache(TtsModelEnum modelEnum) {
+        modelMap.remove(modelEnum);
+    }
+
+
+
+}

speech/src/main/java/cn/smartjavaai/speech/tts/model/SherpaTtsModel.java

@@ -0,0 +1,132 @@
+package cn.smartjavaai.speech.tts.model;
+
+import ai.djl.modality.audio.Audio;
+import ai.djl.modality.audio.AudioFactory;
+import cn.smartjavaai.common.entity.R;
+import cn.smartjavaai.common.enums.DeviceEnum;
+import cn.smartjavaai.speech.asr.exception.AsrException;
+import cn.smartjavaai.speech.asr.pool.WhisperStatePool;
+import cn.smartjavaai.speech.tts.config.TtsModelConfig;
+import cn.smartjavaai.speech.tts.entity.SherpaTtsParams;
+import cn.smartjavaai.speech.tts.entity.TtsParams;
+import cn.smartjavaai.speech.tts.exception.TtsException;
+import cn.smartjavaai.speech.tts.factory.SherpaOfflineTtsModelConfigFactory;
+import com.k2fsa.sherpa.onnx.GeneratedAudio;
+import com.k2fsa.sherpa.onnx.OfflineTts;
+import com.k2fsa.sherpa.onnx.OfflineTtsCallback;
+import com.k2fsa.sherpa.onnx.OfflineTtsConfig;
+import io.github.givimad.whisperjni.WhisperJNI;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+
+/**
+ * @author dwj
+ */
+@Slf4j
+public class SherpaTtsModel implements TtsModel{
+
+    private TtsModelConfig config;
+
+    private OfflineTts offlineTts;
+
+
+    @Override
+    public void loadModel(TtsModelConfig config) {
+        this.config = config;
+        if(StringUtils.isBlank(config.getModelPath())){
+            throw new TtsException("modelPath is null");
+        }
+        if(StringUtils.isBlank(config.getModelName())){
+            throw new TtsException("modelName is null");
+        }
+        Path testModelPath = Paths.get(config.getModelPath());
+        if(!testModelPath.toFile().exists()){
+            throw new TtsException("modelPath does not exist: " + testModelPath.toAbsolutePath());
+        }
+        if(Objects.isNull(config.getLibPath())){
+            throw new TtsException("libPath is null");
+        }
+        if(!config.getLibPath().toFile().exists()){
+            throw new TtsException("libPath does not exist: " + testModelPath.toAbsolutePath());
+        }
+        try {
+            //加载依赖库
+            System.setProperty("sherpa_onnx.native.path",config.getLibPath().toAbsolutePath().toString());
+            OfflineTtsConfig offlineTtsConfig = SherpaOfflineTtsModelConfigFactory.createConfig(config);
+            offlineTts = new OfflineTts(offlineTtsConfig);
+            log.debug("Sherpa tts init success");
+        } catch (Exception e) {
+            throw new TtsException(e);
+        }
+    }
+
+    private GeneratedAudio generateCore(String text, TtsParams params) {
+        if(offlineTts == null){
+            throw new TtsException("模型未初始化");
+        }
+        SherpaTtsParams sherpaTtsParams =null;
+        if(params == null){
+            sherpaTtsParams = new SherpaTtsParams();
+        }else{
+            if(params instanceof SherpaTtsParams){
+                sherpaTtsParams = (SherpaTtsParams) params;
+            }else{
+                throw new TtsException("params参数类型不是 SherpaTtsParams");
+            }
+        }
+        try {
+            int sid = 100;
+            float speed = 1.0f;
+            if(params != null){
+                sid = sherpaTtsParams.getSpeakerId() > 0 ? sherpaTtsParams.getSpeakerId() : 100;
+                speed = sherpaTtsParams.getSpeed() > 0.0f ? sherpaTtsParams.getSpeed() : 1.0f;
+            }
+            GeneratedAudio audio = null;
+            if(sherpaTtsParams.getCallback() != null){
+                audio = offlineTts.generateWithCallback(text, sid, speed, sherpaTtsParams.getCallback());
+            }else{
+                audio = offlineTts.generate(text, sid, speed);
+            }
+            float audioDuration = audio.getSamples().length / (float) audio.getSampleRate();
+            log.debug("-- audio duration: {} seconds", String.format("%.3f", audioDuration));
+            return audio;
+        } catch (Exception e) {
+            throw new TtsException(e);
+        }
+    }
+
+    @Override
+    public R<Audio> generate(String text, TtsParams params) {
+        GeneratedAudio audio = generateCore(text, params);
+        return R.ok(new Audio(audio.getSamples()));
+    }
+
+    @Override
+    public void generate(String text, TtsParams params, String savePath) {
+        GeneratedAudio audio = generateCore(text, params);
+        audio.save(savePath);
+    }
+
+    private boolean fromFactory = false;
+
+    @Override
+    public void setFromFactory(boolean fromFactory) {
+        this.fromFactory = fromFactory;
+    }
+    public boolean isFromFactory() {
+        return fromFactory;
+    }
+
+    @Override
+    public void close() throws Exception {
+        if(offlineTts != null){
+            offlineTts.release();
+        }
+    }
+}

speech/src/main/java/cn/smartjavaai/speech/tts/model/TtsModel.java

@@ -0,0 +1,32 @@
+package cn.smartjavaai.speech.tts.model;
+
+import ai.djl.modality.audio.Audio;
+import cn.smartjavaai.common.entity.R;
+import cn.smartjavaai.speech.tts.config.TtsModelConfig;
+import cn.smartjavaai.speech.tts.entity.TtsParams;
+
+/**
+ * tts模型
+ * @author dwj
+ */
+public interface TtsModel extends AutoCloseable{
+
+    /**
+     * 加载模型
+     * @param config
+     */
+    void loadModel(TtsModelConfig config);
+
+    default R<Audio> generate(String text, TtsParams params){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    default void generate(String text, TtsParams params, String savePath){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    default void setFromFactory(boolean fromFactory){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+}

speech/src/main/java/cn/smartjavaai/speech/utils/AudioUtils.java

@@ -1,5 +1,6 @@
 package cn.smartjavaai.speech.utils;
 
+import ai.djl.modality.audio.Audio;
 import cn.hutool.core.lang.UUID;
 import cn.smartjavaai.speech.asr.exception.AsrException;
 import ws.schild.jave.Encoder;
@@ -9,6 +10,7 @@ import ws.schild.jave.encode.AudioAttributes;
 import ws.schild.jave.encode.EncodingAttributes;
 import ws.schild.jave.info.MultimediaInfo;
 
+import javax.sound.sampled.AudioFileFormat;
 import javax.sound.sampled.AudioFormat;
 import javax.sound.sampled.AudioInputStream;
 import javax.sound.sampled.AudioSystem;
@@ -208,5 +210,62 @@ public class AudioUtils {
     }
 
 
+    /**
+     * 将 float[] 音频数据保存为 WAV 文件
+     */
+    public static void saveToWav(float[] floats, AudioFormat format, String savePath) throws IOException {
+        // 1. 转换为 16-bit PCM
+        byte[] bytes = floatsToPCM16(floats);
+        // 2. 使用 ByteArrayInputStream 封装为音频流
+        try (ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
+             AudioInputStream ais = new AudioInputStream(bais, format, floats.length)) {
+            // 3. 保存到本地文件
+            AudioSystem.write(ais, AudioFileFormat.Type.WAVE, new File(savePath));
+        }
+    }
+
+    /**
+     * 将 float[] 音频数据保存为 WAV 文件
+     */
+    public static void saveToWav(float[] floats, String savePath) throws IOException {
+        // 1. 转换为 16-bit PCM
+        byte[] bytes = floatsToPCM16(floats);
+        // 2. 使用 ByteArrayInputStream 封装为音频流
+        try (ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
+             AudioInputStream ais = new AudioInputStream(bais, getDefaultAudioFormat(), floats.length)) {
+            // 3. 保存到本地文件
+            AudioSystem.write(ais, AudioFileFormat.Type.WAVE, new File(savePath));
+        }
+    }
+
+    public static AudioFormat getDefaultAudioFormat(){
+        return new AudioFormat(
+                AudioFormat.Encoding.PCM_SIGNED,
+                16000,
+                16,
+                1,
+                2,
+                16000,
+                false
+        );
+    }
+
+    /**
+     * 将 float[] 转为 16bit PCM (little endian)
+     */
+    private static byte[] floatsToPCM16(float[] floats) {
+        byte[] bytes = new byte[floats.length * 2];
+        int i = 0;
+        for (float sample : floats) {
+            // 裁剪范围 [-1, 1]
+            sample = Math.max(-1.0f, Math.min(1.0f, sample));
+            short s = (short) (sample * 32767);
+            bytes[i++] = (byte) (s & 0xFF);
+            bytes[i++] = (byte) ((s >> 8) & 0xFF);
+        }
+        return bytes;
+    }
+
+
 
 }

translate/pom.xml

@@ -25,7 +25,7 @@
         </dependency>
     </dependencies>
 
-    <version>1.0.25</version>
+    <version>1.0.26</version>
     <name>translate</name>
     <description>SmartJavaAI</description>
     <url>https://github.com/geekwenjie/SmartJavaAI</url>

vision/pom.xml

@@ -6,11 +6,11 @@
     <parent>
         <groupId>cn.smartjavaai</groupId>
         <artifactId>smartjavaai-parent</artifactId>
-        <version>1.0.25</version>
+        <version>1.0.26</version>
     </parent>
 
     <artifactId>vision</artifactId>
-    <version>1.0.25</version>
+    <version>1.0.26</version>
     <name>vision</name>
     <description>SmartJavaAI</description>
     <url>https://github.com/geekwenjie/SmartJavaAI</url>

vision/src/main/java/cn/smartjavaai/clip/config/ClipModelConfig.java

@@ -0,0 +1,45 @@
+package cn.smartjavaai.clip.config;
+
+import cn.smartjavaai.action.enums.ActionRecModelEnum;
+import cn.smartjavaai.clip.enums.ClipModelEnum;
+import cn.smartjavaai.common.config.ModelConfig;
+import cn.smartjavaai.common.enums.DeviceEnum;
+import lombok.Data;
+
+import java.util.List;
+
+/**
+ * CLIP模型参数配置
+ *
+ * @author dwj
+ */
+@Data
+public class ClipModelConfig extends ModelConfig {
+
+    /**
+     * 模型
+     */
+    private ClipModelEnum modelEnum;
+
+
+    /**
+     * 模型路径
+     */
+    private String modelPath;
+
+
+
+    public ClipModelConfig() {
+    }
+
+    public ClipModelConfig(ClipModelEnum modelEnum, DeviceEnum device) {
+        this.modelEnum = modelEnum;
+        setDevice(device);
+    }
+
+    public ClipModelConfig(ClipModelEnum modelEnum) {
+        this.modelEnum = modelEnum;
+    }
+
+
+}

vision/src/main/java/cn/smartjavaai/clip/enums/ClipModelEnum.java

@@ -0,0 +1,26 @@
+package cn.smartjavaai.clip.enums;
+
+/**
+ * CLIP模型枚举
+ * @author dwj
+ */
+public enum ClipModelEnum {
+
+    OPENAI;
+
+
+
+    /**
+     * 根据名称获取枚举 (忽略大小写和下划线变体)
+     */
+    public static ClipModelEnum fromName(String name) {
+        String formatted = name.trim().toUpperCase().replaceAll("[-_]", "");
+        for (ClipModelEnum model : values()) {
+            if (model.name().replaceAll("_", "").equals(formatted)) {
+                return model;
+            }
+        }
+        throw new IllegalArgumentException("未知模型名称: " + name);
+    }
+
+}

vision/src/main/java/cn/smartjavaai/clip/exception/ClipException.java

@@ -0,0 +1,29 @@
+package cn.smartjavaai.clip.exception;
+
+/**
+ * CLIP异常
+ * @author dwj
+ */
+public class ClipException extends RuntimeException{
+
+    public ClipException() {
+        super();
+    }
+
+    public ClipException(String message, Throwable cause, boolean enableSuppression, boolean writableStackTrace) {
+        super(message, cause, enableSuppression, writableStackTrace);
+    }
+
+    public ClipException(String message, Throwable cause) {
+        super(message, cause);
+    }
+
+    public ClipException(String message) {
+        super(message);
+    }
+
+    public ClipException(Throwable cause) {
+        super(cause);
+    }
+
+}

vision/src/main/java/cn/smartjavaai/clip/model/ClipModel.java

@@ -0,0 +1,133 @@
+package cn.smartjavaai.clip.model;
+
+import ai.djl.modality.cv.Image;
+import cn.smartjavaai.clip.config.ClipModelConfig;
+import cn.smartjavaai.common.entity.R;
+
+/**
+ * @author dwj
+ * @date 2025/10/20
+ */
+public interface ClipModel extends AutoCloseable{
+
+    /**
+     * 加载模型
+     * @param config
+     */
+    void loadModel(ClipModelConfig config); // 加载模型
+
+    /**
+     * 图片特征提取
+     * @param image
+     * @return
+     */
+    default R<float[]> extractImageFeatures(Image image){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    /**
+     * 图片特征提取
+     * @param imagePath
+     * @return
+     */
+    default R<float[]> extractImageFeatures(String imagePath){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    /**
+     * 文本特征提取
+     * @param inputs
+     * @return
+     */
+    default R<float[]> extractTextFeatures(String inputs){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    /**
+     * 文本和图片特征比较
+     * @param image
+     * @param text
+     * @return
+     */
+    default R<Float> compareTextAndImage(Image image, String text){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    /**
+     * 图片特征比较
+     * @param image1 图1
+     * @param image2 图2
+     * @return
+     */
+    default R<Float> compareImage(Image image1, Image image2){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+
+    /**
+     * 图片特征比较
+     * @param image1 图1
+     * @param image2 图2
+     * @return
+     */
+    default R<Float> compareImage(Image image1, Image image2, float scale){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    /**
+     * 图片特征比较
+     * @param imagePath1 图1
+     * @param imagePath2 图2
+     * @return
+     */
+    default R<Float> compareImage(String imagePath1, String imagePath2){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    /**
+     * 图片特征比较
+     * @param imagePath1 图1
+     * @param imagePath2 图2
+     * @return
+     */
+    default R<Float> compareImage(String imagePath1, String imagePath2, float scale){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+
+    /**
+     * 文本特征比较
+     * @param input1 文本1
+     * @param input2 文本2
+     * @return
+     */
+    default R<Float> compareText(String input1, String input2){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    /**
+     * 文本特征比较
+     * @param input1 文本1
+     * @param input2 文本2
+     * @return
+     */
+    default R<Float> compareText(String input1, String input2, float scale){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    /**
+     * 文本特征比较
+     * @param feature1 文本1
+     * @param feature2 文本2
+     * @return
+     */
+    default R<Float> compareFeatures(float[] feature1, float[] feature2, float scale){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    default void setFromFactory(boolean fromFactory){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+
+}

vision/src/main/java/cn/smartjavaai/clip/model/ClipModelFactory.java

@@ -0,0 +1,124 @@
+package cn.smartjavaai.clip.model;
+
+import cn.smartjavaai.action.model.ActionRecModelFactory;
+import cn.smartjavaai.action.model.CommonActionRecModel;
+import cn.smartjavaai.clip.config.ClipModelConfig;
+import cn.smartjavaai.clip.enums.ClipModelEnum;
+import cn.smartjavaai.common.config.Config;
+import cn.smartjavaai.objectdetection.exception.DetectionException;
+import lombok.extern.slf4j.Slf4j;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * @author dwj
+ * @date 2025/10/20
+ */
+@Slf4j
+public class ClipModelFactory {
+
+    // 使用 volatile 和双重检查锁定来确保线程安全的单例模式
+    private static volatile ClipModelFactory instance;
+
+    private static final ConcurrentHashMap<ClipModelEnum, ClipModel> modelMap = new ConcurrentHashMap<>();
+
+    /**
+     * 模型注册表
+     */
+    private static final Map<ClipModelEnum, Class<? extends ClipModel>> registry =
+            new ConcurrentHashMap<>();
+
+
+    // 私有构造函数，防止外部创建实例
+    private ClipModelFactory() {}
+
+    // 双重检查锁定的单例方法
+    public static ClipModelFactory getInstance() {
+        if (instance == null) {
+            synchronized (ClipModelFactory.class) {
+                if (instance == null) {
+                    instance = new ClipModelFactory();
+                }
+            }
+        }
+        return instance;
+    }
+
+    /**
+     * 获取模型（通过配置）
+     * @param config
+     * @return
+     */
+    public ClipModel getModel(ClipModelConfig config) {
+        if(Objects.isNull(config) || Objects.isNull(config.getModelEnum())){
+            throw new DetectionException("未配置模型");
+        }
+        return modelMap.computeIfAbsent(config.getModelEnum(), k -> {
+            return createFaceDetModel(config);
+        });
+    }
+
+    /**
+     * 使用ModelConfig创建模型
+     * @param config
+     * @return
+     */
+    private ClipModel createFaceDetModel(ClipModelConfig config) {
+        Class<?> clazz = registry.get(config.getModelEnum());
+        if(clazz == null){
+            throw new DetectionException("Unsupported model");
+        }
+        ClipModel model = null;
+        try {
+            model = (ClipModel) clazz.newInstance();
+        } catch (InstantiationException | IllegalAccessException e) {
+            throw new DetectionException(e);
+        }
+        model.loadModel(config);
+        model.setFromFactory(true);
+        return model;
+    }
+
+
+    /**
+     * 注册模型
+     * @param modelEnum
+     * @param clazz
+     */
+    private static void registerAlgorithm(ClipModelEnum modelEnum, Class<? extends ClipModel> clazz) {
+        registry.put(modelEnum, clazz);
+    }
+
+
+
+    // 初始化默认算法
+    static {
+        registerAlgorithm(ClipModelEnum.OPENAI, OpenAIClipModel.class);
+        log.debug("缓存目录：{}", Config.getCachePath());
+    }
+
+    /**
+     * 关闭所有已加载的模型
+     */
+    public void closeAll() {
+        modelMap.values().forEach(model -> {
+            try {
+                model.close();
+            } catch (Exception e) {
+                e.printStackTrace();
+            }
+        });
+        modelMap.clear();
+    }
+
+    /**
+     * 移除缓存的模型
+     * @param modelEnum
+     */
+    public static void removeFromCache(ClipModelEnum modelEnum) {
+        modelMap.remove(modelEnum);
+    }
+
+}

vision/src/main/java/cn/smartjavaai/clip/model/OpenAIClipModel.java

@@ -0,0 +1,311 @@
+package cn.smartjavaai.clip.model;
+
+import ai.djl.Device;
+import ai.djl.MalformedModelException;
+import ai.djl.engine.Engine;
+import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer;
+import ai.djl.inference.Predictor;
+import ai.djl.modality.cv.Image;
+import ai.djl.ndarray.NDList;
+import ai.djl.repository.zoo.Criteria;
+import ai.djl.repository.zoo.ModelNotFoundException;
+import ai.djl.repository.zoo.ZooModel;
+import ai.djl.translate.NoopTranslator;
+import ai.djl.util.Pair;
+import cn.smartjavaai.clip.config.ClipModelConfig;
+import cn.smartjavaai.clip.exception.ClipException;
+import cn.smartjavaai.clip.pool.ClipImagePredictorFactory;
+import cn.smartjavaai.clip.pool.ClipImageTextPredictorFactory;
+import cn.smartjavaai.clip.pool.ClipTextPredictorFactory;
+import cn.smartjavaai.clip.translator.ImageTextTranslator;
+import cn.smartjavaai.clip.translator.ImageTranslator;
+import cn.smartjavaai.clip.translator.TextTranslator;
+import cn.smartjavaai.common.cv.SmartImageFactory;
+import cn.smartjavaai.common.entity.R;
+import cn.smartjavaai.common.enums.DeviceEnum;
+import cn.smartjavaai.common.enums.SimilarityType;
+import cn.smartjavaai.common.pool.PredictorFactory;
+import cn.smartjavaai.common.utils.DJLCommonUtils;
+import cn.smartjavaai.common.utils.ImageUtils;
+import cn.smartjavaai.common.utils.SimilarityUtil;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.pool2.impl.GenericObjectPool;
+
+import javax.sound.sampled.Clip;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+
+/**
+ * openai clip 模型
+ * @author dwj
+ */
+@Slf4j
+public class OpenAIClipModel implements ClipModel{
+
+    private ClipModelConfig config;
+
+    private ZooModel<NDList, NDList> model;
+
+    private HuggingFaceTokenizer tokenizer;
+
+    private GenericObjectPool<Predictor<Image, float[]>> imageFeaturePredictorPool;
+
+    private GenericObjectPool<Predictor<String, float[]>> textFeaturePredictorPool;
+
+    private GenericObjectPool<Predictor<Pair<Image, String>, float[]>> imgTextPredictorPool;
+
+    @Override
+    public void loadModel(ClipModelConfig config) {
+        if(Objects.isNull(config)){
+            throw new ClipException("config为null");
+        }
+        if(StringUtils.isBlank(config.getModelPath())){
+            throw new ClipException("modelPath为空");
+        }
+        this.config = config;
+        try {
+//            Device device = null;
+//            if(!Objects.isNull(config.getDevice())){
+//                device = config.getDevice() == DeviceEnum.CPU ? Device.cpu() : Device.gpu(config.getGpuId());
+//            }
+            boolean isUrl = DJLCommonUtils.hasSupportedProtocol(config.getModelPath());
+            Criteria<NDList, NDList> criteria =
+                    Criteria.builder()
+                            .setTypes(NDList.class, NDList.class)
+//                            .optModelUrls("https://resources.djl.ai/demo/pytorch/clip.zip")
+                            .optModelUrls(isUrl ? config.getModelPath() : null)
+                            .optModelName("clip.pt")
+                            .optModelPath(isUrl ? null : Paths.get(config.getModelPath()))
+                            .optTranslator(new NoopTranslator())
+                            .optEngine("PyTorch")
+//                            .optOption("mapLocation", "true")
+                            .optDevice(Device.cpu()) // torchscript model only support CPU
+                            .build();
+            model = criteria.loadModel();
+            Path modelCachePath = model.getWrappedModel().getModelPath();
+            Path tokenizerPath = modelCachePath.resolve("tokenizer.json");
+            tokenizer = HuggingFaceTokenizer.newInstance(tokenizerPath);
+            // 创建池子：每个线程独享 Predictor
+            imageFeaturePredictorPool = new GenericObjectPool<>(new ClipImagePredictorFactory(model));
+            textFeaturePredictorPool = new GenericObjectPool<>(new ClipTextPredictorFactory(model, tokenizer));
+            imgTextPredictorPool = new GenericObjectPool<>(new ClipImageTextPredictorFactory(model, tokenizer));
+            int predictorPoolSize = config.getPredictorPoolSize();
+            if(config.getPredictorPoolSize() <= 0){
+                predictorPoolSize = Runtime.getRuntime().availableProcessors(); // 默认等于CPU核心数
+            }
+            imageFeaturePredictorPool.setMaxTotal(predictorPoolSize);
+            textFeaturePredictorPool.setMaxTotal(predictorPoolSize);
+            imgTextPredictorPool.setMaxTotal(predictorPoolSize);
+            log.debug("当前设备: " + model.getNDManager().getDevice());
+            log.debug("当前引擎: " + Engine.getInstance().getEngineName());
+            log.debug("模型推理器线程池最大数量: " + predictorPoolSize);
+        } catch (IOException | ModelNotFoundException | MalformedModelException e) {
+            throw new ClipException("模型加载失败", e);
+        }
+
+    }
+
+    @Override
+    public R<float[]> extractImageFeatures(Image image) {
+        Predictor<Image, float[]> predictor = null;
+        try {
+            predictor = imageFeaturePredictorPool.borrowObject();
+            return R.ok(predictor.predict(image));
+        } catch (Exception e) {
+            throw new ClipException("特征提取错误", e);
+        }finally {
+            if (predictor != null) {
+                try {
+                    imageFeaturePredictorPool.returnObject(predictor); //归还
+                } catch (Exception e) {
+                    log.warn("归还Predictor失败", e);
+                    try {
+                        predictor.close(); // 归还失败才销毁
+                    } catch (Exception ex) {
+                        log.error("关闭Predictor失败", ex);
+                    }
+                }
+            }
+        }
+    }
+
+    @Override
+    public R<float[]> extractImageFeatures(String imagePath) {
+        Image image = null;
+        try {
+            image = SmartImageFactory.getInstance().fromFile(imagePath);
+            return extractImageFeatures(image);
+        } catch (IOException e) {
+            throw new ClipException(e);
+        } finally {
+            ImageUtils.releaseOpenCVMat(image);
+        }
+    }
+
+    @Override
+    public R<float[]> extractTextFeatures(String inputs) {
+        Predictor<String, float[]> predictor = null;
+        try {
+            predictor = textFeaturePredictorPool.borrowObject();
+            return R.ok(predictor.predict(inputs));
+        } catch (Exception e) {
+            throw new ClipException("特征提取错误", e);
+        }finally {
+            if (predictor != null) {
+                try {
+                    textFeaturePredictorPool.returnObject(predictor); //归还
+                } catch (Exception e) {
+                    log.warn("归还Predictor失败", e);
+                    try {
+                        predictor.close(); // 归还失败才销毁
+                    } catch (Exception ex) {
+                        log.error("关闭Predictor失败", ex);
+                    }
+                }
+            }
+        }
+    }
+
+    @Override
+    public R<Float> compareTextAndImage(Image image, String text) {
+        Predictor<Pair<Image, String>, float[]> predictor = null;
+        try {
+            predictor = imgTextPredictorPool.borrowObject();
+            float[] imageFeatures = predictor.predict(new Pair<>(image, text));
+            if (imageFeatures == null || imageFeatures.length == 0){
+                return R.fail(R.Status.Unknown.getCode(), "特征为空");
+            }
+            return R.ok(imageFeatures[0]);
+        } catch (Exception e) {
+            throw new ClipException("特征提取错误", e);
+        }finally {
+            if (predictor != null) {
+                try {
+                    imgTextPredictorPool.returnObject(predictor); //归还
+                } catch (Exception e) {
+                    log.warn("归还Predictor失败", e);
+                    try {
+                        predictor.close(); // 归还失败才销毁
+                    } catch (Exception ex) {
+                        log.error("关闭Predictor失败", ex);
+                    }
+                }
+            }
+        }
+    }
+
+    @Override
+    public R<Float> compareFeatures(float[] feature1, float[] feature2, float scale) {
+        float similarity = SimilarityUtil.calculate(feature1, feature2, SimilarityType.COSINE, false);
+        return R.ok(similarity * scale);
+    }
+
+    @Override
+    public R<Float> compareImage(Image image1, Image image2) {
+        return compareImage(image1, image2, 1.0f);
+    }
+
+    @Override
+    public R<Float> compareImage(Image image1, Image image2, float scale) {
+        R<float[]> features1 = extractImageFeatures(image1);
+        R<float[]> features2 = extractImageFeatures(image2);
+        if(!features1.isSuccess()){
+            return R.fail(features1.getCode(), features1.getMessage());
+        }
+        if(!features2.isSuccess()){
+            return R.fail(features2.getCode(), features2.getMessage());
+        }
+        return compareFeatures(features1.getData(), features2.getData(), scale);
+    }
+
+    @Override
+    public R<Float> compareImage(String imagePath1, String imagePath2) {
+        return compareImage(imagePath1, imagePath2, 1.0f);
+    }
+
+    @Override
+    public R<Float> compareImage(String imagePath1, String imagePath2, float scale) {
+        Image image1 = null;
+        Image image2 = null;
+        try {
+            image1 = SmartImageFactory.getInstance().fromFile(imagePath1);
+            image2 = SmartImageFactory.getInstance().fromFile(imagePath2);
+            return compareImage(image1, image2, scale);
+        } catch (IOException e) {
+            throw new ClipException(e);
+        } finally {
+            ImageUtils.releaseOpenCVMat(image1);
+            ImageUtils.releaseOpenCVMat(image2);
+        }
+    }
+
+    @Override
+    public R<Float> compareText(String input1, String input2) {
+        return compareText(input1, input2, 1.0f);
+    }
+
+    @Override
+    public R<Float> compareText(String input1, String input2, float scale) {
+        R<float[]> features1 = extractTextFeatures(input1);
+        R<float[]> features2 = extractTextFeatures(input2);
+        if(!features1.isSuccess()){
+            return R.fail(features1.getCode(), features1.getMessage());
+        }
+        if(!features2.isSuccess()){
+            return R.fail(features2.getCode(), features2.getMessage());
+        }
+        return compareFeatures(features1.getData(), features2.getData(), scale);
+    }
+
+    @Override
+    public void close() throws Exception {
+        try {
+            if (imageFeaturePredictorPool != null) {
+                imageFeaturePredictorPool.close();
+            }
+        } catch (Exception e) {
+            log.warn("关闭 predictorPool 失败", e);
+        }
+        try {
+            if (textFeaturePredictorPool != null) {
+                textFeaturePredictorPool.close();
+            }
+        } catch (Exception e) {
+            log.warn("关闭 predictorPool 失败", e);
+        }
+        try {
+            if (imgTextPredictorPool != null) {
+                imgTextPredictorPool.close();
+            }
+        } catch (Exception e) {
+            log.warn("关闭 predictorPool 失败", e);
+        }
+        try {
+            if (model != null) {
+                model.close();
+            }
+        } catch (Exception e) {
+            log.warn("关闭 model 失败", e);
+        }
+        try {
+            if (tokenizer != null) {
+                tokenizer.close();
+            }
+        } catch (Exception e) {
+            log.warn("关闭 tokenizer 失败", e);
+        }
+    }
+
+    private boolean fromFactory = false;
+    public boolean isFromFactory() {
+        return fromFactory;
+    }
+
+    @Override
+    public void setFromFactory(boolean fromFactory) {
+        this.fromFactory = fromFactory;
+    }
+}

vision/src/main/java/cn/smartjavaai/clip/pool/ClipImagePredictorFactory.java

@@ -0,0 +1,36 @@
+package cn.smartjavaai.clip.pool;
+
+import ai.djl.Model;
+import ai.djl.inference.Predictor;
+import ai.djl.modality.cv.Image;
+import cn.smartjavaai.clip.translator.ImageTranslator;
+import org.apache.commons.pool2.BasePooledObjectFactory;
+import org.apache.commons.pool2.PooledObject;
+import org.apache.commons.pool2.impl.DefaultPooledObject;
+
+/**
+ * @author dwj
+ * @date 2025/10/20
+ */
+public class ClipImagePredictorFactory extends BasePooledObjectFactory<Predictor<Image, float[]>> {
+    private final Model model;
+
+    public ClipImagePredictorFactory(Model model) {
+        this.model = model;
+    }
+
+    @Override
+    public Predictor<Image, float[]> create() {
+        return model.newPredictor(new ImageTranslator());
+    }
+
+    @Override
+    public PooledObject<Predictor<Image, float[]>> wrap(Predictor<Image, float[]> predictor) {
+        return new DefaultPooledObject<>(predictor);
+    }
+
+    @Override
+    public void destroyObject(PooledObject<Predictor<Image, float[]>> p) {
+        p.getObject().close();
+    }
+}

vision/src/main/java/cn/smartjavaai/clip/pool/ClipImageTextPredictorFactory.java

@@ -0,0 +1,42 @@
+package cn.smartjavaai.clip.pool;
+
+import ai.djl.Model;
+import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer;
+import ai.djl.inference.Predictor;
+import ai.djl.modality.cv.Image;
+import ai.djl.util.Pair;
+import cn.smartjavaai.clip.translator.ImageTextTranslator;
+import org.apache.commons.pool2.BasePooledObjectFactory;
+import org.apache.commons.pool2.PooledObject;
+import org.apache.commons.pool2.impl.DefaultPooledObject;
+
+/**
+ * @author dwj
+ */
+public class ClipImageTextPredictorFactory extends BasePooledObjectFactory<Predictor<Pair<Image, String>, float[]>> {
+
+    private final Model model;
+
+    private final HuggingFaceTokenizer tokenizer;
+
+    public ClipImageTextPredictorFactory(Model model, HuggingFaceTokenizer tokenizer) {
+        this.model = model;
+        this.tokenizer = tokenizer;
+    }
+
+    @Override
+    public Predictor<Pair<Image, String>, float[]> create() {
+        return model.newPredictor(new ImageTextTranslator(tokenizer));
+    }
+
+    @Override
+    public PooledObject<Predictor<Pair<Image, String>, float[]>> wrap(Predictor<Pair<Image, String>, float[]> predictor) {
+        return new DefaultPooledObject<>(predictor);
+    }
+
+    @Override
+    public void destroyObject(PooledObject<Predictor<Pair<Image, String>, float[]>> p) {
+        p.getObject().close();
+    }
+
+}

vision/src/main/java/cn/smartjavaai/clip/pool/ClipTextPredictorFactory.java

@@ -0,0 +1,40 @@
+package cn.smartjavaai.clip.pool;
+
+import ai.djl.Model;
+import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer;
+import ai.djl.inference.Predictor;
+import ai.djl.modality.cv.Image;
+import ai.djl.modality.nlp.preprocess.Tokenizer;
+import cn.smartjavaai.clip.translator.ImageTranslator;
+import cn.smartjavaai.clip.translator.TextTranslator;
+import org.apache.commons.pool2.BasePooledObjectFactory;
+import org.apache.commons.pool2.PooledObject;
+import org.apache.commons.pool2.impl.DefaultPooledObject;
+
+/**
+ * @author dwj
+ */
+public class ClipTextPredictorFactory extends BasePooledObjectFactory<Predictor<String, float[]>> {
+    private final Model model;
+    private final HuggingFaceTokenizer tokenizer;
+
+    public ClipTextPredictorFactory(Model model, HuggingFaceTokenizer tokenizer) {
+        this.model = model;
+        this.tokenizer = tokenizer;
+    }
+
+    @Override
+    public Predictor<String, float[]> create() {
+        return model.newPredictor(new TextTranslator(tokenizer));
+    }
+
+    @Override
+    public PooledObject<Predictor<String, float[]>> wrap(Predictor<String, float[]> predictor) {
+        return new DefaultPooledObject<>(predictor);
+    }
+
+    @Override
+    public void destroyObject(PooledObject<Predictor<String, float[]>> p) {
+        p.getObject().close();
+    }
+}

vision/src/main/java/cn/smartjavaai/clip/translator/ImageTextTranslator.java

@@ -0,0 +1,49 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package cn.smartjavaai.clip.translator;
+
+import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer;
+import ai.djl.modality.cv.Image;
+import ai.djl.ndarray.NDArray;
+import ai.djl.ndarray.NDList;
+import ai.djl.translate.NoBatchifyTranslator;
+import ai.djl.translate.TranslatorContext;
+import ai.djl.util.Pair;
+
+public class ImageTextTranslator implements NoBatchifyTranslator<Pair<Image, String>, float[]> {
+
+    private ImageTranslator imgTranslator;
+    private TextTranslator txtTranslator;
+
+    HuggingFaceTokenizer tokenizer;
+
+    public ImageTextTranslator(HuggingFaceTokenizer tokenizer) {
+        this.imgTranslator = new ImageTranslator();
+        this.txtTranslator = new TextTranslator(tokenizer);
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public float[] processOutput(TranslatorContext ctx, NDList list) throws Exception {
+        NDArray logitsPerImage = list.get(0);
+        return logitsPerImage.toFloatArray();
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public NDList processInput(TranslatorContext ctx, Pair<Image, String> input) throws Exception {
+        NDList imageInput = imgTranslator.processInput(ctx, input.getKey());
+        NDList textInput = txtTranslator.processInput(ctx, input.getValue());
+        return new NDList(textInput.get(0), imageInput.get(0), textInput.get(1));
+    }
+}

vision/src/main/java/cn/smartjavaai/clip/translator/ImageTranslator.java

@@ -0,0 +1,49 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package cn.smartjavaai.clip.translator;
+
+import ai.djl.modality.cv.Image;
+import ai.djl.modality.cv.util.NDImageUtils;
+import ai.djl.ndarray.NDArray;
+import ai.djl.ndarray.NDList;
+import ai.djl.translate.NoBatchifyTranslator;
+import ai.djl.translate.TranslatorContext;
+
+public class ImageTranslator implements NoBatchifyTranslator<Image, float[]> {
+
+    /** {@inheritDoc} */
+    @Override
+    public float[] processOutput(TranslatorContext ctx, NDList list) {
+        NDArray array = list.singletonOrThrow();
+        return array.toFloatArray();
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public NDList processInput(TranslatorContext ctx, Image input) {
+        NDArray array = input.toNDArray(ctx.getNDManager(), Image.Flag.COLOR);
+
+        float percent = 224f / Math.min(input.getWidth(), input.getHeight());
+        int resizedWidth = Math.round(input.getWidth() * percent);
+        int resizedHeight = Math.round(input.getHeight() * percent);
+
+        array =
+                NDImageUtils.resize(
+                        array, resizedWidth, resizedHeight, Image.Interpolation.BICUBIC);
+        array = NDImageUtils.centerCrop(array, 224, 224);
+        array = NDImageUtils.toTensor(array);
+        NDArray placeholder = ctx.getNDManager().create("");
+        placeholder.setName("module_method:get_image_features");
+        return new NDList(array.expandDims(0), placeholder);
+    }
+}

vision/src/main/java/cn/smartjavaai/clip/translator/TextTranslator.java

@@ -0,0 +1,49 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package cn.smartjavaai.clip.translator;
+
+import ai.djl.huggingface.tokenizers.Encoding;
+import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer;
+import ai.djl.ndarray.NDArray;
+import ai.djl.ndarray.NDList;
+import ai.djl.translate.NoBatchifyTranslator;
+import ai.djl.translate.TranslatorContext;
+
+import java.io.IOException;
+import java.nio.file.Paths;
+
+public class TextTranslator implements NoBatchifyTranslator<String, float[]> {
+
+    HuggingFaceTokenizer tokenizer;
+
+    public TextTranslator(HuggingFaceTokenizer tokenizer) {
+        this.tokenizer = tokenizer;
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public float[] processOutput(TranslatorContext ctx, NDList list) {
+        return list.singletonOrThrow().toFloatArray();
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public NDList processInput(TranslatorContext ctx, String input) {
+        Encoding encoding = tokenizer.encode(input);
+        NDArray attention = ctx.getNDManager().create(encoding.getAttentionMask());
+        NDArray inputIds = ctx.getNDManager().create(encoding.getIds());
+        NDArray placeholder = ctx.getNDManager().create("");
+        placeholder.setName("module_method:get_text_features");
+        return new NDList(inputIds.expandDims(0), attention.expandDims(0), placeholder);
+    }
+}

vision/src/main/java/cn/smartjavaai/cls/config/ClsModelConfig.java

@@ -0,0 +1,62 @@
+package cn.smartjavaai.cls.config;
+
+import cn.smartjavaai.action.enums.ActionRecModelEnum;
+import cn.smartjavaai.cls.enums.ClsModelEnum;
+import cn.smartjavaai.common.config.ModelConfig;
+import cn.smartjavaai.common.enums.DeviceEnum;
+import lombok.Data;
+
+import java.util.List;
+
+/**
+ * 分类模型参数配置
+ *
+ * @author dwj
+ */
+@Data
+public class ClsModelConfig extends ModelConfig {
+
+    /**
+     * 模型
+     */
+    private ClsModelEnum modelEnum;
+
+
+
+    /**
+     * 模型路径
+     */
+    private String modelPath;
+
+
+    /**
+     * 允许的分类列表
+     */
+    private List<String> allowedClasses;
+
+    /**
+     * 置信度阈值
+     */
+    private float threshold = 0.3f;
+
+    /**
+     * 检测结果数量
+     */
+    private int topK;
+
+
+
+    public ClsModelConfig() {
+    }
+
+    public ClsModelConfig(ClsModelEnum modelEnum, DeviceEnum device) {
+        this.modelEnum = modelEnum;
+        setDevice(device);
+    }
+
+    public ClsModelConfig(ClsModelEnum modelEnum) {
+        this.modelEnum = modelEnum;
+    }
+
+
+}

vision/src/main/java/cn/smartjavaai/cls/criteria/ClsCriteriaFactory.java

@@ -0,0 +1,73 @@
+package cn.smartjavaai.cls.criteria;
+
+import ai.djl.Device;
+import ai.djl.modality.Classifications;
+import ai.djl.modality.cv.Image;
+import ai.djl.repository.zoo.Criteria;
+import ai.djl.training.util.ProgressBar;
+import ai.djl.translate.Translator;
+import cn.smartjavaai.action.config.ActionRecModelConfig;
+import cn.smartjavaai.action.enums.ActionRecModelEnum;
+import cn.smartjavaai.action.exception.ActionException;
+import cn.smartjavaai.action.model.CommonActionTranslator;
+import cn.smartjavaai.cls.config.ClsModelConfig;
+import cn.smartjavaai.cls.enums.ClsModelEnum;
+import cn.smartjavaai.cls.translator.YoloClsTranslator;
+import cn.smartjavaai.common.enums.DeviceEnum;
+import cn.smartjavaai.common.utils.DJLCommonUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import java.nio.file.Paths;
+import java.util.Objects;
+
+/**
+ * 分类模型Criteria工厂
+ * @author dwj
+ */
+public class ClsCriteriaFactory {
+
+
+    /**
+     * 创建动作识别Criteria
+     * @param config
+     * @return
+     */
+    public static Criteria<Image, Classifications> createCriteria(ClsModelConfig config) {
+        Device device = null;
+        if(!Objects.isNull(config.getDevice())){
+            device = config.getDevice() == DeviceEnum.CPU ? Device.cpu() : Device.gpu(config.getGpuId());
+        }
+        Translator<Image, Classifications> translator = getTranslator(config);
+        //检查模型路径
+        if (StringUtils.isBlank(config.getModelPath())){
+            throw new ActionException("请指定模型路径");
+        }
+        boolean isUrl = DJLCommonUtils.hasSupportedProtocol(config.getModelPath());
+        Criteria<Image, Classifications> criteria =
+                Criteria.builder()
+                        .setTypes(Image.class, Classifications.class)
+                        .optModelUrls(isUrl ? config.getModelPath() : null)
+                        .optModelPath(isUrl ? null : Paths.get(config.getModelPath()))
+                        .optTranslator(translator)
+                        .optDevice(device)
+                        .optProgress(new ProgressBar())
+                        .optEngine(config.getModelEnum().getEngine())
+                        .build();
+        return criteria;
+    }
+
+    /**
+     * 获取分类模型Translator
+     * @param config
+     * @return
+     */
+    public static Translator<Image, Classifications> getTranslator(ClsModelConfig config) {
+        Translator<Image, Classifications> translator = null;
+        if(config.getModelEnum() == ClsModelEnum.YOLOV11
+                || config.getModelEnum() == ClsModelEnum.YOLOV8){
+        YoloClsTranslator.Builder builder = YoloClsTranslator.builder().optSynsetArtifactName("synset.txt");
+            translator = builder.build();
+        }
+        return translator;
+    }
+}

vision/src/main/java/cn/smartjavaai/cls/enums/ClsModelEnum.java

@@ -0,0 +1,58 @@
+package cn.smartjavaai.cls.enums;
+
+/**
+ * 分类模型枚举
+ * @author dwj
+ */
+public enum ClsModelEnum {
+
+    YOLOV8("OnnxRuntime",224,224),
+    YOLOV11("OnnxRuntime",224,224);
+
+    /**
+     * 模型输入尺寸：宽
+     */
+    private final int inputWidth;
+
+    /**
+     * 模型输入尺寸：高
+     */
+    private final int inputHeight;
+
+    /**
+     * 模型引擎
+     */
+    private final String engine;
+
+    /**
+     * 根据名称获取枚举 (忽略大小写和下划线变体)
+     */
+    public static ClsModelEnum fromName(String name) {
+        String formatted = name.trim().toUpperCase().replaceAll("[-_]", "");
+        for (ClsModelEnum model : values()) {
+            if (model.name().replaceAll("_", "").equals(formatted)) {
+                return model;
+            }
+        }
+        throw new IllegalArgumentException("未知模型名称: " + name);
+    }
+
+
+    ClsModelEnum(String engine, int inputWidth, int inputHeight) {
+        this.inputWidth = inputWidth;
+        this.inputHeight = inputHeight;
+        this.engine = engine;
+    }
+
+    public int getInputWidth() {
+        return inputWidth;
+    }
+
+    public int getInputHeight() {
+        return inputHeight;
+    }
+
+    public String getEngine() {
+        return engine;
+    }
+}

vision/src/main/java/cn/smartjavaai/cls/exception/ClsException.java

@@ -0,0 +1,29 @@
+package cn.smartjavaai.cls.exception;
+
+/**
+ * 分类模型异常
+ * @author dwj
+ */
+public class ClsException extends RuntimeException{
+
+    public ClsException() {
+        super();
+    }
+
+    public ClsException(String message, Throwable cause, boolean enableSuppression, boolean writableStackTrace) {
+        super(message, cause, enableSuppression, writableStackTrace);
+    }
+
+    public ClsException(String message, Throwable cause) {
+        super(message, cause);
+    }
+
+    public ClsException(String message) {
+        super(message);
+    }
+
+    public ClsException(Throwable cause) {
+        super(cause);
+    }
+
+}

vision/src/main/java/cn/smartjavaai/cls/model/ClsModel.java

@@ -0,0 +1,45 @@
+package cn.smartjavaai.cls.model;
+
+import ai.djl.modality.Classifications;
+import ai.djl.modality.cv.Image;
+import cn.smartjavaai.action.config.ActionRecModelConfig;
+import cn.smartjavaai.cls.config.ClsModelConfig;
+import cn.smartjavaai.common.entity.R;
+
+/**
+ * 图像分类模型
+ * @author dwj
+ */
+public interface ClsModel extends AutoCloseable{
+
+
+    /**
+     * 加载模型
+     * @param config
+     */
+    void loadModel(ClsModelConfig config);
+
+    /**
+     * 分类
+     * @param image
+     * @return
+     */
+    default R<Classifications> detect(Image image){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+    /**
+     * 分类
+     * @param imagePath
+     * @return
+     */
+    default R<Classifications> detect(String imagePath){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+
+    default void setFromFactory(boolean fromFactory){
+        throw new UnsupportedOperationException("默认不支持该功能");
+    }
+
+}

vision/src/main/java/cn/smartjavaai/cls/model/ClsModelFactory.java

@@ -0,0 +1,124 @@
+package cn.smartjavaai.cls.model;
+
+import cn.smartjavaai.action.model.CommonActionRecModel;
+import cn.smartjavaai.cls.config.ClsModelConfig;
+import cn.smartjavaai.cls.enums.ClsModelEnum;
+import cn.smartjavaai.common.config.Config;
+import cn.smartjavaai.objectdetection.exception.DetectionException;
+import lombok.extern.slf4j.Slf4j;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * 图像分类 模型工厂
+ * @author dwj
+ */
+@Slf4j
+public class ClsModelFactory {
+
+    // 使用 volatile 和双重检查锁定来确保线程安全的单例模式
+    private static volatile ClsModelFactory instance;
+
+    private static final ConcurrentHashMap<ClsModelEnum, ClsModel> modelMap = new ConcurrentHashMap<>();
+
+    /**
+     * 模型注册表
+     */
+    private static final Map<ClsModelEnum, Class<? extends ClsModel>> registry =
+            new ConcurrentHashMap<>();
+
+
+    // 私有构造函数，防止外部创建实例
+    private ClsModelFactory() {}
+
+    // 双重检查锁定的单例方法
+    public static ClsModelFactory getInstance() {
+        if (instance == null) {
+            synchronized (ClsModelFactory.class) {
+                if (instance == null) {
+                    instance = new ClsModelFactory();
+                }
+            }
+        }
+        return instance;
+    }
+
+    /**
+     * 获取模型（通过配置）
+     * @param config
+     * @return
+     */
+    public ClsModel getModel(ClsModelConfig config) {
+        if(Objects.isNull(config) || Objects.isNull(config.getModelEnum())){
+            throw new DetectionException("未配置模型");
+        }
+        return modelMap.computeIfAbsent(config.getModelEnum(), k -> {
+            return createFaceDetModel(config);
+        });
+    }
+
+    /**
+     * 使用ModelConfig创建模型
+     * @param config
+     * @return
+     */
+    private ClsModel createFaceDetModel(ClsModelConfig config) {
+        Class<?> clazz = registry.get(config.getModelEnum());
+        if(clazz == null){
+            throw new DetectionException("Unsupported model");
+        }
+        ClsModel model = null;
+        try {
+            model = (ClsModel) clazz.newInstance();
+        } catch (InstantiationException | IllegalAccessException e) {
+            throw new DetectionException(e);
+        }
+        model.loadModel(config);
+        model.setFromFactory(true);
+        return model;
+    }
+
+
+    /**
+     * 注册模型
+     * @param modelEnum
+     * @param clazz
+     */
+    private static void registerAlgorithm(ClsModelEnum modelEnum, Class<? extends ClsModel> clazz) {
+        registry.put(modelEnum, clazz);
+    }
+
+
+
+    // 初始化默认算法
+    static {
+        registerAlgorithm(ClsModelEnum.YOLOV8, CommonClsModel.class);
+        registerAlgorithm(ClsModelEnum.YOLOV11, CommonClsModel.class);
+        log.debug("缓存目录：{}", Config.getCachePath());
+    }
+
+    /**
+     * 关闭所有已加载的模型
+     */
+    public void closeAll() {
+        modelMap.values().forEach(model -> {
+            try {
+                model.close();
+            } catch (Exception e) {
+                e.printStackTrace();
+            }
+        });
+        modelMap.clear();
+    }
+
+    /**
+     * 移除缓存的模型
+     * @param modelEnum
+     */
+    public static void removeFromCache(ClsModelEnum modelEnum) {
+        modelMap.remove(modelEnum);
+    }
+}
+

vision/src/main/java/cn/smartjavaai/cls/model/CommonClsModel.java

@@ -0,0 +1,160 @@
+package cn.smartjavaai.cls.model;
+
+import ai.djl.MalformedModelException;
+import ai.djl.engine.Engine;
+import ai.djl.inference.Predictor;
+import ai.djl.modality.Classifications;
+import ai.djl.modality.cv.Image;
+import ai.djl.modality.cv.output.DetectedObjects;
+import ai.djl.repository.zoo.Criteria;
+import ai.djl.repository.zoo.ModelNotFoundException;
+import ai.djl.repository.zoo.ZooModel;
+import cn.smartjavaai.action.criteria.ActionRecCriteriaFactory;
+import cn.smartjavaai.action.model.ActionRecModel;
+import cn.smartjavaai.action.model.ActionRecModelFactory;
+import cn.smartjavaai.cls.config.ClsModelConfig;
+import cn.smartjavaai.cls.criteria.ClsCriteriaFactory;
+import cn.smartjavaai.cls.exception.ClsException;
+import cn.smartjavaai.common.cv.SmartImageFactory;
+import cn.smartjavaai.common.entity.DetectionResponse;
+import cn.smartjavaai.common.entity.R;
+import cn.smartjavaai.common.pool.PredictorFactory;
+import cn.smartjavaai.common.utils.FileUtils;
+import cn.smartjavaai.common.utils.ImageUtils;
+import cn.smartjavaai.common.utils.OpenCVUtils;
+import cn.smartjavaai.objectdetection.exception.DetectionException;
+import cn.smartjavaai.vision.utils.ClassificationFilter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.pool2.impl.GenericObjectPool;
+import org.opencv.core.Mat;
+
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.Objects;
+
+/**
+ * 通用图像分类模型
+ * @author dwj
+ */
+@Slf4j
+public class CommonClsModel implements ClsModel {
+
+
+    private ClsModelConfig config;
+
+    private ZooModel<Image, Classifications> model;
+
+    private GenericObjectPool<Predictor<Image, Classifications>> predictorPool;
+
+    @Override
+    public void loadModel(ClsModelConfig config) {
+        if(Objects.isNull(config.getModelEnum())){
+            throw new DetectionException("未配置模型枚举");
+        }
+        Criteria<Image, Classifications> criteria = ClsCriteriaFactory.createCriteria(config);
+        this.config = config;
+        try {
+            model = criteria.loadModel();
+            // 创建池子：每个线程独享 Predictor
+            this.predictorPool = new GenericObjectPool<>(new PredictorFactory<>(model));
+            int predictorPoolSize = config.getPredictorPoolSize();
+            if(config.getPredictorPoolSize() <= 0){
+                predictorPoolSize = Runtime.getRuntime().availableProcessors(); // 默认等于CPU核心数
+            }
+            predictorPool.setMaxTotal(predictorPoolSize);
+            log.debug("当前设备: " + model.getNDManager().getDevice());
+            log.debug("当前引擎: " + Engine.getInstance().getEngineName());
+            log.debug("模型推理器线程池最大数量: " + predictorPoolSize);
+        } catch (IOException | ModelNotFoundException | MalformedModelException e) {
+            throw new DetectionException("模型加载失败", e);
+        }
+    }
+
+
+    @Override
+    public R<Classifications> detect(Image image) {
+        Classifications classifications = detectCore(image);
+        // 过滤
+        if(Objects.nonNull(classifications) && !classifications.items().isEmpty()){
+            classifications = new ClassificationFilter(config.getAllowedClasses(), config.getThreshold()).filter(classifications);
+        }
+        return R.ok(classifications);
+    }
+
+
+    @Override
+    public R<Classifications> detect(String imagePath) {
+        if(!FileUtils.isFileExists(imagePath)){
+            return R.fail(R.Status.FILE_NOT_FOUND);
+        }
+        Image img = null;
+        try {
+            img = SmartImageFactory.getInstance().fromFile(Paths.get(imagePath));
+            return detect(img);
+        } catch (IOException e) {
+            throw new ClsException("无效的图片", e);
+        } finally {
+            ImageUtils.releaseOpenCVMat(img);
+        }
+    }
+
+    /**
+     * 模型核心推理方法
+     * @param image
+     * @return
+     */
+    public Classifications detectCore(Image image) {
+        Predictor<Image, Classifications> predictor = null;
+        try {
+            predictor = predictorPool.borrowObject();
+            return predictor.predict(image);
+        } catch (Exception e) {
+            throw new DetectionException("动作识别错误", e);
+        }finally {
+            if (predictor != null) {
+                try {
+                    predictorPool.returnObject(predictor); //归还
+                    log.debug("释放资源");
+                } catch (Exception e) {
+                    log.warn("归还Predictor失败", e);
+                    try {
+                        predictor.close(); // 归还失败才销毁
+                    } catch (Exception ex) {
+                        log.error("关闭Predictor失败", ex);
+                    }
+                }
+            }
+        }
+    }
+
+    @Override
+    public void close() throws Exception {
+        if (fromFactory) {
+//            ActionRecModelFactory.removeFromCache(config.getModelEnum());
+        }
+        try {
+            if (predictorPool != null) {
+                predictorPool.close();
+            }
+        } catch (Exception e) {
+            log.warn("关闭 predictorPool 失败", e);
+        }
+        try {
+            if (model != null) {
+                model.close();
+            }
+        } catch (Exception e) {
+            log.warn("关闭 model 失败", e);
+        }
+    }
+
+    private boolean fromFactory = false;
+
+    @Override
+    public void setFromFactory(boolean fromFactory) {
+        this.fromFactory = fromFactory;
+    }
+    public boolean isFromFactory() {
+        return fromFactory;
+    }
+}

vision/src/main/java/cn/smartjavaai/cls/translator/YoloClsTranslator.java

@@ -0,0 +1,386 @@
+package cn.smartjavaai.cls.translator;
+
+import ai.djl.Model;
+import ai.djl.modality.Classifications;
+import ai.djl.modality.cv.Image;
+import ai.djl.modality.cv.transform.*;
+import ai.djl.modality.cv.util.NDImageUtils;
+import ai.djl.ndarray.NDArray;
+import ai.djl.ndarray.NDList;
+import ai.djl.ndarray.NDManager;
+import ai.djl.ndarray.types.DataType;
+import ai.djl.translate.*;
+import ai.djl.util.Utils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+
+public class YoloClsTranslator implements Translator<Image, Classifications> {
+
+
+    protected float threshold;
+    protected List<String> classes;
+    protected boolean applyRatio;
+    protected Pipeline pipeline;
+    private Image.Flag flag;
+    private Batchifier batchifier;
+    protected int width;
+    protected int height;
+    protected int topk;
+
+    private SynsetLoader synsetLoader;
+
+    @Override
+    public void prepare(TranslatorContext ctx) throws IOException {
+        if (this.classes == null) {
+            this.classes = this.synsetLoader.load(ctx.getModel());
+        }
+    }
+
+
+    /**
+     * Constructs an ImageTranslator with the provided builder.
+     *
+     * @param builder the data to build with
+     */
+    protected YoloClsTranslator(Builder builder) {
+        this.threshold = builder.threshold;
+        this.synsetLoader = builder.synsetLoader;
+        this.applyRatio = builder.applyRatio;
+        this.flag = builder.flag;
+        this.pipeline = builder.pipeline;
+        this.batchifier = builder.batchifier;
+        this.width = builder.width;
+        this.height = builder.height;
+    }
+
+    /**
+     * Creates a builder to build a {@code YoloV8Translator} with specified arguments.
+     *
+     * @return a new builder
+     */
+    public static Builder builder() {
+        return new Builder();
+    }
+
+    /**
+     * Creates a builder to build a {@code YoloV8Translator} with specified arguments.
+     *
+     * @param arguments arguments to specify builder options
+     * @return a new builder
+     */
+    public static Builder builder(Map<String, ?> arguments) {
+        Builder builder = new Builder();
+        builder.configPreProcess(arguments);
+        builder.configPostProcess(arguments);
+        return builder;
+    }
+
+    @Override
+    public NDList processInput(TranslatorContext ctx, Image input) throws Exception {
+        NDManager manager = ctx.getNDManager();
+        NDArray array = input.toNDArray(manager, Image.Flag.COLOR);
+        //中心裁剪
+        array = NDImageUtils.centerCrop(array);
+        array = NDImageUtils.resize(array, width, height);
+        // 转为 float32 且归一化到 0~1
+        array = array.toType(DataType.FLOAT32, false).div(255f); // HWC
+        // HWC -> CHW
+        array = array.transpose(2, 0, 1); // CHW
+        return new NDList(array);
+    }
+
+    @Override
+    public Classifications processOutput(TranslatorContext ctx, NDList list) throws Exception {
+        NDArray probabilitiesNd = list.singletonOrThrow();
+//        probabilitiesNd = probabilitiesNd.softmax(0);
+        return new Classifications(classes, probabilitiesNd, 5);
+    }
+
+
+
+    public static class Builder {
+
+
+        protected float threshold = 0.2F;
+        protected boolean applyRatio;
+        protected boolean removePadding;
+
+        protected int width = 224;
+        protected int height = 224;
+        protected Image.Flag flag;
+        protected Pipeline pipeline;
+        protected Batchifier batchifier;
+        protected int topk = 5;
+
+        protected SynsetLoader synsetLoader;
+
+        public Builder() {
+        }
+
+        /**
+         * Builds the translator.
+         *
+         * @return the new translator
+         */
+        public YoloClsTranslator build() {
+            if (pipeline == null) {
+                addTransform(
+                        array -> array.transpose(2, 0, 1).toType(DataType.FLOAT32, false).div(255));
+            }
+//            validate();
+            return new YoloClsTranslator(this);
+        }
+
+        protected Builder self() {
+            return this;
+        }
+
+        public Builder addTransform(Transform transform) {
+            if (this.pipeline == null) {
+                this.pipeline = new Pipeline();
+            }
+
+            this.pipeline.add(transform);
+            return this.self();
+        }
+
+        public Builder optApplyRatio(boolean value) {
+            this.applyRatio = value;
+            return this.self();
+        }
+
+        public Builder optFlag(Image.Flag flag) {
+            this.flag = flag;
+            return this.self();
+        }
+
+        public Builder setPipeline(Pipeline pipeline) {
+            this.pipeline = pipeline;
+            return this.self();
+        }
+
+        public Builder setImageSize(int width, int height) {
+            this.width = width;
+            this.height = height;
+            return this.self();
+        }
+
+
+        public Builder optBatchifier(Batchifier batchifier) {
+            this.batchifier = batchifier;
+            return this.self();
+        }
+
+        public Builder optThreshold(float threshold) {
+            this.threshold = threshold;
+            return this.self();
+        }
+
+        public Builder optTopk(int topk) {
+            this.topk = topk;
+            return this.self();
+        }
+
+        /**
+         * Sets the name of the synset file listing the potential classes for an image.
+         *
+         * @param synsetArtifactName a file listing the potential classes for an image
+         * @return the builder
+         */
+        public Builder optSynsetArtifactName(String synsetArtifactName) {
+            synsetLoader = new SynsetLoader(synsetArtifactName);
+            return self();
+        }
+
+        /**
+         * Sets the URL of the synset file.
+         *
+         * @param synsetUrl the URL of the synset file
+         * @return the builder
+         */
+        public Builder optSynsetUrl(String synsetUrl) {
+            try {
+                this.synsetLoader = new SynsetLoader(new URL(synsetUrl));
+            } catch (MalformedURLException e) {
+                throw new IllegalArgumentException("Invalid synsetUrl: " + synsetUrl, e);
+            }
+            return self();
+        }
+
+        /**
+         * Sets the potential classes for an image.
+         *
+         * @param synset the potential classes for an image
+         * @return the builder
+         */
+        public Builder optSynset(List<String> synset) {
+            synsetLoader = new SynsetLoader(synset);
+            return self();
+        }
+
+        /** {@inheritDoc} */
+        protected void configPostProcess(Map<String, ?> arguments) {
+            if (ArgumentsUtil.booleanValue(arguments, "optApplyRatio") || ArgumentsUtil.booleanValue(arguments, "applyRatio")) {
+                this.optApplyRatio(true);
+            }
+            this.threshold = ArgumentsUtil.floatValue(arguments, "threshold", 0.2F);
+            String centerFit = ArgumentsUtil.stringValue(arguments, "centerFit", "false");
+            this.removePadding = "true".equals(centerFit);
+            String type = ArgumentsUtil.stringValue(arguments, "outputType", "AUTO");
+        }
+
+        protected void configPreProcess(Map<String, ?> arguments) {
+            if (this.pipeline == null) {
+                this.pipeline = new Pipeline();
+            }
+
+            this.width = ArgumentsUtil.intValue(arguments, "width", 224);
+            this.height = ArgumentsUtil.intValue(arguments, "height", 224);
+            if (arguments.containsKey("flag")) {
+                this.flag = Image.Flag.valueOf(arguments.get("flag").toString());
+            }
+
+            String pad = ArgumentsUtil.stringValue(arguments, "pad", "false");
+            if ("true".equals(pad)) {
+                this.addTransform(new Pad(0.0));
+            } else if (!"false".equals(pad)) {
+                double padding = Double.parseDouble(pad);
+                this.addTransform(new Pad(padding));
+            }
+
+            String resize = ArgumentsUtil.stringValue(arguments, "resize", "false");
+            int w;
+            int shortEdge;
+            if ("true".equals(resize)) {
+                this.addTransform(new Resize(this.width, this.height));
+            } else if (!"false".equals(resize)) {
+                String[] tokens = resize.split("\\s*,\\s*");
+                w = (int)Double.parseDouble(tokens[0]);
+                if (tokens.length > 1) {
+                    shortEdge = (int)Double.parseDouble(tokens[1]);
+                } else {
+                    shortEdge = w;
+                }
+
+                Image.Interpolation interpolation;
+                if (tokens.length > 2) {
+                    interpolation = Image.Interpolation.valueOf(tokens[2]);
+                } else {
+                    interpolation = Image.Interpolation.BILINEAR;
+                }
+
+                this.addTransform(new Resize(w, shortEdge, interpolation));
+            }
+
+            String resizeShort = ArgumentsUtil.stringValue(arguments, "resizeShort", "false");
+            if ("true".equals(resizeShort)) {
+                w = Math.max(this.width, this.height);
+                this.addTransform(new ResizeShort(w));
+            } else if (!"false".equals(resizeShort)) {
+                String[] tokens = resizeShort.split("\\s*,\\s*");
+                shortEdge = (int)Double.parseDouble(tokens[0]);
+                int longEdge;
+                if (tokens.length > 1) {
+                    longEdge = (int)Double.parseDouble(tokens[1]);
+                } else {
+                    longEdge = -1;
+                }
+
+                Image.Interpolation interpolation;
+                if (tokens.length > 2) {
+                    interpolation = Image.Interpolation.valueOf(tokens[2]);
+                } else {
+                    interpolation = Image.Interpolation.BILINEAR;
+                }
+
+                this.addTransform(new ResizeShort(shortEdge, longEdge, interpolation));
+            }
+
+            if (ArgumentsUtil.booleanValue(arguments, "centerCrop", false)) {
+                this.addTransform(new CenterCrop(this.width, this.height));
+            }
+
+            if (ArgumentsUtil.booleanValue(arguments, "centerFit")) {
+                this.addTransform(new CenterFit(this.width, this.height));
+            }
+
+            if (ArgumentsUtil.booleanValue(arguments, "toTensor", true)) {
+                this.addTransform(new ToTensor());
+            }
+
+            String normalize = ArgumentsUtil.stringValue(arguments, "normalize", "false");
+            if ("true".equals(normalize)) {
+                float[] MEAN = new float[]{0.485F, 0.456F, 0.406F};
+                float[] STD = new float[]{0.229F, 0.224F, 0.225F};
+                this.addTransform(new Normalize(MEAN, STD));
+            } else if (!"false".equals(normalize)) {
+                String[] tokens = normalize.split("\\s*,\\s*");
+                if (tokens.length != 6) {
+                    throw new IllegalArgumentException("Invalid normalize value: " + normalize);
+                }
+
+                float[] mean = new float[]{Float.parseFloat(tokens[0]), Float.parseFloat(tokens[1]), Float.parseFloat(tokens[2])};
+                float[] std = new float[]{Float.parseFloat(tokens[3]), Float.parseFloat(tokens[4]), Float.parseFloat(tokens[5])};
+                this.addTransform(new Normalize(mean, std));
+            }
+
+            String range = (String)arguments.get("range");
+            if ("0,1".equals(range)) {
+                this.addTransform((a) -> {
+                    return a.div(255.0F);
+                });
+            } else if ("-1,1".equals(range)) {
+                this.addTransform((a) -> {
+                    return a.div(128.0F).sub(1);
+                });
+            }
+
+            if (arguments.containsKey("batchifier")) {
+                this.batchifier = Batchifier.fromString((String)arguments.get("batchifier"));
+            }
+
+        }
+    }
+
+
+
+    protected static final class SynsetLoader {
+
+        private String synsetFileName;
+        private URL synsetUrl;
+        private List<String> synset;
+
+        public SynsetLoader(List<String> synset) {
+            this.synset = synset;
+        }
+
+        public SynsetLoader(URL synsetUrl) {
+            this.synsetUrl = synsetUrl;
+        }
+
+        public SynsetLoader(String synsetFileName) {
+            this.synsetFileName = synsetFileName;
+        }
+
+        public List<String> load(Model model) throws IOException {
+            if (synset != null) {
+                return synset;
+            } else if (synsetUrl != null) {
+                try (InputStream is = synsetUrl.openStream()) {
+                    return Utils.readLines(is);
+                }
+            }
+            return model.getArtifact(synsetFileName, Utils::readLines);
+        }
+    }
+
+
+
+}

vision/src/main/java/cn/smartjavaai/objectdetection/stream/StreamDetector.java

@@ -69,8 +69,11 @@ public class StreamDetector implements AutoCloseable{
     //空帧数量
     private int nullFrameCount = 0;
 
+
     // 连续多少次空帧认为断联
-    private static final int MAX_NULL_FRAMES = 5;
+    private static final int MAX_NULL_FRAMES = 10;
+
+
 
     public static Builder builder() { return new Builder(); }
 
@@ -161,7 +164,8 @@ public class StreamDetector implements AutoCloseable{
         while (!grabberFinished && isRunning) {
             try {
                 Frame frame = grabber.grabFrame();
-                if (frame == null || frame.image == null) {
+                //空帧
+                if (frame == null) {
                     if(sourceType == VideoSourceType.FILE){
                         log.debug("视频检测结束");
                         grabberFinished = true;
@@ -181,8 +185,13 @@ public class StreamDetector implements AutoCloseable{
                         }
                         continue;
                     }
+                }else{
+                    nullFrameCount = 0; // 只要拿到正常帧就清零
+                    //非视频帧
+                    if(frame.type != Frame.Type.VIDEO){
+                        continue;
+                    }
                 }
-                nullFrameCount = 0; // 只要拿到正常帧就清零
                 frameCount++;
                 if (frameCount % frameDetectionInterval != 0) continue;
                 Frame currentFrame = frame.clone();