qiuruiandClaude Opus 4.6
eb6e4329d4
Architecture: - Remove Chrome extension (project fully device-side, no target-machine deps) - Update all docs from "three-layer" to "two-layer" privacy (video redact + network intercept) - Add comprehensive architecture docs (overview, subsystem designs) Services: - kvm_agent: hybrid planner (template/local/cloud), screen state detection, mouse-first architecture, app launcher, visual workflow tests - privacy_gateway: upload scanner, privacy LLM integration, REST API - doc_processor: new document processing service Deployment: - Add kvm-bridge, kvm-meta, kvm-privacy deb package definitions - New systemd services (doc-processor, kvm-gateway, rkllm-server) - Network deploy configs, journald forwarding - Remove secrets.env templates from packages Plans & Docs: - HDMI-TX DRM local output + OSD design (drm_output.c, VOP2 multi-plane) - AI Agent token optimization plan (72% savings via caching/pruning/fingerprint) - Model sync: all RKNN/ONNX models now in project directory - Native H.264 adaptive bitrate plan Submodules updated: - deps/KVM: WebUI i18n, RBAC, DDNS, Agent API, OCR models (LFS) - deps/embedding: models synced (LFS), benchmarks, Ollama backend - deps/info-privacy-rs: regex PII detection, face detection integration Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3.6 KiB
3.6 KiB
RK3588 硬件优化架构
平台: NanoPC-T6 (RK3588) | 8核 ARM64 | 6 TOPS NPU | RGA 2D
1. 硬件资源
CPU 布局
Core 0-3: ARM Cortex-A55 (低功耗, 1.8 GHz)
Core 4-7: ARM Cortex-A76 (高性能, 2.4 GHz)
核心分配方案
| 核心 | 类型 | 分配 | 服务 |
|---|---|---|---|
| 0-1 | A55 | info-privacy-rs + OCR 后处理 | PII 检测 |
| 2-3 | A55 | mem-bridge | 记忆 + 路由 |
| 4 | A76 | HID Dispatcher (SCHED_RR) | 实时 HID |
| 5-6 | A76 | Go runtime (GOMAXPROCS=3) | kvm-server |
| 7 | A76 | 视频编码回调线程 | V4L2→MPP |
systemd 亲和性配置
# kvm-server.service
CPUAffinity=4 5 6 7
# info-privacy.service
CPUAffinity=0 1
# mem-bridge-memory.service / mem-bridge-router.service
CPUAffinity=2 3
Go 进程级别
// main.go
func pinToA76BigCores() {
var mask [16]byte // 128 cores max
mask[0] = 0xF0 // bits 4,5,6,7
syscall.RawSyscall(SYS_SCHED_SETAFFINITY, 0, 16, uintptr(unsafe.Pointer(&mask[0])))
runtime.GOMAXPROCS(3)
}
2. NPU (Neural Processing Unit)
三核调度
NPU Core 0: OCR text detection (RKNN_NPU_CORE_0)
NPU Core 1: OCR text recognition (RKNN_NPU_CORE_1)
NPU Core 2: Embedding vectorization (RKNN_NPU_CORE_2)
之前的问题
embedding 使用 NPU_CORE_ALL（全部 3 核），与 OCR 的 CORE_0/CORE_1 争用，导致推理延迟抖动。
修复
# deps/embedding/src/embed_db/embedder.py
ret = rknn.init_runtime(core_mask=RKNNLite.NPU_CORE_2) # 独占 Core 2
验证
cat /sys/kernel/debug/rknpu/load
# 应显示三核独立负载，无争用
3. RGA (2D 图形加速器)
用途
- 色彩转换: BGR24 → NV12 (im2d.h: imcvtcolor)
- 缩放: 任意分辨率 → 编码目标分辨率 (imresize)
- 隐私遮蔽: NV12 区域黑色填充 (memset, RGA imfill 未验证)
色彩转换路径
V4L2 BGR24 帧
→ wrapbuffer_virtualaddr(src, w, h, RK_FORMAT_BGR_888)
→ wrapbuffer_virtualaddr(dst, w, h, RK_FORMAT_YCbCr_420_SP)
→ imcvtcolor(src, dst, src_format, dst_format)
→ NV12 数据
4. MPP (Media Process Platform)
H.264 编码
- 编码器类型: MPP_VIDEO_CodingAVC
- 帧率: 1-60 fps (动态可调)
- 码率: 100Kbps - 20Mbps (动态可调)
- GOP: 与帧率相同 (每秒一个关键帧)
关键操作
mpp_encoder_force_idr() // 立即产生 IDR 关键帧
mpp_encoder_update_rc() // 动态更新码率+帧率
DMA-buf 零拷贝
V4L2 buffer → VIDIOC_EXPBUF → DMA-buf fd
→ mpp_buffer_import_with_tag(fd) // MPP 直接使用 V4L2 buffer
→ 编码 (零 CPU memcpy)
启用条件:
- V4L2 输出 NV12 格式
- 无缩放 (capture == encode 分辨率)
- 无隐私遮蔽 (遮蔽会修改 buffer)
- DMA-buf 导出成功
节省: 1080p@30fps 约 90MB/s 内存带宽
5. OCR 内联化
之前: exec 模式
Go → exec.Command("kvm-ocr", args...) → fork+exec
延迟: 800-2000ms (含进程启动 + RKNN 模型加载)
之后: CGo 内联
Go → CGo → libkvm_ocr.so → RKNN (常驻, 无 fork)
延迟: 60-100ms
实现
- C 层:
src/audit/ocr_analyzer.cpp新增extern "C"API - 共享库: CMakeLists.txt 新增
libkvm_ocr SHARED目标 - Go 桥接:
ocr_bridge_cgo.go(build tag:cgo && rknn) - 接口:
OCRAnalyzer统一 exec/CGo 两种实现
6. 构建集成
# 构建顺序 (build-deb.sh):
1. 前端 (npm build)
2. 视频 C 库 (cmake, 需要 rockchip_mpp)
3. OCR 共享库 (cmake, 需要 rknn_api) ← 必须在 Go 之前!
4. Go 后端 (自动检测 .so → 设置 CGo flags + build tags)
5. OCR 独立二进制 (可选, --ocr flag)
6. dpkg-buildpackage