Support Yolov5(4.0)/Yolov5(5.0)/YoloR/YoloX/Yolov4/Yolov3/CenterNet/CenterFace/RetinaFace/Classify/Unet. use darknet/libtorch/pytorch/mxnet to onnx to tensorrt
Torchserve server using a YoloV5 model running on docker with GPU and static batch inference to perform production ready and real time inference.
Analyze and generate unstructured data using LLMs, from quick experiments to billion token jobs.
Batch LLM Inference with Ray Data LLM: From Simple to Advanced
RWKV7 推理服务 | 高并发批量推理，动态扩缩容，兼容OpenAI API，CUDA优化，低显存部署大模型推理 RWKV linear attention batch inference server — auto-queue, dynamic capacity, OpenAI compatible.
PipelineScheduler optimizes workload distribution between servers and edge devices, setting optimal batch sizes to maximize throughput and minimize latency amid content dynamics and network instability. It also addresses resource contention with spatiotemporal inference scheduling to reduce co-location interference.
Fast self-hosted embedding engine for search, RAG, and reindexing workloads on NVIDIA GPUs. Built in Rust + TensorRT for teams that care about scale, cost, and control.
Production-ready batch inference for Cohere’s Arabic/English ASR model, with optimized Silero VAD, multi-file GPU batching, subtitles, and optional word-level timestamps.
Ray Saturday Dec 2022 edition
Torchfusion is a very opinionated torch inference on datafusion.
Serve pytorch inference requests using batching with redis for faster performance.
Support batch inference of Grounding DINO. "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection"
Multi-actor RL runtime for high-concurrency adaptive system modeling
Self-hosted batch LLM pipeline for analyzing customer feedback from Excel. Upload xlsx, describe the task, configure output fields — get structured results. Works with any OpenAI-compatible API and Ollama.
LightGBM Inference on Datafusion
FastMCP fleet MCP server for diffusion LMs (dLLM). DiffusionGemma on Goliath RTX 4090 — batch inference, HLE-shaped reasoning, ~200–400 tok/s. Doc phase; llama-diffusion-cli sidecar next. Complements local-llm-mcp.
简单的 Ollama JSONL 批量推理工具 / Simple Ollama JSONL batch inference tool.
Production-ready native Ruby batch inference for Cohere’s Arabic/English ASR model, with CPU/CUDA/Metal execution, Silero VAD, subtitles, and optional word-level timestamps.
Single-binary batch-inference coordinator — fan a JSONL across N OpenAI-compatible endpoints, survive spot kills, resume with zero loss
Queue-driven, scale-from-zero GPU inference for any Kubernetes — bursts to cross-region VMs when GPUs run dry
Add a description, image, and links to the batch-inference topic page so that developers can more easily learn about it.
To associate your repository with the batch-inference topic, visit your repo's landing page and select "manage topics."