LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
-
Updated
Apr 23, 2026 - Python
#
inference-server
Here are 46 public repositories matching this topic...
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
-
Updated
Apr 23, 2026 - Python
Turn any computer or edge device into a command center for your computer vision projects.
python docker machine-learning computer-vision deployment inference classification object-detection vit agents inference-server jetson tensorrt instance-segmentation onnx inference-api yolov5 yolov8 yolo11 yolov12
-
Updated
Apr 23, 2026 - Python
The simplest way to serve AI/ML models in production
open-source machine-learning packaging artificial-intelligence falcon easy-to-use whisper inference-server model-serving inference-api stable-diffusion wizardlm
-
Updated
Apr 23, 2026 - Python
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
python nlp data-science machine-learning ai computer-vision deep-learning tensorflow transformers inference pytorch artificial-intelligence inference-server predict paddlepaddle model-deployment model-serving serving huggingface modelserver
-
Updated
Feb 14, 2023 - Python
This is a repository for an nocode object detection inference API using the Yolov3 and Yolov4 Darknet framework.
api docker dockerfile computer-vision deep-learning neural-network gpu rest-api inference yolo deeplearning inference-server bounding-boxes no-code yolov3 alexeyab-darknet yolo-gui detection-inference-api inference-gui yolov4
-
Updated
Jun 28, 2022 - Python
This is a repository for an nocode object detection inference API using the Yolov4 and Yolov3 Opencv.
api docker opencv deep-neural-networks cpu computer-vision deep-learning neural-network rest-api inference object-detection inference-server bounding-boxes no-code yolov3 detection-inference-api cpu-inference-api inference-gui yolov4 yolov4-darknet
-
Updated
Jun 28, 2022 - Python
This is a repository for an object detection inference API using the Tensorflow framework.
api docker cpu computer-vision deep-learning docker-container tensorflow rest-api docker-image inference deeplearning object-detection tensorflow-framework predictions inference-server bounding-boxes computervision docker-ce inference-engine detection-inference-api
-
Updated
Jun 28, 2022 - Python
Deploy DL/ ML inference pipelines with minimal extra code.
python docker deep-learning websocket gunicorn pytorch falcon http-server triton gevent inference-server tensorflow-serving streaming-audio model-deployment model-serving serving tf-serving torchserve triton-inference-server triton-server
-
Updated
Feb 10, 2026 - Python
[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI
ai ml inference gpt inference-server mistral inference-engine serving mlops gpt3 llm stable-diffusion llms generative-ai llmops llm-serving llm-inference llama2 llm-ops
-
Updated
Jun 25, 2025 - Python
Advanced inference pipeline using NVIDIA Triton Inference Server for CRAFT Text detection (Pytorch), included converter from Pytorch -> ONNX -> TensorRT, Inference pipelines (TensorRT, Triton server - multi-format). Supported model format for Triton inference: TensorRT engine, Torchscript, ONNX
inference pytorch text-detection nvidia-docker inference-server tensorrt inference-engine onnx onnx-torch tensorrt-conversion triton-inference-server text-detection-from-image
-
Updated
Aug 18, 2021 - Python
Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)
-
Updated
Jun 28, 2023 - Python
Inference Server Implementation from Scratch for Machine Learning Models
-
Updated
Dec 31, 2020 - Python
Roboflow's inference server to analyze video streams. This project extracts insights from video frames at defined intervals and generates informative visualizations and CSV outputs.
-
Updated
May 21, 2024 - Python
Session Based Real-time Hotel Recommendation Web Application
-
Updated
Jan 6, 2021 - Python
A networked inference server for Whisper speech recognition
-
Updated
Jun 9, 2025 - Python
Vision and vision-multi-modal components for geniusrise framework
-
Updated
Oct 10, 2024 - Python
Serve pytorch inference requests using batching with redis for faster performance.
-
Updated
Apr 11, 2021 - Python
🤖 Optimize LLM inference on Mac with continuous batching and SSD caching managed from your menu bar for efficient performance.
-
Updated
Apr 23, 2026 - Python
Text components powering LLMs & SLMs for geniusrise framework
-
Updated
Oct 10, 2024 - Python
Improve this page
Add a description, image, and links to the inference-server topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the inference-server topic, visit your repo's landing page and select "manage topics."