kindi
A custom LLM inference engine for low-cost NVIDIA GPUs: Quadro K420 and K1200, two K1200s joined into one 8 GB card, GTX 1080 and RTX 3080. Runs Bonsai 2B, 4B, 9B and 27B models with an OpenAI-compatible server.
Take a look →OPEN SOURCE, IN FOCUS
runanywhereai/trnrt
The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.
Saved projects stay in this browser. No account needed.
TAKE A CLOSER LOOK
The repository is the place to check installation steps, supported platforms, the license, and current maintenance before choosing a tool.
Read the documentation on GitHub ↗Find projects in the same areas, or search the archive for a different approach.