HN · GitHub Signals

OPEN SOURCE, IN FOCUS

llm-inference-bench

local-inference-lab/llm-inference-bench

LLM inference decode throughput benchmark with Rich TUI dashboard. Measures token generation speed across concurrency levels and context lengths. Supports SGLang and vLLM engines.

Explore on GitHub ↗

Saved projects stay in this browser. No account needed.

TAKE A CLOSER LOOK

Start with the source.

The repository is the place to check installation steps, supported platforms, the license, and current maintenance before choosing a tool.

Read the documentation on GitHub ↗

Keep exploring

Find projects in the same areas, or search the archive for a different approach.

ANOTHER GOOD FIND

Explore related projects

View the archive ↗

Connected by topics and descriptions.

cloudrift-ai

emmy

Benchmark and deploy optimized LLM models on GPU servers with vLLM or SGLang. Chose from a list of optimized recipes for popular models or create your own with custom configurations. Run benchmarks across different GPU types and configurations, track results, and share experiments with the community.

Take a look →
jmuncor

tokentap

Intercept LLM API traffic and visualize token usage in a real-time terminal dashboard. Track costs, debug prompts, and monitor context window usage across your AI development sessions. - GitHub - jmuncor/tokentap: Intercept LLM API traffic and visualize token usage in a real-time terminal dashboard. Track costs, debug prompts, and monitor context window usage across your AI development sessions.

Take a look →
sergiiob

intel-arc-pro-b70-inference-cookbook

Open recipes, engine patches, and benchmark harnesses for LLM inference on Intel Arc Pro B60/B70 (Battlemage, Xe2). MoE 35B at 160 t/s decode / 7.5K t/s prefill single-stream, 27B at 50~ t/s decode / 1.7K t/s prefill single stream. vLLM XPU MTP unlocked. Muse Glimmer recipe added!!

Take a look →