HN · GitHub Signals

OPEN SOURCE, IN FOCUS

bw24

avifenesh/bw24

From-scratch Rust+CUDA LLM inference engine for RTX 5090 (sm_120a) — NVFP4, MoE, MTP speculative decoding, tuned against measured hardware limits

Explore on GitHub ↗

Saved projects stay in this browser. No account needed.

TAKE A CLOSER LOOK

Start with the source.

The repository is the place to check installation steps, supported platforms, the license, and current maintenance before choosing a tool.

Read the documentation on GitHub ↗

Keep exploring

Find projects in the same areas, or search the archive for a different approach.

ANOTHER GOOD FIND

Explore related projects

View the archive ↗

Connected by topics and descriptions.

avifenesh

memra

memra — from-scratch LLM inference engine for NVIDIA RTX 50-series (Rust + CUDA)

Take a look →
dicklesworthstone

franken_ocr

Pure-Rust, CPU-only OCR engine for Baidu Unlimited-OCR (a DeepSeek-OCR-derived 3B MoE VLM). One fixed model, custom int4/int8 kernels, no ML framework, no GPU. Pre-Phase-0 scaffold.

Take a look →
giannisanni

pulsar

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.

Take a look →
jakeatx

llamampere

llama.cpp fork for significantly improved performance on Ampere (especially RTX 3090 / 3090 Ti): TurboQuant KV cache, MTP speculative decoding with a 64K draft-vocabulary shortlist, custom SM86 + Qwen kernels. 90 tok/s over a 100K-token generation at temperature 1.

Take a look →