AbdelrhmanEbied/Tiny-MoE

Tiny-MoE is a lightweight Mixture-of-Experts language model built entirely from scratch in native PyTorch and trained end-to-end on Kaggle using free 2× NVIDIA T4 GPUs. The project implements modern LLM techniques—including MLA, RoPE, YaRN, streaming pre-training, and efficient inference—without relying on existing model implementations.