femto-gpt
choose model
Narrative Model
Built for coherent storytelling, scene continuity, and expressive long-form generation.
Dataset: Tiny-stories (Fiction passages and screenplay-style samples)
Fine-tuned: No
Tokens in training dataset: 473.8 million
Present epoch loss: train: 1.4308 | validation: 1.4368
Perplexity (shortlisted next words): e^1.43 (shortlists 4 out of 50,257 tokens)
WEBedu Model
Optimized for educational web text (like simple science, history)
Dataset: Fineweb-Edu (Curated educational websites and technical guides)
Fine-tuned: No
Tokens in training dataset: 400 million
Present epoch loss: train: 3.6942 | validation: 3.7563
Perplexity (shortlisted next words): e^3.75 (shortlists 42 out of 50,257 tokens)
LoRA Fine Tuned
U. S. Affairs Model
Specialized in U.S. political, civic, and public affairs language understanding.
Dataset: Synthetically made. U.S. affairs (presidential, wars etc.)
Fine-tuned: Yes (LoRA)
Fine-tuned Tokens in training dataset: 39,490
Present epoch loss: train: 3.0140 | validation: 3.8601
Perplexity (shortlisted next words): e^3.86 (shortlists 47 out of 50,257 tokens)
Model Architecture Details
| Number of embeddings | 512 |
|---|---|
| Number of tokens | 50257 |
| Attention heads | 8 |
| Transformer block layers | 8 |
| Max token length | 256 |