Posts

Hello! This is part 2 in a TBD part series on creating an LLM from scratch! You can see part 1, the roadmap, here and part 3, building vectors, here. Making a list The first things that pop into my head when I’m thinking about vectors and matrices are lists. A vector is just a list of numbers, and a matrix is a list of vectors. Therefore, we should start with a list and then build our way up from there. Here are the features I want out of this specific list implementation: ...

This is part 1 in a TBD part series on creating an LLM from scratch! Part 2 can be found here. In this document I will be talking about how I’ll be approaching this. This project is currently in progress, so for now it’ll be mostly high-level, and as I go through it I’ll update it with more detail and links to relevant articles. Math & Linear Algebra Foundations Lists, vectors, and matrices Randomness Neurons, Chain Rule, and Autodiff Optimization & Training Mechanics Probabilistic Language Modeling Embeddings and a (Tiny) Transformer