What the hell is a matrix, anyways? When I say matrix, you probably don’t have a great idea of what a matrix actually is, and that’s what I aim to solve. What the hell is a vector, really? This section draws heavily on the textbook Linear Algebra Done Right by Sheldon Axler In the previous issue I didn’t go that deeply into the mathematical definition behind a vector. Sadly, before we can truly understand what a matrix is, we must first understand what a vector is. ...
Hello! This is part 3 in a TBD part series on creating an LLM from scratch! You can see part 2, creating lists, here, and part 4, creating matrices, here. What is a vector? For our purposes, a vector is an ordered list of numbers. Here’s an example of a 2-dimensional vector: [5, 7] And here’s an example of a 5-dimensional vector: [3.5, 4, 2, 1.2, 943.89] We use multi-dimensional vectors to represent things like words and sentences and paragraphs, because the hope is that each vector will allow us to encode multiple independent aspects of a token at once. At a high-level, we’re guessing that the vector will begin to associate different numbers with different attributes: maybe we want the first number to roughly correlate to how dog-like the word is, the second number to correlate to royalty, the third number to correlate with age, etc. For more information, see this paper on word representations in vector space. ...
Hello! This is part 2 in a TBD part series on creating an LLM from scratch! You can see part 1, the roadmap, here and part 3, building vectors, here. Making a list The first things that pop into my head when I’m thinking about vectors and matrices are lists. A vector is just a list of numbers, and a matrix is a list of vectors. Therefore, we should start with a list and then build our way up from there. Here are the features I want out of this specific list implementation: ...
This is part 1 in a TBD part series on creating an LLM from scratch! Part 2 can be found here. In this document I will be talking about how I’ll be approaching this. This project is currently in progress, so for now it’ll be mostly high-level, and as I go through it I’ll update it with more detail and links to relevant articles. Math & Linear Algebra Foundations Lists, vectors, and matrices Randomness Neurons, Chain Rule, and Autodiff Optimization & Training Mechanics Probabilistic Language Modeling Embeddings and a (Tiny) Transformer