Posts

Via slashdot. The news of the layoffs themselves aren’t that interesting, but the most interesting part was the news about them recording their employees to train their models. There’s more about this here, overall I think while I wouldn’t like to work somewhere where I’d be recorded for the entire day, for the purposes of replacing me no less, it is in interesting approach to gathering data. Overall though it feels that Meta is losing the AI race. Despite Llama being one of the first open-weight LLMs to gain a lot of mindshare, they’ve sort of stagnated. ...

Hello, I am Amitav! This is actually my daily update to be first posted on my personal website, amitav.net, before being syndicated to substack (see this). The biggest reason for this is that I have an admin portal now on my website, which gives me a snazzy interface with which to write my posts. You can see this for yourself here, though I will warn you that this is password restricted so that I don’t get random folks posting stuff on my website. Anyways, back to the task at hand, my daily update(s)! ...

Hello! This is my first admin portal post, my previous posts have been uploaded manually using scp. The admin portal was created by my openclaw and allows me to write posts in a nice interface and is password protected.

The disconnect between Canada’s intellectual strength and its lack of private innovation fascinates and concerns me. Canada has produced pioneers like Geoffrey Hinton and Yoshua Bengio, the Godfathers of AI, and innovations like the Canadarm and CANDU reactor. Yet few of these breakthroughs translate to jobs and prosperity here at home. Research is essential, but you can’t build an economy on research alone: research proposes value, while business creates value. ...

I’ve been thinking a lot lately about how one teaches the concepts of functions that operate at higher-dimensions, like a function that takes two inputs and returns two outputs. This is hard, because most of our… Note: This post is a work in progress.

Hello! This page is on the rules of engagement on this website. “I want to get more content from you, but I’m worried I’ll forget about your existence” Has this ever happened to you? We have a revolutionary new technology to ease your woes: RSS. I use an RSS reader for most of my browsing, and I would highly recommend you do as well! The one I use is Bubo Reader, which I use because it’s extremely simple, and can be deployed on your own server. You can also deploy it on Netlify, as seen in this demo. ...

What the hell is a matrix, anyways? When I say matrix, you probably don’t have a great idea of what a matrix actually is, and that’s what I aim to solve. What the hell is a vector, really? This section draws heavily on the textbook Linear Algebra Done Right by Sheldon Axler In the previous issue I didn’t go that deeply into the mathematical definition behind a vector. Sadly, before we can truly understand what a matrix is, we must first understand what a vector is. ...

Hello! This is part 3 in a TBD part series on creating an LLM from scratch! You can see part 2, creating lists, here, and part 4, creating matrices, here. What is a vector? For our purposes, a vector is an ordered list of numbers. Here’s an example of a 2-dimensional vector: [5, 7] And here’s an example of a 5-dimensional vector: [3.5, 4, 2, 1.2, 943.89] We use multi-dimensional vectors to represent things like words and sentences and paragraphs, because the hope is that each vector will allow us to encode multiple independent aspects of a token at once. At a high-level, we’re guessing that the vector will begin to associate different numbers with different attributes: maybe we want the first number to roughly correlate to how dog-like the word is, the second number to correlate to royalty, the third number to correlate with age, etc. For more information, see this paper on word representations in vector space. ...

Hello! This is part 2 in a TBD part series on creating an LLM from scratch! You can see part 1, the roadmap, here and part 3, building vectors, here. Making a list The first things that pop into my head when I’m thinking about vectors and matrices are lists. A vector is just a list of numbers, and a matrix is a list of vectors. Therefore, we should start with a list and then build our way up from there. Here are the features I want out of this specific list implementation: ...

This is part 1 in a TBD part series on creating an LLM from scratch! Part 2 can be found here. In this document I will be talking about how I’ll be approaching this. This project is currently in progress, so for now it’ll be mostly high-level, and as I go through it I’ll update it with more detail and links to relevant articles. Math & Linear Algebra Foundations Lists, vectors, and matrices Randomness Neurons, Chain Rule, and Autodiff Optimization & Training Mechanics Probabilistic Language Modeling Embeddings and a (Tiny) Transformer