Posts

(This is a now page, and if you have your own site, you should make one too.) (Last updated: 2025-10-30) Right now I’m working on building an LLM from scratch. I’m also trying to use transformers to denoise quantum sensor readings. Heavily inspired by this paper using CNNs to denoise quantum sensor readings and this paper on creating physics-informed networks. Finished the first model with 10,000 circuits, which underperformed the current SOTA, so now I’m scaling it up to a million circuits.

I’ve been thinking a lot lately about how one teaches the concepts of functions that operate at higher-dimensions, like a function that takes two inputs and returns two outputs. This is hard, because most of our… Note: This post is a work in progress.

The disconnect between Canada’s intellectual strength and its lack of private innovation fascinates and concerns me. Canada has produced pioneers like Geoffrey Hinton and Yoshua Bengio, the Godfathers of AI, and innovations like the Canadarm and CANDU reactor. Yet few of these breakthroughs translate to jobs and prosperity here at home. Research is essential, but you can’t build an economy on research alone: research proposes value, while business creates value. ...

Hello! This page is on the rules of engagement on this website. “I want to get more content from you, but I’m worried I’ll forget about your existence” Has this ever happened to you? We have a revolutionary new technology to ease your woes: RSS. I use an RSS reader for most of my browsing, and I would highly recommend you do as well! The one I use is Bubo Reader, which I use because it’s extremely simple, and can be deployed on your own server. You can also deploy it on Netlify, as seen in this demo. ...

What the hell is a matrix, anyways? When I say matrix, you probably don’t have a great idea of what a matrix actually is, and that’s what I aim to solve. What the hell is a vector, really? This section draws heavily on the textbook Linear Algebra Done Right by Sheldon Axler In the previous issue I didn’t go that deeply into the mathematical definition behind a vector. Sadly, before we can truly understand what a matrix is, we must first understand what a vector is. ...

Hello! This is part 3 in a TBD part series on creating an LLM from scratch! You can see part 2, creating lists, here, and part 4, creating matrices, here. What is a vector? For our purposes, a vector is an ordered list of numbers. Here’s an example of a 2-dimensional vector: [5, 7] And here’s an example of a 5-dimensional vector: [3.5, 4, 2, 1.2, 943.89] We use multi-dimensional vectors to represent things like words and sentences and paragraphs, because the hope is that each vector will allow us to encode multiple independent aspects of a token at once. At a high-level, we’re guessing that the vector will begin to associate different numbers with different attributes: maybe we want the first number to roughly correlate to how dog-like the word is, the second number to correlate to royalty, the third number to correlate with age, etc. For more information, see this paper on word representations in vector space. ...

Hello! This is part 2 in a TBD part series on creating an LLM from scratch! You can see part 1, the roadmap, here and part 3, building vectors, here. Making a list The first things that pop into my head when I’m thinking about vectors and matrices are lists. A vector is just a list of numbers, and a matrix is a list of vectors. Therefore, we should start with a list and then build our way up from there. Here are the features I want out of this specific list implementation: ...

This is part 1 in a TBD part series on creating an LLM from scratch! Part 2 can be found here. In this document I will be talking about how I’ll be approaching this. This project is currently in progress, so for now it’ll be mostly high-level, and as I go through it I’ll update it with more detail and links to relevant articles. Math & Linear Algebra Foundations Lists, vectors, and matrices Randomness Neurons, Chain Rule, and Autodiff Optimization & Training Mechanics Probabilistic Language Modeling Embeddings and a (Tiny) Transformer