Zotero Catalogue

Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pretraining DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.

At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on auto-regressive objectives is closely related to gradient-based meta-learning formulations. We start by providing a simple weight construction that shows the equivalence of data transformations induced by 1) a single linear self-attention layer and by 2) gradient-descent (GD) on a regression loss. Motivated by that construction, we show empirically that when training self-attention-only Transformers on simple regression tasks either the models learned by GD and Transformers show great similarity or, remarkably, the weights found by optimization match the construction. Thus we show how trained Transformers become mesa-optimizers i.e. learn models by gradient descent in their forward pass. This allows us, at least in the domain of regression problems, to mechanistically understand the inner workings of in-context learning in optimized Transformers. Building on this insight, we furthermore identify how Transformers surpass the performance of plain gradient descent by learning an iterative curvature correction and learn linear models on deep data representations to solve non-linear regression tasks. Finally, we discuss intriguing parallels to a mechanism identified to be crucial for in-context learning termed induction-head (Olsson et al., 2022) and show how it could be understood as a specific case of in-context learning by gradient descent learning within Transformers. Code to reproduce the experiments can be found at https://github.com/google-research/self-organising-systems/tree/master/transformers_learn_icl_by_gd .

Graham shares the four truths that Alpine is using to guide its decisions as we navigate this tech revolution. This was speech was recorded at Alpine's 2025’s Growth Summit.

De facto exclusion of vulnerable populations from markets for energy-efficient technologies can result in multiple barriers to access. For example, exclusion can lead to limited knowledge about available products, an inability to distinguish high-quality from low-quality devices, and limited options for financing, making products seem unobtainable. However, behaviorally informed interventions can offer promising solutions in such contexts, even where exclusion is the result of structural causes. This paper uses a randomized control trial to consider the potential of such interventions for refugees in Uganda in the context of certified solar markets. We evaluate a behaviorally-informed information and savings session embedded in Village Savings and Lending Association (VSLA) meetings, finding evidence for increased pursuit of certified solar products in the treatment group two months later. Results manifest through the barriers described, with increased knowledge, trust in solar companies, financial inclusion through savings group support, and aspirations mediating effects.

💡Putting yourself out there increases the surface area for potential opportunities

"The amount of serendipity that will occur in your life is directly proportional to the degree to which you do something you're passionate about combined with the total number of people to whom this is effectively communicated."

An illustrated therapy session between Humanity, Capitalism, and Post-Capitalism.

We are still in the "touching the elephant" phase of the "ideological mindsets" that emerge from "The Transition" (away from late-stage-capitalism at the birth of the internet). In other words, we don't yet know what the most coherent and powerful mindsets are. https://twitter.com/iiterature/status/1048301983109668864We can view

In-depth report on Building the Wisdom Age and creating the Wisdom Economy.

1/ Maps are one of the most effective ways to communicate Here's a 🧵of maps, starting with everyone's favorite — memespace maps https://xkcd.com/802/ https://www.ribbonfarm.com/you-are-here/ https://slatestarcodex.com/2018/07/19/sentimental-cartography/ https://www.lesswrong.com/posts/WzPJRNYWhMXQTEj69/a-map-of-bay-area-memespace

Tech as a system of values, and not just an industry, is heavily driven by its subcultures and their ideologies. Where do these ideologies come from, and how do they influence what’s accomplished?

Almost everything in life must be done with others. If you learn to relate well to others, you increase your professional leverage and the emotional health of your relationships. I. Understanding Others The first step to relating well to others is understanding them. The first step to understanding others is

1. Stop Comparing Yourself Against Old Models You might be unconsciously following the path of your parents like a doctor or lawyer. Don't. Optimize for rate-of-learning [https://rooteco.notion.site/What-Roote-Hires-For-f21facf226e248e6af61551ef182c933] and consciously reject the paths given to you. Patrick Collison gives this advice [https://patrickcollison.com/advice]: > If you're

The philosopher Kwame Appiah writes that “in life, the challenge is not so much to figure out how best to play the game; the challenge is to figure out what game you’re playing.” When I try to figure out what game I’m playing, I see that for the last 25 years I have been playing […]

I’ve observed thousands of founders and thought a lot about what it takes to make a huge amount of money or to create something important. Usually, people start off wanting the former and end up...

One of the most powerful ways to build agency is to have "almost too much self-belief." But this can go badly quickly: look at how SBF and Elizabeth Holmes built reality distortion fields around themselves, defrauded everyone, and ended up in jail. Some reality distortion is good—we want to

My answer:

In summary: * Write for a specific reader. * Write concisely. * When editing, look to cut. * Write in chunks. * Have fun with it. * Leverage the computer with tools like Grammarly, hotkeys, and ergonomic keyboards. 1. Alex Danco's 5 Writing Tips Write in parallel tracks using "meanwhile". Meanwhile establishes parallel tracks of thought.

For the first 30 years of my life, I coasted and procrastinated often. I am pretty ashamed of it. 😔 In the last few months, I've beaten procrastination. I'm proud of this! ❤️ Here's how I did it. I hope it helps you. 1. Procrastination Is A Harmful Positive Feedback Loop Procrastination

I've previously written a piece on Why [https://www.rhyslindmark.com/anki/] You Should Use Anki. [https://www.rhyslindmark.com/anki/] This piece is how. 1. Understand your why. Why do you want to remember anything? For me, it was reading lots of books but forgetting them. What's your goal?

If you're a knowledge worker doing Tyler Cowen calls an intellectual athlete training [https://marginalrevolution.com/marginalrevolution/2019/07/learn-like-an-athlete-knowledge-workers-should-train.html] , you should use Anki or some other form of spaced repetition. See the responses to Michael Nielsen's tweet here for all the other great folks that do this: > The

1. First, remind yourself that the goal is not to simply to consume, but to contribute back to society. Build, write, teach, help others. That's why you learn in the first place. 2. Use Anki. Here's why. Here's how. 3. Join Twitter. Use these tips. Join these communities. Follow the

Why it's worth trying to make the world more optimistic

It is wrong to think these three statements contradict each other. We need to see that they are all true to see that a better world is possible.

More than half of young people think "humanity is doomed" due to climate change. We need to reframe the narrative from doom and sacrifice, to one of opportunity.

I have edited and expanded this in a newer post, you should read that instead: School is Not Enough

Some examples of people with high stamina. Inspired by /fast. * Beyoncé: As a child: "When some of us wanted to go to the movies, when we did have our off time, she was in the studio sitting there by herself writing a record. I’ll never forget, were at an

Don't "work harder." Instead: Cut away distractions. Stop doing fake work. Ask for help. Plan. Solicit criticism. Automate what can be automated. Engage your mentors. Exercise. Get more sleep. Whatever you do, don't "work harder." It's pretty much never the answer.

A list of things you're allowed to do that you thought you weren't, or didn't even know you could.

Nostalgia for another way of working

Yesterday I had lunch with a former Ph.D student of mine, who is now highly successful and tenured at a very good school.  I was reminded that, over twenty years ago, I was Graduate Director of Admissions.  One of my favorite strategies was to take strong candidates who applied for Masters and also offer them […]

// This was originally a Google Doc where I gathered hard-won life lessons; eventually I open sourced it. It got a great response, including a shoutout from Tim Ferriss, so here it is on Substack.

@_TamaraWinter: Recently I’ve gotten a lot of inbound from new grads asking for career advice, so I wrote down a list of things I wish I’d internalized sooner. The key thing: being precocious has an expiration...…

I give you permission. Permission to be ambitious. Permission to lead. Permission to apply for that fellowship, drop out of school, and start that startup.

Education is the kindling of a flame, not the filling of a vessel. — Socrates Compounding is everything. Compound interest dominates financial markets, compounding network effects dominate business, and compounding yourself dominates all other self-work you can do. So how do you compound yourself? We know what compounding looks like—an

An interesting note: I asked many people I respect to also share their advice, and most of it ended up being completely different from what both I wrote and within the group of respondents. So, I...

Ideas for getting into compilers and programming languages