Claude Code + Karpathy's Obsidian = New Meta

Imagine if Claude Code remembered everything and got smarter every time you spoke to it. In this video, I'm going to show you Andrew Karpathy's Obsidian RAG system that gives Claude a self-updating memory so that Claude Code never forgets, gets smarter, and every conversation actually knows who you are and how to fix its massive limitation that 99% of people aren't even talking about. So by the end of this video, you'll have a Claude Code memory system that will work for the long term. And if you don't know who I am, my name is Jack Roberts. I built and sold my last tech startup to over 60,000 customers, and now I run several AI automation businesses and teach thousands of people stuff like this every day. So if you haven't already, grab that coffee and let's dive straight in. Andrew Karpathy, famous founder of OpenAI. So Andrew Karpathy just gone viral by talking about Claude Code plus Obsidian to create what we call an Obsidian memory system. Now, the way that it works effectively is it will have a lot of files that Claude will create or you'll download and Claude will organize and create the system for you. And the idea is that it's very easy for Claude to edit these if you want to update the information and you can even view them visually in Obsidian. It is essentially like your personal Wikipedia. So anything you want to remember, we can actually just do by having this kind of index that Claude can cycle through based on whatever information you want to, meaning that you don't lose important context whenever you're using Claude. Now, this went viral over 18 million views talking about Claude Code and Obsidian because it solves a real problem, and that is context loss. In other words, the fact that Claude has amnesia for all its wonderful perks, sometimes it just forgets it's a model and it will hallucinate. We expect at least 1%, sometimes up to 10% of the time, depending on how long the conversations happen for and it can be confidently incorrect. And the problem is we don't know when it is confidently incorrect and when it's actually just being accurate. So with this system, Claude can read and write its own memory and it's supplementary to and supports alongside Claude's basic context. So it's like this adds a fourth memory system, which I'll explain in this video. And it gets better, crucially, the more that you use it because we index more the information and we build out over time. However, 90% of the coverage that I've seen on this, I would describe realistically as node porn. And it does actually, the system have one huge, actually several huge limitations, but don't worry, I got you. I can explain what those limitations are, which I haven't seen anyone else talk about in this kind of hype cycle and how you can actually fix it. So you get a very long-term memory system that will work in any situation and give you the accurate data. So let's begin with the source of this, actually download it, show you a real use case of how this will work in your new system. So Andrej Karpathy basically explained this in about four or five different points. Effectively, it's using a large language model to build personal knowledge bases for different topics of interest. The best way to visualize this is like you are essentially building a personal Wikipedia. When you're talking about an obsidian memory system, it's your own personal Wiki. So data ingestion happens at point of obsidian web clipping, or we can give it data. There is a best way to do this, I think, and a not great way to do this, which I'll show you. But effectively, we create this wonderful raw directory and the LLM will incrementally compile a Wiki, which is just a collection of MD files and a directory structure. That's it. And then it has a directory. When you ask it questions, it goes through and it finds the relevant things, clicks on them, and then good things happen. So in terms of ID, this is basically what are you looking at to do this? Now, obsidian itself is also a visual layer, but you could use this with any project you're doing. It could be anti-gravity, it could be cursor, you name it. And he's put some additional things in here. He's got this concept of LinkedIn to do health checks, but I'll come on to that later in the video when we go through everything. So it's actually super-duper, and I say duper because it is super-duper easy to set up. So what we're going to do, first of all, is download Obsidian to sharpen our thinking. Cool story. These guys, not many people, built a very successful company. We're going to go ahead and download it on the computer. I don't have it on this one yet, so we're going to go through the process cleanly together. I also want to touch on this nerd view as well. I think it's interesting. I know a lot of people are, look how I can visualize it. In reality, I haven't seen anyone make any informed decisions off it. I think it's just something that looks cool, but it's not really necessarily going to help you do anything better. Although the relations between certain things on your memory system are really important, which we can touch on later. So we're down to applications. Now we're just going to open this guy up. Obsidian, just think of it as a way to visualize files. That is literally it. It will show you and help you visualize files. So you can create a new vault, open folders vault, whatever you want to. I'm going to do a quick start and show what it looks like. We come in. If you click on this one here, this is a graph view, which will show you basically the relation between all of the key things in your database. But firstly, let's go to dark mode, which you can do by clicking on Obsidian and then file and settings. This exists on appearance. And then you can flip between light mode and dark mode here. Dark mode is usually preferred. So now we are ready to rock and roll. Now we just need to add some important information. The way we do this actually is we're going to head back over to Andrej Karpathy's Twitter. Now, if I come over here, for example, one of the things he did is built this little GitHub, which is really simple. It's very vague. And on this LLM wiki, he basically explains the core premise a bit. The idea is that instead of just retrieving raw documents at query time, the LLM incrementally builds and maintains a persistent wiki, a structured intellect collection of markdown files that sits between you and the raw sources. And you know what guys, all we're going to do literally is come down. I'm going to do a classic control A and C. So we're going to come down and grab all this stuff here. Beautiful. And then we're going to head over to our environment of choice and we're going to be using anti-gravity to do this. Of course, this could be done in the terminal. It could be done in cursor. It doesn't really matter. I'm just using this as an environment. Let me open up a brand new folder. So I've called this one Obsidian. Now I'm going to open up a terminal. I'm going to run Claude Code just because I prefer that environment. I'm going to come up and let's run Claude. And one little hack, by the way, which I highly recommend you do is you can download these images locally. So if you go to Obsidian, settings, file and links, set attachment folder path to a fixed directory, e.g. raw assets. Then in settings, you can hotkeys and click download to find attachments for the current file, which means that when you grab information from a page, it will save it there, which is going to be super duper handy. So come back over then. I'm going to give it the final prompt, which is, hey there, I would like to create a structure in line with the below. Please ask me any questions if you wish to, but just follow this along and create it for me, please. Beautiful. And before we actually run it, it's best to understand exactly what the capacity Obsidian system actually is. So most rack is what we call re-derivation. In other words, that we're constantly going and getting the same information every time we ask the question. But with the LLM wiki, the way it's going to work is it will compound over time. So it's the same question every time you start from scratch with traditional rack, but this system, it gets smarter with every source that you add. So traditional rack, this is how it works. Query, chunks, answer. Claude finds chunks, stitches an answer, and then forgets it. Tomorrow, it's the same process. But with this system, the way that it works is one new source will update 10 to 15 different wiki pages. The knowledge stays and every future query reads from it. So it's self-building in that sense, which is cool. So this has what we call the compounding effect. So one source, 15 pages are updated. You drop a new article into your folder, Claude reads it, it finds every page in the wiki it relates to. Say for example, you and I want to grow a LinkedIn. And we say, hey, this is a great article on growing LinkedIn. When we add that to the system, this will then update every page that it's linked to, which is really cool. It flags anything. This one's really cool. The contradicters, any contradictions are getting ironed out in the process, stuff that are new. One source in, 15 files updated in a single pass, which is why it's so super cool. You're building a personal Wikipedia and it's only good if you can update it, right? So three layers, no database. Everything lives in three places, a folder for the sources you feed into, a folder for the wiki that Claude writes and a single file that tells Claude the rules, the rules of the game. So we've got raw sources, the wiki and the schema. You can see what that looks like. So the sources might be articles, PDFs, transcripts, web clips, you name it. We've got summaries for the wiki. Claude writes it and you kind of read it. And then the rule book, which is your conventions, your workflows, and it turns Claude into a maintainer of your wiki. Now, the way it works is you can ingest. So we're going to drop in things. These are the three kind of operations you're going to be able to query. So one is like asking it questions. And then this process called linting that Karl Parth mentioned, which effectively is every couple of weeks, find contradictions, orphan pages, stale claims, and the kind of maintenance that humans would naturally forget in the whole process. Awesome. So these three files do a lot of heavy lifting, the index.md and the log.md. Effectively though, you're going to curate, the LLM does the bookkeeping and the wiki will compound and value. So what kind of things could we build with this? Well, we could do a book companion. We could do research deep dive, a team or business wiki and personal life wiki. How does it actually work? Four rules, really simple. I'm going to pick one domain. So not everything like one book, one project, one research question. It works better when you have it as a set project. So one project for one thing. So growing on LinkedIn or I don't know, managing this merger and acquisition, that kind of thing. Think of them as individual mini Wikipedia bases that you can call upon and question for different topics. You're going to invest in the first section. So Claude's going to ask you questions, answer them properly. This is where your rule book gets written. And then you're going to run two screens. So Claude code on one side and then your obsidian graph view on the other. If you want to take a look at it, you can see the nodes lining up as Claude does various different things. And then every couple of weeks, you're going to go ahead and purge it. So first session, you're going to paste that prompt I gave you. Claude's going to ask you questions. Hey Jack, what's this wiki for? And then you're going to co-write the rules with Claude in that process. Then every session after that, the loop is going to begin again. You're going to drop a new source. Claude's going to integrate it, update the 10 to 15 related pages, and then you can see it expand and blow up in obsidian. You create it, it maintains, and the wiki compounds. So let's head back over now to antigravity and see how Claude is getting on with this. So it's got a few questions. Now it needs to know what the domain is about. We have everything now. And again, Claude.md, it's got all the different files or systems. It's really cool, really handy. So I think the first one we should be, should be a personal life assistant that we're going to use to leverage to make big decisions about our life. And we can just feed it that information. So let's come down and give Claude that information. So tell me the first domain, drop any file you want to into raw or ingest, and do a real first run. Want me to get in it to get the version ready to rock and roll. So I'm going to say something like, hey, the dude, what I would like to do is do a file, basically just on making better decisions and running my personal life. For example, there's an article I read on high agency and I would like it to be, we're going to call this one a general productivity life decision-making wiki. I'm going to drop in information about me, who I am, what I'm about, just relevant stuff. And then also things and mechanisms of thinking that I find that are valuable that I want you to integrate for different things. What I'm first of all going to do is just drop in a URL to you. And I want you to go ahead and scrape everything in there, images and also text. But obviously in the future, you can use web clippers for that. Let me know when you're ready for the first URLs. Awesome. So I'm going to send this one off and I'll show you that hack in just a second, but this is an example of an article on high agency. It is a fantastic read. My buddy Nick shared this with me, really, really highly recommend it. What I'm going to do is grab this link here, come back over. I'm going to drop it in right there. It's ready for it. LifeOS sounds like a great name. Let's start building up that personal wiki. Awesome. I'm going to drop that in there as well. And then I could probably just get Claude as well to say, oh, and by the way, just add in all the information about me to this life wiki, personal folders. If you go to my business folder, it has lots of information about me, business information, add all that in there as well. Wonderful thing. If you've got code on your laptop and different folders that cover this, it can just go and grab that data. Now the hack, by the way, and it's something that Carpath you mentioned is good, is that Opus itself has a web clipper. So what we're going to do for that is click on these dots at the top right, come down to extensions. And in fact, you're looking for the web store itself. And basically we're going to search for Obsidian and this is the Obsidian web clipper like that. Can you come down, add this to brave, uh, we go and just add that as an extension. And then personally what I like to do, if you come on these dots, come down over to extensions, click on manage extensions, find your Obsidian web clicker, click on details and just make sure you pin this to the toolbar like so. And now you can use that web clicker when you want to. So for instance, if I was here, if I click on a little Obsidian, you can see, I can now actually add this to Obsidian and you can basically specify everything that you want to. Now, one thing to call out with the Obsidian web clipper is if you click on the Obsidian logo here, for example, and then you click on this, this will then be the settings. So effectively what we can do is add all of the vaults here that we want to save to. So I might have a personal operating system. I might have a YouTube growth, LinkedIn growth, whatever it is. And what you can do is create a brand new template, give what you want to. So I could say personal operating system, whatever it is, you go through this whole thing, you can save it and you can add, this will then actually add it directly to your actual folder itself. You just name the location of the folder and then that's how you dynamically add everything over there. So now if I come back over to anti-gravity and now Cloud Code's done everything for me, I've got all the raw files. You can see it on my left-hand side now. It's not only basically grabbed the thing that I wanted, but it's also added loads of specifics. So if I click on assets, for example, it's got images. These are all from the article. So I've got all these downloaded, all saved. Very, very cool. And in fact, if I come over now to Obsidian, which I have right there, you can see I've got the high agency George Mac. Now I can vectorize everything that I want to. And if I want to see the whole node view, all I would simply do is come over here, check this out, open a graph, and I can actually do anything I want to. So with this in mind, I can now physically start asking it a question. And bear in mind, for me, this has got everything about me, decisions, mental models, people you literally know. Okay. These are mental models. So it's kind of adapted for it to go ahead and use and do various different things. It's got decisions here, which are going to be all the decisions that I make about me. Loads of really cool things that we can then piece together. And the more we use it, the better it gets. Now I've come back over here. Now you can, at this point, build a search engine where you can ask it questions, make it a beautiful interface, do whatever you want to. But honestly, I don't think it's worth your time to get started. You have a search engine. His name is Claude. He lives here. So you can have your master folder open up in your environment with all of your subfolders within it and just chat to it here. And then once you exceed to a certain level and you want to add different bells and whistles, we can go ahead and we can do that. And you get to a point where something like this, which is what I built my community. If I ask, for example, what is anti-gravity? How do I set up this is all, but this is all trained on every YouTube video I've ever had or my school community, everything you name it. So you can start building fancy stuff like this for yourself with your personal operating system, but it just wouldn't do it off the bat. Just interest to save me a time. You can just ask it to do that. You can give it API keys. You can do anything that you want to. So what we can do now is just test that it works. So I'm going to say, Hey, based on everything in this folder, um, I am considering how I can be more effective at my job, how I can help my community and to do that, I would love to know strategies of how to create more time in a day. Um, let me know your thoughts. And in addition to that, um, any of the files that you referenced on the left hand side in making that decision. So I'm just asking a question based on its knowledge, based on what's called the more information that we add to this, the better and richer that it physically gets. And if I come out and I say something like, Hey, this thing isn't true anymore, it will start to iron out those contradictions. And you can see, it's got a list of files of reference down there. So now you start to build up this wiki and just follow that process for So the system is awesome, but it has limitations limitations. I haven't seen anyone else talk about. And if you don't understand where this falls down, you're not going to get the best out of this memory system. So let's go through the obsidian rag limitation that I've not seen anyone really cover and how you actually build this into a system that is more robust and give you better long-term memory. I've thought very long and hard about this and I'll put all the data and research into it. So first of all, if you're just using obsidian rag with cloud code, it breaks in five places, or you need to be aware of a stress test. Number one is your cloud.md grows over time. This is also true for your index file. So if you think that every single thing you have in your index is like 75 tokens, at 10 files, 750, at 10,000 files, you can see how your token window just for looking through the index page grows exponentially. And so does your token cost for using this system. So there's a tax you must pay when going through that, that I think is just important to be aware of. There's no semantic search. It doesn't do it semantically. So it goes by the topic of the thing rather than the semantics around it. Some reason can go stale with callback drift. Your million token window can fill up a bit quicker this way. And it's not really built for very large data sets. This is built for more small to I'd say early medium sized data sets. Said another way, it can feel magical at a hundred, but eat you alive at 10,000. Well, Jack, I don't think I have 10,000 files. Cool. Are you sure? I mean, with these things, if you're going to invest in a memory system, it's really important that you have something that can scale with you over time. And it's just worth bearing in mind on what your tokens per query might look like. These are just figurative examples, but you can see the kind of difference. The more files you add, the more expensive it can actually get. Obsidian scales linearly, Pinecone stays flat. So these are rough order of magnitude estimates for this capacity pattern, not measured benchmarks. And the shape of the curve is what matters. So it's just worth bearing in mind and calling out just so you understand what's happening. So here's an interesting one. If I had YouTube transcripts, this is something that is better off in my view, actually living in a Pinecone database. So a good way to think about it is that Obsidian is for the stuff that changes and Pinecone is for the stuff that doesn't. You actually don't have to use Obsidian if you don't want to. I think one of the core benefits is the fact that you can clip and bring it in, but I'll explain more on that in a second. So let's say for example, we have 184 YouTube transcripts. Giving that to Obsidian, I think is not a good idea. That's way better off on Pinecone and I'll explain why that's the case. So think about this. If you're putting it through Obsidian and using Claude Opus 4.6, Claude has to read, even Apple's getting involved now, the entirety of the context window. First of all, to understand it, then it has to type it back out to the various different sections and summaries. Pinecone, for example, runs a script. It runs a query where Claude's not even involved and the embedding cost is like a hundred times cheaper based on the embedding model that you're using. So your tokens at point of ingestion is way lower. So it's saving you a lot to put it into Pinecone rather than Obsidian in this case. So two systems, different physics. So Obsidian RAG, and this is just a great way to physically understand what happens when you're physically using it. And this might sound technical. This is so helpful to understand because when you understand these systems work, you can build anything. So when we're using Obsidian, the session begins, Claude code loads the Claude.md automatically. And that's the schema rulebook for how everything works. You're going to ask it a question. Claude reads the index.md. Okay. And it's got a catalog of every page that you got, one line summary. And it just says, look, well, Jack's asked me about the cloak of invisibility. Well, you know, Harry Potter probably sounds like great one to read. So does this, so does this, so does this. And it goes so on and so on and so forth. Claude uses the retool to load the specific files. The page is cross-referenced with other video obsidian-style wiki links, and then Claude reasons over what is loaded and writes back updating notes, adding new links and appending to your log. So Obsidian lets Claude reason over the structure of how you've set this up. It's a graph. It can answer, you know, which hypothesis did I revise because of customer five. Pinecone doesn't do that. By contrast, Pinecone ingestion is one time. Vectors are stored in Pinecones and index. You attach the exact original text. You can ask it a question. You know, Claude just sends it over there. Pinecone runs an approximate, what we call nearest neighbor search, which finds the top K. Don't worry what that means. It's not, it's just like a technical gobbledygook for like what's most similar very, very quickly. And it returns the exact text. We always discuss this as a librarian and rather than pulling the entire text for Harry Potter, it just grabs a specific paragraphs that you need to actually understand it. So Pinecone gives Claude perfect recall of massive archives, exact text, flat cast, but it's similarity only no graph, no reasoning over the relationships. So think of it like this. Claude.md is who I am. Pinecone is what I've said. Obsidian is how I think. Now there are three layers to this, not two, because you'll have your Claude.md file in your projects based on how you're physically using this, unless you're just querying it like a search engine. Now, what are they best for? Basically Claude.md is your identity, voice rules, read first instructions, Obsidian rag, active projects, decision logs, idea gardens, anywhere that structure matters. And then Pinecone is going to be transcripts, research archives, books, anywhere exact recall physically matters. Said another way, the magic word for Obsidian is reasoning, not memory. Claude.md is you, your identity. Pinecone remembers you. It's how you recall and Obsidian thinks with you reasons over structure. So if you imagine, for example, we're going to work, one of the best ways to think about this is Claude is like your identity badge, your Claude.md. It's your name, your role, your rules, read before anything else, and it never changes midday. Your workshop is where you actively build, where you're doing hard work. You know, notes are linked by string, reasoning happens there. And then warehouse is Pinecone in this example, where we put all our big stuff, all our big files, all our video transcripts, all our archive of every email we've ever sent, we'd throw and let that live inside the warehouse. So with this in mind, what do you actually do? What is the complete structure to run this sustainably so that you have a system that never ever forgets? Well, the way that you understand this relationally, okay, you have your identity in your Claude.md files. If you're using the Claude app, that's the about me section. Again, you can have that based on projects. Obsidian can have all the little mini things, the little projects that you're thinking about, that is how you think. Pinecone is your full archive of all your history. And then NetworkLM is something we bring in to do deep dive and research. So for example, if we're in Claude or a project and we're learning about the manufacturing process of white bottles, I could say, Hey, it's been a notebook with 200 resources explaining everything. Then we can bring that back over to our laptop and even throw that into our long-term memory. If we ever want to recall it and anything that we're physically doing and understanding how these four connect relationally is really important. So for example, if I was taking my entire YouTube transcripts, I'd saw that in Pinecone, my warehouse, my long-term storage, the place where the information is never ever going to change. And the key thing to understand here is that you can connect Pinecone so easily now to Claude. It is ridiculous to the point where you can just ask it a question and Claude will reach over to Pinecone, find the answer and bring it back. And for every project that I've physically done, and I can even show you this now, I showed you earlier in my AI with Jack dashboard. Like I find stuff like this where I can ask questions and I have processes here that will automatically on a daily basis, scrape the transcript for everything inside my YouTube channel. And it also works inside my community, right? Where people saying, Jack, I need to find something like, I don't know, let's say pricing AI services, whatever the thing is, this scrapes everything daily. And this all lives in Pinecone and it's great. And you can click on this, come and check out the community. And again, it's like, there's a lot of different stuff that exists inside the community that you can come and grab. And all of that is linked in Pinecone. So I find it on a day-to-day basis, wonderful. But the thing about Obsidian and the Obsidian index system is it brings a completely different value set. You just need to understand what you're using it for and where you realistically want to introduce Pinecone to support that. And so without a long-term memory system, we're leaving value on the table, which is why the next thing that we need to do is learn how to integrate Pinecone into our system so we can secure information forever, which we're going to do by watching this video right here.