When a Knowledge Base Isn't Enough — You Need a Layer

Andrej Karpathy recently posted about how he used LLMs to build a personal research knowledge base. In short, he drops source material into a folder, has an LLM compile it into structured markdown, queries it through an agent, and uses an LLM to keep the whole thing maintained.
Several CEO friends followed Karpathy’s recipe and started building personal knowledge bases – either for themselves or for their companies. They found these knowledge bases incredibly valuable and asked me how they can implement them in their organizations.
The thing is, a knowledge base like Karpathy’s works when you only have one person contributing, curating, and accessing it.
Once you have a content repo that you want to share with others, what you actually need is a knowledge layer. A knowledge layer sits between your content and your AI tools. It provides the validation and governance that addresses four critical areas: contribution, control, visibility, and privacy.
Contribution: What’s in the knowledge base and who put it there?
Control: Who can access what information? How can we improve the quality? Where is the content stored?
Visibility: What questions are being asked? What answers are being given? Where do the answers come from?
Privacy: How sensitive is the content being added? Is it proprietary? Is it leaking into public models?
For example, if I’m putting in the content, I’m the only person who can assess if the system is returning bad answers. As soon as someone else starts using the system, they have no ability to judge the quality of the answers.
These insights come from something we experienced firsthand building agentic systems at AWS. AI tools are only as accurate as the knowledge they're grounded in. But good content isn’t enough. You need the governance items mentioned above to scale beyond single-user systems.
A Knowledge Layer Makes Your Knowledge Base Even Better
Not only is a knowledge base unable to scale beyond a single user, it also risks becoming less accurate with every document you add.
The reason is conflict. All organizations have documents that disagree. A 2021 procedure contradicts a 2024 update. The Vancouver office's policy differs from Miami's. The official handbook says one thing, actual practice says another. A knowledge base doesn't know this. It retrieves whatever is semantically nearest to the query, hands the contradicting passages to the LLM, and lets the model average them into a confident, yet often wrong answer.
The more you feed it, the more conflicts surface, and the worse the answers can get.
A knowledge layer surfaces conflicts and loops in your experts to resolve them. If there is insufficient knowledge on a certain topic, the knowledge layer says so and automatically routes the question to a designated expert to review.
That’s Why I’m Working on Trovia

A knowledge layer sits between your knowledge base and your AI tools.
Trovia is a knowledge layer which makes your knowledge base actually work for your organization. With Trovia, you can have multiple contributors, control permissions, and get visibility into questions asked and answers given. Moreover, the content is secured in a way that public models can't access it.
Karpathy's setup works for one person. It doesn’t work at scale where you need governance to roll out an AI system your organization can trust. As he says in his post, "there is room here for an incredible new product instead of a hacky collection of scripts."
Trovia is that product.
If you're building a knowledge base and you want to go to the next level with a knowledge layer, drop me a note or learn more at trovia.ai.
Aaron Rallo is a multi-exit startup CEO, Board Member and Forbes contributor. He is co-founder and CEO of Trovia.