While I agree that training your own model weight is a terrible idea, I need to point out a false assumption.
“There just is no way to reliably inject knowledge in the weights of already trained LLMs”
I developed an inference engine that uses a sparse auto encoder to identify and classify clusters in a model’s activation forest. I use this to detect a lack of knowledge for a given domain using a small classifier model to graft virtual knowledge domains when the backbone model doesn’t have the knowledge required to respond. I developed this in an effort to increase intelligence in small models for robots. Robots need small models for speed, but need to be savants for specific knowledge domains. I additionally added a second way using LoRa to graft knowledge domains on the fly as well.
The good news is that it works. The bad news is while it adds knowledge, it does not add reasoning. To increase a model’s intelligence, you need more layers.
More importantly, I learned exactly how hard and expensive it is to even train a single knowledgeable domain.
RAG and harness engineering are the shortest path to what most organizations can get a cost effective benefit.
I’ve taken your advice to publish. See greenhaw.net and take a look at CALM Horizon under Artificial Intelligence and also the Artificial Moral Agent. I have a number of academic style research papers in review and will be releasing the code for both to open source in GitHub soon. Feedback and questions are welcome!
I plan to release it as MIT open source. Would you be willing to take a look at what I’ve documented and or built? In addition to virtual knowledge domains, the inference engine supports Chunked Attention (4 tokens per forward pass), a hazard probe and artificial moral agent (safety during inference, not a guardrail), tokenized context management, comparable to Ollama/llama.cpp for speed with everything turned on, and a workbench for interpretability and steering using j-lens and what I call the Activation Forest Editor. I think it is important work, but publishing is not my strength.
If post-training doesn't work that well, could companies do midtraining on an already pretrained base model? I think cursor did something like that (before getting acquired by grok)
This is a great piece Devansh. I would say another factor to consider when training your own model is the heavier governance work it comes with, including the more onerous obligations under data protection law and the EU AI Act. I am thinking of writing something on that this week.
Refresh cost is the line that sinks most of these business cases. A fine-tune frozen on last quarter's policy docs needs another run every time the docs change, while the retrieval baseline picks up the new doc on the next index build. Teams that price the first training run and not the fourth get surprised.
The Gekhman line makes the retrieval case for me. If facts mostly come from pretraining, a search index may be the cheapest place to keep them current.
Facts versus behavior is the part I'd underline. A fact is cheap to hand over at question time — change one document and you're done — while baking it into the model means reprinting the library whenever a price changes, then rechecking everything that shifted along with it. That's also why beating a model with no documents in front of it proves so little.
While I agree that training your own model weight is a terrible idea, I need to point out a false assumption.
“There just is no way to reliably inject knowledge in the weights of already trained LLMs”
I developed an inference engine that uses a sparse auto encoder to identify and classify clusters in a model’s activation forest. I use this to detect a lack of knowledge for a given domain using a small classifier model to graft virtual knowledge domains when the backbone model doesn’t have the knowledge required to respond. I developed this in an effort to increase intelligence in small models for robots. Robots need small models for speed, but need to be savants for specific knowledge domains. I additionally added a second way using LoRa to graft knowledge domains on the fly as well.
The good news is that it works. The bad news is while it adds knowledge, it does not add reasoning. To increase a model’s intelligence, you need more layers.
More importantly, I learned exactly how hard and expensive it is to even train a single knowledgeable domain.
RAG and harness engineering are the shortest path to what most organizations can get a cost effective benefit.
You should publish your work, it seems interesting
I’ve taken your advice to publish. See greenhaw.net and take a look at CALM Horizon under Artificial Intelligence and also the Artificial Moral Agent. I have a number of academic style research papers in review and will be releasing the code for both to open source in GitHub soon. Feedback and questions are welcome!
I plan to release it as MIT open source. Would you be willing to take a look at what I’ve documented and or built? In addition to virtual knowledge domains, the inference engine supports Chunked Attention (4 tokens per forward pass), a hazard probe and artificial moral agent (safety during inference, not a guardrail), tokenized context management, comparable to Ollama/llama.cpp for speed with everything turned on, and a workbench for interpretability and steering using j-lens and what I call the Activation Forest Editor. I think it is important work, but publishing is not my strength.
If post-training doesn't work that well, could companies do midtraining on an already pretrained base model? I think cursor did something like that (before getting acquired by grok)
It's basically the same thing. New term, but still no proven ROI.
This is a great piece Devansh. I would say another factor to consider when training your own model is the heavier governance work it comes with, including the more onerous obligations under data protection law and the EU AI Act. I am thinking of writing something on that this week.
Refresh cost is the line that sinks most of these business cases. A fine-tune frozen on last quarter's policy docs needs another run every time the docs change, while the retrieval baseline picks up the new doc on the next index build. Teams that price the first training run and not the fourth get surprised.
The Gekhman line makes the retrieval case for me. If facts mostly come from pretraining, a search index may be the cheapest place to keep them current.
Facts versus behavior is the part I'd underline. A fact is cheap to hand over at question time — change one document and you're done — while baking it into the model means reprinting the library whenever a price changes, then rechecking everything that shifted along with it. That's also why beating a model with no documents in front of it proves so little.