Fantastic! There was a time when articles about "tell the model to X" were frequent. The latest of that kind is "make no mistakes". One of my favorites was the suggestion to add "ruminate" to a prompt to get more "thoughtful" responses. Magic? Nope.
I vaguely saw an L1 norm route change through a mind-bogglingly high dimensional space. Can't describe how much seeing that vague notion laid out so well here has cleared that up. How much of carefully constructed prompting is simply noise? 🤯
This reminds me of an old chemistry simulation paper "Diffusion maps, reduction coordinates and low dimensional representation of stochastic systems" by Coifman&al that took flattening distributions very seriously.
Looking forward to trying this out, especially on my smaller local models. They definitely consume a lot of thinking tokens on startup in loops, so this would be great.
Fantastic! There was a time when articles about "tell the model to X" were frequent. The latest of that kind is "make no mistakes". One of my favorites was the suggestion to add "ruminate" to a prompt to get more "thoughtful" responses. Magic? Nope.
I vaguely saw an L1 norm route change through a mind-bogglingly high dimensional space. Can't describe how much seeing that vague notion laid out so well here has cleared that up. How much of carefully constructed prompting is simply noise? 🤯
This is such a good question. I really want to explore this much further
This reminds me of an old chemistry simulation paper "Diffusion maps, reduction coordinates and low dimensional representation of stochastic systems" by Coifman&al that took flattening distributions very seriously.
I'll have to look into it, but flattening representations is a huge problem across the board
Looking forward to trying this out, especially on my smaller local models. They definitely consume a lot of thinking tokens on startup in loops, so this would be great.