Day 1 of being an imposter 😛
RoPE (Rotary Positional Embedding) is crazy good way to reduce the dimensional space, and compact more tokens in same context,
but at what cost?
The cost is – You risk losing the distance between tokens, leading to distint entities being misinterpreted as more related to each other than they actually are.
But if your data is prepared in clear boundaries, this might not cause issues you know ;D
Any better ideas to fit more context in same space? Drop in comments, would love to explore.
Stay Tuned for more ML talks from Not Yet ML Expert 😛
hashtag#LLM hashtag#Context hashtag#RoPE