twitterragllmembeddingtokenizer
Traditional Tokenizers Are Dea
Traditional Tokenizers Are Dead. Here's Why.
DeepSeek-OCR just dropped something huge: it compresses documents into visual tokens with 7-60x efficiency while preserving ALL information—charts, formulas, layouts intact.
More importantly, it reveals why current RAG architectures are fundamentally broken.
Fonte originale
https://x.com/milvusio/status/1981741219383517686