Another (simpler?) approach is to also split your chunks into sentences. So you’ll end up with chunk embeddings and sentence embeddings. Now you can do sentence level search. And also distill chunks down to their most relevant sentences at query time before you dump ‘em into your LLM’s context window. If you use Sentence Transformers, you get your chunk embeddings for free, because they are just the np.mean of the embeddings of all the sentences in that chunk.
I'm not sure how it's going to solve the same issue. If your chunk is too short for the context, it's still going to miss the meaning if I understand the description correctly. As in "it's" 2 paragraphs in is still going to miss context, whether you embed chunks or sentences
This is an interesting development. A couple blog posts exposing on late chunking
https://weaviate.io/blog/late-chunking
https://jina.ai/news/late-chunking-in-long-context-embedding...
The late chunking idea originates from ColBERT, an embedding technique from 2020 https://arxiv.org/pdf/2004.12832
Another (simpler?) approach is to also split your chunks into sentences. So you’ll end up with chunk embeddings and sentence embeddings. Now you can do sentence level search. And also distill chunks down to their most relevant sentences at query time before you dump ‘em into your LLM’s context window. If you use Sentence Transformers, you get your chunk embeddings for free, because they are just the np.mean of the embeddings of all the sentences in that chunk.
I'm not sure how it's going to solve the same issue. If your chunk is too short for the context, it's still going to miss the meaning if I understand the description correctly. As in "it's" 2 paragraphs in is still going to miss context, whether you embed chunks or sentences