New ask Hacker News story: Why large context windows are a good thing for vector DBs (and vice versa)
Why large context windows are a good thing for vector DBs (and vice versa)
7 by bobvanluijt | 1 comments on Hacker News.
In a bunch of conversations, people asked me how large context windows supposedly pose a threat to vector databases. Some thoughts from somebody who works on a vector DB why this is not only _not_ a threat but actually makes vector DBs _better_. 1) storing the whole dataset in the context window (as the “pessimists” describe it) is quite impractical because the larger the dataset, the more complex it becomes to operate. Especially for bigger datasets, you need some form of selection through retrieval (even if that turns into a 100k token corpus, which probably is not the case). 2) a bigger context window is excellent for use cases like generative feedback loops (where the generative output is feedback looped back into the vector DB). 3) it seems that people sometimes forget what it means to operate/create a DB. It's way more than just storing a bunch of blobs and -in this case- vectors, let alone shoehorning data into a context window. 4) A 100k context window is not much when it comes to serious workloads. Also, the larger the context window, the harder to maintain all the ops _around_ it. A solution to manage this is (you've guessed it) is a vector DB. 5) vector DBs and LLMs (or any other model for that matter) excel _together_, I.e., easier-to-operate models (e.g., CPU inference), and larger context windows increase the capability of the vector DB rather than decrease it (see the generative feedback loops example above). So, no worries, we're all good, and can get back to work...
7 by bobvanluijt | 1 comments on Hacker News.
In a bunch of conversations, people asked me how large context windows supposedly pose a threat to vector databases. Some thoughts from somebody who works on a vector DB why this is not only _not_ a threat but actually makes vector DBs _better_. 1) storing the whole dataset in the context window (as the “pessimists” describe it) is quite impractical because the larger the dataset, the more complex it becomes to operate. Especially for bigger datasets, you need some form of selection through retrieval (even if that turns into a 100k token corpus, which probably is not the case). 2) a bigger context window is excellent for use cases like generative feedback loops (where the generative output is feedback looped back into the vector DB). 3) it seems that people sometimes forget what it means to operate/create a DB. It's way more than just storing a bunch of blobs and -in this case- vectors, let alone shoehorning data into a context window. 4) A 100k context window is not much when it comes to serious workloads. Also, the larger the context window, the harder to maintain all the ops _around_ it. A solution to manage this is (you've guessed it) is a vector DB. 5) vector DBs and LLMs (or any other model for that matter) excel _together_, I.e., easier-to-operate models (e.g., CPU inference), and larger context windows increase the capability of the vector DB rather than decrease it (see the generative feedback loops example above). So, no worries, we're all good, and can get back to work...
Comments
Post a Comment