Indexing Billions of Text Vectors, Hacker News

re (******************************************** (s)t(a) ******************************* (u) ****************************** (r) ****************************** (r) ************************************ (a) ******************************** (n) ****************************** (t) [ibest​,irestaurant​,iof​,imunich​] ********************************** (************************************** ( ) ************************************** (v) ******************************************** (****************************** (i) **************************** (i) ************************** (o) ****************************** (f) ****************************** (******************************************** (************************************** ( ) **************************************$(**************************************[ibest​,irestaurant​,iof​,imunich​] ********************** [ ibest​,irestaurant​,iof​,imunich​] ********************** ************************ [10] ************************** () ************************************************** [ ibest​,irestaurant​,iof​,imunich​] **********************i [10] ************ [ ibest​,irestaurant​,iof​,imunich​] **************************** [10] ************************** () **************************************** (b) ************************************** (e) ************************************* (s)t (************************************(************************************** (**************************** [ ibest​,irestaurant​,iof​,imunich​] ************************************[ibest,irestaurant,iof,imunich] ************************** [ibest,irestaurant,iof,imunich] **************************[ibest,irestaurant,iof,imunich] **************** (****************************************************************************** (************************************** ********************************** (************************************** [ibest​,irestaurant​,iof​,imunich​] ************************** (v) ************************************ [10] ************************** () ****************************************[ibest​,irestaurant​,iof​,imunich​] ********************** ************************** i) ************************************* [10] ************************** () ***************************************************************** ************************* (r)es (t) ************************************ (a) ******************************************** (u)ra (n) ************************************ (************************************(************************** ****************** ************************************** (************************************************************************** [ibest,irestaurant,iof,imunich] **************************** () ************************************************** ( )********************************** () **************************************** [ibest​,irestaurant​,iof​,imunich​] ********************** (v) ************************************** ************************ [10] ************************************** ********************** () [10] (i) ************************************ [ ibest,irestaurant,iof,imunich] **************** ********************** (o) f (************************************(************************************** (**************************** [ ibest​,irestaurant​,iof​,imunich​] ************************************(**************************************************(************************************************************************** [10] ************************************************************* (************************************** (**************************** ( ) [ibest​,irestaurant​,iof​,imunich​] ********************** (************************************ [ibest​,irestaurant​,iof​,imunich​] ************************** (v) ************************************ [10] ************************** () ****************************************[ibest​,irestaurant​,iof​,imunich​] ********************** ************************** i) ************************************* [10] ************************** () ***************************************************************** ************************* (m) u [10] **********************n (i) ************************************ [7] ******************************** (h) (************************************[10] ********************** [ibest​,irestaurant​,iof​,imunich​][ibest,irestaurant,iof,imunich] **************************** [ibest,irestaurant,iof,imunich] **************************** ******************************************** [10] ******************************************** [ibest,irestaurant,iof,imunich] ************************** (************************************ the final size of the query representation naturally depends on the combined number of words in all queries, but for 4 billion of our queries the total size ends up being around GB (including the word vectors). In other words, we see a reduction in size by more than (******************************************************************************************************************************************************************** There is one concern that still needs to be addressed. For a single search, we need to visit around 728 - 800 nodes in the graph. Each node has 75 - neighbors. Consequently, we need to compute the distance from the input query vector to - 09320 of the vectors in the index and before that, we need to generate the vectors. Is the time penalty for creating the query vectors on the fly too high? It turns out that, with a reasonably new CPU, it can be done in a few milliseconds. For a query that earlier took a millisecond, we will now need around 5 ms. But at the same time, we are reducing the RAM usage for the vectors by (*********************************************************************************************************************************************************************So far, we have focused solely on the memory footprint of the vectors. However, after the significant size reduction above, the limiting factor becomes the index structure itself, and we thus need to consider its memory requirements as well. [11] The The graph structure in Granne is compactly stored as adjacency lists with a variable number of neighbors per node. Thus, almost no space is wasted on metadata. The size of the index structure depends a lot on the construction parameters and the properties of the graph. Nevertheless, in order to get some idea about the index size, it suffices to say that we can build a useable index for the 4 billion vectors with a total size of around GB. This might be acceptable to use in-memory on a large server, but we can actually do even better. (Figure 3: Two different configurations of RAM / SSD placement) ***************************************************************************** An important feature of Granne is its ability to memory map [ibest​,irestaurant​,iof​,imunich​] [ibest​,irestaurant​,iof​,imunich​] ************************ the index and the query vectors. This enables us to lazily load the index and share the memory between processes. The index and query files are actually split into separate memory mapped files and can be used with different configurations of SSD / RAM placement. If latency requirements are a bit less strict, by placing the index file on a SSD and the query file in RAM, we still obtain a reasonable query speed without paying for an excessive amount of RAM. At the end of this post, we will see what this trade-off looks like. [i_{mathrm{best}}, i_{mathrm{restaurant}}, i_{mathrm{of}}, i_{mathrm{munich}}] ******************************************************************** Improving Data Locality In our current configuration, where the index is placed on a SSD, each search requires up to 200 - 800 read operations from the SSD. We can try to increase the data locality by ordering elements whose vectors are close so that their HNSW nodes are also closely located in the index. Data locality improves performance because a single read operation (usually fetching 4 kB or more) becomes more likely to contain other nodes needed for the graph traversal. This, in turn, reduces the number of times we need to fetch data during a single search. (********************************************************************************(************************Figure 4: Data locality reduces the number of fetches It should be noted that reordering the elements does not change the results, but is merely a way of speeding up the search. This means that any ordering is fine, but not all of them will give a speed up. It is likely very difficult to find the optimal ordering. However, a heuristic that we have used sucessfully consists in sorting the queries based on the mostimportant word in each query. [ibest,irestaurant,iof,imunich] At Cliqz, we are using Granne to build and maintain a multi-billion query vector index for similar query search with relatively low memory requirements. Table 1. summarizes the requirements for the different methods. The absolute numbers for the search latencies should be taken with a grain of salt, since they are highly dependent on what is considered an acceptable recall, but they should at least give a hint about the relative performance between the methods.(**************************************************** [ibest,irestaurant,iof,imunich] ************************************ [4] Baseline$