I’m writing a new vector search SQLite Extension

Would this implement indexing strategies like HNSW? Linear scan is obviously great to start out with, and can definitely be performant, especially if your data is in a reasonable order and under (say) 10MB, so this shouldn't block a beta release.

Also do you build with sqlite-httpvfs? This would go together great https://github.com/phiresky/sql.js-httpvfs

I originally added sqlite-vss (your original vector search implementation) on Langchain as a vectorstore. Do you think this one is mature enough to add on Langchain, or should I wait a bit?

Love your work by the way, I have been using sqlite-vss on a few projects already.

This is awesome! We used Qdrant vector DB for a local AI RAG app https://recurse.chat/blog/posts/local-docs#vector-database, but was eyeing up sqlite-vss as a lightweight embedded solution. Excited to see you are making a successor. Would be interested to learn about latency and scalability benchmarks.

Looks really nice, but the only concern I had — how does the perf compare to more mature libraries like faiss?

Interesting idea.

Do you intend to compress the vector storage in any way and do you intend to implement your own vector search algorithms or reuse some already optimized libraries like usearch?

Could this be implemented in Rust? Does that project (sqlite-loadable-rs) support WASM?

https://observablehq.com/@asg017/introducing-sqlite-loadable...

Really nice to see Wasm there, usually vector search in sqlite wasn't available in the browser.

Did you ever consider making it syntax compatible with pgvector, in order to have a common SQL vector DSL? I am sure the benefits are much smaller than the disadvantages, but I'm curious if it's possible.

Very exciting and I can’t wait to try it. Dependency issues are the reason I am currently not using sqlite-vss (i.e. not using an index for my vector searches), but man do I wish for better performance.

At the risk of fulfilling a HN stereotype, may I ask why you ditched Rust for this extension?

Awesome work!!

* Which distance functions does this support? Looks like it supports binary vectors already -- is Hamming distance supported?

* How does the performance compare with sqlite-vss? I'm curious about the profiling numbers -- both in terms of query speed, as well as memory usage.

Overall, this looks absolutely fantastic, and I love the direction you're heading with all of this.

Though initially, sqlite-vec will only support exhaustive full-scan vector search. There will be no "approximate nearest neighbors" (ANN) options. But I hope to add IVF + HNSW in the future!

I think this is 1000% the correct approach -- kudos for not over-complicating things initially! I've shipped on-device vector search (128-bit binary vectors, Hamming distance) and even with a database size of 200k+ entries, it was still fast enough to do full brute-force distance search on every camera frame -- even running on crappy phones it was fast enough to get 10+ fps, and nicer phones were buttery-smooth. It's amazing how frequently brute-force is good enough.

That said, for implementing ANN algorithms like HNSW and whatnot, my first thought is that it would be slick if these could be accomplished with a table index paradigm -- so that switching from brute-force to ANN would be as simple as creating an index on your table. Experimenting with different ANN algorithms and parameters would be accomplished by adjusting the index creation parameters, and that would let developers smoothly evaluate and iterate between the various options. Maybe that's where your mind is going with it already, but I figured I would mention it just in case.

Overall, awesome writeup, and fantastic project!!

Binary, quantize each dimension to +1 or -1

You can try out binary vectors, in comparison to quantize every pair of vectors to one of four values, and a lot more, by using a FAISS index on your data, and using Product Quantization (like PQ768x1 for binary features in this case) https://github.com/facebookresearch/faiss/wiki/The-index-fac...

Author here - ya "binary vectors" means quantizing to one bit per dimension. Normally it would be 4 * dimensions bytes of space per vector (where 4=sizeof(float)). Some embedding models, like nomic v1.5[0] and mixedbread's new model[1] are specifically trained to retain quality after binary quantization. Not all models do tho, so results may vary. I think in general for really large vectors, like OpenAI's large embeddings model with 3072 dimensions, it kindof works, even if they didn't specifically train for it.

[0] https://twitter.com/nomic_ai/status/1769837800793243687

[1] https://www.mixedbread.ai/blog/binary-mrl

BTW, the "curse of dimensionality" technically refers to the relative sparsity of high-dimensional space and the need for geometrically increasing data to fill it. It has nothing to do with storage. And typically in vector databases the data are compressed/projected into a lower dimensionality space before storage, which actually improves the situation.

Only public SQLite APIs! So no need to append to amalgamation.

I'm a big fan of your wazero SQLite bindings! I actually plan on providing 1) CGO bindings to sqlite-vec and 2) a custom WASI build sqlite-vec that can be used in go-sqlite3 directly. My plan was to build a sqlite3.wasm file using the build scripts in your repo. If you did want to support it in your project directly, I think you could just drop the sqlite-vec.c/h file in go-sqlite3/sqlite3 and be good to go

Re incremental Blob I/O: I learned that the hard way! It's definitely the limiting factor on query speed for sqlite-vec. I've found that keeping the chunks relatively low in size (low MB's) and increasing the page_size strikes a good balance, but page_size in particular has consequences. PRAGMA mmap_size also helps out a ton, since it seems to keep pages in memory and makes overflow lookups faster, but that of course means a ton more memory usage. It's a difficult balance!

Definitely interested in this for your wasm Go bindings!

Keep in mind, a presentation (text, or otherwise) is an art. Culture, background, experiences and ideologies influences art.

... and to address your tone, did you know, you can give feedback with empathy?

Why do people write comments like this? What does "grow up, people" add to anyone's understanding of the topic? It's so annoying. Is it supposed to make you look better? It doesn't.

Language evolves. Grow up yourself.

It could! It's based on the official SQLite WASM build, so you can use the same persistent options[0] that are offered there. Not sure if IndexedDB is specifically supported, but localStorage/OPFS VFS is available.

[0] https://sqlite.org/wasm/doc/trunk/persistence.md#kvvfs

Thanks for the kind words!

I started with code first — the extension itself is already mostly written[0]. But it's one of those "20% effort for 80% of the work," where the last 20% I need to write (error handling, fuzzy tests, correctness testing) will take 80% of the time. But people already have questions about the current status of `sqlite-vss`, so I figured this "work-in-progress" blog post could answer some questions.

Though I do like the idea of starting with docs first! Especially with SQLite extensions, where it all really matters what the SQL API looks like (scalar functions, virtual tables, etc.). I definitely did a lot of sketching of what the SQL part of sqlite-vec should look like before writing most of the code.

[0] https://github.com/asg017/sqlite-vec/blob/main/sqlite-vec.c

Yes it is! Sorry for not following up there. Actually, when I first read that ticket, it started me down the rabbit-hole of "how can I make sqlite-vss" better, which eventually turned into "I should make sqlite-vec." So thanks for helping me go down this path!

With sqlite-vec's builtin binary quantization, you should be able to do something like:

  CREATE VIRTUAL TABLE vec_files USING vec0 (
    contents_embedding bit[1536]
  );

  INSERT INTO vec_files(rowid,contents_embedding) 
    VALUES (
      (1, vec_quantize_binary( /* 1536-dimension float vector here*/))
    )

Thanks for the kinds words! Really cool to hear that people were able to use sqlite-vss, lol.

Oooh I haven't thought of seperating the SQLite stuff and the vector search stuff. Good idea, will keep that in mind!

As of now for this beta release not really, but after v0.1.0 is released, will definitely have contributing guides + issues that people could tackle!

I definitely plan to! I have a much larger list of SQLite extensions I've built here: https://github.com/asg017/sqlite-ecosystem

Here's a few other references you may enjoy if you wanna learn more about SQLite extensions:

- The single source file for sqlite-vec: https://github.com/asg017/sqlite-vec/blob/main/sqlite-vec.c

- sqlean, a project from Anton Zhiyanov which is good base of great SQLite extensions: https://github.com/nalgeon/sqlean

- The official SQLite docs: https://www.sqlite.org/loadext.html

- The "hello world" SQLite extension example: https://www.sqlite.org/src/file/ext/misc/rot13.c

I plan to have v0.1.0 in about a month or so! Definitely will include a ton of docs and a quickstart guide. There's an undocumented pip package "sqlite-vec" that you might be able to use now, if you wanted to call it from your Python "Agent Backend" directly.