Move from annoy to Voyager?

[aarmey]

annoy seems to no longer be maintained, in favor of Voyager, with better performance:

https://github.com/spotify/annoy
https://spotify.github.io/voyager/python/

Would there be interest in moving PaCMAP to Voyager? I am happy to assemble a PR, if so.

[mathematicalmichael]

it's maintained, a PR for generating wheels for windows + other OS's for Python 3.12 + 3.13 has been tested and merged recently, and just needs to be released to pypi. I've published the wheels myself and have been using them in pacmap under a fork. There's nothing that needs "active maintaining" in annoy beyond releases / wheels - the nearest neighbor computation is implemented in C++ and hasn't changed in many years.

[aarmey]

I bring it up because this bug has been an issue for quite some time: spotify/annoy#682

This is the underlying issue causing #94

[mathematicalmichael]

I think this is just wheels vs source, since "macOS" seems to be a common theme, but I may be wrong.

In any case, voyager looks promising, and they have even more wheels for macOS than what I built for annoy (though fewer linux variants). All for the increased recall as well. I'd be happy to volunteer to review + test a PR.

[patcon]

While I wouldn't normally be about chasing the next shiny thing, it is a very valid point if even the maintainers are off using their newer project: https://github.com/spotify/voyager

Presumably all the speed-ups are happening in the other repo, and pacmap/localmap being faster for free, that would be nice (I say as someone who has some use-cases that take 5-10 minutes, and which are intended to drive a live web application, so speedup is meaningful improvement :) )

[aarmey]

I now have a working branch.

One potential downside is that voyager only supports Euclidean, inner product, or cosine distances. If Manhattan and Hamming distances are used, then this might not be a viable alternative...

[hyhuang00]

Thanks for the effort! This is indeed very interesting and perhaps a great solution to move things forward. @williamsyy Any thoughts on Manhattan distance? IMO Hamming has been the least used distance for a while.

[williamsyy]

Thanks @aarmey for the effort and really appreciate all the discussions! @hyhuang00 From my perspective, I am not too worried about the Manhattan distance missing, as I haven't seen a use case using Manhattan distance for a while, but I do see some Hamming distance onces (even though it should be totally the same as using Manhanttan distance) while dealing with binarized data visualization, but that could be a relatively rare case from my experience. I would prefer to keep annoy option as default since it supports more distance metrics, and add an alternative option as a backup (maybe provide a warning while using annoy).

[patcon]

Thanks for the consideration here -- before migrating to a new library that limits distance metric options, is it worth considering the bigger picture? I understand pynndescent to be a very well-supported library with tons of metrics.

Would it make sense to consider the larger playing field of libraries/algos

(Voyager uses hnswlib: https://ann-benchmarks.com/index.html#hnswlib )

[williamsyy]

Thanks for all the discussions here. The new version of PaCMAP: v0.9.0 with FAISS and Voyager as backend has been released in pypi.

[patcon]

THIS IS THE BEST

thanks @aarmey @williamsyy @hyhuang00!! Super stoked about these movements -- I need to step up my contribution plans, as I am building democracy-supporting things on top of pacmap and localmap that I really care about

https://patcon.github.io/valency-anndata/tutorials/aufstehen/
https://nodescription.substack.com/p/valence-swarming-and-perspective