zeekay
f5aaeb712e
Derived from Dgraph (github.com/dgraph-io/dgraph) at commit 64804bd2, Apache-2.0. See NOTICE. WHAT THIS IS. The graph engine as a Go library: posting lists, the DQL parser, the type and tokenizer system, the index and schema layers, and the wire types. 39,170 lines of the upstream's 166,337. THE STORE is github.com/luxfi/zapdb, a Badger fork that kept the package name, the subpackages (options, pb, y) and the API — Open, OpenManaged, DB, Options — so the 74 import sites are aliased rather than rewritten. THE CODEC is ZAP. zapdb serializes with ZAP, not protobuf: its pb package builds the ZAP types by default and the protobuf ones only under `-tags grpc`, a path that does not compile (table/builder.go calls pb.Marshal, which that tag removes). One call site in posting needed converting; every other type here is Hanzo's own protobuf and is untouched. NO TRANSPORT. An embedded engine dials nothing, so nothing here imports google.golang.org/grpc. Removed: 20 transport functions in x, the generated service stubs for Raft, Zero and Worker (1,657 lines), and the JWT helper — identity belongs to IAM. The core called none of them. NOT KEPT: worker, conn, raftwal, the cluster commands, the GraphQL layer and the test harnesses. `query`, the DQL execution engine, went with them because it depends on worker; parsing DQL is available here, executing it is not. The history of the upstream project lives at github.com/dgraph-io/dgraph. This repository begins at the derivation because three quarters of that history describes code this fork does not carry.
1.7 KiB
In Dgraph, we have the abiltity to specify an @index directive for the type vfloat. Here, we want to expand that capability to allow us to specify something like:
myAttribute @index("hnsw-euclidean","maxNeighbors":"5","maxLevels":"5","exponent":"6")
Roughly, this should be read as: I want to specify an HNSW-type vector index using euclidean distance as a metric with maxNeighbors, maxLevels, and exponent being declared as written, and with all other parameters taking on default values. (In the case of specifying an exponent, this should affect the defaults for the other options, but that is something that can be decided by the factory for the hnsw index)!
But this leads to some natural questions: How do I know what option types are allowed? And how do I interpret their types? For example, how do I know that I should interpret "5" as an integer rather than a string?
The solution will be to allow each factory to specify a set of "allowed option types". Hence, we get the AllowedOptions class, which specifies a set of named options and corresponding parsers for the given name.
Now, if we want to populate an Options instance based on the content of myAttribute @index(hnsw(metric:"euclidean",exponent:"6")),
we collect the key-value pairs: "metric":"euclidean" "exponent":"6"
And we can now invoke: allowedOpts := hnswFactory.AllowedOpts() myAttributeIndexOpts := NewOptions()
val, err := allowedOpts.GetParsedOption("exponent", "6") if err != nil { return ErrBadOptionNoBiscuit } myAttributeIndexOpts.SetOpt("exponent", val)
The final resolution of the "puzzle" in fact is to invoke allowedOpts.PopulateOptions(pairs, myAttributeIndexOpts) based on pairs being built from the collection of key-value pairs.