This is retrieval, not cloud captioning
Queryable does not ask a remote language model to inspect a photo library. Its image encoder converts each available photo into a numeric embedding, while its text encoder places a natural-language query in the same vector space. Search is then a ranking problem: find the photo vectors closest to the query vector.
The current model interface produces 512 Float32 values for an image or text prompt. Those values are useful because related visual and language concepts tend to occupy nearby directions. They are not captions, object labels, or probabilities, and an embedding cannot reconstruct the original photo.
Turn thousands of comparisons into one matrix operation
A direct implementation can loop through a dictionary, calculate cosine similarity for every embedding, create a photo-ID-to-score result, and sort the entire collection. That is simple, but it repeats Swift-level work and temporary allocation for every query.
Queryable’s current GPU path instead arranges the photo index as an N × 512 matrix and the query as a 512 × 1 column vector. Their matrix product returns one similarity score per photo. MPSGraph schedules that operation on the Apple GPU through a Metal command queue.
scores[N, 1] = normalizedPhotos[N, 512] × normalizedQuery[512, 1]Evidence boundary: The public code contains a 65,000-item engineering path, but this article does not claim a universal benchmark. Device, library composition, thermal state and available originals all affect observed performance.
Normalize once and keep the index compact
Cosine similarity compares vector direction. If both sides are L2-normalized, cosine similarity becomes a dot product. During index construction, Queryable uses Accelerate’s vDSP routines to calculate the squared norm and multiply the source values by its inverse.
The normalized Float32 row is converted in bulk to Float16 with vImage. Halving the bytes per scalar reduces the storage and transfer footprint of the matrix. A small precision trade-off is acceptable for ranking candidates, while model inference can continue to produce Float32 output.
- Read the Core ML output through its data pointer.
- Calculate the L2 norm with vectorized Accelerate operations.
- Normalize the 512 values.
- Convert the completed row from Float32 to Float16.
Reuse the expensive parts across searches
The photo matrix changes much less often than the user’s query. Queryable therefore creates a shared MTLBuffer when the index changes and keeps a cached MPSGraph description for that matrix size. A normal search only needs to normalize and upload the small query vector before graph execution.
When photos are added or removed, the app updates its contiguous CPU-side representation and rebuilds the GPU buffer and graph. That makes index mutation more expensive than an ordinary query, which matches the product’s usage pattern: searches are frequent, library topology changes are comparatively infrequent.
Select top-k without building a giant dictionary
The GPU produces one score per indexed photo. Queryable reads that Float16 score buffer and keeps a minimum heap containing only the strongest k candidates. The weakest retained candidate sits at the heap root; a stronger score replaces it, while weaker scores are ignored.
This avoids hashing every photo ID into a full score dictionary and avoids sorting the entire result set. A date-filter closure is evaluated during selection, so excluded rows never occupy a top-k slot and starve the filtered result list.
for score in scores: if allowed(photo) and score > heap.minimum: replaceMinimum(score)Search speed is only half of the local-index problem
Rewriting every embedding whenever a new photo arrives would make ongoing maintenance unnecessarily expensive. Queryable’s EmbeddingStore uses a main binary file, an append-only journal for new records, and tombstones for deletions. It compacts the files after the journal grows or deletions accumulate.
The app also tracks capture dates beside the vectors so a time window can restrict candidates. The index is an operational data structure rather than a one-time demo: it needs startup loading, incremental writes, deletion safety, compaction and a path for unavailable iCloud originals.
What this architecture guarantees—and what it does not
The architecture keeps the semantic index, query encoding and similarity ranking on the iPhone. It eliminates the need for a Queryable server to receive the photo library for AI search, and it allows repeat queries after required originals and the index are available locally.
It does not guarantee that every semantic match is correct. MobileCLIP similarity can miss small details, text, identities or rare concepts. The index request keeps network access disabled, so cloud-only items need usable image data made local through Apple Photos first. A ranking score should be treated as a retrieval signal, not a factual description of the image.
Source code
The implementation links below are the primary sources for this article.