Query

The Query page (DIVE Desktop) searches many datasets at once. Pick the datasets, build a search index over them, then search from an image, a frame of a video, or a text description. Hits come back as the same chip grid the Review page uses, and a similarity search can be refined by marking hits correct or incorrect.

Open it from the Query tab, or select datasets in the Library and click Index to open the Index panel with those sequences selected.

Index panel

Add datasets with the same picker as the Training and Pipelines pages: search the library, add rows one at a time or with Select all, and drop them with Remove all. Each selected dataset shows whether it is in the search index. Build index indexes every selected dataset that is not indexed yet (each row also has its own build button), choosing how the index is made:

• Around generic detections: run the generic object detector and describe its boxes.
• Detection and tracking: detect and track, then describe the tracks.
• Around existing annotations: describe the dataset’s current annotations.
• Whole frames: describe each frame as a whole, with no detector.

Indexing runs as jobs on the Jobs page; the rows update as they finish. Build all not indexed queues every unindexed dataset. All indexed datasets share one index, so a dataset indexed here is also searchable from the viewer’s Image Query panel, and the reverse.

Query panel

The panel is on the left, results on the right.

Image. Choose an image file. Drag a box on it to search for one object, or leave the whole image as the exemplar. Press Search.

Video. Choose one of the listed datasets or a video file, enter a frame number and press Show frame, then optionally drag a box on the frame. Press Search.

Both run a similarity search over every indexed dataset (limited to the listed datasets unless the switch is turned off). Mark results correct or incorrect and press Refine to re-rank; Hide reviewed hides the marked ones. Save model keeps the refined classifier as a trained pipeline. A saved .svm model can also start a search (Start from a saved model).

Results can be turned into annotations without leaving the page: accepted results, and results given a type or an edited box, are written to their datasets by Save on the results toolbar (rejected results only when typed). Results that overlap existing annotations prompt for whether to keep the originals, replace the overlapped ones, replace every annotation in those sequences, or discard the results; see the Review page docs for details.

Text. Type what to find. Sampled frames of every listed dataset (every N frames, up to a per-dataset cap) are searched with the SAM3 text model, which must be installed as a VIAME add-on. Hits show in the grid with their label and score. Each hit offers Search the index for objects like this one, which turns it into an image query, and Open in the annotation viewer.

The results toolbar can switch from the grid to a 3D view (the cube button; off by default) that places the top results around the query in descriptor space: the query exemplar sits at the center, each result hangs off it on a line, and results with similar descriptors cluster together, with the more similar ones closer to the center. Positions are the results’ descriptor offsets from the query projected onto the three principal axes of the shown set. Choose how many top results to place (up to 1000; this needs a VIAME whose query service returns that many), drag to orbit, drag with the middle button to slide the view, use the wheel to zoom, and press the reset button to bring the query back to the center. Results are shown cropped to their box alone. Click a result to mark it correct or incorrect, and double click to open it. To work on several at once, hold shift and drag a box over them (or turn on the box tool), then mark them all correct or incorrect, give them all a type, or delete them together; shift click adds or removes a single result.

Double clicking any result opens its dataset in the viewer at that frame. Coming back to Query afterwards resumes the page as it was left: the exemplar, the results and their marks, unsaved annotation edits, and the grid page, so adjudication carries on where it stopped.

Index builds appear in Jobs while preparing and running, with their dataset, indexing method, live stdout/stderr, and final result. Expand the job’s output to diagnose failures; the process log is also saved as runlog.txt in its working directory. Builds waiting for the GPU appear under Queued Jobs as indexing jobs. Startup failures remain in job history.

Selecting a stereo or multicamera sequence uses its first camera in the configured display order. Query shows that camera by name and indexes its media and, for the existing-detections method, its annotations. The other cameras are not indexed automatically. Results refer to the indexed camera, keeping thumbnails and frame numbers aligned with its media.

Inside an annotation sequence, the Image Query panel lists the available indexed sequences, with an option to search all of them. Selecting an indexed sequence limits the ranked results and result grid to that sequence; the underlying similarity search still uses the shared database. Changing this selection clears the previous results and feedback so the next query starts fresh.

Use Create index when no index exists, or Build a new index below the selector, to open Query’s Index panel with the current sequence selected. Index type, building, and removal are managed on that page.

The Index list survives navigation and restores saved index membership and indexing jobs on return. Rows show in-progress builds, successful indexes, or failed builds with their job error, even when the job finished on another page.

The top of Index lists successfully generated entries in the shared search index, independently of the selection used to queue new builds below. There is one entry per video or sequence; rebuilding replaces that entry. Remove from index deletes one sequence’s search data, while Delete entire index removes the whole shared index. Both ask for confirmation and keep source media and annotations. Index deletion is disabled while index builds are queued or running.

The sequence editor’s Image Query panel is a launch point. Its large search buttons open Query and run the selected annotation, image, or saved-model search there. The chosen index and annotation crop are preserved; results and refinement appear on Query rather than in the sidebar. Leaving the editor still follows the normal unsaved-annotation checks.

Search from selected track queries with the whole selected track instead of one frame: up to six frames sampled along the track each contribute a descriptor, and all of them join the query as separate positive examples. Query shows the current frame’s crop; redrawing the box or choosing Use this frame only returns to a single-frame query.

Web image and video queries

The web application’s Query tab supports image similarity search and feedback refinement. A pipeline worker with VIAME’s configs/index.py, file-backed search index support, and viame.core.query_service must be installed, including the query and descriptor models. Existing older workers need to be upgraded before using this page.

1. Choose an indexing method and Choose dataset and build index. Select a video or image sequence; for multicamera data, choose an individual camera. Index builds appear in Jobs, with progress, logs and cancellation.
2. Select one or more completed index snapshots. Rebuilding makes a new snapshot; select the version you want to search. Snapshots are private to their creator and do not change source media or annotations.
3. Choose an image or video file. For a video, seek with its controls and click Use current video frame. Alternatively, Use a dataset frame selects media already in DIVE. Drag a crop on the exemplar or search the whole image.
4. Search queues a job. Results include cropped thumbnails, scores and links into the source sequence at the matching frame. Mark hits Correct or Incorrect, then Refine using feedback to queue another iteration.

All model execution, including index building, exemplar descriptors, similarity search and refinement, runs in the existing pipeline job queue with the worker’s GPU environment. The browser decodes media and captures still frames; it does not run a query model. Users with private worker queues use those queues.

Each job downloads selected index snapshots into a separate temporary directory and starts its own query process. No live process, GPU allocation, PostgreSQL server or writable index is shared between users or retained between jobs. Refinement replays the original search and previous feedback against the same snapshots, matching feedback by result identity rather than process-local IDs. This adds startup and replay work to each iteration, but lets other jobs run between iterations and allows later iterations to run on another worker.

Source read permissions are checked on submission, when the job starts and before publishing results. Returning from a result’s viewer restores the last query, exemplar, feedback marks and result page in the same browser tab. Input PNGs are limited to 10 MB, searches to 32 index snapshots, and refinement to 20 rounds.

Saved SVM models, text queries and saving result annotations remain desktop features. Query artifacts are stored in private Query … folders under the user’s account; delete unused folders through the Data browser to reclaim their storage. Deleting an index needed by a later refinement requires starting a new search with another snapshot.