Built on our visual similarity model, Custom Search is a robust tool to search your own image and video database for visual matches to a query image or video. Customers have the ability to load relevant images and videos to an individualized index (custom index). Following a query task, the API will return a list of any images/videos in your custom index classified as visual matches to the query image/video.

The flexibility to maintain a hand-picked custom index enables a wide variety of use-cases for Custom Search – from identifying fake profile photos on a social platform or scam listings on marketplaces to copy detection and spam control in content moderation applications.

Visual Similarity Model

Hive's visual similarity model is trained to generate a similarity score – normalized between 0 and 1 – for any two image frames. A similarity score of 1.0 indicates an exact match between two images, while lower values indicate that the query image has been modified to some extent.

This model will capture both near duplicates and more noticeable alterations. This includes manual image manipulations like rotations and text overlays, as well as more subtle augmentations such as introduction of noise, filters, and other pixel-level changes that may change the image hash but are visually imperceptible to humans.

Custom Index Search API

The Custom Index Search API has three different endpoints:

• "Add" Endpoint: used to add images/videos to your custom index
• "Query" Endpoint: used to search your custom index for visual matches to a query image/video
• "Remove" Endpoint: used to remove images/videos from your custom index that you no longer wish to compare against

NOTE: The API Key used to authenticate each type of request will be the same for all three endpoints.

Adding to your Index

Submitting a POST request to the "Add" endpoint will add the specified image/video to your custom index to be checked against in future searches. You can also choose to append descriptive metadata to be stored alongside the uploaded file. This could be an ID for the file, tags, source information, or anything else that would be useful to receive back in a query response.

Hive will then assign a unique identifier to the uploaded file (task_id) and store the file alongside any metadata included in the "Add" request (tags, descriptors, your own ID, etc.).

For direct uploads, we support most standard image and video file formats including but not limited to: .png, .jpg, .webp, .mp4, .mov, .mkv.

📘

ADD ENDPOINT

Synchronous: POST to https://api.thehive.ai/api/v2/custom_index/add/sync
Asynchronous: POST to https://api.thehive.ai/api/v2/custom_index/add/async

For syntax and more information on submitting requests to Hive APIs, please see our API reference materials. Here, you'll also find a more detailed explanation of synchronous versus asynchronous submission.

Example API response objects for the "Add" endpoint are available here.

Querying your Index

Submitting a GET request to the query index enables you to access model-based comparisons between a query image/video and the image/videos stored in your custom index.

📘

QUERY ENDPOINT

Synchronous: GET to https://api.thehive.ai/api/v2/custom_index/query/sync
Asynchronous: GET to https://api.thehive.ai/api/v2/custom_index/query/async

When a task is submitted to the "Query" endpoint, Custom Search will compare the input image/video file against every file stored inside your custom index.

For querying images against an index of images, Hive will simply return the images in your custom index that have the highest visual similarity to your query image/video.

For querying videos against an index of videos, Hive first converts the query video and index videos into a sequence of image frames sampled at a rate of 1 frame per second. For each frame extracted from the query video, it finds the highest matching frame extracted from the index video.

Querying videos against images and vice versa are also supported. In these two flows, images can be thought of as a video containing just a single frame.

Response Fields

The response object, task.status.output will include the following fields:

• task_id : a unique identifier generated and returned for the matching image/video when it was added to the index
• metadata : any metadata that was included with the matching image/video when it was added to the index
• matches : a list enumerating the timestamps that were visually similar between the query file and the matching index file. Each item in the list includes three fields: query_timestamp, matching_timestamp, and similarity_score. A response with query_timestamp: 0, matching_timestamp: 10, similarity_score: 0.9 indicates that the 0th second of the query video matched with the 10th second of the matched video with a similarity score of 0.9. Note that within a single match, a query_timestamp can only show up once and that we only return matches where the similarity_score is above a certain threshold. For querying images against images, query_timestamp and matching_timestamp will always be 0.

For each query we return up to 5 videos/images that we deem to be the closest matches: the ordering and selection of these matches is based on a combination of the similarity_scores of the matching frames and the total number of frames that matched between the query and matching index file.

Example API response objects for the "Query" endpoint are available here.

Using the Response

For image vs. image use-cases, we recommend that customers pick a threshold (0.9 is a great place to start) and take action on matches where the similarity_score under matches exceeds a certain threshold.

For video vs. video use-cases, we recommend customers use a combination of the similarity_score of the matching frames in conjunction with the query_timestamps that are returned. Customers have found success using the following two workflows:

• Overall Ratio : In this scenario, customers flag content when the total number of frames in the query video with a similar_match divided by the total number of frames extracted (typically, the number of seconds in the query video) exceeds a certain threshold.
• Continuous Time Ranges : In this scenario, customers flag content when there is an extended period of consecutive seconds from the query video that have matching video frames with frame(s) in an index video.

Removing from your Index

If you no longer need to query against a particular reference image/video in your index, you can remove it with a DELETE request to the "Remove" endpoint. When calling this endpoint, you'll need to provide the task_id (returned when the file was initially added) of the file you want to remove.

Example API response objects for the "Delete" endpoint are available here.

📘

REMOVE ENDPOINT

Synchronous: DELETE to https://api.thehive.ai/api/v2/custom_index/remove/sync
Asynchronous: DELETE to https://api.thehive.ai/api/v2/custom_index/remove/async