Hive’s Text Moderation response format is an instantiation of the general classification response with additional fields to support the pattern-matching algorithms and the optional splitting of larger text inputs into sentence chunks.
Pattern-Matching Algorithm Response
The pattern-matching algorithm results for profanity are returned in the text_filter object. Similarly, pattern-matching algorithm results for PII are returned in the pii_entities object. Each pattern match will return an object describing matched substring in the value field, the start and end index of the pattern match in the start_index and end_index field, respectively, and the type (profanity, email, phone number, etc.).
Deep Learning Model Classifications
The classified language is returned in the language object. Based on the classified language, and depending on the currently supported Text Moderation model classes, the moderated_classes field will indicate which classes have been moderated. If the classified language is "UNSUPPORTED", the moderated_classes array will be empty. The output array will contain the deep learning model results for each supported class.
Note: We are aware of an issue where start_index and end_index may be offset or misaligned relative to the text input in some cases. This can occur if the text input is significantly distorted with non-alphabetic characters, if pattern-matching occurs on a subword, or if many characters are repeated on both ends of the text input. We are working to optimize our solution to this issue.
{
"id": "19fa9f60-5b06-11ed-8fc4-659b38d4c6a6",
"code": 200,
"project_id": 41392,
"user_id": 3121654,
"created_on": "2022-11-02T23:28:48.546Z",
"status": [
{
"status": {
"code": "0",
"message": "SUCCESS"
},
"response": {
"input": {
"hash": "18320b1e0a343a0ddf7d595f6e64b683",
"inference_client_version": "6.0.13",
"model": "...",
"model_type": "TEXT_CLASSIFICATION",
"model_version": 1,
"text": "...",
"id": "19fa9f60-5b06-11ed-8fc4-659b38d4c6a6",
"created_on": "2022-11-02T23:28:48.214Z",
"user_id": 3121654,
"project_id": 41392,
"charge": 0.003
},
"custom_classes": [
{
"value": "HO",
"start_index": 139,
"end_index": 141,
"class": "badwords_subwords"
},
{
"value": "HO",
"start_index": 178,
"end_index": 180,
"class": "badwords_subwords"
},
{
"value": "HIV",
"start_index": 226,
"end_index": 229,
"class": "badwords_subwords"
},
{
"value": "PORN",
"start_index": 276,
"end_index": 280,
"class": "badwords_no_subwords"
},
{
"value": "PORN",
"start_index": 276,
"end_index": 280,
"class": "badwords_subwords"
},
{
"value": "KILL",
"start_index": 285,
"end_index": 289,
"class": "badwords_no_subwords"
},
{
"value": "KILL",
"start_index": 285,
"end_index": 289,
"class": "badwords_subwords"
}
],
"text_filters": [
{
"value": "DAMN",
"start_index": 16,
"end_index": 20,
"type": "profanity"
},
{
"value": "PORN",
"start_index": 276,
"end_index": 280,
"type": "profanity"
}
],
"pii_entities": [
{
"value": "[email protected]",
"start_index": 38,
"end_index": 62,
"type": "Email Address"
},
{
"value": "123 YERBA BUENA LN, SAN FRANCISCO, CA 94103",
"start_index": 81,
"end_index": 124,
"type": "U.S. Mailing Address"
},
{
"value": "617-768-2274",
"start_index": 152,
"end_index": 164,
"type": "U.S. Phone Number"
},
{
"value": "+91-92342-43234",
"start_index": 190,
"end_index": 205,
"type": "International Phone Number"
}
],
"urls": [
{
"value": "thehive.ai/projects/99999/settings",
"base_domain": "thehive.ai",
"start_index": 215,
"end_index": 257
}
],
"language": "EN",
"moderated_classes": [
"sexual",
"hate",
"violence",
"bullying",
"spam",
"promotions",
"gibberish",
"child_exploitation",
"phone_number"
],
"output": [
{
"time": 0,
"start_char_index": 0,
"end_char_index": 307,
"classes": [
{
"class": "spam",
"score": 3
},
{
"class": "sexual",
"score": 1
},
{
"class": "hate",
"score": 3
},
{
"class": "violence",
"score": 3
},
{
"class": "bullying",
"score": 3
},
{
"class": "promotions",
"score": 3
},
{
"class": "gibberish",
"score": 0
},
{
"class": "child_exploitation",
"score": 0
},
{
"class": "phone_number",
"score": 3
}
]
}
]
}
}
],
"from_cache": false
}| Name | Description |
|---|---|
| classes | List of dictionaries of all output classes. Each dictionary contains the class name and its respective score. Scores represent the content's severity on a scale of 0 (benign) to 3 (most severe). If the language of the input is not currently supported for a specific class, the class will receive a score of -1. |
| class | Name of predicted class. |
| score | Score of predicted class. |
| start_char_index | First character processed. |
| end_char_index | Last character processed. |