Hive’s Text Moderation response format is an instantiation of the general classification response with additional fields to support the pattern-matching algorithms and the optional splitting of larger text inputs into sentence chunks.

Pattern-Matching Algorithm Response

The pattern-matching algorithm results for profanity are returned in the text_filter object. Similarly, pattern-matching algorithm results for PII are returned in the pii_entities object. Each pattern match will return an object describing matched substring in the value field, the start and end index of the pattern match in the start_index and end_index field, respectively, and the type (profanity, email, phone number, etc.).

Deep Learning Model Classifications

The classified language is returned in the language object. Based on the classified language, and depending on the currently supported Text Moderation model classes, the moderated_classes field will indicate which classes have been moderated. If the classified language is "UNSUPPORTED", the moderated_classes array will be empty. The output array will contain the deep learning model results for each supported class.

Note: We are aware of an issue where start_index and end_index may be offset or misaligned relative to the text input in some cases. This can occur if the text input is significantly distorted with non-alphabetic characters, if pattern-matching occurs on a subword, or if many characters are repeated on both ends of the text input. We are working to optimize our solution to this issue.

{
    "id": "19fa9f60-5b06-11ed-8fc4-659b38d4c6a6",
    "code": 200,
    "project_id": 41392,
    "user_id": 3121654,
    "created_on": "2022-11-02T23:28:48.546Z",
    "status": [
        {
            "status": {
                "code": "0",
                "message": "SUCCESS"
            },
            "response": {
                "input": {
                    "hash": "18320b1e0a343a0ddf7d595f6e64b683",
                    "inference_client_version": "6.0.13",
                    "model": "...",
                    "model_type": "TEXT_CLASSIFICATION",
                    "model_version": 1,
                    "text": "...",
                    "id": "19fa9f60-5b06-11ed-8fc4-659b38d4c6a6",
                    "created_on": "2022-11-02T23:28:48.214Z",
                    "user_id": 3121654,
                    "project_id": 41392,
                    "charge": 0.003
                },
                "custom_classes": [
                    {
                        "value": "HO",
                        "start_index": 139,
                        "end_index": 141,
                        "class": "badwords_subwords"
                    },
                    {
                        "value": "HO",
                        "start_index": 178,
                        "end_index": 180,
                        "class": "badwords_subwords"
                    },
                    {
                        "value": "HIV",
                        "start_index": 226,
                        "end_index": 229,
                        "class": "badwords_subwords"
                    },
                    {
                        "value": "PORN",
                        "start_index": 276,
                        "end_index": 280,
                        "class": "badwords_no_subwords"
                    },
                    {
                        "value": "PORN",
                        "start_index": 276,
                        "end_index": 280,
                        "class": "badwords_subwords"
                    },
                    {
                        "value": "KILL",
                        "start_index": 285,
                        "end_index": 289,
                        "class": "badwords_no_subwords"
                    },
                    {
                        "value": "KILL",
                        "start_index": 285,
                        "end_index": 289,
                        "class": "badwords_subwords"
                    }
                ],
                "text_filters": [
                    {
                        "value": "DAMN",
                        "start_index": 16,
                        "end_index": 20,
                        "type": "profanity"
                    },
                    {
                        "value": "PORN",
                        "start_index": 276,
                        "end_index": 280,
                        "type": "profanity"
                    }
                ],
                "pii_entities": [
                    {
                        "value": "JONABCDE.SMITH@GMAIL.COM",
                        "start_index": 38,
                        "end_index": 62,
                        "type": "Email Address"
                    },
                    {
                        "value": "123 YERBA BUENA LN, SAN FRANCISCO, CA 94103",
                        "start_index": 81,
                        "end_index": 124,
                        "type": "U.S. Mailing Address"
                    },
                    {
                        "value": "617-768-2274",
                        "start_index": 152,
                        "end_index": 164,
                        "type": "U.S. Phone Number"
                    },
                    {
                        "value": "+91-92342-43234",
                        "start_index": 190,
                        "end_index": 205,
                        "type": "International Phone Number"
                    }
                ],
                "urls": [
                    {
                        "value": "thehive.ai/projects/99999/settings",
                        "base_domain": "thehive.ai",
                        "start_index": 215,
                        "end_index": 257
                    }
                ],
                "language": "EN",
                "moderated_classes": [
                    "sexual",
                    "hate",
                    "violence",
                    "bullying",
                    "spam",
                    "promotions",
                    "gibberish",
                    "child_exploitation",
                    "phone_number"
                ],
                "output": [
                    {
                        "time": 0,
                        "start_char_index": 0,
                        "end_char_index": 307,
                        "classes": [
                            {
                                "class": "spam",
                                "score": 3
                            },
                            {
                                "class": "sexual",
                                "score": 1
                            },
                            {
                                "class": "hate",
                                "score": 3
                            },
                            {
                                "class": "violence",
                                "score": 3
                            },
                            {
                                "class": "bullying",
                                "score": 3
                            },
                            {
                                "class": "promotions",
                                "score": 3
                            },
                            {
                                "class": "gibberish",
                                "score": 0
                            },
                            {
                                "class": "child_exploitation",
                                "score": 0
                            },
                            {
                                "class": "phone_number",
                                "score": 3
                            }
                        ]
                    }
                ]
            }
        }
    ],
    "from_cache": false
}

Name	Description
classes	List of dictionaries of all output classes. Each dictionary contains the class name and its respective score. Scores represent the content's severity on a scale of 0 (benign) to 3 (most severe). If the language of the input is not currently supported for a specific class, the class will receive a score of -1.
class	Name of predicted class.
score	Score of predicted class.
start_char_index	First character processed.
end_char_index	Last character processed.