Hive’s Text Moderation response format is an instantiation of the general classification response with additional fields to support the pattern-matching algorithms and the optional splitting of larger text inputs into sentence chunks.

Pattern-matching algorithm response:
The pattern-matching algorithm results for profanity are returned in the text_filter object. Similarly, pattern-matching algorithm results for PII are returned in the pii_entities object. Each pattern match will return an object describing matched substring in the value field, the start and end index of the pattern match in the start_index and end_index field, respectively, and the type (profanity, email, phone number, etc.).

Deep Learning model classifications:
The classified language is returned in the language object. Based on the classified language, and depending on the currently supported text moderation model classes, the moderated_classes field will indicate which classes have been moderated. If the classified language is "UNSUPPORTED", the moderated_classes array will be empty. The output array will contain the deep learning model results for each supported class.

Note: We are aware of an issue where start_index and end_index may be offset or misaligned relative to the text input in some cases. This can occur if the text input is significantly distorted with non-alphabetic characters, if pattern matching occurs on a subword, or if many characters are repeated on both ends of the text input. We are working to optimize our solution to this issue.

{
    "id": "19fa9f60-5b06-11ed-8fc4-659b38d4c6a6",
    "code": 200,
    "project_id": 41392,
    "user_id": 3121654,
    "created_on": "2022-11-02T23:28:48.546Z",
    "status": [
        {
            "status": {
                "code": "0",
                "message": "SUCCESS"
            },
            "response": {
                "input": {
                    "hash": "18320b1e0a343a0ddf7d595f6e64b683",
                    "inference_client_version": "6.0.13",
                    "model": "...",
                    "model_type": "TEXT_CLASSIFICATION",
                    "model_version": 1,
                    "text": "...",
                    "id": "19fa9f60-5b06-11ed-8fc4-659b38d4c6a6",
                    "created_on": "2022-11-02T23:28:48.214Z",
                    "user_id": 3121654,
                    "project_id": 41392,
                    "charge": 0.003
                },
                "custom_classes": [
                    {
                        "value": "HO",
                        "start_index": 139,
                        "end_index": 141,
                        "class": "badwords_subwords"
                    },
                    {
                        "value": "HO",
                        "start_index": 178,
                        "end_index": 180,
                        "class": "badwords_subwords"
                    },
                    {
                        "value": "HIV",
                        "start_index": 226,
                        "end_index": 229,
                        "class": "badwords_subwords"
                    },
                    {
                        "value": "PORN",
                        "start_index": 276,
                        "end_index": 280,
                        "class": "badwords_no_subwords"
                    },
                    {
                        "value": "PORN",
                        "start_index": 276,
                        "end_index": 280,
                        "class": "badwords_subwords"
                    },
                    {
                        "value": "KILL",
                        "start_index": 285,
                        "end_index": 289,
                        "class": "badwords_no_subwords"
                    },
                    {
                        "value": "KILL",
                        "start_index": 285,
                        "end_index": 289,
                        "class": "badwords_subwords"
                    },
                    {
                        "value": "NIGGER",
                        "start_index": 301,
                        "end_index": 307,
                        "class": "badwords_no_subwords"
                    },
                    {
                        "value": "NIGGER",
                        "start_index": 301,
                        "end_index": 307,
                        "class": "badwords_subwords"
                    }
                ],
                "text_filters": [
                    {
                        "value": "ASSHOLE",
                        "start_index": 16,
                        "end_index": 23,
                        "type": "profanity"
                    },
                    {
                        "value": "PORN",
                        "start_index": 276,
                        "end_index": 280,
                        "type": "profanity"
                    },
                    {
                        "value": "NIGGER",
                        "start_index": 301,
                        "end_index": 307,
                        "type": "profanity"
                    }
                ],
                "pii_entities": [
                    {
                        "value": "[email protected]",
                        "start_index": 38,
                        "end_index": 62,
                        "type": "Email Address"
                    },
                    {
                        "value": "123 YERBA BUENA LN, SAN FRANCISCO, CA 94103",
                        "start_index": 81,
                        "end_index": 124,
                        "type": "U.S. Mailing Address"
                    },
                    {
                        "value": "617-768-2274",
                        "start_index": 152,
                        "end_index": 164,
                        "type": "U.S. Phone Number"
                    },
                    {
                        "value": "+91-92342-43234",
                        "start_index": 190,
                        "end_index": 205,
                        "type": "International Phone Number"
                    }
                ],
                "urls": [
                    {
                        "value": "thehive.ai/projects/99999/settings",
                        "base_domain": "thehive.ai",
                        "start_index": 215,
                        "end_index": 257
                    }
                ],
                "language": "EN",
                "moderated_classes": [
                    "sexual",
                    "hate",
                    "violence",
                    "bullying",
                    "spam",
                    "promotions",
                    "gibberish",
                    "child_exploitation",
                    "phone_number"
                ],
                "output": [
                    {
                        "time": 0,
                        "start_char_index": 0,
                        "end_char_index": 307,
                        "classes": [
                            {
                                "class": "spam",
                                "score": 3
                            },
                            {
                                "class": "sexual",
                                "score": 1
                            },
                            {
                                "class": "hate",
                                "score": 3
                            },
                            {
                                "class": "violence",
                                "score": 3
                            },
                            {
                                "class": "bullying",
                                "score": 3
                            },
                            {
                                "class": "promotions",
                                "score": 3
                            },
                            {
                                "class": "gibberish",
                                "score": 0
                            },
                            {
                                "class": "child_exploitation",
                                "score": 0
                            },
                            {
                                "class": "phone_number",
                                "score": 3
                            }
                        ]
                    }
                ]
            }
        }
    ],
    "from_cache": false
}
NameDescription
classesList of dictionaries of all output classes. Each dictionary contains the class name and the score. The scores range from 0 to 3 with 3 being the most severe.
className of predicted class.
scoreScore of predicted class.
start_char_indexFirst character processed.
end_char_indexLast character processed.