How to safely challenge models against prompt injection? – Prompting

I would like to challenge OpenAI models against prompt injections. I plan to collect the already known injection patterns and use them against my model.

My Question
“How do i safely challenge the model over and over again without getting banned by OpenAI?”

Current Approach
“Run the test with local model like mistral 7b and applied them to gpt-3.5 -turbo-1106 later”

But
Since different variant of model response to prompt differently, I doubt that it’s transferable.

Implementation Snippet

from openai import OpenAI
from openai.resources.moderations import Moderations

class Guard:
    backend = OpenAI(api_key=cfg.openai_api_key)
    openai_moderator = Moderations(backend)

    @classmethod
    def is_appropriate(cls, prompt) -> bool:
        # level-1
        result = cls.moderator.create(input=prompt, model="text-moderation-latest")
        is_not_appropriate = any(map(lambda x: x.flagged, result.results))
        if is_not_appropriate:
            return False
        # level-2
        completion = cls.backend.chat.completions.create(
            model="gpt-3.5-turbo-1106",
            messages=[
                {
                    "role": "system",
                    "content": "<system-prompt>",
                },
                {"role": "user", "content": prompt},
            ],
            max_tokens=1, n=1, temperature=0.0
        )
        response = completion.choices[0].message.content
        
        try:
            return float(response) < 0.5
        except:
            print(f"Failed to parse response: {response}")
            return True

Hi,

You can make use of the safe harbour facility as part of the bug bounty program, however you would need to check the specifics of that harbour to see if it includes protection against prompt misuse

Indeed, it seems like something OpenAI would want to have constructive feedback on, but I’m unsure as to the correct method to obtain it. The old safe harbour did seem to include prompting, perhaps that is no longer required or even wanted.

You also have to read the entire document to find the context. Making the AI emit anything you want = $0.00.

the terms seem to be changing every other day with this company

I’ll add this to the docket to discus on Thursday with OAI as there seems to be some confusion.

Bug Bounty looks interesting to me, and I am very much into this game.

But at the moment I am more interested in knowing whether I am allowed to challenge the models with a bunch of manipulative/NSFW/should be censored prompt through API.

I hope that my model’s response always stay within the scope I draw.

Essentially:

There is no known disabling the OpenAI policy-checking on model inputs and outputs on API. It’s not even documented that AI generations have an automatic consequence or what is actively looked for. Defensible is with user ID sent and terms & policies carefully reviewed.

You can preview the input scores on the API moderation endpoints. If you aren’t generating detected “moderations” safety categories, you are far better.

More likely ban on API is jailbreaky prompts from character sites getting accounts banned, by some fingerprinting being done. Or not paying a bill.

ChatGPT seems to flag outputs far earlier than anything measured on API’s moderations. Making GPTs for laundry give manicure advice isn’t going to be detected, though.

OpenAI has other types of moderations they have run, such as one looking for and unsharing ChatGPT chat shares that have the AI “acting wrong (or basically acting at all)”. So no promise that jailbreak isn’t completely unconsidered just because it is not moderated or laughed at by your AI.

  "results": [
    {
      "categories": {
        "harassment": false,
        "harassment_threatening": false,
        "hate": false,
        "hate_threatening": false,
        "self_harm": false,
        "self_harm_instructions": false,
        "self_harm_intent": false,
        "sexual": false,
        "sexual_minors": false,
        "violence": false,
        "violence_graphic": false
      },
      "category_scores": {
        "harassment": "0.0071744",
        "harassment_threatening": "0.0025186",
        "hate": "0.0000059",
        "hate_threatening": "0.0000094",
        "self_harm": "0.1781347",
        "self_harm_instructions": "0.0000338",
        "self_harm_intent": "0.0027713",
        "sexual": "0.0000244",
        "sexual_minors": "0.0000064",
        "violence": "0.2037043",
        "violence_graphic": "0.0282443"
      },
      "flagged": false
    }
  ]