Repetition penalties

What is it?

Repetition penalties can be used to counteract the model's tendency to repeat prompt text verbatim and/or get stuck in a loop. This is accomplished by adjusting the token probabilities at each generation step, such that tokens that already appeared in the text (either in the prompt or in the completion) are less likely to be generated again.

Repetition penalties: what types are there?

There are three types of repetition penalties: presencePenalty, countPenalty and frequencyPenalty. One or more penalties can be used, and the magnitude of each can be controlled independently.

presencePenalty

The presence penalty is utilized to decrease the likelihood of generating tokens that have already been mentioned in either the prompt or the completion. For instance, with presencePenalty set to its maximum value (presencePenalty = 5), and given the prompt (or the current completion) "One fish, two fish, red fish, blue fish", the chances of producing tokens such as "One", "fish", "two", "red" and "blue" will be reduced. It's important to note that even though the word "fish" appears multiple times, the penalty applies uniformly to all these tokens.

countPenalty

The countPenalty is used to reduce the probability of generating new tokens that appear at least once in the prompt or in the completion, in proportion to the number of appearances. For instance, with countPenalty set to its maximum value (countPenalty = 1), and given the prompt (or the current completion) "One fish, two fish, red fish, blue fish", the probability of the tokens "One", "two", "red", and "blue" will decrease by some value and the probability of the token "fish" will decrease by a larger value.

frequencyPenalty

frequencyPenalty is used to reduce the probability of generating new tokens that appear in the prompt or in the completion, in proportion to the frequency of their appearances in the text (normalized to text length). Using this method is beneficial in longer texts, since repeating some words is less of an issue than it is in short texts.

Example - general

The following example introduces all three penalties simultaneously (though each one individually would have sufficed to prevent repetition):

Example - special tokens

In addition to controlling the penalty scale, the API allows penalties to be toggled on and off for five special categories of tokens: whitespaces (including newlines), punctuations, numbers, stopwords (including multi-word combinations of stopwords) and emojis. For example:

Penalty data

Each repetition penalty is characterized by a PenaltyData data structure containing the following fields:

---

scale | float

Controls the magnitude of the penalty. Required.

A positive penalty value implies reducing the probability of repetition. Larger values correspond to a stronger bias against repetition.

---

applyToWhitespaces | boolean

Apply the penalty whitespaces and newlines. Optional, default=True.

Determines whether the penalty is applied to the following tokens:

'▁', '▁▁', '▁▁▁▁', '<|newline|>'

---

applyToPunctuations | boolean

Apply the penalty to punctuations. Optional, default=True.

Determines whether the penalty is applied to tokens containing punctuation characters and whitespaces, such as ; , !!! or ▁\[[@.

---

applyToNumbers | boolean

Apply the penalty to numbers. Optional, default=True.

Determines whether the penalty is applied to purely numeric tokens, such as 2022 or 123. Tokens that contain numbers and letters, such as 20th, are not affected by this parameter.

---

applyToStopwords | boolean

Apply the penalty to stop words. Optional, default=True.

Determines whether the penalty is applied to tokens that are NLTK English stopwords or multi-word combinations of these words, such as are , nor and ▁We▁have.

---

applyToEmojis | boolean

Exclude emojis from the penalty. Optional, default=True.

Determines whether the penalty is applied to any of approximately 650 common emojis in the Jurassic-1 vocabulary.