Metrics

Quality metrics to evaluate the results

Basics

Responses from the style transform endpoint include quality metrics when the include_quality parameter is set to true. The quality metrics show differences between the unpersonalized text, which is the AI-generated text without Vaero styling, the personalized text, which is the text after Vaero styling, and ground_truth text, which is the dataset used for fine-tuning the model. Many metrics also include a personalization_improvement value that indicates the relative improvement from styling.

• ROUGE scores — measures n-gram overlap between the output text and the training data

  • ROUGE-1 — unigram overlap (precision, recall, F-measure)
  • ROUGE-2 — bigram overlap (precision, recall, F-measure)
  • ROUGE-L — longest common subsequence (precision, recall, F-measure)

• Sentence length statistics — sentence length measured in words

  • Mean — the mean sentence length in the text
  • Median — the median sentence length in the text
  • Std dev — the standard deviation in sentence length in the text

• Word length statistics — word length measured in characters

  • Mean — the mean word length in the text
  • Median — the median word length in the text
  • Std dev — the standard deviation in word length in the text

• Readability

  • Flesch Reading Ease — higher values indicate easier-to-read text
  • Flesch-Kincaid Grade Level — the U.S. school grade level needed to understand the text

• Function word frequency — measures similarity of function word distributions to the training data using Jensen-Shannon distance

• Part-of-speech frequency — measures similarity of part-of-speech distributions to the training data using Jensen-Shannon distance

• Lexical density — the ratio of content words to total words, indicating how informationally dense the text is

• Sentence structure frequency — distribution of sentence structure types (simple, compound, complex, compound-complex, fragment/unknown) with Jensen-Shannon distance

• Sentence voice frequency — distribution of active vs. passive voice with Jensen-Shannon distance

• Punctuation — density of punctuation types (comma, semicolon, colon, period, question mark, exclamation mark, parenthetical) with Jensen-Shannon distance