How brands can stay relevant and avoid search engine penalties.
INK created a classifier to detect if text was written by AI against 14 large language models. This includes commercial ones such as ChatGPT and open source ones such as T5.
The INK classifier is not fully reliable. Our reliability increases as the length of the evaluated text increases, and is optimized for detecting generated text in the English language. Our work to improve the classification technology continues as long as new generative AI technology hits the market.
While there is no fool-proof method for detecting generative AI, our aim at INK is to protect our users and their brands to a very high degree from platforms cracking down on content that can be perceived as written by AI.
Because of our belief that online protection for brands is so critically important, we have included AI and Plagiarism Shield in all our INK plans.
Limitations
Our primary goal is not to authoritatively answer the question if a human wrote the text, but if a search engine or platform could reasonably believe that a human was not the source.
Therefore, the INK classifier should not be used as the primary decision making tool, but rather be part of a holistic review.
- Sometimes, our classifier will believe that human text was created by AI.
- Our classifier currently is optimized for English, and is not reliable for non-English text.
- It is possible for our classifier to mis-classify text with high confidence if the subject of the text was not part of our AI training data.
- Very predictable text will almost always be flagged as AI written, since a human and AI would provide the exact same output.
- The longer the text, the more reliable our classifier becomes. The minimum text we accept for analysis is 50 words, but we recommend checking text of at least 250 words long for the best results.
- Editing AI generated text is one of the different ways our detection can be defeated. For this reason, we have invented a way to show the parts of the article that are most at risk of being identified as AI.
How to Interpret Classifier Results?
We clearly label each document based on how confident the classifier is that content is AI written:
- Human Written - AI written is 0-10%
- Likely Human Written - AI written is 10-45%
- Inconclusive - AI written is 45-90%
- Likely AI Written - AI written is 90-98%
- AI Written - AI written is 98-100%
In addition, our classifier will do a paragraph-by-paragraph analysis and identify the paragraphs that most contribute to the impression of AI written score.
Our goal is to help users identify the text most at risk, and take appropriate action to protect their content and brand.
Use of our Classifier for Writer Evaluation
To evaluate if a writer uses AI, we recommend looking at classification scores on a series of English-written articles on different topics, and excluding content less than 250 words from evaluation.
A similar approach works for evaluating whether a website is at risk from search engine penalties.
Use of our Classifier for Education
Our primary goal with the classifier is for use in marketing contexts, and protecting online brands. INK AI shield should neither be used by students to avoid getting caught cheating on assignments, nor for educators to accuse students.
What is Google's Stance on AI Generated Content?
Google only cares about the quality of the content, and not whether AI was used in the process.
However, general AI language models are by definition not able to create helpful content because they fail at including personal experience.
Google has clearly said that it considers using AI for the sole purpose of gaming SEO results as spam, which can put your site at risk of suspension, blacklisting and other penalties.
John Mueller (Google) said this on the topic of AI generated content:
“For us these would, essentially, still fall into the category of automatically generated content which is something we’ve had in the Webmaster Guidelines since almost the beginning.
And people have been automatically generating content in lots of different ways. And for us, if you’re using machine learning tools to generate your content, it’s essentially the same as if you’re just shuffling words around, or looking up synonyms, or doing the translation tricks that people used to do. Those kind of things.
My suspicion is maybe the quality of content is a little bit better than the really old school tools, but for us it’s still automatically generated content, and that means for us it’s still against the Webmaster Guidelines. So we would consider that to be spam.”
We are fully aware that advanced "black hat" marketers will exploit the gray area that technological advancements create to evade best practices in exchange for short term gains.
We caution our customers that we have seen this script play out before with link spam, and there is absolutely no reason to believe that Google will act any different when it comes to AI spam.
- Our user interface actively encourages a human-in-the-loop approach. We warn you if you overuse AI.
- INK is the only AI is trained to co-write like a human, by adopting your style, and learning from your highest performing competitors. As such, our patented technology is able to include topics that general models can't write about, find consensus in opinion, writing style, tone, voice, and write with emotional intelligence.
- All our plans include Content Shield as a standard feature. We actively scan your content for AI safety, and identify which paragraphs need additional human editing.
- All our plans feature plagiarism checking to ensure your content is original.
How Does Our Classifier Work?
INK analyzes randomness (entropy), perplexity and variance density with a linear probabilistic model, along with a classifier trained on LLM and human writing. We also employ a patent pending technology to identify paragraphs which require human editing.
Our training data
Prior to generative AI, we scraped large portions of the web for analysis in creating our semantic SEO score. We believe it is reasonably assumed that this content is human written.
AI large language models provide an easy source of content to compare with, and train our classifier.
Tokenization
We use GPT3 style tokenization.