Retrieved literature indicates that while watermarking algorithms and detection tools have been developed and studied for large language models including ChatGPT, the evidence only supports the feasibility or development of such tools rather than confirming that standard ChatGPT outputs contain active algorithmic watermarks.
Existing watermarked generation algorithms employ token-level designs and therefore, are vulnerable to paraphrase attacks. To address this issue, we introduce watermarking on the semantic representation of sentences. We propose SemStamp, a robust sentence-level semantic watermarking algorithm that uses locality-sensitive hashing (LSH) to partition the semantic space of sentences. The algorithm encodes and LSH-hashes a candidate sentence generated by a language model, and conducts rejection sampling until the sampled sentence falls in watermarked partitions in the semantic embedding space. To test the paraphrastic robustness of watermarking algorithms, we propose a “bigram paraphrase” attack that produces paraphrases with small bigram overlap with the original sentence. This attack is shown to be effective against existing token-level watermark algorithms, while posing only minor degradations to SemStamp. Experimental results show that our novel semantic watermark algorithm is not only more robust than the previous state-of-the-art method on various paraphrasers and domains, but also better at preserving the quality of generation.
Experimental results (§3) demonstrate that our pro- posed semantic watermark remains effective while token-level watermarks suffer significantly from the bigram attack. We summarize our main contributions as follows. First, we propose a sentence-level semantic water- mark for LLMs and show that it is robust to para- phrasing and more quality-preserving than a token- 4067 1110 0100 Lucy smiled. It was genuine. Her eyes crinkled. She was happy. She felt delighted.
4068 Algorithm 1 SEMSTAMP text generation algorithm Input: language modelPLM, prompts(0), number of sentences to generateT . Params: sentence embedding modelMembd with embedding dimensionh, maxout numberNmax, marginm> 0, valid region ratioγ∈ (0, 1), LSH dimensiond, a large prime numberp. Output: generated sequences(1)...s (T ). procedure SEMSTAMP init LSH(·), randomly initialized vectorsn(1)...n (d)∈ Rh, to create 2d semantic subspaces. fort = 1, 2,...,T do 1.
Next, we pseudoran- domly divide the LSH partitions into a set of “valid” regionsG(t) and a set of “blocked” regions R(t), where the masking is seeded
In the paraphrase attack phase, we paraphrase generations by SEMSTAMP and KGW and compare their post-hoc detection rates after attacks. We also experiment with a distortion- free watermark by Kuditipudi et al.
(2023) proposes a distortion-free watermark that preserves the orig- inal distribution of LM during watermarking. Yoo et al. (2023) embeds multi-bit information into wa- termark and enhances performance against corrup- tion through a robust infilling model. They inject the watermark via word replacement after initial generation, which is incorporated into one-stage watermarked generation by Wang et al. (2023). Christ et al. (2023) propose a watermarking scheme that is computationally undetectable without the se- cret key in theory.
To the best of our knowledge, our work is the first sentence-level semantic watermark algorithm targeted against paraphrase attacks. Post-Hoc Detection of Machine-Generated Text In post-hoc methods, applying binary classifica- tion models is the most straightforward approach (Zellers et al., 2019; Jawahar et al., 2020; Liu et al., 2022; Mireshghallah et al., 2023; Pu et al., 2023). These methods are applicable to black-box gen- erators but need sufficiently large corpus for fine- tuning.
The difficulty of reverse en- gineering can also be increased by using a larger LSH dimension, while the watermark could be less robust to paraphrase attack. Bigram Paraphrase Attack Control We control the “intensity” degree of bigram paraphrase attack by constraining the paraphrase candidate selection with a BERTScore constraint. Removing the con- straint will more forcefully lower AUROC at the expense of paraphrase quality. Finally, due to lack of space we defer discussions on ethical impacts to §F.
Ashish Venugopal, Jakob Uszkoreit, David Talbot, Franz Och, and Juri Ganitkevitch. 2011. Watermark- ing the outputs of structured prediction with an appli- cation in statistical machine translation. In Proceed- ings of the 2011 Conference on Empirical Methods in Natural Language Processing. Lean Wang, Wenkai Yang, Deli Chen, Haozhe Zhou, Yankai Lin, Fandong Meng, Jie Zhou, and Xu Sun. 2023. Towards codable text watermarking for large language models. ArXiv, abs/2307.15992. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra 4077 Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models.
(2023) fo- cus on protecting the copyright of prompts through inserting the secret key into the prompt optimiza- tion stage. These works mainly apply watermark techniques for data copyright protections , whereas our work focuses on exploring the robustness of watermark against paraphrasing. Contrastive Learning in NLP Contrastive learn- ing (Hadsell et al., 2006) aims at improving the distinguishability of representation by pulling over positive pairs and pushing off negative pairs.
(2021) propose a self-guided contrastive learning between embed- dings from a fixed model and a fine-tuned model. B Watermark Detection Kirchenbauer et al. (2023a) proposes using a one- proportionz-test on the number of green list tokens to detect watermarks, assuming the following null hypothesis: H0 : The text is not generated (or written) knowing a watermarking green list rule.
Although current LLMs are often exposed to users as API endpoints, ma- licious users can still postprocess and paraphrase the API-generated response to escape the injected watermark. This motivates us to study watermark robustness against paraphrasing in this work. We hope that the proposed SEMSTAMP algorithm can mitigate the risk of LLM misuse by providing a reliable method to counter paraphrasing attacks on watermarked generations.
Text generation tools, often presented as a form of generative artificial intelligence, have the potential to pose a threat to the integrity of the educational system. They can be misused to afford students marks and qualifications that they do not deserve. The emergence of recent tools, such as ChatGPT, appear to have left the educational community unprepared, despite the fact that the computer science community has been working to develop and improve such tools for years. This paper provides an introduction to text generation tools intended for a non-specialist audience, discussing the types of assessments that students can outsource, showing the type of prompts that can be used to generate text, and illustrating one possible watermarking technique that may allow generated text to be detected. A small-scale study into watermarking suggests that this technique is feasible and show technical promise but should not be relied on as a solution to widespread use of artificial intelligence based tools by students. Alternative solutions are needed, including encouraging the educational community to work with artificial intelligence rather than against it. As such, the paper concludes by discussing seven potential areas for further exploration.
To mitigate potential risks associated with language models (LMs), recent AI detection research proposes incorporating watermarks into machine-generated text through random vocabulary restrictions and utilizing this information for detection. In this paper, we show that watermarking algorithms designed for LMs cannot be seamlessly applied to conditional text generation (CTG) tasks without a notable decline in downstream task performance. To address this issue, we introduce a simple yet effective semantic-aware watermarking algorithm that considers the characteristics of conditional text generation with the input context. Compared to the baseline watermarks, our proposed watermark yields significant improvements in both automatic and human evaluations across various text generation models, including BART and Flan-T5, for CTG tasks such as summarization and data-to-text generation. Meanwhile, it maintains detection ability with higher z-scores but lower AUC scores, suggesting the presence of a detection paradox that poses additional challenges for watermarking CTG.
Meanwhile, it maintains detection ability with higher z 𝑧 z -scores but lower AUC scores, suggesting the presence of a detection paradox that poses additional challenges for watermarking CTG. 1 1 1 https://github.com/FYYFU/semantic-watermark Figure 1: The outputs with the original watermark (OW) (Kirchenbauer et al. 2023 ) and our proposed semantic-aware watermark (SW) on a test example from DART – a data-to-text generation benchmark – with parameters γ = 0.1 𝛾 0.1 \gamma=0.1 and δ = 5 𝛿 5 \delta=5 . We expect ∼ similar-to \sim 90% of human-generated texts from the red list, whereas AI primarily utilizes the green list.
Figure 1 illustrates an example where the randomly bifurcated red list (Kirchenbauer et al. 2023 ) contains key entities from the source that has to be generated for the data-to-text generation task; the mismatch between context and watermark not only impairs detection but also introduces 12 hallucinated words in a 30-token generation. To enhance the effectiveness of watermarks for CTG, we propose a simple yet effective semantic-aware watermarking algorithm that leverages hash function to embed watermarks, while also taking into account the input context and the distinctive characteristics of conditional generation tasks.
We then apply the original watermark and randomly bifurcate the remaining vocabulary. Algorithm 1 Semantic-Aware Watermark Input : Input sequence 𝐱 = { x 1 , x 2 , … , x | 𝐱 | } 𝐱 subscript 𝑥 1 subscript 𝑥 2 … subscript 𝑥 𝐱 \mathbf{x}=\{x_{1},x_{2},\dots,x_{|\mathbf{x}|}\} Parameter : Conditional model p θ subscript 𝑝 𝜃 p_{\theta} , green list size: γ ∈ ( 0 , 1 ) 𝛾 0 1 \gamma\in(0,1) , hardness parameter: δ > 0 𝛿 0 \delta>0 cluster parameter: k ∈ [ 1 , 2 , 5 , 10 ] 𝑘 1 2 5 10 k\in[1,2,5,10] Output : Watermarked text y 𝑦 y 1: Get word embeddings and compute the | V | × | V | 𝑉 𝑉 |V|\times|V| word similarity matrix 𝐌 𝐌 \mathbf{M} .
10: end for To implement this approach, we tokenize the input sequence 𝐱 𝐱 \mathbf{x} to 𝐱 ^ = { x ^ 1 , x ^ 2 , … , x ^ | 𝐱 ^ | } ^ 𝐱 subscript ^ 𝑥 1 subscript ^ 𝑥 2 … subscript ^ 𝑥 ^ 𝐱 \hat{\mathbf{x}}=\{\hat{x}_{1},\hat{x}_{2},\dots,\hat{x}_{|\mathbf{\hat{x}}|}\} . Next, the tokenized sequence 𝐱 ^ ^ 𝐱 \hat{\mathbf{x}} is transformed into contextualized vector representations using the model’s embedding layer. Integrating input information into the watermark’s green list is a direct and crucial step (step 2 in Algorithm 1), consistent with the requirements of CTG tasks where the output is dependent on the input.
• Enhanced Output Quality: Including semantically related tokens in the green list elevates the relevance and quality of the generated text, aligning it more effectively with the CTG task objectives. Assume the word embeddings for a specific model have a size of | V | × d emb 𝑉 subscript 𝑑 emb |V|\times d_{\text{emb}} , where | V | 𝑉 |V| and d emb subscript 𝑑 emb d_{\text{emb}} denote the vocabulary size and the dimension of the model’s embeddings, respectively. Each row in this embedding matrix contains the
Our proposed semantic-aware watermarking method exhibits significant improvements in comparison to the original watermarking method across all datasets and models. Additionally, we observe that hard watermarks invariably cause a greater decline in CTG performance compared to soft watermarks (especially ROUGE-2 for summarization and BLEU for data-to-text generation). The hard watermarks designed for language models (Kirchenbauer et al. 2023 ) essentially completely forbid generation from the red list that might contain key input context, potentially leading to near-ineffective generations with almost no overlap with the reference generations.
With a confidence level of 95% and one-sided A/B tests, the semantic-aware watermark exhibits a significantly higher preference according to human judges ( p = 0.0358 𝑝 0.0358 p=0.0358 ). Specifically, the preference for the semantic-aware watermark (55.33%) surpasses that of the original watermark (44.67%) by a substantial margin of 10.66%. Moreover, pairwise inter-annotator agreement was assessed, resulting in agreement percentages of 70%, 66%, and 54% for the respective evaluations. These findings strongly support the effectiveness of the semantic-aware watermarking method, highlighting its ability to enhance the quality of summarization outputs.
2023 ) . However, as Figure 3 illustrates, the AUC curve for the original watermarking method surpasses ours, as our constructed green lists incorporate more input tokens that humans would commonly use. Consequently, human-generated text also contains more green list tokens. The disparity between the z 𝑧 z -scores and AUC scores of semantic-aware watermarks highlights an additional challenge in applying watermarks for CTG: the common human practice of utilizing input-similar tokens in CTG introduces complexity to the watermark detection process .
Text watermarking aims to subtly embed statistical signals into text by controlling the Large Language Model (LLM)'s sampling process, enabling watermark detectors to verify that the output was generated by the specified model. The robustness of these watermarking algorithms has become a key factor in evaluating their effectiveness. Current text watermarking algorithms embed watermarks in high-entropy tokens to ensure text quality. In this paper, we reveal that this seemingly benign design can be exploited by attackers, posing a significant risk to the robustness of the watermark. We introduce a generic efficient paraphrasing attack, the Self-Information Rewrite Attack (SIRA), which leverages the vulnerability by calculating the self-information of each token to identify potential pattern tokens and perform targeted attack. Our work exposes a widely prevalent vulnerability in current watermarking algorithms. The experimental results show SIRA achieves nearly 100% attack success rates on seven recent watermarking methods with only 0.88 USD per million tokens cost. Our approach does not require any access to the watermark algorithms or the watermarked LLM and can seamlessly transfer to any LLM as the attack model, even mobile-level models. Our findings highlight the urgent need for more robust watermarking.
Our approach does not require any access to the watermark algorithms or the watermarked LLM and can seamlessly transfer to any LLM as the attack model even mobile-level models. Our findings highlight the urgent need for more robust watermarking. The source code is available at SIRA . 1 Introduction Large language models (LLMs), exemplified by ChatGPT (OpenAI, 2024 ) and Claude (Anthropic, 2024 ) , have demonstrated remarkable capabilities in generating coherent, human-like text.
Watermarking techniques for large language models are designed to embed identifiable patterns in model outputs, allowing for the traceability of generated text back to its originating source. These watermarks serve as an essential tool for ensuring accountability and ownership, particularly in scenarios where identifying the specific model or version that produced the content is crucial. LLM watermark methods can be broadly classified into two primary categories: the KGW Family and the Christ Family. Each family employs distinct mechanisms that are integral to the internal workings of LLMs.
We denote Y h Y_{h} is human written text space, Y u Y_{u} is the machine generated unwatermarked text, Y w Y_{w} is the machine generated watermarked text. Definition 2 (Watermark Algorithm): A watermark algorithm consists of a watermarking function W W , a secret key k k , and a detector D D . The watermarking function W W , parameterized by the key k k , denoted as W k W_{k} , modifies the output y y to embed a watermark, given an input prompt x ∈ X x\in X resulting in a watermarked output y w y_{w} which M ( x , W k ) → y w ∈ Y w M(x,W_{k})\rightarrow y_{w}\in Y_{w} .
The detector D D , using the same key k k , can then verify whether a given output y ^ ∈ Y \hat{y}\in Y contains the embedded watermark. The detector D D operates as a binary classifier with the following output behavior: D ( W k , y ^ ) = { 1 if y ^ is detected as watermarked 0 otherwise D(W_{k},\hat{y})=\begin{cases}1&\text{if }\hat{y}\text{ is detected as watermarked}\\ 0&\text{otherwise}\end{cases} (1) The detector D D contains a parameter θ \theta , where the θ \theta is the z-score threshold.
The goal of detector D D can represente as: max θ D \displaystyle\max_{\theta_{D}}\quad 𝔼 y h ∼ Y h [ log ( 1 − D θ D ( W k , y h ) ) ] \displaystyle\mathbb{E}_{y_{h}\sim Y_{h}}\left[\log\left(1-D_{\theta_{D}}(W_{k},y_{h})\right)\right] + 𝔼 y w ∼ Y w [ log ( D θ D ( W k , y w ) ) ] \displaystyle+\mathbb{E}_{y_{w}\sim Y_{w}}\left[\log\left(D_{\theta_{D}}(W_{k},y_{w})\right)\right] (2) For attacker, the perturbation function P P is defined to minimize the probability that the detector D D successfully identifies the watermark in the perturbed output y p y_{p} , while ensuring semantic preservation.
Our attack is based on a common principle of watermarking algorithms, as discussed in the KGW (Kirchenbauer et al., 2023 ; Liu et al., 2023 ) work: since the watermark must remain imperceptible to the user, high-entropy tokens are ideal candidates for embedding. High-entropy tokens exhibit a more uniform distribution of probabilities, this uniformity means that when logits are adjusted to increase the likelihood of green tokens, it is easier to embed watermarks effectively without significantly compromising the quality of the output. Meanwhile this also implied high-entropy token has lower probability thus higher self-information.
The placeholders serve as cues, maintaining the text’s structure, indicating where tokens have been masked which providing the LLM with hints about the original text’s length and the likely number of words, allowing for more high quality reconstructions. Algorithm 1 Pseudocode for Self-information rewrite attack Input: Watermarked token sequence 𝐲 = { y 1 , y 2 , … , y n } \mathbf{y}=\{y_{1},y_{2},\dots,y_{n}\} , language model M a t t a c k M_{attack} , self-information percentile ϵ \epsilon , instruction 𝐬 \mathbf{s} Output: \State \EndIf \EndFor \State Response token sequence 𝐲 𝐩 \mathbf{y_{p}} without watermark.
Therefore, log P ( x w ) + 1 \log P(x_{w})+1 is still negative, and the product with the small δ \delta results in a tiny Δ H \Delta H (decrease in logarithmically). This attribute makes the watermark algorithm need to embed patterns in high-entropy tokens , otherwise it will significantly compromise the quality of the output.
Figure 4 : Comparison of different paraphrasing methods on KGW watermarks. Each word’s color indicates whether it is a green or red token. Fewer green words/lower z-scores suggest a more effective paraphrasing approach. The unwatermarked text represents the
Another option allows ChatGPT to recall old conversations. GPT-based moderation classifiers are used to reduce the risk of harmful outputs being presented to
ChatGPT is a generative artificial intelligence chatbot developed by OpenAI. Originally released on November 30, 2022, the product uses large language models—specifically generative pre-trained transformers (GPTs)—to generate text, speech, and images in response to user prompts. ChatGPT accelerated the AI boom, an ongoing period marked by rapid investment and public attention toward the field of a
Nonsense and misinformation presented as fact by ChatGPT and other LLMs are often referred to as hallucinations. A 2023 analysis estimated that ChatGPT hallucinates around 3% of the time. The term "hallucination" as applied to LLMs is distinct from its meaning in psychology, and the phenomenon in chatbots is more similar to confabulation.
Journalists and scholars have commented on ChatGPT's tendency to output false information. When CNBC asked ChatGPT for the lyrics to "Ballad of Dwight Fry", ChatGPT supplied invented lyrics rather than the actual lyrics.
Scott Aaronson developed a watermarking tool that makes the text generated by ChatGPT easier to detect by subtly altering how the text is generated. The watermarking was claimed to be 99.9% effective on sufficiently long passages and was found not to degrade performance. It was of particular interest for teachers seeking to mitigate cheating. In surveys, respondents favored the release of such a tool by a four-to-one margin, but nearly 30% of users declared that they would use ChatGPT less often if it watermarked outputs and while rival…
Everything we examined (5)
This check searched the claim as stated. It did not run a separate search for evidence against it.