End-symbols are mathematically necessary in n-gram models
the verdict
SUPPORTED
the evidence backs this
refutedsupported
the weight of evidence
2 sources for · 0 against
AS REPORTEDno primary record reached; this is what the reporting says
Linguistic and computational resources establish that end-symbols are required in n-gram models so that sentence probabilities sum to one and form a true probability distribution.
# Jurafsky Speech & language processing Chapter 3 "end-symbol to make the bigram grammar a true probability distribution"
- Tags: probability, formal-languages, hidden-markov-models
- Score: 0
- Views: 33
- Answers: 0 (unanswered)
- Asked by: alpharing (19 rep)
- Asked on: Mar 12, 2025
- Last active: Mar 12, 2025
- License: CC BY-SA 4.0
---
## Question
In Chapeter 3 of Speech & language processing, Jurafsky says
"We need the end-symbol to make the bigram grammar a true probability distribution. Without an endsymbol,instead of the sentence probabilities of all sentences summing to one, the sentence probabilities for all sentences of a given length would sum to one. This model would define an infinite set of probability distributions, with one distribution per sentence length. See Exercise 3.5."
For the unigram case, I proved the sum of probabilities of all sentences are sum to 1 by following arguments. Let $P\_s$ as the probability of the end-symbol. Then the probability of length $n$ sequence $P\_n$ would be
\\begin{equation} P\_n = P\_s \\times (1-P\_s)^{n} \\end{equation}
Then, we have
\\begin{equation} \\sum\_{n=0}^{\\infty} P\_s \\times (1-P\_s)^{n-1} = 1 \\end{equatio
# What's the real need for an end-symbol in n-gram models?
Tags: nlp, n-grams
- Score: 9
- Views: 4017
- Answers: 3
- Answered: yes
- Asked by: mcrisc (191 rep)
- Asked: 2015-02-24
- Edited: 2015-02-24
- Site: linguistics
## Question
There's a footnote in Jurafsky & Martin (2008, p.89) pointing out that, without an end-symbol, an n-gram model would not be "a true probability distribution". Even after seeking the paper they've referenced, I could not get the point. Here is the example and the footnote:
<s> I am Sam </s>
<s> Sam I am </s>
<s> I do not like green eggs and ham </s>
About the end-symbol, </s>, the authors say:
"We need the end-symbol to make the bigram grammar a true probability distribution. Without an end-symbol, the sentence probabilities for all sentences of a given length would sum to one, and the probability of the whole language would be infinite."
Could someone help me to interpret the quoted part above? Or, does anybody know about some resource that could help me understand what is the need for an end-symbol in n-gram models?
## Answers
### Answer by Draconis (score: 5)
Without loss of generality, let's consider a bigram model (looking at two words a
Everything we examined (2) — 1 independent source
This check searched the claim as stated. It did not run a separate search for evidence against it.