Language models ignore word order in the context.
Is it true that language models ignore word order in the context?
Partly. The extent to which language models ignore word order remains a subject of debate in recent literature. Some studies find that altering word order has surprisingly little impact on certain downstream tasks, while others demonstrate that models actively utilize positional encoding and internal mechanisms sensitive to the sequence of tokens.
Verdict: CONTESTED · 2 sources for, 2 against · Last checked
The retrieved papers present conflicting evidence regarding whether transformer language models ignore or rely on word order. Papers [1] and [6] suggest that word order shuffling has minimal impact on model performance in certain contexts, supporting the idea that models can largely bypass fine-grained word order. Conversely, papers [2] and [9] demonstrate specific model sensitivities to token order and show that incorporating positional encodings significantly improves performance, indicating that word order is tracked and used. Because strong evidence is present on both sides, the verdict is CONTESTED.
The evidence we hold leans evenly split
How this was weighed
official record 3x · fact-check 2x · hedged 1x · crowd & reference 1x
- What Context Features Can Transformer Language Models Use? · peer-reviewed · supports · weight 1.6 · 2021
- Shaking Syntactic Trees on the Sesame Street: Multilingual P · peer-reviewed · supports · weight 1.3 · 2021
- Talking Heads: Understanding Inter-layer Communication in Tr · peer-reviewed · refutes · weight 1.3 · 2024
- Improving Part-of-Speech Tagging with Relative Positional En · peer-reviewed · refutes · weight 1.05 · 2025
Get citation
O’Connor, J. & Andreas, J. (2021). What Context Features Can Transformer Language Models Use?. doi.org. https://doi.org/10.18653/v1/2021.acl-long.70
Get citation
Merullo, J., Eickhoff, C., & Pavlick, E. (2024). Talking Heads: Understanding Inter-layer Communication in Transformer Language Models. doi.org. https://doi.org/10.48550/arXiv.2406.09519
See more details
Get citation
Taktasheva, E., Mikhailov, V., & Artemova, E. (2021). Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations. doi.org. https://doi.org/10.18653/v1/2021.mrl-1.17
Get citation
Achir, J. A., Mohammad, A., & Abdullahi, M. (2025). Improving Part-of-Speech Tagging with Relative Positional Encoding in Transformer Models and Basic Rules. doi.org. https://doi.org/10.56705/ijodas.v6i2.184
- Is it true that grammatical number changes word order in certain languages? PARTIAL LEAN
- Is it true that word meanings are determined by their patterns of usage in linguistic contexts? SUPPORTED
- Is it true that modern English contains non-V2 word order patterns? SUPPORTED
- Is it true that Germanic languages share specific word order typological features? PARTIAL LEAN
- Is it true that a single word can be classified into two different word classes depending on context? SUPPORTED
- Is it true that languages exist where every character's pronunciation depends entirely on its word context? REFUTED
Receipt History
Challenge the receipt
Citation formatting by citeproc-js (Frank Bennett) and the Citation Style Language project. Source and licenses.
Terms · Privacy · How verdicts work