Despite its empirical success, Reinforcement Learning from Human Feedback (RLHF) has been shown to violate almost all the fundamental axioms in social choice theory -- such as majority consistency, pairwise majority consistency, and Condorcet consistency. This raises a foundational question: why does RLHF perform so well in practice if it fails these seemingly essential properties? In this paper, we resolve this paradox by showing that under mild and empirically plausible assumptions on the preference profile, RLHF does satisfy pairwise majority and Condorcet consistency. These assumptions are frequently satisfied in real-world alignment tasks, offering a theoretical explanation for RLHF's strong practical performance. Furthermore, we show that a slight modification to the reward modeling objective can ensure pairwise majority or Condorcet consistency even under general preference profiles, thereby improving the alignment process. Finally, we go beyond classical axioms in economic and social choice theory and introduce new alignment criteria -- preference matching, preference equivalence, and group preference matching -- that better reflect the goal of learning distributions over responses. We show that while RLHF satisfies the first two properties, it fails to satisfy the third. We conclude by discussing how future alignment methods may be designed to satisfy all three. Finally, we go beyond classical axioms in economic and social choice theory and introduce new alignment criteria— preference matching , preference equivalence , and group preference matching —that better reflect the goal of learning distributions over responses. We show that while RLHF satisfies the first two properties, it fails to satisfy the third. We conclude by discussing how future alignment methods may be designed to satisfy all three. In such cases, the reward modeling objective only approximates the true preference structure by fitting the best possible BT representation. This naturally leads to the question: how does the reward behave under non-BT preferences? Answering this question requires revisiting classical results in social choice theory (Brandt et al., 2016 ) , which studies how to aggregate individual preferences into a collective decision. 3 Pairwise Majority Consistency and Condorcet Consistency of RLHF Answering the above question requires delving into social choice theory, which studies how to aggregate individual preferences within a group to arrive at an optimal collective decision. We begin by briefly introducing the basic concepts of social choice theory. Definition 3.1 (Aggregation Rule) . An aggregation rule (or social choice function) is a function f : 𝒯 m → 𝒪 ​ ( 𝒴 ) , f:\mathcal{T}^{m}\to\mathcal{O}(\mathcal{Y}), where 𝒯 m \mathcal{T}^{m} denotes the set of all possible preference profiles over m m voters, and 𝒪 ​ ( 𝒴 ) \mathcal{O}(\mathcal{Y}) denotes the set of strict rankings over the response set 𝒴 \mathcal{Y} . Given a preference profile 𝝉 = ( τ 1 , … , τ m ) \boldsymbol{\tau}=(\tau_{1},\ldots,\tau_{m}) , the function f ​ ( 𝝉 ) f(\boldsymbol{\tau}) returns an aggregated ranking over 𝒴 \mathcal{Y} that represents the collective preference of the group. Reward Modeling as an Aggregation Rule. Based on Definitions  2.1 and  3.1 , the maximum likelihood estimation (MLE) of the objective of reward modeling (Equation ( 2 )) can be interpreted as an aggregation rule: it takes However, we argue that the fact that RLHF satisfies pairwise majority and Condorcet consistency in practice is largely incidental, driven by the cost constraints that limit each comparison to a single labeler. In future settings, where developers may collect comparisons from multiple labelers, care must be taken—Copeland RLHF may provide a more principled and robust alternative. 4 Axioms for Preserving Diverse Human Preference In this section, we move beyond the traditional frameworks of economics and social choice theory to propose new axioms that alignment methods for LLMs should satisfy in order to preserve diverse human preferences. 7 7 7 Probabilistic aggregation rules have also been studied in social choice theory, typically focusing on the probability of each candidate being selected as the winner. This differs from our setting. Definition 4.1 (Probabilistic Aggregation Rule) . A probabilistic aggregation rule (or probabilistic social choice function) is a function ρ : 𝒯 m → Δ ​ ( 𝒴 ) , \rho:\mathcal{T}^{m}\to\Delta(\mathcal{Y}), where 𝒯 m \mathcal{T}^{m} denotes the set of all possible preference profiles over m m voters, and Δ ​ ( 𝒴 ) \Delta(\mathcal{Y}) denotes the set of distributions over the response set 𝒴 \mathcal{Y} . We now proceed to define the strongest desirable property. Axiom 3 (Group Preference Matching) . Given a complete preference profile 𝛕 \boldsymbol{\tau} , the probabilistic aggregation rule ρ \rho must return the group preference matching distribution 𝐩 ∗ \boldsymbol{p}^{*} , i.e., ρ ​ ( 𝛕 ) = 𝐩 ∗ \rho(\boldsymbol{\tau})=\boldsymbol{p}^{*} . This represents a strong requirement for AI systems to faithfully preserve human preferences. Although we have shown that RLHF is consistent with traditional social choice theory under certain conditions, it does not satisfy this property. Proposition 4.1 . In Section  4 , we introduce axioms aimed at faithfully preserving human preferences, which contribute positively to the fairness and representational equity of LLMs. As for potential negative impacts, this is primarily a theoretical study and does not pose direct societal risks. Acknowledgments This work was supported in part by NIH grant U01CA274576, ARPA-H Award D24AC00253, NSF grant DMS-2310679, a Meta Faculty Research Award, and Wharton AI for Business. References Anthropic [2024] AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card , 2024. Arrow [2012] Kenneth J Arrow. Social choice and individual values , volume 12. Yale university press, 2012. Azar et al.