Given an input \(x\) with a pair of responses \(\{y^+, y^-\}\), where \(y^+ \succ y^- | x\), we align \(\pi_\theta\) to favor \(y^+\) over \(y^-\). RRPO training objective is defined as:

\[ \mathcal{L}_\text{RRPO}(\pi_{\theta};\pi_\text{ref}) = -\mathbb{E}_{(x,y^+,y^-) \sim \mathcal{D}} \left[ \log \sigma (u) + \alpha \cdot \mathbb{D}_{\text{TKL}} \big(x,y^+\big) \right] \]

the total reward margin \(u\) is defined as: \[ u = \sum\limits_{i=1}^{N} u_i = \sum\limits_{i=1}^{N} \bigl( r_\theta(x, y^+_i) - r_\theta(x, y^-_i) \bigr) \]

the reward for \(i^{th}\) phrase \(r_\theta(x, y_i)\) is defined as: \[ r_\theta(x, y_i) = \beta \log \left( \frac{ \prod\limits_{j=s_i}^{e_i} \pi_\theta(t_j \mid x, t_{\lt j}) } { \prod\limits_{j=s_i}^{e_i} \pi_{\text{ref}}(t_j \mid x, t_{\lt j}) } \right) \] where \({s_i}\) and \({e_i}\) are the start and end token indices of \(i^{th}\) phrase

the token-wise KL regularizer \(\mathbb{D}_{\text{TKL}}\) is defined as: \[ \mathbb{D}_{\text{TKL}} \big(x,y^+;\pi_{\text{ref}} \,\|\, \pi_{\theta} \big) = \sum\limits_{t=1}^{|y^+|} \mathbb{D}_{\text{KL}} \left( \pi_{\text{ref}} (\cdot \mid [x, y^+_{\lt t}]) \,\|\, \pi_{\theta} (\cdot \mid [x, y^+_{\lt t}]) \right) \]