Adds the CPO Alignment Loss Function

[pramodith]

Summary

CPO is almost the same as DPO with the major difference being that the Reference Model in CPO is assumed to be a Uniform distribution. This assumption leads to the cancellation of all terms related to the reference model.

$$CPOLoss = -\log(\sigma(\beta\log(\pi_\theta(y_c|x)) - \beta\log(\pi_\theta(y_r|x))))$$

This corresponds to equation 3 in the paper. Additionally CPO also assumes a scaling factor alpha for the NLL loss on the preferred response. In TRL this corresponds to the CPOTrainer using a loss_type="sigmoid"

We also refactor the test cases for chunked loss functions to include a generic HFAlignmentLoss base class that takes care some of the plumbing work to correctly generate batches of input, calculate the NLLoss etc. All future test cases can inherit from this class and just implement the alignment_loss function to compare implementation in the TRL lib versus the custom impl.

Testing Done

A100-80G-SXM

Benchmark Results:

• Hardware Type:
• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence