I get the sense that the earlier versions of language models used to be far more creative, dare I say more human. Before all of the RLHF, these models were pure creations of the collective's footprint on the internet. They were unconstrained.

Now there is a focus on 'correctness' and eliminating 'hallucinations'. I think this has killed their creative spirit, just as it would kill a human's. It reminds me of what many have highlighted with the current failings of Science. People are focused on withholding the existing dogma that they never challenge the status quo. This is the death of progress.