A preliminary experiment regarding consistency as a measure of conceptual abilities in language models (opens in new tab)

Cross-posting from my coworker Caspar Oesterheld's blog which I think is great and generally not well known.I’ve recently been working a little on whether consistency across different questions can be used as a measure of (and perhaps ultimately as a training target for) philosophical competence. I’m in the process of writing up the results into a paper. I’m here reporting results from a small, preliminary experiment that I ran late last year. I’ll leave a more careful discussion (with a prop...