pull down to refresh
I guess a new framework will emerge where papers will be triaged by AI before sending them out for human review.
I'm doing this for code. It's very time consuming though, because you have to do multiple rounds. I was complaining to k00b last night that 2 bots flagged up a design decision in my code after literally 2 weeks of work, with on average 2-4 bot reviews per day, but I woke up this morning realizing that what it really means is that I'm starting to have code that is free of quirks now, so that they have to make up really nasty shit to deliver on the "find issues" part of the instruction.
I wish they did that in the beginning though. lol
if you ask ai for feedback, does it ever stop nitpicking? that's a big issue i find with using ai for review, it doesn't know when to stop nitting. If it doesn't have a reasonable stopping rule, you can't really trust the feedback
does it ever stop nitpicking?
It used to but not anymore. There's been something added to training data, I guess to Claude 4.7, that then propagates through traces of Claude executions to all the other models. The solution is usually to scope the review out by section demarcation and then asking it to review each section in subtasks, bring it all together, and do a final scan of the entirety.
This way, you can get a lot of "no findings" on individual sections, which is more reliable. I never instruct literally for a no findings case though, that goes at the cost of completeness because you're giving the bot an out - the "cheating" is trained in too.
If it doesn't have a reasonable stopping rule, you can't really trust the feedback
I disagree with that. In code, you can address a finding with a clarifying comment too.
"Looks good to me"... had to look that one up.
Yeah, that's a fair point. It's not like human referee reports are known to be always right or correct, either. With proper prompting by the editor, he may even remove several of the biases we have been struggling with for years (with attempts at resolving them through double-blind refereering, etc)--while introducing new ones, for sure[1].
I guess a new framework will emerge where papers will be triaged by AI before sending them out for human review... with unintended dirty side effects (#1551663).
Coz AI is so woke, except for Grok. \s ↩