CES Journal Design Review
Goals: Build the field, improve signal to noise, align incentives transparently.
respect reviewer attention; focus on constructive new value, not gatekeeping.
aim for substance over form, good rather than perfect, experiments.
We’re starting with a simple experiment: doing what we know from CS and computer security; peer review chosen from academics we know.
Peer review ideas
Assign reviewers from at least 2 different fields
Foreword from the reviewers: Ask reviewers to collab on a summary to publish alongside the paper as supplemental value
Validation reviews: focus on explaining standards for eval in each field, including different core facets of validation
Q&A (town hall, asking one another)
Q: How will you iterate on feedback and change the journal+process?
A: at least once a year; performance metrics before each conference like this. Want to poll authors after the fact, have other workflows decided in advance to see what’s working.
Q: How do we try things, fail, and iterate? How do we publish and review in a way that helps more meaningfully develop a body of facts and understanding in this [noisy] enviroment?
A: Ex: some PoS discussions start w/ induction on a fallacy (that’s not a distributed system; there are other circular arguments too). I was an editor for the Ledger journal, a pervious one in this space. The quality of submissions were abysmal. Why? This community may have more people from distrib systems and cryptography; but we had mainly economists who misunderstood a lot of things, leading to ‘not even wrong’ ideas, frameworks, proposals.
A: Don’t allow logical fallacies; you can’t get people out of circular arguments.
Q: I publish an OA econ journal; focused on short letters, 7pp or less. The stanrdard was ‘correct, original, at least of interest to a specialist’. We initially had an ‘up/down’ review model, but this didn’t work. Asking people to improve papers was difficult: why take the reviewer’s view of what is interesting, if you have correct claims to share? Encourage a light touch.
A: we have a diamond OA agreement; noone pays to publish; you can upload papers to eprint or arxiv; we expect that. You won’t need our review process to publish your work. Some communities worry that publishing on arxiv may tamper w/ double blinding.
Q: What do you think can be the collab w/from industry? Wishlists, problems, use cases, [reviewers]?
[list of 7 things a citation could mean]