I think of probability as the primitive (we are surprised *because* of the rarity). So the main question for me is whether S-values serve a human factors purpose.
When I teach significance testing, I talk about surprise (I think I initially got this from Dempster, re: significance tests?) so the general idea of surprise is important to me in expressing (part of) what a significance test does. I also think we probably agree that experienced data analysts/statisticians think of p values on a log scale *anyway*, so in that sense surprisal values are natural to us.
I've learned, though, in years of teaching, that identifying something that seems to have helped us, as experts (e.g. thinking in logarithms), and then applying it trying to improve non-experts' reasoning doesn't always work (the best example I have is geometric interpretations of ANOVA...).
I think that as soon as we switch to S-values, they'll be dichotomized. The reason I think this is that probability is itself not usually dichotomized - it is only in the context of p values that they tend to be. In other contexts people reason about rare events (and other continuous things like time, distance, etc) quite flexibly (if not exactly correctly). In my view the dichotomization of p values occurs for three main reasons:
1. Obviously, probabilistic reasoning IS hard, but...
2. ...not impossible. Students can reason to some extent about probability, but aren't sure about p values specifically when they learn them because they are often contextless (for them). When we think about the size and use of a probability, it is always in a context. We use that context to decide whether something is suitably "rare"/"surprising" or not (something that could kill you might get a different assessment than something occuring in a game, say). Contextless numbers will generate dichotomous thinking, because...what else are we to do with them? Students learn about significance testing WAY too early in their scientific education. Their discomfort (our fault) leads to simplistic thinking and superstition, which is carried forward.
3. The broader "gatekeeping" role of results in science causes dichotomous *behaviour*, which leads to dichotomous thinking.
S-values don't really fix these problems, even though I think they're natural. It would be interesting to assess probability, odds, and S-values experimentally to see which ones people find easier to reason with.
Importantly, all of these are to some extent empirical questions. In this paper (https://richarddmorey.github.io/Morey_Hoekstra_StatCognition/extras/manuscript.pdf) we show that scientists can handle the general idea of comparing to a null distribution and making a decision, even (especially?) when we throw them for a loop and don't give them any numbers to work with. Our conclusion is that the human factors critiques of significance testing are overblown, but that we can't tell whether the issue is their expression (e.g. p values), the broader scientific context, or both.
