Metakgp:SCAMP

SCAMP stands for "Sanity Checking Any Maintainer Proposal". It's a simple algorithm for generating a somewhat reproducible score for any proposal, based on Kahneman's idea that algorithms are more noise-free than people's subjective judgements, and often perform better in low-validity situations.

SCAMP is not the final word on the proposal. It need not even be important - after scoring a proposal, you're free to discard the score and then vote whichever way you want. Its most important goal is to make you think harder about each proposal.

Measurable

This score represents how easy it is for anyone to agree after-the-fact whether someone succeeded in their proposal, or failed.

1. Impossible to measure: there's no way to say if the proposal succeeded.
2. Difficult to measure: it might be possible to measure this, but < 50% people agree on how, or it'll take > 2 days
3. Possible to measure: > 50% people can agree whether the proposal succeeded or failed if they spend < 2 days trying
4. Easy to measure: > 75% people can agree whether the proposal succeeded or failed if they spend < 1 hour trying

Calibrated

This score represents how carefully the proposal considers possibilities for failure, and how to mitigate them.

1. Blind to failure: almost no discussion about why the proposal could fail (or has failed in the past)
2. Admits possibility of failure: mentions some failure modes (or past failures), but says almost nothing to address them
3. Seriously considers failure: devotes < 25% of the proposal to giving a clear picture of how the proposal could fail (or has failed in the past)
4. Nate Silver: devotes >25% of the proposal to explaining how the proposal has failed in the past, and admits a significant (> 25%) possibility of failure this time, despite best efforts.

Aligned

Assuming this proposal succeeded, this score represents how much you agree with the objectives of the proposal, and the reasoning behind it.

1. Actively harmful: if the proposal worked, it would have been better to do nothing at all
2. Unaligned: if the proposal worked, it would make so little positive impact that just spending that time doing the current highest impact activity (eg. writing wiki articles, contributing to metakgp github projects) is better
3. Impactful: if the proposal worked, it would have enough impact that it's worth doing this instead of the current highest impact activity.
4. Awesome: if the proposal worked, it would have at least as much impact as the current highest impact project (eg. wiki, mftp, mfqp, naarad, or whatever you think is highest impact).

Coherence

This represents how well defined and well reasoned is the proposal.

1. All air: The goal is very vague, the steps are unclear and there is no time line. Basically the proposal doesn't tell what you'll do, how will you do it and when will you do it.
2. Largely vague: The goal is clear but step to take and time line are vague.
3. Sketchy: The goal is clear, the steps to take and time line is broadly defined.
4. Rock solid: Everything, the goal, the step and the time line is very well defined.

Magic

This score represents what you personally, subjectively, feel about the proposal, irrespective of any objective criterion (including the ones above).

1. Don't like it
2. It's okay
3. Like it
4. Love it

How to score

Pick a single number from each of the sections. Half points (eg. 3.5) are okay, but no other decimals allowed. Add the score from each of the sections. The final score can be from 5 to 20.