On impact-effort prioritization

For a lot of people, the impact-effort matrix is a quick way to work out what they should – or shouldn't – focus their efforts on.

This is how it's typically done:

  1. Make a list of things you want to work on
  2. Get a bunch of people together.
  3. Ask everyone to estimate how much effort it will take to do each item in the list, and how much impact it will have.
  4. Pick the things that most people agree have the most impact for the least effort.
  5. “Just do it”

On the face of it, this seems straightforward enough. Here's what doesn't happen between steps 3 and 4:

3a. Ask everyone to put a confidence number on their estimate for the effort (because planning is hard) 3b. Ask everyone to present data that supports their idea of how much impact each thing will have (because without data, all you have is an opinion) 3c. Ask everyone to specify who will be impacted, and how (because if you're comparing things that impact completely different sets of audiences, you'll end up comparing apples to oranges) 3d. Consider whether, over time, the effort required to produce the desired value will increase, go down, or stay the same (because when was the last time you planned for a feature to be delivered in 2 months, and it got delivered in 2 months?)

If you aren't doing 3a – 3d, what are you really doing with your prioritization?