No announcement yet.

Process: Why I Can't Take More Playtest Data Now

  • Filter
  • Time
  • Show
Clear All
new posts

  • Process: Why I Can't Take More Playtest Data Now

    Hey all. I'm seeing some (I assume well-meaning) people who are still posting playtest responses and information for Pirates of Pugmire. I've posted in a few places that I can't accept any more data, and I wanted to go into why a bit more.

    First off, this is the first time I'm using an open playtest process for Pugmire, but it's not the first time I've done open development. I did a few books like that back when I was in charge of Vampire: The Masquerade between 2011 and 2016, and on those I tried to make sure I got every scrap of information I could from everywhere I saw it. However, the problems with that method were two-fold:

    1) People will start to post everywhere, assuming I can see it. Not only does it make it difficult for me to see any one piece of data, but it's even more difficult to track it. Books are often out of my hands for weeks or even months at a time, and trying to remember where I saw that Dominate had a number wrong three months later is maddening. The alternative was to devote time to transcribing every piece of data I got, which takes way too much time away from me actually making books, and I often lose the context the comment was made in as well.

    2) It gets outdated very quickly. Easily 80% of the errata I've seen since I closed the form are things I've already addressed before the manuscript went to editing, and the other 15% are the kinds of things the editing process is designed to catch. I can't keep sharing iterative versions of every chapter, because again it takes a lot of time and causes a lot of confusion. And I can't say what the end result it, because it might change again (or several times!) between now and the time the PDF is laid out.

    In addition, how I develop is to look at information as a collective. One person saying one thing feels wrong isn't data -- it's an anecdote. Six different people saying it in slightly different ways gives me a much larger picture, and helps me to assess what's truly wrong. And sometimes, what appears to be a small change reveals a larger, more systemic concern. This occurred in Aberrant playtesting, for example, where an entire subsystem was rewritten before the Kickstarter because the numbers are a bit off in a few of the Edges.

    That's why I went with one point of contact for this round of playtest data, and why it's "blind" (i.e., other players can't see the results). The downside to that, naturally, is that everyone assumes their concern is fresh and new, when I might have dealt with it months ago. Once in a while -- that other 5% -- something does come up. And back when I was doing open development, that was a genuine concern. But now we have a good, solid errata process, where people can comment on the PDF for a week or two before it goes to press. It's a chance to catch that last 5%, while still giving me that blind, comprehensive picture I need to make good design choices.

    So to be clear, I am really pleased that lots of you are excited and eager. It means the world to me that you're passionate about the world of Pugmire! However, I really do have good reasons for the process I detail. If you want to chat among yourself about the manuscript, go ahead! But please don't tag me on it or expect me to wade in with an official answer, because the official answer will be "wait until you can see the errata PDF."

    I hope this helps!

  • #2
    Thank you for bringing this up, Eddy. Both from my WW/CCP/OPP history and my indie stuff now... this is really important, and applies to most creators/publishers. Given an audience of any real size, there's only so much we can do to process the incoming information. And despite the newfound openness and flexibility of recent years, all of these books are crafted in phases... and on deadlines.

    Feedback and enthusiasm are a big part of what keeps us going, but the process of dealing with anything that might change a book has to be timed and focused.

    Cavaliers of Mars Creator
    Retired CofD Lead

    Check out my guides to Vampire and my indie games!