Wednesday, October 27, 2010

GDS2 Multiple Choice Test

The GDS2 multiple choice test has several questions that I believe deserve a second opinion. It's a bit hard to claim this has wider application, so I won't. I'm only going to talk about a few of the questions for which I believe there is room for debate around the answer WotC has claimed is correct. I haven't read all of the debate about them, so I don't know exactly what's been said elsewhere.
5) If Design wanted to change this card's rules text but keep it a hybrid card, which of the following options would not work?
a) flash; shroud
b) double strike
c) 2 Mana: Switch Deflecting Mage's power and toughness until end of turn. 
d) Tap: Target player gains 2 life. 
e) Tap: Target player discards a card. Activate this ability only any time you could cast a sorcery.– CORRECT


Number 5 I don't have a big problem with, but I wanted to point out an easy mistake you could make in thinking about it. They want a hybrid card, not a gold card. A black 1/4? yuck. But a black / white 1/4 that makes you discard... well it's acceptable, at least. Meanwhile a flash shroud 1/4, or a doubestrike 1/4, those are both too monocolored to be gold. Same can go for c or d. They are all really monocolored cards as 1/4s. (blue, white, blue, white, respectively). So while I can't argue strongly that e is wrong, the question overall seems pretty weak to me, as it's too easy to consider a, b, c, and d as monocolored cards. Hybrid treads a thin line between monocolored and gold, so they're some of the hardest cards to design.


6) Design often makes creatures that have flash and "enters the battlefield" triggered abilities like Deflecting Mage. Which of the following abilities would we least likely pair with flash?
a) Counter target spell.
b) The next time target instant or sorcery spell would deal damage, it deals double that damage instead.   
c) CARDNAME deals 2 damage to target attacking or blocking creature. 
d) Prevent all combat damage that would be dealt this turn. 
e) Target creature gets +3/+3 until end of turn. – CORRECT

The claim WotC makes here is that "e) is the only "enter the battlefield" effect you would ever see of the above on a creature that didn't have flash" and therefore the least likely to be paired with flash. I don't think that's accurate at all. Exhibit A: Briarhorn - the likelyhood of e+flash = 100%. It was done, and not in ancient history, but recently, in a block I would consider part of modern design. Answers a and d are seen often enough, both as recent or more recently than e in one form or another. This leaves b and c. Neither has been done yet, though Bogarden Hellkite is a superset of c. Clearly they cannot be done without flash, but that doesn't mean any design team should hand them off to development. Their "likelihood" seems pretty low to me. I could see b as a cool new uncommon or rare cheap red creature. If I wanted to design c I would be much more likely to design a white creature with flash, 2 power, and that either has first-strike or gives itself first-strike when it enters the battlefield.

Overall I can see the argument for b or c, but not e. Also b and c are so close for me that I wouldn't put them together in this question.

19 and 20 use this card:

Take It and Suffer
Enchantment – Aura
Enchant creature
When Take It and Suffer enters the battlefield, destroy enchanted creature. Take It and Suffer deals damage to target player equal to that creature's toughness.
19) If this card is black, which change is Design most likely to make to this card?
a) Add "nonblack" to the enchant ability
b) Change the damage so it is equal to the creature's power rather than its toughness.
c) Change the destruction to a sacrifice effect.
d) Change the damage to loss of life.
e) Make the card an instant or sorcery. – CORRECT 
My disagreement here comes from an insider view. It's quite interesting: Submitting it as an enchantment is so ridiculous that there must already be an established reason for it. Clearly overthinking, but there would never be anyone on a design team that would submit this as an enchantment unless that was a prerequisite. Design doesn't take card ideas from the outside, so there must be an important reason to keep it an enchantment. Perhaps it's for an enchantment themed block. With e out of the way, d is the next most likely change, followed by a. Don't try this at home kids.

20) Could Design change this card into an Equipment with only minor tweaking?
a) yes
b) no – CORRECT

What?! You're kidding! Design is way more powerful than that:
When Take It and Suffer becomes attached, destroy equipped creature. Take It and Suffer deals damage to target player equal to that creature's toughness. equip:5
Super easy. Is this card a good idea? It's no worse than the original aura version. Mind you, I'd still answer b to the question because it's obvious that what they are asking is "should design make this into an equipment" not "could." It's pretty clear that nobody from the templating team looked at the questions before they went out. Also, Mark's explanation is completely wrong. It's perfectly useful for flinging your creatures at the opponent. Fling itself is in M11, so clearly design has no issue with fling effects.


45) The philosophy of four of the five colors is stated below. Which color is missing?
  • "Morality? There's no such thing as morality. It's a construct of the weak to justify their actions."
  • "What value is there in thinking about tomorrow? Who knows if we'll even be alive tomorrow?"
  • "Everybody is trying so hard to change everything that they sometimes miss that things don't need to be changed."
  • "Any problem that is understood can be solved."
And the choices where the 5 colors, of course. My issue here is with the 4th quote - the one about change. I actually believe that is a white philosophy and not a green philosophy. Green is about life, and growth, and the natural way of things. All of those are about change. Life is change, growth is change, and the natural world is constantly changing. Green loves change more than any other color! This philosophy cannot be green's, and therefore green is missing. Preventing change is somewhat white - as white likes a ruled, ordered society. Those in power stay there, rules are not broken, things do not change. I was totally shocked by the answer to this question more than any other on the test.

46) I'm not going to quote the question here. Just a cute tidbit: I got this one wrong because I designed the correct answer card and was blinded by my love for my own card. "It's so cool" does not make it Johnny. Heh.


48) One of R&D's ongoing concerns is board complexity. We've coined the term "virtual vanilla" to refer to a creature that, after the first turn it enters the battlefield, functions as a simple vanilla creature for purposes of evaluating the board state. (Avoid getting hung up on obscure combinations of cards that could make the card not function as a vanilla.)
Here are ten creatures:
WotC seems to have admitted that this question was a bit of a mistake. I think it's a big mistake. It's very bad for a test when you can be graded wrong when you are right, but much worse when you can get credit when you are wrong (don't ask me to explain that). The issue here, by the way, is that "virtual vanilla" is clearly a superset of actual vanilla and therefore in any reasonable analysis there are 8 virtual vanillas here. This is a clear and simple question that turned out was a trick, and the kind of trick the GDS2 participants were promised would not appear on this multiple choice test.


There are always going to be some questions you can argue about, like number 6. However, I believe that 45 & 48 are so egregious that the tests should be regraded on those two questions. This test is taken very seriously by a lot of people desperate to get a chance as a designer. It's sad that some participants missed the cutoff because of serious errors like those made in the design of questions 45 & 48.

11 comments:

  1. Further comment on #6: If it was "which card in this cycle should we remove flash from" it would be more obviously E. But the question feels much more like "which of these are we least likely to print" and that leads to a more historical outlook.

    ReplyDelete
  2. I think you have made a mistake in your analysis of question 48. There are seven Virtual Vanillas there, which includes the one Actual Vanilla. Did you accidentally include Squadron Hawk in your count?

    ReplyDelete
  3. Disagree completely :-)

    5 is absolutely unambiguous. The first four abilities each appear in two colors; it's okay to pay only red mana or only white mana for double-strike, for instance. Discard, however, is mono-black. It breaks the color pie to be able to make an opponent discard as an activated ability by paying only green, red, blue, or white.

    6 was a poorly phrased question, and people misunderstood it. However, the question was intended to ask: "Given that me are printing creatures with the following abilities, which is most likely to not also have flash." That's how I read the question, and I nailed it; given a proper understanding of the question, there is only one answer. The first four abilities all do NOTHING unless the creature has flash. The Brianhorn ability is certainly stronger as an instant, but is entirely playable at sorcery speed.

    I'm completely boggled by your overthinking of 19; it is a one time effect and there is no conceivable reason to make it an enchantment. 20, you missed the obvious: by making it an equipment you can no longer play it on an opponents creatures, completely changing the effect of the card.

    45 is also quite clear; enjoying the past, and the recurring cycle of nature, is clearly concerned with things saying the same. MaRo made this clear in the philosophy of colors articles he linked to for homework. #33, however, was the color question that was sketchiest--in those articles, 'harmony' is mentioned in the white one and not the green one. Apparently we were expected to search Gatherer for Harmonize.

    #48, however, was a bad question. The 'virtual vanilla' including 'vanilla' was obviously a trick question; and, because it was asking for a number, people who mis-categorized two cards could have lucked into the opposite answer.

    Still, thanks for you ideas--they're though-provoking, at least!

    ReplyDelete
  4. Monty: Oh hey, you're right. Lolz @ me. Why were people complaining about this question? Once I saw the complaints I figured it was because everyone thought there are 8, not 7, but really everyone thought there were 6? That's just silly!

    Eli: for #5 I didn't mean to say I disagree with the answer, I just wanted to talk about the difference between gold and hybrid. I agree with what you're saying.

    I agree with you on #6, but that's not how the question was worded, see my other comment.

    For 20, skipping to the equipment version, I did not miss that you could no longer target opponent's guys. I also kind of thought my flippant semi-sarcastic "you're kidding" would be more obvious in writing than it was, sorry.

    Still, I made the argument, and for that I guess I was thinking about "changing the wording" but "changing the effect" is pretty big too. Design often makes changes about what a card can target and they always seem like small changes to me, even if they have a big effect on how the card is used. Perhaps I was thinking too much about "can we get a card out of this" and not enough "what is this trying to do."

    For 45, I just plain disagree with what Mark is claiming about green and change. The cycle of life is a series of major changes. It's a repeating loop of changes! Birth, growth, hunting, death, eating. All life does is change. Trust me, I'm a biologist, or at least I was, before I was a game designer.

    Thanks for your comments. It's fun to talk about.

    ReplyDelete
  5. The point about 48 is that the dictionary definition of the word "virtual" is the antonym of "actual" -- so it is technically and literally accurate from the given definition that actual vanilla is a subset of virtual vanilla, but it makes no plain-language sense.

    Thus, especially given the ambiguities elsewhere on the test (demonstrating the sloppy wording job on the exam), the taker is left to guess whether the question was plain-language inaccurate or technically inaccurate. This is not a good choice to which to put a test-taker, and it's not at all silly for people to complain about it.

    I also agree that #6 was a bad question and that #33 was the worse of the two "color philosophy" questions.

    ReplyDelete
  6. Hi Greg! :)

    I am annoyed. I haven't read through all of this but just read #6 and it is wrong. The correct answer is B not E.

    There are a few reasons why B is the correct answer.

    First, of the five abilities listed, B is the text box we do the last. Therefore it is much better to as an instant spell, not tacked onto a flash creature. The reason +3/+3 (answer E) is a great fit for a flash creature CIP, is that we need to do a lot of giant growths, and finding new "interesting" ways to make them is important. This "need to find new interesting ways to do it" does not apply to things we rarely make.

    Secondly, E is likely to be made because it can be cast on opponent's turn when they are attacking and target itself, taking down a much larger creature. There is a nice "oh cool, this card is better than I first thought" moment there. Really a "great" design, one I would expect from a "great designer".

    Thirdly, B's effect on a flash creature plays clunky. If the instant that does this costs N, then the flash creature that does this costs N+1 if it is a 1/1 or possibly N+2 if it is larger. If I want to target my lava axe, and this creature costs 3 or 4, that means I must have 8 or 9 lands in play? That is frustrating. Not a "great" design. One I would expect from a "not great designer". (All the others can be cast and make use of their CIP on the turn you have enough lands to cast them, much less clunky).

    Fourthly, (this one is good), A,C,D and E HAVE ALL BEEN MADE!!! If the question is least likely, I think it is safe to pick the only one that HAS NOT BEEN MADE!!! :)

    (FYI Rosewater is the nuts, ie. please don't hate me Mark!) :)

    Matt Place

    ReplyDelete
  7. I haven't played Magic regularly in quite a while. Your strange new terms and abilities startle and confuse me, so take my opinion with a grain of salt...

    That said, I would have immediately picked out that quote in question #45 as belonging to White and not to Green. As an "anti-change" sentiment, it fits with White, which represents order as well as righteousness, and is particularly opposed to the life-and-change-themed Green and the war-and-chaos-themed Red. I would think Black and Blue would be more neutral to it.

    ReplyDelete
  8. I agree that the Green philosophy is badly defined in that question. To put it crudely, sometimes it feels like green is more about hating blue (in this question, blue = change) than about anything it actually does - like turning lands into other lands, upgrading creatures, animating trees...

    ps. at matt place: b literally doesn't function without flash, and you can target lighting bolt with it just fine. the question asks, IF the listed abilities are printed *word for word* on a creature spell, which ability is least likely to also have the flash ability? Since four of the options have zero chance of not having flash, and E has non-zero chance of not having flash, the answer has to be E.

    ReplyDelete
  9. Philip,

    You just wrote stuff that isn't in the question while implying that it is. :) I had heard earlier that we weren't supposed to do that? I'll find out for sure and let you know.

    Matt

    ReplyDelete
  10. Phillip,

    Matt's right. You and many others added words to the question that are simply not there. It may be that R&D intended to write the question you describe for #6... but they did not write that. They wrote a very different question; one that asks which of the choices is most likely to exist. If they every printed B it would be a mistake, and they have printed (or nearly printed) A,C,D, and E.

    In my graduate school days as a TA of premed students I learned that multiple choice questions have to be much more clear than this. R&D got away with some rather loose question writing because nobody can hold them accountable for it.

    (R&D people are all totally awesome, but they're not infallible.)

    "be paired with" is the phrase that's causing this problem. What does that mean? It's very ambiguous. As this is a design test, I thought it meant "which of these would least likely be submitted as a design" - which is clearly B, for the reasons Matt stated.

    When they wanted to force you to do something, they made it very explicit "you must move an ability from one color to another" for example. If they wanted that sort of result, they should have said: "4 of these 5 creatures will be printed with flash, one will not. Given that choice, which one would you print without flash?"

    If, after releasing a test, you realize a mistake like this, holding your line is not the right approach. You have to admit the ambiguity or mistake, and regrade the test appropriately.

    ReplyDelete
  11. The advantage of a well written multiple choice test is that it is very easy to grade. As subjective as the questions may or may not be the answers are clear. Either they gave the correct (desired) answer or they didn't.

    The inherent disadvantages are twofold: 1) it is hard to write questions clearly and unambiguously such that there is one, and ONLY one right answer.
    2)it fails to thoroughly appraise the test-taker's knowledge of the subject, because a person with no knowledge whatsoever can select answers at random and still score higher than zero, and also because knowing which single answer is correct is often less important than knowing WHY it is correct, and which incorrect answers are closer to being correct than others.


    I feel a way to improve questions would be to phrase questions in the form of "which of the below is MOST/LEAST correct" or, much better: "rank the following from MOST accurate to LEAST accurate". (or vice versa)


    In regards to this specific test It would have been wise to have a field for people to add comments on specific questions.


    It's possible that what is currently a WRONG answer could become a RIGHT answer retroactively if the reasoning behind it changes R & D policy. I guarantee that if MaRo himself retook this test in 10 years some of the answers would change.

    As regards to 48, I'm not fond of the concept of virtual vanilla with ETB triggered abilities. With the increasing number of "blink" effects, as well as simple unsummon/boomerang variants, I find these to be much less vanilla than, say, a 5/5 trample.

    ReplyDelete