Judgment Day in STC Competition Land

Yes, today was the last day for us judges (for the STC Carolina Chapter competition) to have our evaluation forms submitted. I got my forms in just in time and our team leader is probably glad that’s over with. I’m done with judging.

The Form of the Form
I joined a very respectable team of fellow judges and we really spent some time looking at our entries. Do you know what the average time a judge spends evaluating an entry for an STC competition? Well, I don’t know, but I know that I put at least a couple of hours per entry into this whole judging effort. It’s a demanding job that requires experience, a good eye, and articulation abilities that differ from our usual technical communication abilities. Offering both constructive criticism and positive feedback, we need to write our evaluations on particularly unfriendly forms. Several years ago we used forms that asked us to quantify (yes/no) whether an attribute of a good entry was met or not, and then total up the numbers to determine if the entry was worthy of an award. And oh, yes, we were supposed to add some additional comments that would be helpful. I guess a lot of judges were skimping on that last part, so STC decided to change the layout of the evaluation forms. Now, the form consists of rectangles of white space where you have to write comments about every attribute of an entry, which sounds great – lots of freedom to express yourself, but the problem is, who has the time to write complete sentences for 20 to 40 aspects of an entry? Now it takes longer and requires more thought to fill out the evaluation form. And this extra effort is expected from professionals who have seen their workload double and triple in their day job and who have less time to volunteer with STC. Shouldn’t the forms take less time to fill out now then previous years? And who came up with the word or phrase that appears in the otherwise empty box? What do these boxes even mean? Who designs these forms? Never mind. Let’s just say that I don’t think the change to the forms is an improvement.

New Evaluation Form Sample

New Evaluation Form Sample

Old Evaluation Form Sample

Old Evaluation Form Sample

Time is in Short Supply
I must say that it has been a learning experience for me, but it is not as fun as several years ago when I judged for a similar competition but under much different circumstances. Besides things being easier in the old days — when we didn’t have to wear so many hats at work, and your colleagues weren’t getting laid off or downsized, ah, the good old days, when STC was about twice the size that it is today, and there was no shortage of volunteers and professionals had time to meet for lunch or dinner or an evening meeting — but I digress, besides all that, there was value in evaluating each other’s work. Today, who has time to put into this archaic ritual? I mean, outside of STC judging, do people really still fill out boxes in Microsoft Word documents and attach those documents to emails anymore? I felt as if I had been transported back in time to the 1990s but without the luxury of time I had then. For some reason, this year, the entries didn’t get distributed until later than usual, and with the Thanksgiving holiday being more important these days with family time getting pressure from every side, we had very little time to look over our entries, evaluate them, and meet to decide about awards. I started to wonder why were using such old technology for evaluating entries. I mean, for all of STC, can’t someone just make an online web site so we can post our evaluations in the cloud?

Blurring Boundaries
And what IS an entry anyway these days? In the old days there were user manuals and there were sales data sheets. Now there is everything in between and documents that cross boundaries and content that does not even reside in a document. There is no separate printed versus online competition anymore and STC has changed the categories in a vain attempt to keep up with the kaleidoscope of content containers. So entries come in PDF files and I’m not sure if they are meant to be viewed online or read as a printed document. Entries are in this gray area between sales and engineering, between promotion and instruction. And what does “Informational” mean anyway? Isn’t that redundant for anything we do? Of course this blurring of categories is simply mirroring reality, but it poses the question of why we have categories at all. But there’s an even bigger question that comes to mind when one considers the technology of the entries and the technology behind competition judging.

Feedback is Essential, but When?
With so many forms of communication on the Web now, and so much communication happening instantly, who needs judging of entries? The question is – Is this form of peer review and feedback even useful these days? I mean, besides the great back-slapping that goes on as we award fellow professionals for their work, besides the chance to make ourselves feel important and honored, the main value from STC competitions has been the great feedback you get on your entry. And that feedback used to be very valuable before web site analytics came along, before server-based documentation included feedback forms, and before social media became available. Now there are so many forms of feedback that tie directly to the users of the content and that feedback can be collected in almost real-time, that I can’t imagine how useful my judging is if it takes months before the submitter receives the feedback from the judging. Hey, by that time, they are probably on to other projects, moved to other divisions of the company, working on the next generation product, or even looking for another job after being laid off from the one that involved them creating the entry. In an age where Whitney Quesenbery can do an online web site usability study where she remotely connect to my computer and see where I click in her Web site, who needs the delay or the anonymity of these old-fashioned judging events?

Yes, today’s the last day for me. That’s it for me. I don’t think I’ll be judging for STC anymore. I’ve done my time; I’ve judged several competitions now and it’s time to retire. Maybe it’s time to retire the whole notion of STC competitions.

This entry was posted in Practices, Process, Professional Association and tagged , , , , , , . Bookmark the permalink.

11 Responses to Judgment Day in STC Competition Land

  1. Pingback: Tweets that mention Judgment Day in STC Competition Land | Ideas in the Making -- Topsy.com

  2. Sheila Loring says:

    Bill, you pretty much summarized my thoughts on the competitions this year. The forms are not well designed. What does “Evaluation” mean in each category? In the Information Design section, what’s the difference between document organization, document structure, and document/information sequence? They’re all very similar.

    The forms need to be documented. But tech writers volunteering their time to judge don’t have time to read documentation. They’re spending at least 4 hours evaluating each entry and trying to come up with the kind of detailed feedback they’d like to receive on their publications.

    The forms need to be intuitive and obvious, like the previous checklist-based forms. When judges don’t understand a categorization on the new forms (like “Evaluation” or “Creativity” of the purpose), they leave the box blank or type “n/a.” How helpful is that? I’m including myself in this crowd.

    I’m stunned that the new forms made it past a review by any experienced technical communicator. I know that the STC is a volunteer organization, and volunteers are hard to come by. Volunteers are hard to come by. That’s the point. Why spend time developing new forms that complicate a judge’s task and ultimately drive them away from future competitions? If the problem with the old forms was lack of comments, lead judges should ensure that team members fill out that portion of the form.

    I appreciate you participating in the competitions for so many years, Bill. But I completely understand why your days as a judge have ended. Thanks for your, as always, honest blog.

  3. Larry Kunz says:

    I haven’t seen the new forms, so I can’t say whether they’re better or worse than the old ones. The old ones certainly needed updating. They were pretty rigid, and they didn’t do a very good job of distinguishing the important things from the trivial.

    However, I hear you saying “That didn’t go well, so let’s throw the whole thing away.” Just because the execution was flawed, we mustn’t infer that the whole idea of competitions is obsolete. OK, the idea of judging a “book” or a “web site” is obsolete. But the idea of judging a documentation product to see whether it effectively meets its intended purpose — that’s not obsolete.

    You asked “Who needs judging? Is this form of peer review and feedback even useful these days?” I think it is useful. The awards themselves might not be useful. But it’s well worth the entry fee to get constructive feedback from knowledgeable colleagues. Web analytics can’t give me that kind of feedback.

    I’d like to see your thoughts on how the process could be improved, to make the results more meaningful.

  4. Linda Oestreich says:

    I, too, was a judge this year for the STC Southern California Spotlight Awards. And, I agree, the forms were tough to respond to and took much more time than I had planned. However, rather than push back and say that I won’t contribute in the future, I think your comments need to be shared with the STC Staff and Volunteers who lead this effort and give them a chance to modify the forms to improve them. As you said, this was the first year using the new forms and the new categories. I thought the new categorizes were great and it was interesting to judge whether an entry served its purpose and audience within other entries in different media types. Anyway, I will certainly pass your site on to folks at STC. There’s a good chance they’ll listen.

  5. Karen Mulholland says:

    Bill, I agree that the way STC expects people to judge competition entries has utterly failed to either keep pace with our world or bear any resemblance to the way tech comm professionals evaluate the portfolios of the people they consider hiring.

    In principle, though, there is great value in providing a structured way for technical communicators to invite their peers to evaluate their work in a public forum. In principle it’s valuable to be able to say to a prospective employer, “This is how I stack up against my peers.”

    But the STC competitions have become less and less useful as the mechanism to do that, for exactly the reasons you cited.

  6. Bill Albing says:

    Larry, to answer your questions, I don’t think feedback three months later (at the quickest) is at all useful for most documentation projects I’m working on these days.
    As for suggestions for STC, Linda, I have plenty with regard to how we could provide a structured way for technical communicators to invite peers to evaluate their work, so be sure to give them my name. I’m not presently an STC member. But I have a strong hunch that it has be done in close-to-real-time and it has to be done in the cloud (on the web) if at all. There is so many social media mechanisms now that I’m even questioning the value of such feedback from peers through any other media. I really do think the idea of competitions is obsolete, not just because of the form, but because of how the forms are used in a way that is more reminiscent of the day when documentation had months to produce a release. We don’t live there anymore. But I’m open to STC improving what they’re doing, and I’m more than willing to talk through some ideas. Thank you all for your valuable comments.

  7. Bill Albing says:

    To see the new evaluation forms, simply download them from this page of STC Competition Evaluation Forms (see the links in the “Judging Evaluation Forms” section): http://www.stc.org/comp/internationalCompetitions01_forms.asp
    Of course we should be transparent and let everyone know the criteria by which entries are evaluated, so that’s a good thing that they post the forms. I’m just wondering why they don’t have a Web form to fill out – they’ve gone to the trouble of making it in electronic form; now they just need to put it on a Web page with a database behind it instead of in a document.

  8. Jackie Damrau says:

    As the General Manager of the 2010 STC International Summit Awards and having been a member of the Competitions Restructuring committee, I invite anyone to provide comments directly to me at jackie.damrau@comcast.net regarding the forms and other items you would like to see. If you are not an STC member, yet have judged in a local/regional chapter competition, I’ll welcome your input (that’s you, Bill).

    Yes, some of what we do is still stuck in ancient times. We are making strides to move things to new ways. The STC just purchased a software package that will allow us to put the entry form for the STC International Summit Awards online for the 2010-2011 year. Next on the list for this new software is to put up the Judging Evaluation Forms.

    The STC International Summit Awards committee is a volunteer group of STC members that report up to the STC Education Director. Our time is limited in what we can provide as well as the monetary resources needed to buy software, etc. Having just added the software last month, we will be going through training on it and working closely with the STC Office to build more of this online which will ease the burden on the volunteers and the STC Office staff.

    Again, send your comments or “flames” (I can take it) to me at jackie.damrau@comcast.net. You will be heard; your comments will be noted; and the STC International Summit Awards committee will review during the summer of 2011 following the international judging process that runs from 1 March through 1 May 2011.

  9. Sheila Loring says:

    Bill, the competitions give tech commers feedback on our work that we might not otherwise receive. You can take some of the ideas and apply them to your future projects. So, in the long run, it typically doesn’t matter when you receive the feedback.

    Whether the new forms are conducive to helpful feedback is another matter.

  10. Bill, while I agree with you that the evaluation forms aren’t as good as they could be, I think that saying you won’t judge again because of them is kind of like throwing the baby out with the bath water.

    I, for one, enter my work in competitions each year to receive feedback that I wouldn’t receive otherwise that I’m able to take back to my clients and team members as we work on the same and/or new projects. And the older forms, as extensive as they were, also provided me with a quick list of best practices for me to consider as we explored a new/different type of project. The awards themselves are great, but it’s the feedback that gives me a daily ROI.

    For immediate feedback, we use focus groups or user testing services. There’s a LOT that happens in the competition process that prevents immediate turn-around.

    I’m certain that Jackie and the volunteers on the restructuring committee are diligently working to improve the forms. I’m sure they would appreciate your input on how to make them better… for both the judges (new or experienced) and the entrant.

    I think they’ve done an awesome job in reworking the categories to encompass the multiple forms of tech comm, which in and of itself is an undertaking. They just didn’t quite get there with the forms. They’re human, they’re volunteers, they’ve come a long way in a little time.

  11. Bill says:

    Louellen, thanks for the feedback. I’m not just complaining about the forms. I’m suggesting that there are multiple things wrong with how juding is done. It’s more like throwing out the bath tub because there are better ways to wash a baby. I hoped to make clear in my post that the forms are bad, but the lag of time is even worse. I don’t think the roadblocks are insurmountable. And the multiple sources of feedback available now online in real time make judging less useful. I was hoping you’d get a sense of all that. Reworking the categories might have taken effort and been creative, but aren’t they just reconfiguring silos that don’t exist anymore? Yes, we’re volunteers but we are also professionals and we know there are more sophisticated tools and solutions for providing value. I’m providing feedback to STC (ala http://www.keycontent.org/STC+Competition+Suggestions) but I think my judging days are over.

Comments are closed.