Blog

Survey writing woes

By on

I’ve been spending a lot of time lately thinking about survey writing. In 2008, I took a short three day course with Jon Krosnick of Stanford University, which made me think about survey writing in a new way. In particular, I started realizing that the surveys that I wrote were poorly written.

Since then, it seems like I keep finding poorly written surveys everywhere I turn. Here are some examples I’ve found recently in  my everyday life:

The US Postal Service sent me a Postal Customer Questionnaire lately because they were thinking about closing my branch. “If you now receive Post Office box service, you will be able to transfer your remaining box rent credit to another post office, or you may be eligible to receive a partial refund. How would you feel about consolidating the Dinkytown station with other postal stations? Better, Just as Good, No Opinion, or Worse” I answered No Opinion. Then I crossed it out and marked Worse. Then I crossed it out and marked No Opinion and wrote a three sentence explanation in the “Please explain” section. Why was this such a hard question to answer? Well, primarily because they’d never asked me about how I’d feel about the consolidation WITHOUT the refund. So now they were merging my opinions about the consolidation in with my feelings about the refund. Personally, I was mad that they were consolidating and I’d feel cheated if they didn’t refund my money, but really the refund wouldn’t change my opinion at all. No where on the survey did they ask me anything to this effect.

This second example isn’t exactly a survey, but is still getting at some of the problems with survey writing. I’m having problems with allergies and need to go see an allergy specialist. So the clinic sent me my paperwork so I could fill it out before my appointment. Leaving aside many of my other complaints (and there are many!), the first main page has a section entitled “Chief complaints of patient.” For each option you are supposed to check “Yes” or “No.” The options are Asthma, Rhinitis (Hay fever), Urticaria (Hives), Eczema, Sinusitis, Chronic recurrent bronchitis, Nasal polyps, Recurrent otitis media,  Recurrent pneumonia, G.I. disturbances (colic, diarrhea, etc), Insect sting reaction, drug reactions, or blank lines. Now I’m a pretty smart person. I’ve been in school for a grand total of twenty-one years now, but I can’t tell you what many of those things are, and I can’t tell you which ones I should select. I have a runny nose and a cough. I’ve been diagnosed with something, but I forget what it is, and it didn’t include the second two symptoms, just the runny nose. Why on earth is this questionnaire that is obviously for the patient or patient advocate full of doctor jargon instead of patient jargon?

Now that I know better, I want to do my best to avoid writing bad survey questions, but at the same time, it’s incredibly difficult to write good survey questions. So what I’ve been doing is writing my same old, same old questions and then revising…and revising…and revising. Trying to revise them to turn them into good questions isn’t easy, but I try. I also ask for a lot of feedback and am very self-critical. One proof-reading pass doesn’t cut it for a survey, even if it’s only going out to 10 people. That would reflect poorly on me, my advisor, my lab, and my university…so I do more work. Hopefully if you take one of my surveys, you’ll see the result of this work, and if not, I hope you’ll take a moment to let me know.

Stack Overflow *

By on

This article on Read/Write Web describes how Stack Overflow, the tech Q&A site, will let other sties use their software, changing the look and feel, while keeping the Q&A goodness.

John

Designers as Activitists?

By on

Since I started riding the bus to work, I’ve gained about 80 minutes of reading time a day, and lately I’ve been reading recent issues of interactions.  I’ve found many of them quite interesting, probably because they’re rather far afield from my usual concerns. They’re mostly by and for HCI (broadly construed) practitioners, rather than for researchers. One particular article got my attention: Learning from Activists: Lessons for Designers, by Tad Hirsch.

Hirsch talks about how activists have been technology innovators, touches on some examples, and talks about what the design process is like under these conditions. For example, the “immediacy of activist projects, coupled with a perpetual lack of funding, forces a kind of rough-and-tumble innovation”. Sounds right.

Things get more interesting later, as Hirsch says that “Activists willingness to engage in extra-legal activity also enables unique design opportunities”. He hastens to add that he doesn’t mean violence or vandalism, but the “exploit[ation] [of] excess acpacity”, like squatting in abandoned buildings or using wireless networks without their owners’ permission.

Finally, he talks about how “contestational designers” [i.e., those who design for activists] are “openly partisan practitioners who take sides in pressing issues of the day. They are neither objective technicians nor hired guns — images that continue to dominate the technical development community”.

It was this final point that I found most provocative. On the one hand, I too feel that it is imperative for all educated people — and I hope that includes not just designers, but also software professionals, academics, and students — to “take sides in pressing issues of the day”. However, if we do that in our roles as professionals — as designers, as researchers, as academics, etc. — do we lose our professional community? Note that Hirsch isn’t proposing this [that all designers, let alone all HCI researchers, should become “contestational designers”] — I’m just tracing out the implications of his advice.

For example, it’s obvious that the HCI community is heavily liberal and leftist. “Everyone” at CHI 2009 was ecstatic about Obama’s election, but this fact was mostly “informal”. That is, the conference program per se did not reflect it. But what if this changes? What if activist papers play a larger and larger role in our professional community? Would they all be liberal-activist papers? Would that drive out non-liberals? Or would conservative activists, Chinese nationalist activists, anti-abortion activists, gun-rights activists all be represented? Would you be happy about papers about providing technology support for radical environmentalists to shut down a coal plant, or for radical anti-abortion activists to shut down a family planning clinic?

I don’t know… However, as I get older, I am more and more interested in reconciling my personal beliefs and my professional practice. Hirsch’s article made this topic more urgent for me.

The Netflix Prize and the MovieLens Data Sets

By on

Hello all.  We’ve been asked by several of the Netflix Prize teams if they can use the MovieLens datasets in training their algorithms.  The answer is yes! We’re happy to encourage algorithmic experimentation using our datasets — and you don’t even have to share any of your winnings with us :).  We only ask that you credit the MovieLens datasets on your web site, and in any written descriptions you write of the resulting algorithms.

Compete well!

John

Amazon Erases 1984

By on

In a “too delicious to be true” story, Amazon has used one of its Kindle’s features to erase copies of the book 1984 from their customer’s devices. Yes, that 1984, the one about the futuristic society that controls and audits everything their citizens read or speak.

Apparently a third-party seller uploaded an illegal version of 1984 to the Amazon web-site, and some users purchased it.  When Amazon found out the version was illegal, they refunded the purchase price *and deleted the copies of the book from the Kindles*.  Almost too funny to be true.  (One of the users was a 17 year old high school student whose notes on the book were also erased by Amazon when the deleted his copy of the book.)

Amazon has already promised not to do something like this again.  However, the story makes clear the deep danger in aggressive digital rights management.  If the owners of the content can control what you read, when you read it, and how you read it, our access to media becomes only a temporary “right” that can be granted and taken away at a whim.  We need to create a set of rules that ensure that information can never be controlled in this way.

One extreme example of the need for rules to protect the free flow of information is the hubbub over the new version of Hemmingway’s “A Movable Feast”.  Depending on who one talks to, Hemmingway’s grandson Sean has either edited the book to make it truer to how Hemmingway really felt about his first wife or has altered Hemmingway’s text to change history about that relationship.  (It helps muddy the water that the first wife is Sean’s grandmother.)  The publisher is releasing the new version, which will now be compared endlessly by scholars to the 1964 original.  What would happen in the digital world of the future?  Would the publisher be able to change the text of everyone’s original version to the new updated content?  Presumably noone would lobby for such a world … but if we aren’t careful to constrain contracts between publishers and digital device owners, we could accidentally end up living in it!

How wonderful that Amazon made this mistake with the book 1984.  It’s not the greatest of the anti-utopian novels — that’s Huxley’s Brave New World! — but perhaps we were too quick to accuse it of wandering too far from reality …

John