Five Tips for Designing Remote, Unmoderated User Testing Tasks

Carrying out remote, unmoderated user testing is growing more and more popular with tools and services to help popping up all the time. The benefits are obvious – you can test as many people as you can get hold of with no scheduling issues and no need for you to use up part of your busy day actually being there while participants work through the task. All you have to do is send the task out to your participants and wait for the screen and voice recordings to come back, ready to be analysed.

Not being there at the time of the test does have some drawbacks though, and I’ve seen some pretty disappointing results come out of these unmoderated sessions. At best these poor results mean you’ve wasted time and money, but at worst they can lead an inexperienced client or site manager to make changes based on not much more than a throwaway comment.

These are my tips on setting a task to make sure you generate useful, actionable results from your tests.

1) Make sure your site works
This one sounds obvious but a surprising number of test videos come back showing that part of the site doesn’t work as it should. This can really throw participants when there’s no one there to explain it. Some people can get really quite irate, even abandoning the whole test over relatively small bugs.

If you’re carrying out tests on an unfinished site, then it’s probably better to opt for a research technique that lets you be there to explain and encourage participants when any confusing issues arise.

If you’re testing with a work in progress then you’ll need to be extra clear about what you’re providing and what isn’t yet finished. I’ve seen participants completing tasks on clickable wireframes confused into commenting on the Lorem Ipsum content and even the visual design of wireframes. (They were indeed a “bit plain to look at.”)

2) Ask for action not opinion
The best tests are the ones where participants are given a realistic activity to do – to find something out, to buy something, to make a booking. If participants have strong opinions about anything on your site, they’ll mention these as they carry out the task anyway.

Sometimes site owners set tasks along the lines of “Click on all the pages and give feedback.” The results of these types
of task are never particularly useful. For instance, participants may say things like “I love this navigation” or “This menu is cool” as they watch a snazzy bit of jQuery flip about, only to get totally lost when they actually try to use the navigation to do something real.

If you ask people what they think then nine times out of ten, their comments will be focused around look and feel of the site. A certain amount of feedback on this aspect of your site is fine, but it needs to be in a realistic context. “I love this blue” doesn’t mean much. “I’d be reluctant to give this site my card details because it looks cheap” is much more useful.
Of course, choosing a realistic activity can be the main challenge of designing a good task, which brings me to my next tip…

3) Don’t be too prescriptive
You need to make it clear to your participants what you need them to do, but you need to leave them the freedom to act in a natural way. The whole point of user testing is to see what kind of behaviour your site elicits, if you’ve been too strict with your task then all you’ll find out is how well a participant can follow instructions.

It’s often useful to ask participants to imagine they’re using the website to help someone else – to buy a present, to find some information to relay. This way, participants can still act in a natural way even if they don’t feel the particular site or service is totally relevant to their current situation and needs. For example, the participant you’re using to test your hotel-booking website might say, “Of course, I wouldn’t do this in real life because I NEVER stay in hotels. Self-catering all the way, me.” Everything which follows is then not much more than an act, as the participant focuses more on checking the boxes of the task than showing what they’d really do.

If you ask your participant to help someone else, for example “Book a hotel for a friend’s visit” (being sure to ask them to think of a specific friend) then it gives them a genuine person to act for, which is usually much more effective than asking them to second-guess a make-believe version of themselves. Forcing participants to be someone they’re not is never the best way to draw out authentic behaviour.

The other important part of this tip is about not giving too much away. Don’t use any of the wording you’ve used in the site itself in your activity questions. The classic example is the one about asking participants to imagine they have book scattered around the floor of their living room, rather than specifically directing them to the term “bookshelf.”

4) Give participants false details
Lots of site functionality requires participants to register with the site or complete some other kind of form using their personal details. Participants are almost always reluctant to use their real details so they make up dummy names and email addresses on the spot.

Sometimes this skews the testing of forms as validation messages go crazy when faced with names of only one character, invalid email address formats or telephone numbers with the wrong number of digits. Participants can be left feeling much more frustrated than they would be if they’d completed the form with their real details.

It can also cause real problems if participants need to re-use their test details later. I once saw a whole batch of participants abandon a task because they were asked to register and then log in again later in the same activity – every one of them had forgotten the made-up email address they’d stuck in the registration form.

This one’s a pretty easy one to get right – just remember to give your participants dummy details to use.

5) Consider your task’s different parts

This is perhaps a bit more of an obscure one, but I’ve seen it cause a few problems so is still worth mentioning: if your task has multiple activities, then think about how they flow on from one another.

I once saw a task where participants were asked to book an appointment as one activity and cancel it as another. The only problem was that the cancelling activity came right after the booking part, so all the users still had their booking confirmation page on screen. The “cancel this appointment” button was pretty obvious from there, but of course users generally don’t want to cancel an appointment in the same session that they’ve booked it, so this was a bit useless. What the task really needed to be testing was how easy it was to cancel the appointment from the site’s home page.

This kind of thing is easy to guard against but it’s just something to bear in mind – make sure you get participants to close the site between activities.

Overall, I think it’s important to recognise that not everything mentioned in the sessions will be that relevant or helpful. The “think aloud” nature of the tests often means you’ll get a lot of commentary that sometimes won’t be much more than the participant filling the silence. As long as you don’t fall into the trap of reading into everything that comes out of the participants’ mouths, you should get some useful stuff from this technique.

Posted in UX process, UX skills | Leave a comment

Gamification doesn’t exist

The word ‘gamifcation’ is everywhere at the moment – everyone’s trying to grab some game-like elements and stick them into their products to try to make things ‘fun’ and ‘engaging’ – but when you dissect examples of so-called gamifcation, you start to find that it doesn’t really exist at all. All the examples of gamification are actually examples of other things – standard interaction design, bad marketing or just straight-out games.

Let’s take, for example, some of the badge and reward schemes that are often cited as making use of gamification techniques.

On the Treehouse site, users work through tutorials on various web design and development subjects. One they’ve passed the subject’s quiz, they collect a badge confirming they’ve completed that tutorial. Foursquare is another example, letting users earn badges for ‘checking-in’ and announcing their whereabouts to their friends. Both sites offer a range of options for telling other people about the badges you’ve earned.

At the Gamification Summit in 2011, Charlie Kim, CEO of Next Jump, talked about how he’d made use of a similar reward system in his company, convincing his employees to go to the gym more regularly by setting up a point-scoring and leader board system so they could compete with their colleagues.

All of these schemes have been have been undeniably successful, but are they examples of gamifcation? Not really.

As popular as they may be, none of these ideas really has anything to do with games at all. Once you take away the obvious gaming symbols – the neat badges and the physical leader boards – at their core, these ideas are based on well-known principles of human behaviour. People are motivated by progress. People are motivated by social validation. These designs have just taken things people already want to do – learning stuff, going places, getting fit – and motivated people to do them more by making it easier for users to a) track their progess and b) tell other people what they’re doing.

The schemes are examples of good design that’s drawing cleverly on human psychological principles, but they’re not any more like a game than any other piece of good design.

The website The Fun Theory takes a slightly different approach to using games to motivate. The initiative claims to show that making things fun can change people’s behaviour. The kinds of ideas featured in its videos clips include a bottle bank done up like an arcade machine that gives users points when they put bottles in it (designed to encourage recycling) and steps rigged up to make the sounds of musical notes when people walk on them (designed to get people to take the stairs rather than the escalator).

In the videos people do seem to be enjoying these ideas, but that’s probably because it’s the first time they’ve seen them. The people look delighted – and surprised. It wasn’t what they’d expected to see.

The site reminds me of when Mary Poppins was trying to get the kids to tidy the nursery and she said, “In every job that must be done, there is an element of fun. You find the fun and – snap! – the job’s a game!”. After she’d finished singing, she and the kids proceeded to use magic to do a speed-tidy of the nursery, with all the toys flying back into their places with a click of the fingers. This was obviously impressive to the children – and definitely surprising – but not really a game. After a while, the magic tidying up would be as mundane as the normal tidying up.

Mary Poppins’ ‘game’ was really just a novelty. The kids liked it because they weren’t expecting things to fly around the room when they clicked their fingers – and humans are intrigued by and attracted to new and surprising things. The problem is though, things don’t stay new and surprising for long.

The same applies to the flashing bottle bin and the musical stairs. They’re just novelties. Gimmicks. After a week or so of encountering those musical stairs on their commute, people will be back on the escalator. Probably wishing the tuneless racket created by musical stairs next to them would stop.

This is another example of design drawing on a human behaviour – this time playing on our delight at surprises – but again, nothing to do with games.

The most important things about a game is that it offers an experience that is enjoyable in itself. If a game is designed well, people will play it just for the entertainment. Very few gamifcation examples seem to remember this, and so not many focus on creating a fantastic gaming experience as their priority, but there are some.

In his book Playful Design, John Ferrara talks about the game Foldit. The game gives users puzzles to complete based on protein folding and scientists examine the solutions provided by the highest scorers to see if there is anything that can be applied to real-life proteins. One of the solutions helped scientists to decipher the structure of an AIDs-causing monkey virus – remarkably, something they’d been trying to do for 15 years before they got Foldit players on the case.

This, surely, is an example of the power of gamifcation? Still no. It’s an example of the power of a game. The distinction isn’t just semantics. It’s whole different mindset. A game is created first and foremost for the sheer enjoyment of the playing experience. Gamification suggests an add-on to an existing process or interaction. Gamifying is just modifying. To create a game, you need to start with the foundations.

So, although there are a lot of things which claim to be gamification in action, they can actually all be categorised as something else. ‘Gamified’ designs are either:

  1. Examples of interaction design that use psychological principles and behavioural economics to get people to do something. Executed properly, these are examples of good design or clever marketing, but they’re not gamification or anything to do with games.
  2. Badly thought-out, bandwagon-jumping uses of obvious gaming imagery and icons without any serious analytical consideration given to the appeal of the experience. This kind of stuff is like drawing whiskers on yourself a calling yourself a cat. No one’s convinced.
  3. Something that has been constructed with an entertaining player experience as the main priority, and any other goals as secondary. These designs are not gamified – they ‘re games.

Number one is already common place. Number two best left alone. Number three though, is where things get interesting.

Gamificaton doesn’t exist, but games certainly do. If we want to use the power of games, we need to leave the word gamification behind. It suggests that you can just add a bit of ‘game’ to something, sprinkle it on top for a little extra interest. It’s only when the game is put first, and an entertaining experience is the primary goal, that it becomes a powerful motivator.

Posted in Games, Psychology | Leave a comment

Four reasons why I don’t think a ‘designer’s gotta code’

Jared Spool started this debate last summer, but recently his tweet, ‘The ugly truth: Designers who can code will squash designers who can’t code in the marketplace’ got me thinking about it again, mostly because he hashtagged the tweet, ‘designersgottacode’.

I don’t think a designer’s got to code.

I mean, I don’t think there’s anything wrong with a designer who can code. Generally, of course, it’s better to be able to do something than not be able to do it. And let’s face it, the principles behind HTML and CSS are pretty easy, so getting a handle on the basics, at least, probably can only be a benefit in terms of your career.

But I don’t think a designer has got to be able to code, and in fact, regardless of whether or not they’re able to code, I think it’s often better if they don’t do it.

Here’s why:

1) It makes it tempting to cut out UX activities

If you’re recruited for a role where you have to design and code, there’s often a risk you’ll end up focusing on coding.

Project managers (or business owners – anyone who has an interest in the budget) always want to see tangible products. Stuff they can sell, stuff they can deliver to clients. Coding the website has to be done.  If it’s not, then you don’t have anything to show for yourself. All the research and planning are unlikely to impress without a finished product. When budget and time are tight, when the project manager is looking for places where work can be trimmed down, you can bet it will be the UX design activities that are viewed as expendable luxuries.

Of course, start-ups are likely to want someone who can design and code (and project manage, too, probably). One person is cheaper than two or three. And if they hire you as a UX designer who can code, than they tell themselves and their clients that their products are the outcome of a user-centred design process, even if, in reality, the design activities have been cut from the project.

2) Coding a bit can be worse than not coding at all

Most of the developers I’ve worked with would be aghast if a designer attempted to meddle with their code. I showed one Jared Spool’s comment that ‘knowing how to code helps you identify bugs and flaws in the production code’ and he just laughed.

Maybe if you have a solid background in proper software engineering (as Jared Spool does) then this would work. Or maybe if you were working on a simple website, it wouldn’t be too dangerous.  Generally, though, for a product to be robust, it needs to coded by someone who really knows their stuff, and to get to really know your stuff in the world of software engineering takes a long time.  I suspect that someone who had spent time building up their UX expertise would not have had the time, practice or focus to be able to be a really solid developer too. I think there is some truth in what they say about Jacks-of-all –trades.

3) You don’t need to be able to write code to understand it

Many of the arguments around needing designers to be able to code focus on the point that it’s easier to design well if you know the capabilities of the media you’re designing for. That’s true, but you don’t need to be a coder to understand how it works and how something will be built.

As you work with developers and talk to them about the feasibility of various ideas, you pick up more and more information on how things are put together. You get more of a feel for how different elements of the code talk to each other.  This kind of knowledge is great, but it’s not the same as being to write code. It’s possible to understand how code works without being able to spot a missing semi-colon on line 265.

4) It takes your focus away from the user

This is the most important one, I think.

If you’re going to be coding your design yourself, even if just to the prototype stage, it’s going to be much more tempting to taint your design decisions with a little bit of developer-centricity. Often when developers I’ve been working with have made design suggestions, they’re asking to have features designed in a way that’s quicker or easier to code, or that lets them try out something that they think is a particularly neat coding solution.  My experience has been that when you’re as intimately involved with a product to have seen its insides, you’re not in the best position to see if through the eyes of a user.

This leads me to conclude this post in the same way as I did my previous post (on UX developers): Designing a user experience should be focused around people, not technology.

Posted in UX skills, What is UX? | Leave a comment

Is the ‘UX Developer’ job title really a problem?

There’s been a lot talk lately about what the job title ‘UX Developer’ means and whether there is a place for it in the industry.

Leisa Reichelt and others in the ‘yes’ camp think it’s fine – a perfectly valid term for people whose skillset and role includes elements of coding as well as having a hand in design decisions. Andy Budd and the ‘no’ camp think it makes unmerited use of the term UX as if it’s some kind of mark of quality, and waters down the complexity of the discipline of user experience design.

In some ways, I agree with Leisa Reichelt. I don’t really see the need for the absolutist categorisations and the militancy that often seem to crop up when people are discussing this industry’s job titles.  If you have a mixed skillset then why not use your job title to describe what you do as succinctly as possible? Isn’t that what a job title’s for anyway?

I also don’t agree that a developer with an interest in UX activities is just a ‘good developer’ as Andy Budd argued, or as Whitney Hess said, ‘a developer working in 2012’. There are plenty of good developers who know a lot about code and technical engineering, but don’t  have much knowledge of (or interest in) psychology. I don’t think that’s necessarily a bad thing. After all, the UX designer is there to represent the user. I’d rather work with a developer who I could rely on 100% to code a robust product than one with one eye on the design.

However, where I do agree with Andy Budd, and my only problem with the title UX developer, is that in some ways it does cause some damage to the meaning of the phrase ‘user experience’ because it contributes to a trend which seems to be happening anyway:

It makes people think UX means ‘web’.

Nearly all of the UX Developer job descriptions I’ve seen are focused around front-end coding.  Similarly, I’ve noticed many freelance projects looking for a ‘UX Designer’ and even ‘UX Consultant’ are basically looking for a visual designer and front-end coder, who, as a bonus, should give a passing thought to the usability of the thing before they build it. I think this is worrying because it narrows the role of user experience design right down, making it a subset of web design.  It makes it seem as if we can only design for the experience of people using a website and ignores the huge range of non-web-based interactive products and services out there. And in turn, this makes user experience less about strategy, less about people, and more about a particular technology.

Posted in What is UX? | Leave a comment

Psychology is more than just a corner of UX

I was recently reading Dr Susan Weinschenk’s article on The Psychologist’s View of UX Design, (which, by the way, is a great introduction to some of the psychological principals of design) and I couldn’t help but wonder if she was selling short the role of psychology in UX with this introduction:


“A king brings six men into a dark building. They cannot see anything. The king says to them, “I have bought this animal from the wild lands to the East. It is called an elephant.” “What is an elephant?” the men ask. The king says, “Feel the elephant and describe it to me.” The man who feels a leg says the elephant is like a pillar, the one who feels the tail says the elephant is like a rope, the one who feels the trunk says the elephant is like a tree branch, the one who feels the ear says the elephant is like a hand fan, the one who feels the belly says the elephant is like a wall, and the one who feels the tusk says the elephant is like a solid pipe. “You are all correct”, says the king, “You are each feeling just a part of the elephant.”

The story of the elephant reminds me of the different view of design that people of different backgrounds, education, and experience have. A visual designer approaches UX design from one point of view, the interaction designer from another, and the programmer from yet another. It can be helpful to understand and even experience the part of the elephant that others are experiencing.

I’m a psychologist by training and education. So the part of the elephant I experience applies what we know about people and how we apply that to UX design.

Now, I kind of think that applying what we know about people and psychology to the design process is the UX elephant – not just one small corner. Of course, the visual designer, the programmer, the project manager and everyone else involved in the development of a product will approach the project from a slightly different angle. Whilst each of them will (hopefully) have an interest in the user’s experience of the finished product, they all have other things as their primary concern – making it look attractive, making the thing actually work. The UX designer on the project is probably the only person who has the end user’s overall experience as their primary focus.

I’m by no means trying to put down the role of the programmer or the visual designer, by the way (that would be ridiculous – UX would be pretty pointless if we didn’t have anyone to make a design real.) I just mean that when your knee-deep in functions and variables, or vectors and gradients, then it’s a lot easier (and more efficient) to stay there, rather than wading back out to think about what psychological theory says users are likely to make of what you’re doing.

I think I would offer a slightly different take on the elephant analogy and suggest that maybe, like the blind men, lots of the other people involved in the creation of product have to get so close to their particular task that they may have to lose sight of the big picture. And perhaps that’s precisely why we need the role of a UXer on the project – to take that step back and look at the whole elephant, and to use what we know about the way people’s brains work to think about how the elephant will look to them.

Posted in Psychology, What is UX? | Leave a comment