I don't know where to look because you're everywhere. So my talk today is called At the whites of their eyes and the importance with, the point about that is that you can only really understand the hypothesis, is that you can only really understand how well your website is doing from a design point of view if you see the white's of the people's eyes who are actually using it. And of course you can do that through data and that's fantastic and you can point people in the right way as well but there’s nothing actually better than seeing them in the flesh. So, two things to talk about today; something called agile user testing - stunned silence and another thing called campfires.

Great, so, let's get on, I'm from a company called Mix Radio I'm not allowed to call it that anymore because I'm not technically employed by Nokia, I'm now employed by Microsoft we were bought out a few months ago so now I'm technically a Microsoft employee. There's the badge to prove it but we're still called Mix Radio. What we are is a company or a group inside now Microsoft that does streaming music services for mobile phones. It's a reason for you to buy a Nokia and now a Microsoft device because when you buy one, one like this, you get free music streaming on it out of the box. You take it out of the box, you pay however many you know, pounds for it and you take it home, you hit the play button and then we give you free music streaming for the lifetime of that device. That's what we do.

I'm a design- oh god that's big [looking at screen] that's horrible. I'm a principal designer, which means I look after a particular platform and the way the mix radio is put onto that platform and the thing I did last was Windows 8. Which is why we have this tablet here, so that's been my baby. I kind of see it through from inception and kind of us, they totally want to do that on that platform and how it's going to work, all the way through to actually building it. I sit with engineers, well; my team and me sit with engineers. My team and I, we actually work out making sure that the thing that actually gets built is of a good enough quality and that people actually want to use it. But that's a huge process from a design point of view. I'm a designer though, that's the bottom line, yeah? I'm interested in how well people understand what they see, whether it fulfills their purpose, whether it does it for them. And also want to make them a little excited, yeah? In the case of music it's a great place to work because everyone gets excited for music pretty much, so half my job is done for me. Tools to get a good website or a good mobile application or a good mobile website put together are really kind of few and far between. And knowing which ones are good is a quite important thing. So what we're going to do is, we are going to do talk about this to start with.

So this is a Nokia, 920, it's a few years old now. But the point is it's running on MixRadio software. So this is running Windows Phone as well which is a platform, who uses Windows Phone? Just raise your hands even if you don't, thank you, thank you, thank you, that's great. It's more than I normally get unfortunately; we’re working on that. This is what's called a panoramic Windows Phone, it's a big long screen. When you load and application, you can scan through it, it's really rich. There's lot of visual content here, you can see what you need to place right away. That's what we do; my job recently was to actually send that out onto this thing. So this is our first tablet, this is a Lumia 2520 and this is running Windows 8 as well. We wanted to put an application onto this that had the same feel as a mobile device and functionality obviously but works on a bigger screen. So I think a lot of you guys are probably doing it the other way round, going from big screens, maybe desktop website applications down to small screen, so we were doing the other way around which is really odd and so that's what I was looking after. So, how many designers in the room? Oh, really, there are only a few of you? Great, okay. So then I can lie to you as much as I like, that's fantastic.

Yes, so these are a group of engineers. How many engineers in the room? Wow, what a diverse range of people you guys are. So what do you guys all do then? Oh, so these are engineers? This is my motley crew, the motley crew I work with. These are the guys who put together the Windows phone, Windows 8 apps. They do something called agile development. Again, show of hands, who's interested, fantastic. Right, okay so the way it would work is we would build this massive specification document that said exactly what it is the application website product should do, then we'd go and we'd see it sail over the fence and it'd land somewhere in the near department and they would go right, okay, what do we do with this then? And that would take six months, and this would take six months. And at the end of it after year you'd go - does it actually work? Do people, you know, six months ago, do they still think it's relevant now? Because a lot changes in mobile in six months, does this make sense anymore? Am I walking around too much? Sorry, OK, I'll stop walking around. Does that make sense any more? What do you do? So because of these large time scales this thing called agile was kind of invented by a load of Californians as these things usually are and the idea is that you do small tiny little pieces of work. Usually over couple of weeks - they're called sprints then you release them to the public after a few of those sprints that all makes sense when they group together. The point is that after you've done a little bit of development in two weeks that should make sense everything you drop in those two weeks you make sense to go out to the public so you do these really discreet little changes and then you release it and the case of the website that's really, well relatively, easy to do, if you can just update the website straightaway and people start getting your new, shiny stuff. For mobile devices, it's a little bit harder because the application needs to get sent out and people need to update and whatever, but it's still a really good idea.

But, it causes a problem as designers because we are used to going okay, great, we've got six months to work this out. Let's start with the concept let's start by looking at people and deciding it is what they do with other applications or in this space. We can then start you know, creating paper prototypes, doing some sketching. There's loads of stuff that we have to do before we're anywhere near giving the specification. A lot of people only ever see the specification and they go, oh that was easy, I could have told you that. I have to take a deep breath, because it's not that easy. But, we do a lot of this kind of stuff in Nokia. Now obviously Nokia, they've got lots of resources, they're worldwide. Microsoft even more so, this is stuff we do a lot of. We do big lots of big research projects, we go and spend hundreds of thousands of Pounds in South America understanding how people use their you know, 40 dollar mobile phones to listen to music, and then we try and build them something that's even better than that. But, a lot of people don't have a big research budget as we can fully attest, so these are the kind of things we do. We do lots of ethnographic studies, we go off and see people, and we do eye tracking studies.

You know, some of this stuff you guys may have played with or, or tried. We get agencies, large agencies- maybe even like some of you guys in the room here- to do this kind of stuff for us. And then all of that research comes back in and someone writes a really large check and that gets sent off and that's really good. And we sit in a, you know, one-day workshop deciding what of this stuff we can do anything with. And there are product guys, there are design guys, and there are marketing guys all in the room and it all gets a bit heavy. And a bit slow, and a bit un-agile and a little bit like what we reduced to before, where we create this big thing and see what happened to it when it got over there.

So, that's no good for us in the mobile space really anymore. What we want to do is this - this is called Agile User Testing. Who has done this? Or been part of it? Done anything like this before? Right, right, right. This is what I've got basically set up right here. So what we do is we pay a company in Bristol who I can tell you about later who will go and find us people in our demographics, the kind of people were interested in unfortunately their usually Bristolians as well because we can't afford to travel to Guatemala every few weeks and then those people will come into our office and we’ll show them some stuff that we’ve been working on. And we will change it, we will have five people a day and we'll change and between those sessions.

So this is one of our lovely participants this is the other person I'd like to say thank you too. This is Ben Bywater he's on our team he's the guy who heads up our kind of user research side. And he's the influence for a lot of the stuff you're seeing today and I get the opportunity to stand up and talk about it. So, agile user test, that's what this thing is, the point of it is you test early and often. I can't overestimate, or underestimate how important this is - test early with what you think might work and test it often. Change it, iterate it and test it often. Get your team members, so get the people in your groups, like the engineers, this is what we say in our business, get the marketers, get the brand people and other designers, the product guys, the guys who thought this was a great idea to start with, meeting the real users.

And there's a way that this set up works really well for allowing us to do that which I'll talk about in a second. And then it keeps the insight in house and that's the other really important thing for us it means that we get to do the research ourselves and that research doesn't go anywhere else and you know I'm not saying they would but maybe an agency would like to use that information and say oh, one of our other clients whatever but we want to keep that information in house, it keeps it secure for us as well. So, this is what we do, we have rooms by our reception, which means that people don’t have to go to the secure floors and then see the secret prototypes we're working on, and that's one of the ways we facilitate it. We have some recruitment guys who go and find, the people, we pay each individual person forty pounds in cash to come and spend an hour in our office; I don't think that's too bad. We're getting them to sign an NDA, that's a very important one. You can download NDAs off the Internet, teal easy to do. You don't have to pay much for them, that's what we did. And what we do is we stream and we record the sessions. So streaming allows us to send what the person is seeing, the whites of their eyes, out to the product manager who is in Espoo in Finland. So he gets to see exactly what he thought was a great idea and we thought was a great design being tested in front of real people.

So that is the most, one of the most valuable parts of this whole experience. We test in pairs so we have one guy facilitating, which was Ben in that picture. So he's the guy who's actually talking to the person, not really interviewing them but working with them. That is a dark art in itself being able to be a good interviewer, not asking leading questions but you'll learn that, there are plenty of books about it. You can pick it up over time, practice makes perfect and another person takes notes. We can watch from anywhere in the streaming that's a really important thing we did it in front of engineers they love that, they love actually seeing real people. And we do rapid iterative testing and this is one of Ben's slides. So the idea with this is that between sessions and the five people we see everyday we change the design every time and we try and improve it and if less and less issues come out with each person then we know we're getting somewhere. This is kind of rapid iterative test and evaluation, there we go, and I knew my brain would kick in at some point. So five is the magic number five users a day the wisdom goes - is enough for you to figure out the vast majority of your usability issues it looks a bit like this. So usability problems found; so issues with your website your mobile application your whatever your products versus number of users, and for the cost value benefit, I like using those phrases, five users is roughly the right number before you get either too expensive or too long or too slow or too un-agile.

So, let me show you a video really briefly. I'm going to have to fiddle underneath the desk for this for a moment, so please bear with me. Talk amongst yourselves for a bit. Here we go, so this is one of our user tests, going on for real. So this is, [Video plays]

>> Play me is according to the product guides our most, the most important part of our proposition. And this is a guy that uses our application a lot and has never found it yet.

So that's great, because what it shows us, if that take a step back. If I could go into, an agile user test, with every one of the people who bought our mobile phones, I would be a happy man, because everyone who looks at my tests goes, oh, that's really good, yeah, I didn't realize it could do that. So those kinds of insights, you can see there, that was only a minute, less than a minute worth of insight there, incredibly valuable. Right, you can't just design this stuff in isolation or do a lot of pre kind of amble research and then design this stuff and go right we think this is great, let's get it out there. You've got to do this kind of testing, this kind of testing is vitally important. And even if you guys aren't designing this yourself, you're getting other people to design it, make sure they've tested it. You can test this yourself as well because it's not just the design that's being tested here. It's the product composition as well, it's the you can test; you know the way that this thing is branded. There are all kinds of things you can test using this method. This kind of lean approach or agile approach, it applies to all kinds of things. It's just we particularly use it in Nokia in design. Let me give you a little demo so the software we use is called Wirecast, I can see lots of people start scribbling good. I don't have shares in the company or anything like that honest they're a California company for about 70 or 80 dollars you can get piece of software that will do what you saw there. So, it will record camera pointing at the person you're watching and it will record you know, video from an overhead camera. One like one of these, which we've had made but you can pick these up from, from PC World for like, 30 or 40 quid, they're not very expensive.

So, for less than £100, you can set yourself up with a little usability testing laboratory, or testing laboratory really. And then have this stuff streamed out to whoever you want to see it and all they need is QuickTime or there's this thing called VLC. So it's really easy to install this piece of free software on the person that wants to see it and then they can watch this stuff for free. This is what it looks like, we were going to have this projected, but unfortunately, the technical gods weren't with us today, so you're going to have to look over my should and see what this looks like. So, you would have - Oh and it's crashed. That's really good. You would have a - and one I prepared earlier, you have a participant sitting - this feels really awkward sitting with my back to everyone - like this and they would have a setup with a mobile phone that was underneath the camera here and then you would run this piece of software. And this is basically a piece of kind of low grade broadcast software that we use and we've had about 50 people streaming simultaneously onto this, you need quite a powerful machine to do that but that's absolutely possible to do and this broadcasts around the world it can go over the internet it can broadcast quite happily around the world. See, if anyone can see around here. And you can see the picture that's been broadcasted is my face and that's coming out of the webcam and then I've got, the live kind of stream coming from over there, over the top of that as well.

You can even broadcast with this software and these kinds of utilities. The desktop from another computer, so you can have someone doing something on your website that's being broadcast in high definition through to this thing, which then does the streaming and does the composition of picture in picture and that kind of stuff as well. And it all gets very nice and very kind of effective. So that's a little bit about the kind of nuts and bolts of this but of course, like all these things are, it's not a silver bullet. It won't solve all your usability problems, you still need your designers to do something with the results, and you still need guys to go back and think about whether the product proposition was such a great idea, or whether the brand is really working.

The problems with this particular approach for example, is lead-time. So we find that although we want to get stuff tested quite quickly, finding the right people to test it with is almost just as important. And sometimes that can take a while; finding Windows Phone users who have used MixRadio in Bristol can be quite difficult sometimes. So our wonderful recruitment agency that finds these people can sometimes take awhile to get them, which is no fault of their own, they're very good. They do take awhile, if you're testing website on an iPhone, then you're probably good to go; this won't be a problem for you. But lead-time camera problem is worth being aware of. Obviously all of your results become Bristol optimized instantly, which is great if you're selling something in Bristol, and it's local, and that's great. But if you are selling as we are to South America, to Indians, to Chinese or whatever, then that's no good. But you can send someone over for not very much money on easy jet and do exactly this kind of stuff because all of this is portable in that country with those people and there are, just like there's a recruitment there's a recruitment agency here in Bristol that can find you people for not very much money there are agencies around the world that do exactly this kind of thing. So it's really easy for you to take this and keep it somewhere else but if you keep it in Bristol you will get a very Bristol optimized product.

We also have this thing called ethno-checkins that we do, which is another way of doing it. Now, this is, so this setup is fine. But it's a bit laboratory, right? It's a bit, bringing someone into an environment that they're not very used to, sitting them down in a chair, pointing a camera in their face, getting them to sign an NDA that says you are going to show their video in a presentation of 100 people.

It's all that kind of stuff, so it can get a little artificial. Instead, if you can do something called ethnographic research, what that is, is basically taking something like this, this set up but without the camera and any of this stuff, into the actual houses, and we do this as well and we've started doing it more because it's really valuable, and again it's just as cheap really. Take your stuff, on your phone or tablet or whatever, into someone's house and these recruitment people will find, you know, your participants that willing to allow you to do this. And just sit with them on their sofa; you can film them that is fine, that's all part of the process. And film them actually using the software, as they would do normally, you get much more natural results. Sometimes I have to say, the results you get from this are a little artificial. But getting someone to sit there while their kids are playing havoc in a corner and they have to figure out how to change a mix or skip a song or say they like a track while the TV's on and the doorbell rings and the phone goes, that's the reality of what your people are experiencing with your websites. And your applications and getting them actually in the house as well, doing that kind of research in house is really lean, it's really agile, and it's just as good but it takes a bit more time to set up and it can be a tiny bit more expensive so this is kind of what we do a lot of, but lean ethnographic studies, which are really, really interesting and it's an area where as designers we're looking to focus a bit more on.

At the moment, the other thing is this thing called campfires. Now it's something that we've set up specifically in Nokia, but again we know that there are agencies that can help you guys get this stuff settled as well. This is fascinating, right? This is like ethnographic study, but where you don't go anywhere. This is a Facebook page that you set up and you have all these people that are pre-qualified by the agency, and you'll say these people will be the users of your software, and they will be warmed up to it, and they'll have used it for a while, they like it, they don't like it, whatever.

They can be from all around the world, overseas, this is just a Facebook page, and they are also under NDA. So you can show them stuff you don't want to go public yet, which is really important for us, because we have a lot of stuff like that. And then you can, ask them questions directly.
Just like you would do if they were sat here in your laboratory, you would ask them direct questions. What do you think about this, what do you think about the color, what do you think about where that button is, does that copy make sense? What do you think about the product proposition, in a Facebook page and then they can comment and they can talk about it with each other and it's almost like having a kind of focus group but remotely.

Campfires, we call them campfires, which I think is the term for them. But you get some really great stuff coming out of it like. We asked the question, did you know that a lot of the mixes on Nokia mix radio, Microsoft Mix radio are curated by a team of music experts? Because we thought that maybe people weren't really getting that. They thought this stuff just kind of a theory that comes out of nowhere, this music creator we have on our service. And we really want to try to get home that we have real guys and girls in our team who are in every country around the world creating music contents, playlists that go onto these devices. Do you know that idea? And what do you think of that idea as well? So we're kind of testing the abilities, we're testing the product proposition here, we're kind of testing maybe the brand and the awareness of what mixed radio is. And then you get loads and loads of great comments, really detailed stuff. An absolute invaluable gold mine, of course, you then have to go through it and make some sense out of it but you know; that's a great problem to have. It's very light touch though; you don't get an opportunity to react to what they're doing live here or in their house. And you have to kind of post a question and then hope for the best but it's not necessarily always that representative of your typical user as well because these are people that have been warmed up and they're kind of in your environment, and you've paid them a little bit of money and it's not as representative maybe as getting you know, someone that has been, has been more tightly qualified. And of course because it's, whether they are under NDA, there's some certain stuff that you probably don't want to release to the public too widely, that we certainly wouldn't want to release to the public too widely and have them talk about it on Facebook. So you obviously have some problems with secrecy there as well.

Alright, that's it, I've spoken enough.