Remote testing tool feature and price comparison

As previously mentioned, I've been wanting to start doing remote testing, so I put together this table on features offered by the various tools that are out there. I figured it might be handy for others, too, so here you go. I haven't used any of them except for Userlytics (which I do like), so these aren't endorsements, just things I've found.

All of these tools are focused on testing mockups and live sites, which is what I'm most interested in right now.

static or live* what's tested what you get cost recruits
Usability Hub static (but can do a sequence of mockups with navflow test) you can ask questions about first impressions, see where people click in response to a question, or ask people to complete a task. short responses to comments, click heat maps, or heat maps and funnel success rates free if you do tests, subscriptions start at $20/month for 100 testers/month yes (other designers), or recruit your own (which doesn't use up monthly tester budget)
Usabilla static you can ask people to click on things and optionally add comments click heatmaps with participant annotations subscriptions start at $19/month for 20 participants, 30 day trial no
Chalkmark static (but can do a sequence of mockups) static mockups; you can ask people where they would click in the mockup to do some tasks, and putting several in order would let you test a whole series of actions click heatmaps and completion time 10 participants and 3 tasks per test for free, unlimited for $109/month no
TryMyUI live participants try to complete tasks with the site and answer questions video of participant's screen with think-aloud audio and written answers to questions $35/test yes, with requested demographics
Feedback Army live or mockup visible at a url participants answer questions about the site, which might require them to complete tasks or just play with the site text responses to questions starts at $20 for 10 participants yes, all participants are from mechanical turk
YouEye live participants try to complete tasks with the site video of participant and participant's screen with think-aloud audio 3 videos/month for free, 25 videos/month for $49/month yes, but maybe not with custom demographics. $19 each with free plan, $9 each with subscription, or you can recruit your own for no extra cost
Userlytics live or static participants try to complete tasks with the site and answer questions video of participant and participant's screen with think-aloud audio and annotations and transcriptions of key moments $59/participant with a 5 participant minimum yes, with requested demographics (some demographic requests may require custom price quotes)
Loop 11 live participants try to complete tasks with the site and push a button when they're done or answer questions video of participant's screen, aggregated completion rates, time on task, and other stats $350 per project or annual subscription for $1900 (for a non-profit, medium business and large business are higher) no
OpenHallway live participants try to complete tasks with the site video of the participant's screen with think-aloud audio $19/month (but you can get 3 participants for 10 minutes each for free to try it out) no
UserZoom live and static (they have click testing, timeout testing, and full site testing) participants click, answer questions, or complete tasks on a site heatmaps, clickstream charts, task completion rates, word clouds Subscriptions start at $1000/month or $9000/year, but you need to go up to the next level to get click testing and timeout testing and other tools. Basic plan includes live testing, surveys, and cardsort, others include everything but the kitchen sink yes, they integrate with marketing panels (may be additional cost?)
Google Website Optimizer live automates A/B tests for you on your live website goal completion stats based on cohort, presumably integrated with Google Analytics free N/A, it's your existing users

*Note that everything marked "live" except for Google Website Optimizer can probably also be used for interactive wireframes and prototypes available at a public url, for instance something created with Invision.

In making this list I also ran across a few other tools, but I didn't look at them in the same depth:

If you've used any of the tools above, I'd love to hear about your experiences! Likewise, if you have corrections or additions, I'd love to get those too :)

Comments