UsabilityTestingReport
Summer2008
Includes
Pre-testingUsabilityReview
Round1:WordPress2.5andFirstPrototype
Round2:CrazyhorsePrototype
ProjectOverview
BetweenMayandAugustof2008,BallStateUniversity’sCenterforMediaDesign(CMD)
conductedusabilitytestingofseveralWordPressinterfaces.Tworoundsoftestingwere
conductedwitheyetrackingatCMD’sNewYorkfacility,theMediaInsightCenter,with
thefirstroundinMayandthesecondroundinJuly.Theprojectconsistedofseveral
phases,including:
1.Pre-testingUsabilityReview
2.Round1Testing(WordPress2.5andTest1515prototype)
3.Round2Testing(Crazyhorseprototype).
Thisreportoutlinestheprojectfindingsfromeachofthesephases.
WordPress2.5/CrazyhorseUsabilityTestingReport2/25
2.5UsabilityReview
Priortoconductingtestsessions,CMDstaffconductedausabilityreviewoftheexistingWordPress2.5
administrativeinterfacewiththegoalofidentifyingareasmostlikelytopresentusabilitychallengestotest
participants.Thereviewresultedinanumberofrecommendationsforimprovingtheuserexperiencethat
wereconsidered“low-hangingfruit.”
Global
Reducesizeofnavigationfontsandpageheader
tomakebetteruseofverticalrealestate.
ThereisoverarchingambiguityintheWrite/
Managenavigation,inthattheentire
administrationpanelisamanagementtool,and
ineditingcontentusersarealsowriting.The
movebetweenwriteandmanageduringthe
processofpostcreationseemsjarring.
IncludeDashboardonthesamelevelasWrite/
Manage/Designnavigation.
Createavisualseparationbetweenidentity/
logoutandhelplinks.
BlognameshouldlinktoDashboard.
Dashboard:
TheRightNowboxtakesuptoomuchspace
abovethefoldforinformationthatisessentially
asnapshot.
CombineYourStuffwiththeRecentComments
boxtomakeoneRecent-themedboxaccessible
abovethefold.
Manage:
WheneditingapostorpageunderManage,the
screenlabelstillshows"WritePost/Page.”It
shouldchangetosay"EditPage/Post"whenin
theManagesectionsothatitprovidesanother
indicatorthatthishasalreadybeenpublished/
saved.
WritePost/Page:
Previewrequestshouldtriggersavesomost
currentversionappearsinpreview.
DonotcleareditorafterPublish,allowcontinued
editing.
RelatedLinksarenotreallyrelated,but
shortcutstootheradminfunctions.Remove
needforthisbymakingmoreeasilynavigable.
"Keepthispostprivate”andpasswordprotection
arerelatedandshouldbeincloserproximityto
eachothertoshowuserstheprivacyoptions.
AddingMedia:
ReplaceAddMediaiconswithasinglebutton,
anddisplayprogressivemenus/formfieldsbased
onfiletypethatisuploadedratherthan
displayingallfieldsbydefault.
Makedifferencebetweenuploadingand
embeddingmoreclearinthemediauploader.
Maketherelationshipsbetweenuploadingmedia
andtheMediaLibrarymoreclear.
ClarifytheGalleryfunction.
DesignExtras:
GiveexplanationofSnapShots,asisonefor
hidingrelatedlinks.
Widgets:
Showanexampleofeachwidgetsousershavean
ideawhatthey''regettingbeforeclickingadd.
SwaptheplacementofCurrentandAvailable,
andconsiderchangingthenameofCurrent
Widgetstosomethingmorespecific,suchas
"YouAreUsingTheseWidgets."
Comments:
UnderActions,add"Edit"link.Author''sname
shouldNOTbelinktoeditcontent.Authorname
shouldnotactastitle,sincemanytimesan
authorwillleavemultiplecomments.Theseare
twodifferentobjecttypesaltogether.
DetailViewvs.ListViewismisnamed.Listview
showsalltheactualdetails,itjustremovesthe
commentbody.It''snotareallist,buta
collapsedview.Changelabelstobemore
accurate.
SpecifictoWordPress.com:
WordPresstipthatappearsinyellowboxeither
shouldbemovedorhaveanoptiontobeturned
off.
IncludeIncominglinksinthestatsboxinsteadof
inseparatemodule.
TheWhat''sHotboxisnotveryhotifyouhaveto
scrolldownthatfartoseeanyofit.Thisshould
trytoatleaststartabovethefold.
HelpandSupportarenotdistinctlabels.Change
labelstoreflectlinkdestinations.
WordPress2.5/CrazyhorseUsabilityTestingReport3/25
Test1515Prototype
Thepre-testreviewwasconductedtoinformthedevelopmentofaprototypemeanttoprovidea
comparativetestprotocol.Thoserecommendationsthatdidnotrequireextensivecode
restructuringwereincorporatedbytheWordPressdevelopmentteamintoaprototypeknownas
Test1515.ThisversionofWordPressdidnotmakemajorchangessuchasrestructuringtheWrite
Postscreenorresizingthenavigation,butdidincludesmallerinterfaceadjustmentssuchas:
-RemovetheRightNowbox
-RemovetheWordPresstip(WordPress.com)
-Combinespecificdashboardmodulestobringcontentabovefold
-RemovetabwithlinktoDashboard
-Simplifyhelplinks
-InitiateautosaveonPreviewofnewpost
Theintentbehindcreatingthisprototypewastohaveeachtestparticipantuseitaftercompleting
thefirstportionofthesessionusinghis/herownblog,toseeifthesesmallchangesmadean
appreciabledifferenceinusability.
WordPress2.5/CrazyhorseUsabilityTestingReport4/25
Participants
WhiletheWordPressdevelopment
teamworkedondeveloping
Test1515,CMDbeganrecruiting
participantsanddevelopingtest
protocols.CMDpostedascreening
surveytoidentifypotential
participantsfromthosewho
respondedtoacallforvolunteerson
officialWordPressblogs.In
exchangefortheirtime,participants
wereoffereda$75WordPresscredit
toberedeemedforWordPress
merchandiseand/orWordPress.com
upgradecredits.
Participantswerescreenedto
achieveabalancebetweenusersof
thehostedWordPress.comandthe
downloadedWordPress.org,aswell
asscreeningforadditionalcriteria
suchaslengthandfrequencyofblog
use,age,gender,technicalsavvy,
employment,typicalusage(text,
images,videos,etc.),andplatforms
used.
12participantswererecruited,with
anevensplitbetween.comand.org
users.Blogusageamongparticipants
variedwidely,rangingfrom
relativelynewpersonalbloggerswho
postafewtimesaweektopower
userswithfiveormoreblogswho
postmultipletimesperdayand/or
manageblogsforothers.
Amajorityoftheparticipants
frequentlyincludedmediawiththeir
blogposts,andthoughmost
participantstendedtouseimages
only,twoparticipantsfrequently
postedvideosaswell.Mostaccessed
theadminpanelfrombothworkand
home,withamixbetweenPCand
Macusers.Abouthalfalsoblogged
regularlyfromamobiledevice.
Agesrangedfrom19-45,withmost
usersfallinginthemid-twentiesto
earlythirties.Respondentstothecall
forvolunteerswerepredominantly
male,causingtheparticipantpoolto
skewthisway,withfourfemale
participantscomparedtoeightmen.
However,asitislikelythatthe
WordPressuserbaseprobablyskews
alittlebitmale,thisratiowas
consideredacceptable.
Abouthalftheparticipantswere
employedintheinteractiveor
technologyindustries.Thiswas
influencedbyseveralfactors,
includingtheplacementofthecall
forvolunteers,thelocationofthe
testsessions,andparticipant
availability.
Giventhenatureofthetestsessions,
whichbothincludedtheuseof
unfinishedprototypes,theabilityto
imaginehowfeaturesmightwork
whencompletedmadethisskewa
benefitandkeptsessionsmoving
forward,whilelesssavvy
participantstendedtogethungup
byincompletefeatures.
Participantswerealsoscreenedfor
theabilitytoreadscreentextfroma
2-footdistance,willingnesstohave
testsessionsvideotaped,and
availabilityforthetworoundsof
plannedtestsessions.
WordPress2.5/CrazyhorseUsabilityTestingReport5/25
Methodology
Duringthetworoundsoftesting,CMDobtained
usage/behavioraldatathroughacombinationofthe
followingthreemethods.
Talk-aloud:
Participantswereencouragedto“thinkaloud”as
theyusedeachinterfacetogiveanideaofhowthey
madethedecisionsreflectedintheirclickpaths.In
addition,amoderatorguidedeachsession,
observingparticipantresponsesandaskingfollow-
upquestionstoobtainmorecompleteinformation
andfeedback.
Morae:
CMDutilizedMoraesoftwaretorecordscreen
activityandparticipantvideo,aswellasallowing
additionalCMDresearcherstofollowthesessions
remotely,markingtasksandaddingcommentsto
thevideofiles.Thesenotes,videosandtimedtask
segmentationallowedforin-depthanalysisofthe
userexperience.
EyeTracking:
Theeyetrackingequipmentinvolvedtwocameras.
Onewastiedtofacialrecognitionsoftware,while
theotherrecordedthereflectionfromtheeye.
Softwarecombinedthetwosetsofinformationto
giveusarecordofeyemovementrelativetothex,y
coordinatesofthemonitor.Thiswasthebasisfor
outputssuchaseye-trackingvideos,gazetrailsand
lookzonecharts.Sampleoutputsareincludedin
thisreport,asthelargefilesizeswouldmakeit
unfeasibletoincludethefulllibraryinone
document.
WhyEyeTracking?
Themainreasontouseeyetrackingistobemore
granularinunderstandinginterfaceissues.For
example,ifyou’retestinganinterfaceandnooneis
clickingonaparticularlink,youwouldnormallyask
theparticipantiftheysawthelinkorwhythey
hadn’tusedit.Theproblemisthatourbrainsare
cleverandcanrewriteevenourveryrecentmemory,
sotheperson’sanswermaybesubjective.Witheye
tracking,youcanknowforsure.Iftheydidfixateon
thelinkbutdon’trecallseeingit,thatmightbea
cluethatthecontrastofthelinktextistoolow,
whileiftheydidnotactuallylookatit,itmightbe
anindicationthatthelinkplacementfallsoutsideof
normalviewingpatterns.
WordPress2.5/CrazyhorseUsabilityTestingReport6/25
Round1Protocol
Round1testsessionslastedalittleoveranhoureach,witheachparticipantbeginningby
loggingintotheirownblog.Afterperformingtheirusualmaintenancetasks,thesession
moderatorpromptedthemtocompleteadditionaltaskssothateveryonewouldbe
recordedusingallofthefeaturesincludedinthetest,whichincludedalmosteveryscreen
oftheWordPressadminpanel.Participantswerenotrequiredtocompletetasksinany
specificorder,andthemoderatorusedachecklisttoensurethatfeaturesofhighpriorityto
theresearchwereallaccessedbeforeparticipantsloggedoutoftheirblogs.
Afterparticipantscompletedthefirstpartoftheprotocolontheirownblogs,theylogged
intotheTest1515prototype,wheretheyreviewedtherevisedscreensandcompleted
certaintaskstoassesstheeffectivenessofthechanges.
Participantswereaskedtorateallthefeaturesofthe2.5adminpanelforeaseofuseand
usefulnessonascaleof1-5.Inaddition,eachparticipantfilledoutapost-sessionsurvey
thataskedforassessmentssuchasbestthreeandworstthreefeatures,aswellasasking
participantswhatchangestheywouldmaketotheadminpaneliftheywereable.This
informationinconjunctionwiththeusagepatternsrevealedthroughtheMoraerecordings
andeyetrackingdataledtotherecommendationsfordevelopmentoftheRound2
prototype.
WordPress2.5/CrazyhorseUsabilityTestingReport7/25
Round1FindingsandRecommendations
Overall:
Forthemostpart,participantsfeltWordPress2.5waseasytouse,
evenwhentheyhadtroublefindingsomething.Thepositiveoverall
feelingfortheapplicationcausedseveralparticipantstorate
certainfeatures/functionsaseasyevenwhentherecordings(eye,
screen,voice)showedthemhuntingthroughlinksandscreensto
findtheoptionstheywanted.Thissenseofemotionalcapital
carriedthroughallthesessions,makingitclearthatthe
participantsaretrulyfansoftheapplication,evenwhenthey
encounterissues.
Whiletherewereseveralareas/featuresthatdidnotscoreaswell
astherestoftheapplication,indicatingthatspecificfixestargeted
totheseareaswouldimprovetheuserexperience,alargerissue
becameobviousasweobservedthesessions.Thenavigational
schemecurrentlyinusecausedpeopletohavetogofromscreento
screentofindthelinkstoscreenstheyneeded,andthenumberof
navigationallinksonmostscreensprovedoverwhelming,sothat
userswouldscanoverthewordswithoutreallyabsorbingthem.
TheWrite/Manageparadigmalsoprovedtobeincontrasttohow
usersactuallythinkoftheircontent,whichismoreobject-oriented
(i.e.,Posts,Pages,Comments,Media,Links,Settings,etc.).A
numberofuserstrippedoverthemselvesastheymaneuvered
betweenWriteandManagescreens,lookingforpoststheywere
workingon,drafts,orrecentlypublisheditems.
WhiletheparticipantsmadeitclearthattheyloveWordPress,it
alsobecameclearthroughthesessionsthattheexperienceofusing
WordPresswassometimesfrustrating,especiallywhenwaitingfor
newscreenstoloadjusttoseethesubnavigationoptions.
Theparticipantshadagenerallyfavorableopinionofthe2.5design
intermsofvisualappeal,buthadlowopinionsofthestructureand
theamountofverticalscrollingitrequired.
TheoveralllayoutandstructureofWordPress2.5wastoomuch
likeawebsite,andnotenoughlikeanapplication.Alessrigid
interfaceconstructionwouldenableuserstocompletetaskswith
lesswaitingandscrolling.
ThesmallchangesmadeintheTest1515prototypeweresuccessful
inmakingthingsclearertotheparticipants,butdidnotaffectthe
overallfeelingsabouteaseofuse.Itwasclearthatwhilesmall
thingslikelabelsandindividualplacementissuescouldimprove
clarity,abiggershiftinfunctionandstructurewouldbeneededto
addresstheproblemscausedbythecurrentlayout.
CMDrecommendedexploringarestructurednavigationalscheme
forRound2testing.
Ratherthanaformalreport,findingsfromRound1testingwereapproachedinamoreaction-orientedwayto
providethemostusefulfeedbacktothedevelopmentteam.Standoutresultswereoutlinedinanemailalong
withrecommendationsforimprovementinthesecondprototype,withthefindingsandrecommendations
exploredinmoredepththroughconferencecallspriortothebeginningoftheRound2prototype
developmentcycle.
WordPress2.5/CrazyhorseUsabilityTestingReport8/25
Round1Findings:Dashboard
Mostofthetestparticipantssaidtheydidnotspend
muchtimeontheirDashboards,eitherclicking
directlytoanotherscreenorbookmarkinga
differentscreentouseasanadminentrypoint
(Stats,CommentsandNewPostweremost
frequentlyidentified).
Thefactthattimelyinformationsuchasnew
comments,incominglinksornewswaslargely
hiddenbelowthefoldmadeitlesslikelythat
participantswouldseeit.Mostparticipantssaid
thatiftheywereabletochoosetheirown
Dashboardcontentand/orpositioningofmodules
theywouldbemorelikelytopayattentiontothe
Dashboard.
Participantsfeltthemostusefulelementsonthe
currentDashboardwerenewcomments,thelinkto
Drafts(thoughwouldpreferdirectlinksto
individualdrafts),andforWordPress.comusers,
theStatssnapshot.
Recommendation:MaketheDashboardmodule
layoutuser-controlled,andconsideramending
modulesforinclusion.
ThistypicaluserscannedtheDashboardbutdid
notfocusonanyspecificareasofinformation
beforeclickingtoanotherscreen.
WordPress2.5/CrazyhorseUsabilityTestingReport9/25
Round1Findings:WritePost
TheWritePostscreencausedanumberofproblems
forparticipants.Themainissueisthatthelayoutis
structuredasalongcolumn,whichrequiresusersto
scrollupanddowntocreateapostwithmetadata.
Severalparticipantsforgottoaddtagsorcategories
becausethemoduleswerebelowthefold,andwent
backtoaddthemafterviewingthepublishedpost
andseeingthelackofmetadata.Thescrolling
requirementisinterruptive;betteruseoftheright
handcolumnshouldbemadeforabove-the-fold
accesstometadatafields.
RelatedLinksintherighthandcolumnwereseenas
unnecessaryandnotparticularlyrelatedtothetask
athand,andparticipantsdidnotusethem.
InthePublishmodule,thestatusdropdownwasnot
understood.Participantsthoughttheyunderstood
whattheoptionsmeant,butnotwhatwouldhappen
iftheychangedthemenuchoice.Theywerealsonot
sureifthemenuwouldreplaceclickingPublish.
Mostparticipantsdidnotnoticethewordcount
feature,thoughtheysaidtheywishedtheyhad
knownaboutit(toosmall/lowcontrast/poorly
placed).
Recommendation:Adjustlayoutsothattagsand
categoriesarevisibleabovethefold,makemodule
draganddropsothatuserscandecidefor
themselveswhereeachmoduleshouldbelocated.
Thesethreescreensshowtheprogressionofvisualattentionforoneparticipantasshecomposedapost;the
scrollingrequirementmeanttheparticipantforgottoaddmetadatauntilshepreviewedthepostand
noticedtheabsence,atypicalproblem.
WordPress2.5/CrazyhorseUsabilityTestingReport10/25
Round1Findings:Comments
ThelayoutfortheCommentsscreenwasseenasa
stepbackfromversion2.3.Participantsacrossthe
boardfounditconfusingthattoeditthecomment
theyneededtoclickonthecommenter’sname,
whichusedtobelinkedtotheauthorratherthanan
editscreen.Additionalactionlinksfoundinthefar
rightcolumnwerenotinthenaturalgazepath,and
whenaskedtomarkacommentasspamordelete
it,participantsfrequentlyscannedtheareanearthe
commenttextlookingforthelinks.
Severalparticipantswereusingpluginstoallow
easiereditingofcomments,includinginlineediting
ofcommenttext.Mostparticipantssaidtheywould
prefertobeabletoeditcommentsinlinerather
thangoingtoanewscreentofixasimple
typographicalorspellingerror.
Recommendation:Bringactionlinksinto
naturalgazepath,returntopositionbelow
commenttext.
Thischartshowsthegazetrailofatest
participantmoderatingcomments.Notethatthe
actionlinkstomarkasspamordeleteinthe
rightcolumndidnotreceiveanyattention,while
theusernaturallyscannedtheareaaroundthe
commenttext.
WordPress2.5/CrazyhorseUsabilityTestingReport11/25
Round1Findings:MediaUploader
Formanyparticipants,theFlashuploaderdidn''t
alwaysworkcorrectly,includingcrashing.Written
cuesdidnotindicateclearlyenoughthetypeoffile
upload/choicesformethodonceintheuploader.
Becauseofthetoptobottomorientationoftext,the
distinctionbetweenuploadingandembedding
mediawasunclear.
TheGalleryfunctionwasnotunderstood.
Participantswereunsureifitwasarecordof
currentmediaadditions,aFlickr-likealbumthey
couldmanipulate,orasegmentoftheexisting
MediaLibrary.BecausetheGallerydidnotappear
anywhereelseintheadminpanel,they
predominantlyassumeditwasinaccessible.
Whenrankedbyparticipantsforeaseofuse,adding
mediavariedbymethod,withtheeasiestbeing
fromaweblink,secondeasiestfromtheharddrive,
andthemedialibraryrankinglast(andscoringwell
belowaverage).
Recommendation:Makethedifferencebetween
uploadingandembeddingmediamoreclear,
providemorevisibleaccesstogalleriesforediting,
simplifyuploadprocess.
Thisgazetrailshowstheparticipant’sconfusion
whenaddingmediawiththeFlashuploader.
Theparticipant,likeseveralothers,lookedback
andforthbetweentheuploadmechanismand
theURLembedareabeforedecidingwhichone
wasmeanttobeusedtouploadanimagefrom
theharddrive.
WordPress2.5/CrazyhorseUsabilityTestingReport12/25
Round1Findings:SmallerIssues
Otherareasthatdidnotperformwellincludelink
categories,medialibrary,tagmanagementand
settings.
LinkCategories:Participantsfeltthiswastoo
closeinterminologytoLinksandCategories,and
thedifferencebetweenpostcategoriesandlink
categoriesnotentirelycleartomany.
Tagmanagement:Thestructureofthisscreen
wasproblematic,withalmostnoscreenrealestate
devotedtoactualtagmanagement,andmostofit
devotedtowhitespace..Orguserstendedtousea
pluginforthistogetaroundtheclunkydesign.
Settings:ParticipantstendedtoclickonManage,
notnoticingtheSettings/Usersnavigationoverto
therightorthinkingthattheyweregoingto
“managesettings.”Inaddition,thetimezone
settingelicitedcommentsfrommultiple
participants.Thereseemedtobealackofclarity
aroundUTCingeneral,andforWordPress.com
users,severalwereunsurehowtohavethecorrect
timezonedisplaywithoutaffectingtheirstats
(specifically,theycommentedthatstatsendedthe
dayat8pminsteadofmidnight).
MediaLibrary:Allfoundthisscreenconfusing.
Mostwereabletoidentifyhowtomanageitems,
butweren''tconfidentofwhatwouldhappenasa
result,beingunclearonthedifferencebetween
managingmediafromthisscreeninsteadoffroma
specificpost.Allexpectedaccesstomoreediting
featurestobeavailablefromthisscreen.
Userstendedtoscandownthecolumnofimages
ratherthanengagethefullscreenasa
managementtool.
WordPress2.5/CrazyhorseUsabilityTestingReport13/25
Round1Recommendations
Aswasmentionedearlierinthisdocument,attheendofRound1,CMDdidnotprepareaformalreport,
optinginsteadforworkingsessionstodiscussthefindings.Inadditiontoalaundrylistofsmallinterface
issuesthatpresentedsimplefixes,suchaschangingcommentauthorlinks,wewerefacedwithlargerissues
suchasthedesireforuser-determinedhierarchiesonlong/scrollingscreens,ambiguityintheWrite/Manage
navigationparadigm,andadisconnectbetweentheactofaddingmediatoapostandtheabilitytomanageit.
WhataboutthatTest1515prototype?Wasn’tthatsupposedtotestsomechanges?
TheTest1515prototypewasusedbyeachparticipantaftercompletingtheregulartestprotocolwithhis/her
ownblog.Becausethechangeswereoftenminor,participantsdidnothaveastrongreactioneitherway.In
fact,thoughmanyacknowledgedthatcertainaspectsoftheTest1515prototypegavethemquickeraccessto
things(Dashboardlinks,amendedhelpaccess,moreprominentinformationonDashboard,etc.),most
rankedTest1515onthesamelevelas2.5ineaseofuse.Thisledustobelievethatalthoughtherewere
numeroussmallchangesthatcouldbemadetoimprovetheusabilityof2.5,aslongasthebasicstructure
remained,userperceptionwouldfailtochangeappreciably,andbasicproblemsliketheneedforscrollingin
postcreationandhavingtoloadnewscreenstoaccesssubnavigationwouldremain.Thisledtothedecision
tocreateanexperimentalprototypeforRound2ratherthanmakingchangeswithinthe2.5interface.
TheguidingprinciplesoftheRound2prototypedesignweretomaximizeverticalspace,reducescrolling,
increaseaccesstonavigationtoreduceunnecessaryscreenloads,enabledraganddroponscreensthatwould
mostbenefitfromusercontrol,andredesignmanagementscreenstotakeadvantageofnaturalgazepaths.
Ultimately,thegoaloftheRound2prototypewastofeelmorelikeawebapplicationandlesslikeawebsite.
WordPress2.5/CrazyhorseUsabilityTestingReport14/25
CrazyhorsePrototype
BasedontherecommendationsprovidedbyCMD
attheconclusionofRound1testing,theWordPress
teamdevelopedtheRound2prototype,whichwas
builtontheCrazyhorsebranch.Duetothe
extensivenatureofthechanges,whichincludeda
completeoverhaulofthenavigationsystem,new
functionsandnumerousnewlayouts,Round2
testingwasscheduledtoallowenoughtimefor
developmentsothatthetestingcouldbeperformed
withafunctionalprototyperatherthanconceptual
designs.
TheprototypethatwastestedinRound2was
largelyfunctional,butretainedafewconceptual
areasthatwerebasicallysnapshotsmeantto
conveythenewideastopotentialusers.In
addition,thefocuswasonuserexperiencedesign
andfunctionaldevelopment,sotheprototypedid
notreceiveanynewvisualdesigntreatment.The
analysisofRound2takesthisintoaccount.The
prototypewasfinalizedseveraldaysbeforeRound
2testingbegan,atwhichpointthecodewasfrozen
andnofurtherchangesweremade,toensurethat
allparticipantswouldhaveaccesstothesame
experienceduringtheiruseofCrazyhorse.
WordPress2.5/CrazyhorseUsabilityTestingReport15/25
Round2Protocol
ParticipantsweregivenaccesstoCrazyhorseafewdaysbeforetheirsessionsandwereinstructedtocreate
posts,uploadmediaandperformotheraveragetasksathomesothatwhentheycameintheywouldbe
familiarwiththenewlayoutandfunctions.ThiswasdonesothattheirRound2sessionswouldtrackusesof
afamiliarapplicationratherthanexplorationsofanewinterface,allowingustocomparetheresultsof
Rounds1and2.ThetestssessionsagainusedMoraeandeyetracking,andweredividedintotwoparts:atask
flowandadiscussion.Thetaskflowwasdesignedtohaveparticipantsengageeachnewfeatureorscreen
layoutwithinthecourseofthetaskflow.Thetasks,inorder,were:
1.LogintotheCrazyhorsedashboard.
2.Accessadraftpost.
3.Onyourdraft,deleteoneofthetagsorcategories.
4.Addanimagetoyourpostfromtheharddrive.
5.Previewyourpost.
6.Publishyourpost.
7.Viewalltheimagesthathavebeenuploadedtothisblog.
8.AddanimagetoyourMediaLibraryforfutureuse.
9.Accesscommentspeoplehavemadeonyourblog.
10.Markacommentasspam.
11.Changeawordinthetextofacomment.
12.Gobacktoyourdashboard.
13.UseQuickPress.
14.Writeanewpost,notusingQuickPress.
Afterthesetaskswerecompleted,eyetrackingwasdiscontinuedandtheremainingsessiontimewasusedfor
discussionabouteachnewfeatureandscreenlayout.
Weuseda1-5ratingscaleinthepost-sessionsurveysforeaseofuseandusefulnessthatwasthesameasthe
scaleusedduringRound1,butaddedaquestionforeachitemaboutwhichwayparticipantspreferredeach
feature:asitisin2.5,asitisinCrazyhorse,ornoopinion.
WordPress2.5/CrazyhorseUsabilityTestingReport16/25
Round2Findings:Overview
Round2testingwasanunqualifiedsuccess,meaningthateverynewfeatureordesignelementprovidedus
withactionableinformationthatcouldbeusedtoinformthenextversionofWordPress.Mostofthe
responsestothediscussionquestionsandpost-sessionsurveysindicatedastrongpreferenceforthe
individualCrazyhorsescreenlayoutsandfunctions,whileallparticipantspreferredCrazyhorseover2.5
overall.Eyetrackingdatashowedthatparticipantswereabletolocatescreenelementsmoreeasilythanin
2.5,withtheexceptionofthePreviewandPublishlinksontheWriteScreen.
ParticipantComments:
“Iloveit.”“It’sdefinitelybetter.”“Itjustseemstoflowbetter.”“Moreorganized.”“It’smoreextensible.”
“Easiertonavigate.”“Moreefficientuseofspace.”“Seemsmoremodern,friendlier.”“Lookingforwardto
usingsomeofthesefeaturesonadailybasis!”
TheBestandtheWorst:
Inthepost-sessionsurveys,participantswereaskedtochoosethethreebestandworstfeaturesofthe
Crazyhorseprototype.Thetopchoicesfor“BestChange”wereNavigation,QuickPress,MediaUploader,
Inbox.
TherewasverylittlenegativefeedbacktotheCrazyhorseprototype,andmostofitwascenteredonone
feature,theonlyfeaturetoreceivemorethanonevotefor“WorstChange.”Thepersistentbottombaronthe
WritePostscreenwasidentifiedastheworstnewfeaturebytwo-thirdsoftheparticipants.
WordPress2.5/CrazyhorseUsabilityTestingReport17/25
Round2Findings:Navigation
Thelefthandnavigationthatallowedparticipantstoexpandand
collapsenavigationsectionswithoutloadinganewpagewas
universallypopular.Participantsthoughtitwasmucheasiertofind
things,wascleanerlooking,andwouldprovidegreaterextensibility
forplugins.Theobject-orientedorganizationwasseenasagreat
improvement,withseveralpeoplecommentingthatitwould
removethecommonerrorofwritingpagesinsteadofposts.The
abilitytoexploreadministrativeoptionswithoutleavingthe
currentscreenwasseenasbeingespeciallyhelpful,andmost
participantscommentedthatitfeltlikeabetteruseofspace.
ParticipantComments:
“Better.”“Muchcleaner.”“Easiertoaccess.”“Moreroomformy
plugins.”“Easiertoseewhereeverythingis.”“Iwon’tgetlost.”“I
hadfasteraccesstomoreitems.”
Response:EveryparticipantpreferredCrazyhorsenavigation
over2.5.
WordPress2.5/CrazyhorseUsabilityTestingReport18/25
Round2Findings:Dashboard
ThenewDashboardwasconsideredmoreusefulthanthecurrent
model,andparticipantsalsosawalotofopportunitiesfor
customizationwithinthenewframework.TheInboxconceptwas
popularwithmostparticipants,thoughpeoplewhoreceivefew
commentsdidnotknowiftheywouldneedsuchafunction.
However,ifremindersaboutpendingdraftswereincludedinthe
Inbox,allparticipantssaiditwouldbeuseful.Everyonethought
QuickPresswasagoodidea,thoughparticipantswerespliton
whetherornottheywoulduseit.Someonlywritelongpostsor
media-richposts,andthoughtthemselvesunlikelytouseit.Those
whowriteshorterpostsorliketostartdraftstokeeptrackofideas
thoughttheywoulduseitfrequently.Anumberofpeoplecompared
ittoTwitter,andfeltitwouldappealtopeoplewholiketomakefast
andeasyposts.Theabilitytodecidewhichfieldstobedisplayed
wouldincreaseitsutility,andparticipantsmostfrequentlyaskedfor
theabilitytoaddmediaorhyperlinks.Theenhancedstatsmodule
wasseenasanimprovement,butthedisplaystyleofnewsleft
participantscold.Mostwouldappreciatebeingabletodetermine
placementandcontentofmodulesontheirown.
ParticipantComments:
“Muchmoreuseful.”“LovetheadditionofQuickPress.”“Greatfor
jottingdownaquickidea.”“Remindersaboutdraftsintheinbox
wouldbegreat.”“Theextrastatsarecool.”“IwouldcheckthiswhenI
loggedin.”
Response:EveryparticipantpreferredCrazyhorsedashboardover
2.5,thoughalmostallwouldprefertohavecontroloverlocationand
contentofmodules.
OntheCrazyhorseDashboard,
participantswereabletoquickly
focusonareasofinterest,such
asrecentcomments.
WordPress2.5/CrazyhorseUsabilityTestingReport19/25
Round2Findings:Comments
Everyonethoughtitwasmucheasiertoeditandacton
comments.Eventhoughitdidn’tactuallytakethemthat
muchlonger,thefactthatthelinksappearedinamore
logicalplacemadethemfeelliketheyweregettingthere
fasterandhavingtothinkless.Participantsalso
appreciatedtheadjustedscreenlayout,especially
movingtheauthormetadatatotherightandtheclear
linktotheoriginalpost.
ParticipantComments:
“Itstreamlinestheprocessofreviewingcomments.”
“Itmakessensetohavethembelowbecausethenyou
canactonthecommentrightasyoufinishreadingit.”
“Thisismuchbetter.”
Response:EveryparticipantpreferredCrazyhorse
commentsscreenover2.5.
Typicalgazetrailsshowthataction
linksunderthecommenttextare
morereadilyseenbyparticipants,
sincethenaturalgazepathismainly
focusedinthecommentarea,notin
thefarrightcolumnwhereaction
linkswerelocatedin2.5.
WordPress2.5/CrazyhorseUsabilityTestingReport20/25
Round2Findings:WritePost,TheGood
Overall,theparticipantsalllikedthenewWritescreen.
Everyoneappreciatedbeingabletodecidewhichpost
elementsweredeservingofthebestscreenrealestate.
Thedraganddropofmoduleswaspopular,andthe
two-columnlayoutmeantmorefieldsabovethefold.
The“Mediaaddedtothispost”wasconsideredagood
idea,thoughpossiblyinneedofabetter/shortername.
Therewassomequestionoverwhetherdeletingan
imageinthismodulewoulddeleteitfromthepost,or
fromtheMediaLibrary.Havingaccesstopost
commentsonthisscreenwasviewedpositively.The
modulesmostfrequentlyplacedintheupperrightby
participantswereTags,Categories,MediaAddedtothis
Post,PrivacyandAuthor.Participantswhomaintain
multi-userblogsweretheonesmostlikelytoselectthe
Authormodule.
ParticipantComments:
“Iwon’tforgettoaddmytagsbeforeIpublishnow.”
“MuchbettertohavewhatIusenearthetop.”“Ilike
beingabletochoosewhatgoeswhere.”“Thisseems
cleaner.”
Response:Allparticipantspreferredthegeneral
layoutandfunctionsoftheCrazyhorsescreenover2.5.
Theseparticipantsusedthedraganddrop
functionalitytorearrangemodules,and
rememberedtoaddmetadataliketags
andcategoriespriortopublishing.
WordPress2.5/CrazyhorseUsabilityTestingReport21/25
Round2Findings:WritePost,TheBad
TheonlysignificantproblemwiththenewWritescreenwas
thebottombarthatcontainedtheelementsoftheformer
Publishmodule.Thisisanexampleofhowafeaturethatwas
unsuccessfulintermsofperformancewasactuallysuccessful
intermsofthetestinggoals.Eventhoughparticipantsfeltthe
designofitwasnoteffective,itbroughtoutcommentsabout
thedesirabilityofhavingapersistentstatusmodulesothey
wouldn’thavetoscrollupanddowntogettothePublish
button.Ifthiselementhadnotbeenincludedintheprototype,
wemightnothavegainedthisinformation.Allusers,with
manycomparingittoanadbannerorbrowserchrome,
overlookedthebottombar.Mostidentifiedthecolorand
placementasaproblem,andfeltittookuptoomuchroom.
Almostallrequesteditsreturntotheupperright.Withthe
additionofdraganddropexpandablemodules,thereduction
inpagescrollingwouldbelikelytokeepthePublishmodule
relativelypersistent.Evenwhentheywereusedtothebottom
bar,manyforgottolooktherewhenaskedtopreviewor
publishtheirposts,astheeyetrackingdataconfirms.
ParticipantComments:
“Ithoughtitwasbrowserchrome.”“EvenwhenIknewitwas
there,Ikeptforgettingtolookatit.”“Ilookedrightpastita
bunchoftimesbeforerealizingitwasthere.”
Response:Bottombarperformedpoorly,butparticipants
likedtheideaofapersistentstatusmoduleingeneral.
Participantstendedtolookbackandforth
atthebottombarseveraltimesbefore
realizingtheywerelookingrightatthe
Publishbutton.
WordPress2.5/CrazyhorseUsabilityTestingReport22/25
Round2Findings:MediaUploaderandLibrary
Everyparticipantfeltthenewuploaderwasavastimprovement,even
thoughwetestedwithlimitedfunctionality.Theysaiditfeltmoreintegrated
withtheMediaLibrary,whoseredesigntheyalsofeltwasanimprovement.
SeveralwereveryinterestedinseeinghowvideoandURLlinkingwouldbe
integrated,butfeltthegeneralstructurewasapositiveimprovement.The
organizationofthelibrarymadeiteasiertoscanmediacollectionsandact
onspecificpiecesofmedia.However,participantsfelttheadvancedediting
optionswerestilltoohidden,andconsistentmediaaddition/editing
experiencewasdesired.
ParticipantComments:
“It’ssomucheasier.”“Iloveit.”“Somuchbetter.”“Iwouldactuallyuse
this.”“Yes.”
Response:EveryparticipantpreferredCrazyhorsemediauploaderand
libraryover2.5.
WordPress2.5/CrazyhorseUsabilityTestingReport23/25
Conclusions
TheresponsetoRound2testingmadeitabundantlyclearthattherevisednavigationalapproach,new
featuresandadjustedscreenlayoutsoftheCrazyhorseprototypewerepreferredbyparticipantsoverthe
existingWordPress2.5administrativepanel.
WordPress.orgusersraisedtheonlyconcern,whichwashowthechangewouldaffecttheirplugins,aconcern
tobetakenintoaccountifWordPresschoosestodeveloptheCrazyhorsedesignfurther.
Morethoughtshouldbegiventoelementssuchasthedashboardinbox,themediaaddedtothispostmodule
andthepersistentpublishmoduleontheWritePostscreen.
Overall,though,iftheresponseofthetestparticipantsisanyindication,theintroductionofanewadmin
panelintheCrazyhorsestyleislikelytobesuccessfulwithbothnewandexperiencedusers.
WordPress2.5/CrazyhorseUsabilityTestingReport24/25
WordPress2.5/CrazyhorseUsabilityTestingReport25/25
Keeptrackofongoingusabilityeffortsathttp://wordpress.org/development/
|
|
