IMAGESEGMENTATIONBYSPATIALLYADAPTIVECOLORANDTEXTUREFEATURES
JunqingChen,ThrasyvoulosN.Pappas
ElectricalandComputerEngineeringDept.
NorthwesternUniversity,Evanston,IL60208
AleksandraMojsilovic,BerniceE.Rogowitz
IBMT.J.WatsonResearchCenter
Hawthorn,NY10532
ABSTRACT
Wepresentanimagesegmentationalgorithmthatisbasedon
spatiallyadaptivecolorandtexturefeatures.Theproposedal-
gorithmisbasedonapreviouslyproposedalgorithmbutintro-
ducesanumberofnewelements.Weuseanewsetoftexture
featuresbasedonasteerablefilterdecomposition.Thesteerable
filterscombinedwithanewspatialtexturesegmentationscheme
provideafinerandmorerobustsegmentationintotextureclasses.
Theproposedalgorithmincludesanelaborateborderestimation
procedure,whichextendstheideaofPappas’adaptiveclustering
segmentationalgorithmtocolortexture.Theperformanceofthe
proposedalgorithmisdemonstratedinthedomainofphotographic
images,includinglowresolutioncompressedimages.
1.INTRODUCTION
ThefieldofContent-BasedImageRetrieval(CBIR)hasmadesig-
nificantadvancesduringthepastdecade[1,2].ManyCBIRsys-
temsrelyonscenesegmentation.However,imagesegmentation
remainsoneofthemostchallengingproblems.Whilesignificant
progresshasbeenmadeintexturesegmentation(e.g.,[3–6])and
colorsegmentation(e.g.,[7–9])separately,thecombinedspatial
textureandcolorsegmentationproblemremainsquitechalleng-
ing[10,11].
In[12],wepresentedanimagesegmentationalgorithmthatis
basedonspatiallyadaptivecolorandspatialtexturefeatures.The
perceptualaspectsofthisalgorithmwerefurtherdevelopedin[13],
includingtheuseofasteerablefilterdecompositioninsteadofthe
discretewavelettransform.Aswesawin[12],theresolutionof
thespatialtexturesegmentationislimitedbecauseitisdefinedon
afiniteneighborhood,whilecolorsegmentationcanprovideaccu-
rateandpreciseedgelocalization.Inthispaper,weimproveand
refinethealgorithmpresentedin[12,13].Themainstructureof
thealgorithmremainsthesame,i.e.,thecolorandspatialtexture
featuresarefirstdevelopedindependently,andthencombinedto
obtainanoverallsegmentation.Whilethecolorfeaturesalsore-
mainthesame,weuseanewsetofspatialtexturefeaturesbased
onthesteerablefilterdecomposition.Thesteerablefilterscom-
binedwithanewtexturesegmentationschemeprovideafinerand
morerobustsegmentationintodifferenttextureclasses(smooth,
horizontal,vertical,a0a2a1a4a3a4a5,a6a1a4a3a7a5,andcomplex).Akeytothe
proposedmethodistheuseofthe“max”operatortoaccountfor
thefactthatthereissignificantoverlapbetweenthefiltersthatcor-
respondtothedifferentorientations.Thisavoidsmisclassification
problemsassociatedwiththepreviouslyproposedtextureextrac-
tiontechnique[13].The“max”operationisfollowedbyame-
ThismaterialisbaseduponworksupportedbytheNationalScience
Foundation(NSF)underGrantNo.CCR-0209006.Anyopinions,findings
andconclusionsorrecomendationsexpressedinthismaterialarethoseof
theauthorsanddonotnecessarilyreflecttheviewsoftheNSF.
diantypeofoperationwhich,aswepointedoutin[12],responds
totexturewithinuniformregionsandsuppressestexturesassoci-
atedwithtransitionsbetweenregions.Finally,weuseanelaborate
borderrefinementprocedure[13],whichextendstheideaofthe
adaptiveclusteringalgorithm[7]tocolortexture,andresultsin
accurateborderlocations.
Thefocusofthisworkisinthedomainofphotographicim-
ages.Therangeoftopicsisessentiallyunlimited(people,nature,
buildings,textures,indoorscenes,etc.).Animportantassumption
isthattheimagesareofrelativelylowresolution(e.g.,a8a10a9a11a9a13a12a14a8a15a9a11a9)
andoccasionallydegradedorcompressed.Theimagesegmenta-
tionresultscanbeusedtoderiveregion-widecolorandtexture
features.Thesecanbecombinedwithothersegmentinforma-
tion,suchaslocation,boundaryshape,andsize,inordertoex-
tractsemanticinformation.Akeytothesuccessoftheproposed
approachistherecognitionofthefactthatitisnotnecessarytoob-
tainacompleteunderstandingofagivenimage:Inmanycases,the
identificationofafewkeysegments(suchas“sky,”“mountains,”
“people,”etc.)maybeenoughtoclassifytheimageinagiven
category[14].Inaddition,regionsthatareclassifiedascomplex
or“noneoftheabove,”canalsoplayasignificantroleinscene
analysis.
InSection2,wereviewthecolorfeatureextraction.Ournew
approachforspatialtexturefeatureextractionispresentedinSec-
tion3.Section4discussestheproposedalgorithmforcombining
thetextureandcolorfeaturestoobtainanoverallsegmentation.
2.COLORFEATUREEXTRACTION
Inthissection,wereviewthecolorfeatureextractionalgorithm
thatwasusedin[12].Themainideaistousespatiallyadap-
tivedominantcolorsasfeaturesthatincorporateknowledgeofhu-
manperception[12].Thecolorfeaturerepresentationconsistsof
alimitednumberoflocallyadapteddominantcolorsandthecor-
respondingpercentageofoccurrenceofeachcolorwithinacertain
neighborhood:a16
a17a15a18a20a19a22a21a24a23a25a21a24a26a28a27a11a29a30a11a31a33a32a35a34a36a18a20a37a39a38a24a21a20a40a25a38a41a31a42a21a24a43a44a32a46a45a10a21a39a47a39a47a39a47a48a21a50a49a51a21a52a40a4a38a54a53a56a55
a9
a21a57a45a59a58a61a60(1)
whereeachofthedominantcolors,a37a38,isathreedimensionalvec-
torinLabspace,anda40a4a38arethecorrespondingpercentages.a26a28a27a11a29a30
representstheneighborhoodaroundthepixelatlocation(x,y).a49
isthetotalnumberofcolorsintheneighborhooda26a27a11a29a30.
Thespatiallyadaptivedominantcolorsareobtainedbythe
adaptiveclusteringalgorithm(ACA)proposedin[7]andextended
tocolorin[8].TheACAisaniterativealgorithmthatusesspatial
constraintsintheformofMarkovrandomfields(MRF).Theal-
gorithmstartswithglobalestimates(obtainedusingthea62-means
algorithm)andslowlyadaptstothelocalcharacteristicsofeach
region.Wefoundthatagoodchoiceforthenumberof(locally
adapted)dominantcolorsisa49a63a32a65a64.
Fig.1.One-LevelSteerableFilterDecomposition
NotethattheACAwasdevelopedforimagesofobjectswith
smoothsurfacesandnotexture.Intexturedregions,theACAover-
segmentstheimage,butthesegmentsdocorrespondtoactualtex-
turedetails.Thus,someothermechanismisneededtoconsolidate
thesesmallsegmentsintoregions.Suchamechanismisprovided
bythe“localhistograms”wedescribenextandthetextureclasses
wepresentinthenextsection.
ColorFeatures:TheACAprovidesasegmentationoftheimage
intoclasses.IntheexampleofFig.5(b),eachpixelispaintedwith
theaveragecolorofthepixelsinitsneighborhoodthatbelongto
thesameclass[7].Assumingthatthedominantcolorsareslowly
varying,wecanassumethattheyareapproximatelyconstantin
theimmediatevicinityofapixel.Wecanthencountthenumber
ofpixelsineachclasswithinagivenwindow,andaveragetheir
colorvaluestoobtainafeaturevectorthatconsistsofafew(up
tofour)dominantcolorsandtheassociatedpercentages.Thus,
thecolorfeaturevectorateachpixelisessentiallyacrude“local
histogram”oftheimage.
ColorMetric:Tomeasuretheperceptualsimilarityoftwocolor
featurevectors,weusethe“OptimalColorCompositionDistance
(OCCD)”proposedbyMojsilovicetal.[15].TheOCCDwas
designedtoprovidetheoptimalmappingbetweenthedominant
colorsoftwoimages,andthusobtainabettermeasureoftheir
similarity.SinceweareusingOCCDtocomparelocalhistograms
thatcontainonlyfourbins,itsimplementationcanbesimplified
considerably[13].
3.SPATIALTEXTUREFEATUREEXTRACTION
Asin[12],thespatialtexturefeatureextractionisindependent
fromthatofcolor.Thus,weusethegray-scalecomponentofthe
imagetoobtainthespatialtexturefeatures,basedonwhichwe
obtainanintermediatesegmentation,whichisthenusedtogether
withthecolorfeaturesdescribedintheprevioussectiontoproduce
thefinalimagesegmentation.
Asindicatedin[13],thesteerablefilterdecomposition[16]
providesafinerfrequencydecompositionthatmorecloselycor-
respondstohumanvisualprocessing.Weuseaone-levelsteer-
ablefilterdecompositionwithfourorientationswhichprovidefour
textureclasses:horizontal,verticalandtwodiagonaldirections
(a0a64a67a66a5anda6a64a36a66a5),asshowninFig.1.
Asnotedinourearlierwork[12],manytexturesarenotdi-
rectional;thus,itisnecessarytoincludeacomplexor“noneof
above”category.Eventhoughsuchacategorydoesnotprovide
muchinformationaboutagivenregion,itneverthelessplaysan
importantroleintheoverallimageclassification.Notethateven
withtheadditionofthediagonaltexturecategories,thetexturede-
scriptionisstillquitecrude.However,unlikethetexturesynthesis
problemthatrequiresaveryprecisemodelinordertoaccurately
synthesizeawiderangeoftextures,acrudemodelcanbequitead-
equateforsegmentation.Suchasimplemodelisactuallythekey
toobtaininggoodsegmentationsfromlow-resolutioncompressed
images(e.g.,a8a15a9a11a9a68a12a69a8a15a9a11a9pixels).
Fig.2showsacircularcross-sectionfromthesteerablefilter
responses.Thex-coordinatedenotesspatialorientationindegrees.
Thus,thefilterwithpeakata9a5representsthehorizontalsubband
a70a57a71,thefilterwithpeakat
a72a11a9
a5representstheverticalsubband
a70a74a73,
andsoon.Notethatthereisalargeoverlapbetweenneighbor-
ingfilters.In[13],weaccountsuchoverlapbycomparingthe1st
and2ndmaximumamongthefoursubbandcoefficients.However,
thismethodcouldmisclassify,ascomplex,textureswithorienta-
tionsthatfallbetweenthemainorientationsofthesteerablefilters.
That’sbecauseforsuchtexturestheresponsesofthetwofilters
areclose.Thecomplexcategoryshouldinsteadbereservedfor
textureswithmanydifferentorientations.Notethatusingsharper
orientationfilterswillnarrowtherangeofmisclassifiedorienta-
tions,butwillnotentirelyeliminatetheproblem.Aswewillsee
below,wesolvethisproblembyintroducinga“max”operator,and
usingthelocalhistogramoforientationstodeterminethetexture
orientation.
Thefirststepinthetextureclassificationistoidentifyandlo-
catethesmoothregionsintheimage.Weusea70a71a18a20a19a22a21a75a23a7a31,a70a11a76a18a20a19a22a21a24a23a7a31,
a70a74a73
a18a20a19a22a21a75a23a7a31,
a70a57a77
a18a20a19a22a21a75a23a78a31torepresentthesteerablesubbandcoefficients
atlocationa18a20a19a22a21a24a23a7a31thatcorrespondtothehorizontal(a9a5),diagonal
withpositiveslope(a0a64a36a66a5),vertical(a72a11a9a5),anddiagonalwithnega-
tiveslope(a6a64a36a66a5)directions,respectively.Foreachimagelocation
a18a20a19a22a21a24a23a7a31,wefindthemaximumofthefourcoefficients,denotedby
a70a57a79a44a80a75a81
a18a20a19a82a21a75a23a7a31.Thesubbandindex
a70
a38a18a20a19a22a21a75a23a78a31thatcorrespondstothat
maximumisalsostoredforuseinthenextstep.Then,amedian
operationisperformedona70a79a44a80a75a81a18a20a19a22a21a83a23a7a31.Recallthatthevaluesin
a70a57a79a44a80a75a81
a18a20a19a82a21a75a23a7a31comefromfourdifferentsubbands;thus,thecross-
subbandmediancanonlyhelpindeterminingwhetherapixelbe-
longstoasmoothregion,notwhichtextureclassitbelongsto.
Finally,atwo-levelK-meansalgorithm,segmentstheimageinto
smoothandnon-smoothregions.
Aclustervalidationstepisnecessaryatthispoint.Iftheclus-
tersaretooclose,thentheimagemaycontainonlysmoothornon-
smoothregions,dependingontheactualvalueoftheclustercenter.
Wehavealsoexperimentedwithalternativewaystoobtain
smoothvs.non-smoothclassification.Forexample,wetriedan
approachsimilartotheonedescribedin[12],wherebyamedianis
appliedtoeachsubbandfollowedbya2-levelK-means.Apixelis
thenclassifiedassmoothifallsubbandsareclassifiedinthelower
class.Thisleadstosimilarresultsyetinvolvesmuchmorecom-
putation.Anotherapproachistoapplyamediantoeachsubband,
followedbyK-meansappliedtothevectorofthefoursubband
coefficients.Wefoundthattheproposedalgorithmhasthebest
performanceintermsofaccuracyandrobustness.
Thenextstageistofurtherclassifythepixelsinthenon-
smoothregions.Aswediscussedabove,thereissignificantover-
lapbetweenneighboringdirectionalfilters.Thus,eveninatexture
ofasingleorientation(e.g.,horizontal),theresponsesofthetwo
neighboringfilterswillstillbesignificant.Thus,themaximum
ofthefourcoefficientsistheonethatcarriessignificantinforma-
tionaboutthetextureorientation.Basedonthisobservation,we
usetheindexa70a38a18a20a19a22a21a75a23a7a31ofthesubbandwiththemaximumvaluein
ordertodeterminethetextureorientationforeachpixellocation.
Wethenconsiderawindow,andfindthepercentageofindices
foreachorientation.Onlynon-smoothpixelswithinthewindow
areconsidered.Ifthemaximumofthepercentagesishigherthana
giventhreshold(e.g.,36%)andthedifferencebetween1stand2nd
maximumissignificant(e.g.,greaterthan15%),weconcludethat
(a)Originalimage(b)Maxindex(c)TextureClasses
Fig.3.TextureMapExtraction
Fig.2.SteerableFilterFrequencyResponse
thereisadominantorientationinthewindowandthepixelisclas-
sifiedaccordingly.Otherwise,thereisnodominantorientation,
andthepixelisclassifiedascomplex.Thus,ourtextureclassifica-
tionisbasedonthelocalhistogramoftheindicescorrespondingto
maximumsubbandvalues.Thisisessentiallya“median”typeof
operation,whichisnecessary,aswesawin[12],forboostingthe
responsetotexturewithinuniformregionsandtosuppressthere-
sponseduetotexturesassociatedwithtransitionsbetweenregions.
AnexampleisgiveninFig.3.Fig.3(a)showsthegrey-levelcom-
ponentoftheoriginalcolorimage.Fig.3(b)showsthematrixa70a38of
indicesindicatingthemaxima.Thesmoothregionsareshownin
black,whiletheother4orientationsareshowninshadesofgray.
Fig.3(c)showstheresultingtextureclasses,whereblackdenotes
smooth,whitedenotescomplex,andlightgreydenoteshorizontal
textures.Thewindowusedinthisexamplewasa8a15a84a85a12a69a8a15a84.
Thewindowsizeformedianoperationshouldbereasonably
bigtoobtainanaccurateestimateofthehistogramandtosuppress
textureedges.Ontheotherhand,averybigwindowwillresult
intextureclassestoocrudetobeuseful.Ourexperimentsindi-
catethatawindowsizeintherangeofa45a57a86a12a45a74a86toa8a10a72a87a12a69a8a15a72works
well.Sincethetextureclassesareobtainedthroughwindowoper-
ations,weknowthetextureboundariesarenotaccurate.Thus,we
havetorelyonthecolortexturefeaturestoobtainamoreaccurate
segmentation.
4.FINALSEGMENTATION
Wenowdiscussthecombinationoftextureandcolorfeaturesto
obtainthefinalsegmentation.First,weconsiderthesmoothtex-
tureregions.Asin[12],werelyonthecolorsegmentationwhich
providesregionsofdifferentuniformcolors.RecallthattheACA
providesslowlyvaryingcolor.Toavoidoversegmentation,wefind
alltheconnectedsegmentsthatbelongtodifferentcolorclasses,
andthenmergeneighboringsegmentsiftheaveragecolordiffer-
enceacrossthecommonborderisbelowagiventhreshold.Finally,
Fig.4.Illustrationofborderrefinement.
anyremainingsmallregionsneighboringnon-smoothtexturere-
gionsarerelabeledascomplexsothattheycanbeconsideredin
thenextstep.
Next,weconsiderthenon-smoothtextureregions.First,we
usearegiongrowingapproachtoobtainaninitial“crude”seg-
mentationthatisbasedonthegrayscaletextureclassificationand
thecolorfeaturespresentedinSection2.Sincebothofthesefea-
turesareslowlyvarying,weuseamulti-gridapproach.Westart
frompixelslocatedonacoarsegrid.Wesetthewindowsizefor
thecolorfeatureequaltotwicethegridspacing,i.e.,thereisa
50%windowoverlap.Apairofpixelsbelongtothesameregionif
thecolorfeaturesaresimilarintheOCCDsense.Thethresholdis
higherforpixelsthatbelongtothesametextureclass(i.e.,easier
tomerge),andlowerforpixelsindifferenttextureclasses.Inad-
dition,weuseMRF-typespatialconstraints(asin[13]).Thatis,a
pixelismorelikelytobelongtoaregionifmanyofitsneighbors
belongtothesameregion.ThesymmetricMRFconstraintmakes
itnecessarytoiterateafewtimesforagivengridspacing.The
gridspacingisthenreduced,andtheprocedurerepeateduntilthe
gridspacingisequaltoonepixel.
Finally,thecrudesegmentationisrefinedusinganadaptive
algorithmsimilarinnaturetotheACA[7].Fig.4illustratesthe
idea.ThedottedlineinFig.4representstherealboundaryandthe
solidlinedenotestheboundarygivenbyouralgorithmincurrent
iterationoraninitialsegmentationobtainedinthepreviousstep.
Givenasegmentation,weusetwowindowstoupdatetheclas-
sificationofeachpixel.Thelargerwindowprovidesalocalized
estimateoftexturecharacteristicsofeachregionthatoverlapsthe
window.Foreachtexture,withinthesegmentationboundaries,we
findtheaveragecolorandthecorrespondingpercentageforeach
ofthedominantcolors.Thesmallerwindowprovidesanestimate
ofthepixeltexture.Thisconsistsofthedominantcolorscorre-
spondingpercentageswithinthesmallerwindowignoringthecur-
rentboundary.Thenthetextureofthepixeliscomparedwiththe
texturesofthedifferentregionsusingtheOCCDcriterion.The
procedureisrepeatedforeachpixelinarasterscan.Asin[7],
(a)OriginalColorImage(b)ColorSegmentation(ACA)(c)TextureSegmentation
(d)CrudeSegmentation(e)FinalSegmentation(f)FinalSegmentation(onoriginalimage)
Fig.5.ColorandTextureImageSegmentation(a,b,d,e,fshownincolor)
anMRF-typeconstraintisnecessarytoinsureregionsmoothness.
Afewiterationsarenecessaryforconvergence.Aniterationhas
convergedwhenthenumberofpixelsthatchangeclassisbelowa
giventhreshold.Theoverallproceduremustthenbeiteratedfora
seriesofwindowpairsstartingfrom35/5andendingwith11/3.
Oneoftheimportantdetailsintheaboveprocedureisthateach
ofthecandidateregionsinthelargerwindowmustbelargeenough
inordertoobtainareliableestimateofitstextureattributes.Oth-
erwise,theregionisnotavalidcandidate.Areasonablechoice
forthethresholdfordecidingwhetheraregionshouldbeavalid
candidateistousetheproductofthetwowindowsizesdividedby
2.ThecrudeandfinalsegmentationresultsareshowninFigs.5(e)
and(f).
Theuseofbothcolorandtextureinformationtoestimatere-
gionboundariesfindsfurtherjustificationinpsychophysicalex-
periments[17],whichshowedthattheperceivededgelocationisa
combinationofthepositionsignaledbytextureandbyothercues
(motion,luminance,coloretc).
5.REFERENCES
[1]Y.Rui,T.S.Huang,andS.-F.Chang,“Imageretrieval:
Currenttechniques,promisingdirectionsandopenissues,”
J.Vis.Comm.Im.Repr.,vol.10,pp.39–62,Mar.1999.
[2]W.M.Smeulders,etal.,“Content-basedimageretrievalat
theendoftheearlyyears,”IEEETr.PAMI,vol.22,pp.1349–
1379,Dec.2000.
[3]A.KunduandJ.-L.Chen,“TextureclassificationusingQMF
bank-basedsubbanddecomposition,”CVGIP,GMIP,vol.54,
pp.369–384,Sept.1992.
[4]T.ChangandC.-C.J.Kuo,“Textureanalysisandclas-
sificationwithtree-structuredwavelettransform,”IEEE
Tr.Im.Pr.,vol.2,pp.429–441,Oct.1993.
[5]M.Unser,“Textureclassificationandsegmentationusing
waveletframes,”IEEETr.Im.Pr.,vol.4,pp.1549–1560,
Nov.1995.
[6]T.RandenandJ.H.Husoy,“Texturesegmentationusing
filterswithoptimizedenergyseparation,”IEEETronIm.Pr.,
vol.8,pp.571–582,Apr.1999.
[7]T.N.Pappas,“Anadaptiveclusteringalgorithmforimage
segmentation,”IEEETr.SP,vol.40,pp.901–914,Apr.1992.
[8]M.M.Chang,M.I.Sezan,andA.M.Tekalp,“Adaptive
Bayesiansegmentationofcolorimages,”JEI,vol.3,pp.
404–414,Oct.1994.
[9]D.ComaniciuandP.Meer,“Robustanalysisoffeature
spaces:Colorimagesegmentation,”CVPR,June1997,pp.
750–755.
[10]S.Belongie,etal.,“Color-andtexture-basedimagesegmen-
tationusingEManditsapplicationtocontentbasedimage
retrieval,”ICCV,1998,pp.675–682.
[11]Y.DengandB.S.Manjunath,“Unsupervisedsegmenta-
tionofcolor-textureregionsinimagesandvideo,”IEEE
Tr.PAMI,vol.23,pp.800–810,Aug.2001.
[12]J.Chen,T.N.Pappas,A.Mojsilovic,andB.Rogowitz,
“Adaptiveimagesegmentationbasedoncolorandtexture,”
Proc.ICIP2002,Rochester,NewYork,Sept.2002.
[13]J.Chen,T.N.Pappas,A.Mojsilovic,andB.Rogowitz,“Per-
ceptualcolorandtexturefeaturesforsegmentation,”Human
VisionandElectronicImagingVIII,Jan.2003,Proc.SPIE
Vol.5007.
[14]A.MojsilovicandB.Rogowitz,“Capturingimagesemantics
withlow-leveldescriptors,”Proc.ICIP,Oct.2001,pp.18–
21.
[15]A.Mojsilovi′c,J.Hu,andE.Soljanin,“Extractionofpercep-
tuallyimportantcolorsandsimilaritymeasurementforimage
matching,retrieval,andanalysis,”IEEETr.Im.Pr.,vol.11,
no.11,pp.1238–1248,Nov.2002.
[16]E.P.SimoncelliandW.T.Freeman,“Thesteerablepyramid:
Aflexiblearchitectureformulti-scalederivativecomputa-
tion,”Proc.ICIP,vol.III,Oct.1995,pp.444–447.
[17]J.RivestandP.Cavanagh,“Localizingcontoursdefinedby
morethanoneattibute,”VisionResearch,vol.36,pp.53–66,
1996.
|
|
