AndroidMultimedia
FrameworkOverview
LiLi,SolutionandServiceWindRiver
|?2010WindRiver.AllRightsReserved.2
Agenda
?WhatisMultimediainamobiledevice
–MPEGstandard
–Fileformat
–Codec
?AndroidMultimediaFramework
–OpenCORE
–OpenMAX
–Whatwecandowiththis
|?2010WindRiver.AllRightsReserved.3
WhatisMultimediainmobiledevice
?Multimediaisthemostimportantcomponentinmodern
mobiledevicewithmodem
–Multimediaframeworkisusedtoprocessvideo/audioinputand
outputtosatisfycertainfunctionalityincluding
?VideoandAudio
–Storagevideoandaudioinmedia
–Playbackvideoandaudio
–Recordvideoandaudio
Audio,Video
devices/input
Perfect!
Audio
Video
|?2010WindRiver.AllRightsReserved.4
?Whatistheirrelationship
–Containerformatspecifyhowtowrapvariousmeta-data/stream,
mostiscodeddatabycodec
–Theoretically,acontainerformatcouldwrapanykindsofdata,
mostcontainerformatsarespecializedforspecificdata
requirements
–Containerdoesnotdescribehowthedatawarpedisencoded.
Alwaysaprogrambeabletoidentifyandopenafile,butnotbe
abletodecodecontaineddata.Youmaybeweretoldto
downloadrightdecoder.
Codec/DecodeandFileFormat
|?2010WindRiver.AllRightsReserved.5
?Codec/Decode
–deviceorcomputerprogramcapableofencodingand/or
decodingadigitaldatastreamorsignal
–Acodecencodesadatastreamorsignalfortransmission,
storageorencryptionanddecodeitforplaybackorediting.
codec=coder+decoder
–Rawmultimediadataishuge,codeccompressthemtofacilitate
storeandtransfer
Codec/DecodeandFileFormat
|?2010WindRiver.AllRightsReserved.6
?FileFormat
–Alsocallcontainerorwrapperformat
–Specifyhowdifferentdataelementsandmetadatacoexistina
computerfileorstream
–Alwayscontaincodedvideo,codedaudio,subtitles,chapter-
information,maybeadvertisementandsynchronization
informationneededtoplaybackvariousstreamstogether
Codec/DecodeandFileFormat
|?2010WindRiver.AllRightsReserved.7
?Codec-Audio
–AAC,MPEG-4AudioPart3subpart4
–AC-3,DolbyDigitalcodec
–AMR,AdaptiveMulti-RateAudiocodec
–AMR-WB,AdaptiveMulti-RateWideband
–MP2,MPEG1/2AudioLayerII
–MP3,MPEG2AudioLayerIII
–Vorbis,OGGaudio,opensourceproject
–Wma,WindowsMediaAudio
–RealAudio,RealNetwork
–ALAC,AppleLosslessAudioCodec
Codec/DecodeandFileFormat
|?2010WindRiver.AllRightsReserved.8
?Codec-Video
–MPEG–1,MPEG–1Part2
–MPEG–2/H.262,MPEG–2part2
–MPEG–4ASP,MPEG–4Part2
–MPEG–4AVC/H.264,MPEG–4Part10
–VC–1
?TheinformalnameofSMTP421Mvideocodecstandard
?InitiallydevelopedasaproprietyvideoformatbyMicrosoftbeforeitwasreleasedasa
formalSMPTEstandardvideoformatonApril3,2006
–VC–2
?Anopenandroyalty-freevideocompressionformat,Dirac.2010theSMPTE
standardizedDiracProasVC–2.
–VC–3
?DNXHD,DigitalNonlinearExtensibleHighDefinition
?alossyhigh-definitionvideopost-productioncodecengineeredformulti-generation
compositingwithreducedstorageandbandwithrequirements
?TheDNXHDcodecwassubmittedtotheSMPTEorganizationastheframeworkforthe
VC-3familyofstandard.
Codec/DecodeandFileFormat
|?2010WindRiver.AllRightsReserved.9
?StandardMPEG
–MovingPictureExpertsGroup.
–AworkinggroupofISO/IECinchargeofthedevelopmentof
internationalstandardsforcompression,decompression,
processingandcodedrepresentationofmovingpicturesaudio
andtheircombination
?Thestandardforstorageandretrievalofmovingpictures
andaudioonstoragemedia.ApprovedNov.1992
–VCD
–MP2,MPEG-1AudioLayerII
?Thestandardfordigitaltelevision.ApprovedNov.1994
–DVD
–MP3,MPEG-2AudioLayerIII
Codec/DecodeandFileFormat
|?2010WindRiver.AllRightsReserved.10
?MPEG-4
–Thestandardformultimediaapplication
–adevelopingstandard
–dividedintoanumberofparts
?Mostarelefttoindividualdeveloperstodecidedwhetherto
implement
?probablynocompleteimplementationsoftheentrireMPEG-4setof
standards
?Todealwiththis,thestandardincludesconceptof“profiles”and
“levels”
–allowingaspecificsetofcapabilitiestobedefinedinamanner
appropriateforasubsetofapplications
Codec/DecodeandFileFormat
|?2010WindRiver.AllRightsReserved.11
–MPEG-4part2,AdvancedSimpleProfile
?usedbycodecssuchasDiVX,Xvid,NeroDigitaland3ivx,
Quicktime6
–MPEG-4part10,MPEG-4AVC(AdvancedVideoCoding)/H.264
?usedbyx264encoder,NeroDigitalAVC,Quicktime7,andhigh-
definitionvideomedialikeBlu-rayDisc
Codec/DecodeandFileFormat
|?2010WindRiver.AllRightsReserved.12
?Video/Audiocontainer
–MPEGprogramstream
?standardcontainerforMPEG-1andMPEG-2elementarystreams
onreasonablyreliablemediasuchasdisks;usedalsoonDVD-
Videodiscs
–MPEG-2programstream,MPEG-TS
?Standardcontainerfordigitalbroadcastingandfortransportation
overunreliablemedia;usedalsoonBlu-rayDiscVedio;typically
containsmultiplevideoandaudiostream,andanelectronic
programguide
–MP4
?StandardaudioandvideocontainerfortheMPEG-4multimedia
portfolio,basedonMPEG-4Part12andJPEG2000Part12
Codec/DecodeandFileFormat
|?2010WindRiver.AllRightsReserved.13
?Containerformatparsertorecognizeandunwrapfile
?Codectoencode/decodedata.
?Synchronizationamongvariousstream
?Memory/Buffermanagement
?Streamtrackcontrol,playback,backwardsplay,forward
play
?Integratedintovideo/audiooutputsystem
?Takeadvantageofhardwareaccleration
–Hardwarecodec
–Hardwareoverlay
–Hardwareaudioflinger
WhatneedtodowithMultimedia
|?2010WindRiver.AllRightsReserved.14
AndroidMultimediaFramework
|?2010WindRiver.AllRightsReserved.15
OpenCORE–InitialAndroidMultimediaFramework
?Androidmultimediasubsystem
providedbyPacketVideo
–modular,extensibleframework
?Combiningindependentmediaprocessing
components
?fileformats,codecs,streamingprotocol
components
?renderingcompoments
?otherelementsindifferentwaysto
implementawidevarietyofmultimedia
scenarios
–abstractionaconceptionofNODE
?thebaseofthemodulization
?ImplementationinformofNODEfor
recognizermodule,parsermodule,codec
module,sinkmodule
|?2010WindRiver.AllRightsReserved.16
OpenCORE–InitialAndroidMultimediaFramework
|?2010WindRiver.AllRightsReserved.17
OpenCORE–InitialAndroidMultimediaFramework
PaserNode
FileHandler
Linuxhardwarecodecdriver
VDECNodeSinkNodeVMIO
ADECNodeSinkNodeAMIO
Sync
OMXHWCodecOMXSWCodec
Data
Control
|?2010WindRiver.AllRightsReserved.18
OpenCORE–InitialAndroidMultimediaFramework
TypeCodecDecoderEncoderComments
VideoH.263yesyesBaselineProfile
MPEG-4yesyesSimpleProfile
AVC/H.264yesyesConstrainedBaselineProfile
AudioMP3yesno
AACyesnoAAC,HE-AACv1/v2
AMR-NByesyes
AMR-WByesno
ContainerFormatParserComposer
MPEG4yesyes
3GPPyesyes
3GPP2yesno
MP3yesno
AACyesno
AMRyesyes
WAVyesno
|?2010WindRiver.AllRightsReserved.19
OpenMAX–BridgeCodec/DecodetoMultimediaFramework
?theIntegrationLayer(IL)interfacewithOpenCore
–royalty-free,cross-platformAPIfor
?comprehensivestreamingmediacodecandapplicationportability
?enableacceleratedmultimediacomponentstobedeveloped,
integratedandprogrammedacrossmultipleoperatingsystemsand
siliconplatform
–TheOpenMAXAPIwillbeshippedwithprocessorstoenable
libraryandcodec
?Rapidlyandeffectivelymakeuseofthefullaccelerationpotentialof
newsilicon
?regardlessoftheunderlyinghardwarearchitecture
|?2010WindRiver.AllRightsReserved.20
OpenMAX-Framework
|?2010WindRiver.AllRightsReserved.21
OpenMAXPositioninAndroidMultimediaFramework
OMXcore
(Software)
Decoder
Component
H.263
Decoder
Component
H.264
Decoder
Component
AAC
Decoder
Component
H263
Decoder
Component
H263
Decoder
Component
H263
OMXcore
(Hardware)
OpenCore
|?2010WindRiver.AllRightsReserved.22
OpenMAXAPIlist
?ForeachOMXcore,therearefunctionpointsneedtobe
implmented.
–pOMX_Init
–pOMX_Deinit
–pOMX_ComponentNameEnum
–pOMX_GetHandle
–pOMX_FreeHandle
–pOMX_GetComponentsOfRole
–pOMX_GetRoleofComponent
–pOMX_SetupTunel
–pOMX_GetContentPipe
–pOMXConfigParser
|?2010WindRiver.AllRightsReserved.23
OneSampleofOpenmaxDesign
|?2010WindRiver.AllRightsReserved.24
?Functionality
–Doesyousupportenoughcontainerformat
–Doesyousupportnecessarycodec
–Doesyousupportstream,videooverair
?Performance,critical
–Videoperformance.Framepersecond
–Audioperformance.Highqualitysound
–Synchronizationbetweenvideoandaudio
Commercialization
|?2010WindRiver.AllRightsReserved.25
?TakeadvantageofSoCcapability
–DesignOMXILlayerforspecifichardwarecodeccomponent
–DesignOverlaytakeadvantageofhardwaregpuorvideooutput
controller
–Designhighqualityaudiosolution
?Wholesystemvideobuffermanagementscheme
–Sharememorybufferbetweenvarioushardwarecomponent
–Decreasememorycopyaspossible
?Synchronizationscheme.Espforsomefaultfile
–Choosetherightpointtofixsyncproblem.Inparserorcodec.
Commercialization
|?2010WindRiver.AllRightsReserved.26
SurfaceFlingerVideooutput
OverlayDataGralloc
V4L2driverFramebuffer
Commercialization
GPU
OverlayControl
Videodata
VideoControl
Otherdata&control
|?2010WindRiver.AllRightsReserved.27
?AndroidintroduceStagefrighttoreplaceOpenCORE
startfroméclair
?EitherOpenCOREorStagefrightislinkedto
MediaPlayerService,soapplicationlevelneverknow
aboutthem
?Bothofthembasedonsameconecption,buttake
differentimplementation
–Stagefrighttakeparseranddecodeasawhole.Opencoretake
themasindividualnode
–Stagefrighttakeparser,decoder,sink/outputasserialoperation.
Opencoretakethemasparallel
–DifferentSychronizationmechanism
?EasierforStagefrighttodevelopparser
?AdoptOMXfromOpenCORE
OpenCOREandStagefright
|?2010WindRiver.AllRightsReserved.28
?ShortpointandStongpoint
–OpenCOREmorestableandmature,lowrisk
–OpenCOREsupportsmorecontainerformatandcodec
–Stragefrightismoreeasiertodevelopment.
OpenCOREandStagefright
|?2010WindRiver.AllRightsReserved.29
|
|
