android media framework 2

1.media abstraction

stock linux
============
process
v4l
gstreamer alsa

android
=======
activity
mediaserver
surfaceflinger audioflinger

1.An activity makes an association with surfaceflinger and audioflinger(manifested as an object or handle)
2.An activity knows what datasource to use
3.An activity passes this information to media server
4.mediaserver checks the datasource to determine which engine to use(checks extension .mp4 etc)
each engine is a dll
5.mediaserver passes the sf,af context to engine
6.engine gets the frames from the datasource
7.engine uses the sf and af context to post the frames to corresponding objects
8.sf takes care of underlying hardware
9.af takes care of underlying hardware
10.activity knows whether sf will use hw rendering or sw rendering??? it can also be that activity does not know and sf will try available renderers in a chained manner
11.similar to af
12.sf will have additional dll's for hw driver or software driver
13.af will have additonal dll for sw driver or hw driver
14.where does codecs come into picture??? when taking frames from the datasource,mediaserver will
send this to codec engines(dlls) to get a converted buffer.this buffer is ultimately passed to sf,af.
15.again the codecs will be decided based on hw or sw.also there will be a chain of codecs and
a fallback mechanism
16.every hw codec device will expose an interface to mediaserver (/dev/dsp1 etc)

17.whenever there is an interface in java, ...think of the instances it will represent.for MediaPlayer interface ... represents mp4,3gp,vorbis,midi instances.

18.so activity passes the triplet to mediaplayer instance.Each mediaplayer instance is also
registered with mediaserver.And each instance is declared in dll.

19.mediaserver for each actity can have 5 prongs
player prong
aud codec prong .... this can be multiple
vid codec prong ..... this can be multiple
sf prong ... single
af prong ... single


20.in context of activity sf can have 3 prongs
sw renderer
3d render
2d rendere
hw renderer
2d renderer
3d render
overlay renderer

In addition globally an overlay activity can work within the activity context
overlay renderer will punch through the activity surface to render

different renderer dlls register with sf at boot time or dynamically.
sf maintains a lookup table to see which dll to use for current activity context.
activity passes some parameter to sf to tell sf which renderer to use.
it may be a fallback mechanism also.

an activity will render into the display controllers buffer which is global.
overlay component will have direct access to this controller buffer,so it can render into this
irrespective of activity requirements.

this overlay component will have link to sf and external component.sf gives window info,external gives data

22.sf talks with sw renderer dll,(which directly writes to /dev/graphics/fb0)
opengl renderer dll(which talks to hw card supporting opengl maybe ../dev/graphics/hwaccel0)
so sf first calls opengl hw for manipulating buffers
it gets the modified buffer in return which it sends to fb
or first opengl hw, again open glhw ..which internally writes to fb0

23.a view can have multiple surfaces,each surface can have single glcontext,g1 and emulator doesnt
allow more than one glcontext per view.

fb0 is a surface.


24.an activity does its draw on a surface.this surface might be an abstraction for a hw.like an
accelerator.Once this is over it is handed over to surface flinger which will then write this
buffer to screen.before actually rendering to screen sf has join the menu,status bar etc and then write to hardware.content is copied twice before rendering.


25.android has composing api's and rendering api's
rendering is common for activity,movie playback etc

26.android uses gpu through libGLES_android.so
currently generic gui uses 2d rendering using skia.
generic gui doesnt use opengl.
Only explicit activities use opengl and hence gpus.