Graphics | Android Developers

In this document

The Android framework has a variety of graphics rendering APIs for 2D and 3D that interact with your HAL implementations and graphics drivers, so it is important to have a good understanding of how they work at a higher level. There are two general ways that app developers can draw things to the screen: with Canvas or OpenGL.

android.graphics.Canvas is a 2D graphics API and is the most widely used graphics API by developers. Canvas operations draw all the stock android.view.Views and custom android.view.Views in Android. Prior to Android 3.0, Canvas used the Skia 2D drawing library to draw, which could not take advantage of hardware acceleration.

Introduced in Android 3.0, hardware acceleration for Canvas APIs uses a new drawing library called OpenGLRenderer that translates Canvas operations to OpenGL operations so that they can execute on the GPU. Developers had to opt-in to this feature previously, but beginning in Android 4.0, hardware-accelerated Canvas is enabled by default. Consequently, a hardware GPU that supports OpenGL ES 2.0 is mandatory for Android 4.0 devices.

The OpenGLRenderer does not interact with Skia, so we anticipate Skia to be slowly phased out without adverse effects to developers. Skia is currently deprecated and in maintenance mode but will be neccessary for a while because most apps published today still rely on non-hardware accelerated Canvas operations. In addition, not all Skia operations are supported by OpenGL, so some operations are still done in software with Skia, even with hardware acceleration turned on.

The other main way that developers render graphics is by using OpenGL ES 1.x or 2.0 to directly render to a surface. Android provides OpenGL ES interfaces in the android.opengl package that a developer can use to call into your GL implementation with the SDK or with native APIs provided in the Android NDK.

Note:A third option, Renderscript, was introduced in Android 3.0 to serve as a platform-agnostic graphics rendering API (it used OpenGL ES 2.0 under the hood), but will be deprecated starting in the Android 4.1 release.

How Android Renders Graphics

No matter what rendering API developers use, everything is rendered onto a buffer of pixel data called a "surface." Every window that is created on the Android platform is backed by a surface. All of the visible surfaces that are rendered to are composited onto the display by the SurfaceFlinger, Android's system service that manages composition of surfaces. Of course, there are more components that are involved in graphics rendering, and the main ones are described below:

Image Stream Producers: Image stream producers can be things such as an OpenGL ES game, video buffers from the media server, a Canvas 2D application, or basically anything that produces graphic buffers for consumption.
Image Stream Consumers: The most common consumer of image streams is SurfaceFlinger, the system service that consumes the currently visible surfaces and composites them onto the display using information provided by the Window Manager. SurfaceFlinger is the only service that can modify the content of the display. SurfaceFlinger uses OpenGL and the hardware composer to compose a group of surfaces. Other OpenGL ES apps can consume image streams as well, such as the camera app consuming a camera preview image stream.
SurfaceTexture: SurfaceTexture contains the logic that ties image stream producers and image stream consumers together and is made of three parts: SurfaceTextureClient, ISurfaceTexture, and SurfaceTexture (in this case, SurfaceTexture is the actual C++ class and not the name of the overall component). These three parts facilitate the producer (SurfaceTextureClient), binder (ISurfaceTexture), and consumer (SurfaceTexture) components of SurfaceTexture in processes such as requesting memory from Gralloc, sharing memory across process boundaries, synchronizing access to buffers, and pairing the appropriate consumer with the producer. SurfaceTexture can operate in both asynchronous (producer never blocks waiting for consumer and drops frames) and synchronous (producer waits for consumer to process textures) modes. Some examples of image producers are the camera preview produced by the camera HAL or an OpenGL ES game. Some examples of image consumers are SurfaceFlinger or another app that wants to display an OpenGL ES stream such as the camera app displaying the camera viewfinder.
Window Manager: The Android system service that controls window lifecycles, input and focus events, screen orientation, transitions, animations, position, transforms, z-order, and many other aspects of a window (a container for views). A window is always backed by a surface. The Window Manager sends all of the window metadata to SurfaceFlinger, so SurfaceFlinger can use that data to figure out how to composite surfaces on the display.
Hardware Composer: The hardware abstraction for the display subsystem. SurfaceFlinger can delegate certain composition work to the hardware composer to offload work from the OpenGL and the GPU. This makes compositing faster than having SurfaceFlinger do all the work. Starting with Jellybean MR1, new versions of the hardware composer have been introduced. See the hardware/libhardware/include/hardware/gralloc.h Hardware composer section for more information.
Gralloc: Allocates memory for graphics buffers. See the If you are using version 1.1 or later of the hardware composer, this HAL is no longer needed.

The following diagram shows how these components work together:

Figure 1. How surfaces are rendered

What You Need to Provide

The following list and sections describe what you need to provide to support graphics in your product:

OpenGL ES 1.x Driver
OpenGL ES 2.0 Driver
EGL Driver
Gralloc HAL implementation
Hardware Composer HAL implementation
Framebuffer HAL implementation

OpenGL and EGL drivers

You must provide drivers for OpenGL ES 1.x, OpenGL ES 2.0, and EGL. Some key things to keep in mind are:

The GL driver needs to be robust and conformant to OpenGL ES standards.
Do not limit the number of GL contexts. Because Android allows apps in the background and tries to keep GL contexts alive, you should not limit the number of contexts in your driver. It is not uncommon to have 20-30 active GL contexts at once, so you should also be careful with the amount of memory allocated for each context.
Support the YV12 image format and any other YUV image formats that come from other components in the system such as media codecs or the camera.
Support the mandatory extensions: GL_OES_texture_external, EGL_ANDROID_image_native_buffer, and EGL_ANDROID_recordable. We highly recommend supporting EGL_ANDROID_blob_cache and EGL_KHR_fence_sync as well.

Note that the OpenGL API exposed to app developers is different from the OpenGL interface that you are implementing. Apps do not have access to the GL driver layer, and must go through the interface provided by the APIs.

Pre-rotation

Many times, hardware overlays do not support rotation, so the solution is to pre-transform the buffer before it reaches SurfaceFlinger. A query hint in ANativeWindow was added (NATIVE_WINDOW_TRANSFORM_HINT) that represents the most likely transform to be be applied to the buffer by SurfaceFlinger. Your GL driver can use this hint to pre-transform the buffer before it reaches SurfaceFlinger, so when the buffer actually reaches SurfaceFlinger, it is correctly transformed. See the ANativeWindow interface defined in system/core/include/system/window.h for more details. The following is some pseudo-code that implements this in the hardware composer:

ANativeWindow->query(ANativeWindow, NATIVE_WINDOW_DEFAULT_WIDTH, &w);
ANativeWindow->query(ANativeWindow, NATIVE_WINDOW_DEFAULT_HEIGHT, &h);
ANativeWindow->query(ANativeWindow, NATIVE_WINDOW_TRANSFORM_HINT, &hintTransform);
if (hintTransform & HAL_TRANSFORM_ROT_90)
swap(w, h);

native_window_set_buffers_dimensions(anw, w, h);
ANativeWindow->dequeueBuffer(...);

// here GL driver renders content transformed by " hintTransform "

int inverseTransform;
inverseTransform = hintTransform;
if (hintTransform & HAL_TRANSFORM_ROT_90)
   inverseTransform ^= HAL_TRANSFORM_ROT_180;

native_window_set_buffers_transform(anw, inverseTransform);

ANativeWindow->queueBuffer(...);

Gralloc HAL

The graphics memory allocator is needed to allocate memory that is requested by SurfaceTextureClient in image producers. You can find a stub implementation of the HAL at hardware/libhardware/modules/gralloc.h

Protected buffers

There is a gralloc usage flag GRALLOC_USAGE_PROTECTED that allows the graphics buffer to be displayed only through a hardware protected path.

Hardware Composer HAL

The hardware composer is used by SurfaceFlinger to composite surfaces to the screen. The hardware composer abstracts things like overlays and 2D blitters and helps offload some things that would normally be done with OpenGL.

Jellybean MR1 introduces a new version of the HAL. We recommend that you start using version 1.1 of the hardware composer HAL as it will provide support for the newest features (explicit synchronization, external displays, etc). Keep in mind that in addition to 1.1 version, there is also a 1.0 version of the HAL that we used for internal compatibility reasons and a 1.2 draft mode of the hardware composer HAL. We recommend that you implement version 1.1 until 1.2 is out of draft mode.

Because the physical display hardware behind the hardware composer abstraction layer can vary from device to device, it is difficult to define recommended features, but here is some guidance:

The hardware composer should support at least 4 overlays (status bar, system bar, application, and live wallpaper) for phones and 3 overlays for tablets (no status bar).
Layers can be bigger than the screen, so the hardware composer should be able to handle layers that are larger than the display (For example, a wallpaper).
Pre-multiplied per-pixel alpha blending and per-plane alpha blending should be supported at the same time.
The hardware composer should be able to consume the same buffers that the GPU, camera, video decoder, and Skia buffers are producing, so supporting some of the following properties is helpful:
- RGBA packing order
- YUV formats
- Tiling, swizzling, and stride properties
A hardware path for protected video playback must be present if you want to support protected content.

The general recommendation when implementing your hardware composer is to implement a no-op hardware composer first. Once you have the structure done, implement a simple algorithm to delegate composition to the hardware composer. For example, just delegate the first three or four surfaces to the overlay hardware of the hardware composer. After that focus on common use cases, such as:

Full-screen games in portrait and landscape mode
Full-screen video with closed captioning and playback control
The home screen (compositing the status bar, system bar, application window, and live wallpapers)
Protected video playback
Multiple display support

After implementing the common use cases, you can focus on optimizations such as intelligently selecting the surfaces to send to the overlay hardware that maximizes the load taken off of the GPU. Another optimization is to detect whether the screen is updating. If not, delegate composition to OpenGL instead of the hardware composer to save power. When the screen updates again, contin`ue to offload composition to the hardware composer.

You can find the HAL for the hardware composer in the hardware/libhardware/include/hardware/hwcomposer.h and hardware/libhardware/include/hardware/hwcomposer_defs.h files. A stub implementation is available in the hardware/libhardware/modules/hwcomposer directory.

VSYNC

VSYNC synchronizes certain events to the refresh cycle of the display. Applications always start drawing on a VSYNC boundary and SurfaceFlinger always composites on a VSYNC boundary. This eliminates stutters and improves visual performance of graphics. The hardware composer has a function pointer

int (waitForVsync*) (int64_t *timestamp)

that points to a function you must implement for VSYNC. This function blocks until a VSYNC happens and returns the timestamp of the actual VSYNC. A client can receive a VSYNC timestamps once, at specified intervals, or continously (interval of 1). You must implement VSYNC to have no more than a 1ms lag at the maximum (1/2ms or less is recommended), and the timestamps returned must be extremely accurate.

Explicit synchronization

Explicit synchronization is required in Jellybean MR1 and later and provides a mechanism for Gralloc buffers to be acquired and released in a synchronized way. Explicit synchronization allows producers and consumers of graphics buffers to signal when they are done with a buffer. This allows the Android system to asynchronously queue buffers to be read or written with the certainty that another consumer or producer does not currently need them.

This communication is facilitated with the use of synchronization fences, which are now required when requesting a buffer for consuming or producing. The synchronization framework consists of three main parts:

sync_timeline: a monotonically increasing timeline that should be implemented for each driver instance. This basically is a counter of jobs submitted to the kernel for a particular piece of hardware.
sync_pt: a single value or point on a sync_timeline. A point has three states: active, signaled, and error. Points start in the active state and transition to the signaled or error states. For instance, when a buffer is no longer needed by an image consumer, this sync_point is signaled so that image producers know that it is okay to write into the buffer again.
sync_fence: a collection of sync_pts that often have different sync_timeline parents (such as for the display controller and GPU). This allows multiple consumers or producers to signal that they are using a buffer and to allow this information to be communicated with one function parameter. Fences are backed by a file descriptor and can be passed from kernel-space to user-space. For instance, a fence can contain two sync_points that signify when two separate image consumers are done reading a buffer. When the fence is signaled, the image producers now know that both consumers are done consuming.

To implement explicit synchronization, you need to do provide the following:

A kernel-space driver that implements a synchronization timeline for a particular piece of hardware. Drivers that need to be fence-aware are generally anything that accesses or communicates with the hardware composer. See the system/core/include/sync/sync.h file for more implementation details. The system/core/libsync directory includes a library to communicate with the kernel-space
A hardware composer HAL module (version 1.1 or later) that supports the new synchronization functionality. You will need to provide the appropriate synchronization fences as parameters to the set() and prepare() functions in the HAL. As a last resort, you can pass in -1 for the file descriptor parameters if you cannot support explicit synchronization for some reason. This is not recommended, however.
Two GL specific extensions related to fences, EGL_ANDROID_native_fence_sync and EGL_ANDROID_wait_sync, along with incorporating fence support into your graphics drivers.