Upto: Table of Contents of full book "Programming the Raspberry Pi GPU"

Image Processing on the Raspberry Pi

This chapter looks at handling single images using OpenMAX on the Raspberry Pi. It also introduces a pipeline of components and looks at alternative ways of handling communication between them.

Resources

Files

Files used are here

Image components

The Raspberry Pi has a number of OpenMAX components specifically for image processing:

Image formats

OpenMAX has a number of data structures used to get and set information about components. We have seen some of these before

Some of these were discussed in the Components chapter.

We haven't looked at the field OMX_IMAGE_PORTDEFINITIONTYPE which is part of the port definition information it contains the following relevant fields

	
typedef struct OMX_IMAGE_PORTDEFINITIONTYPE {
    OMX_STRING cMIMEType;
    OMX_U32 nFrameWidth;
    OMX_U32 nFrameHeight;
    OMX_S32 nStride;
    OMX_U32 nSliceHeight;
    OMX_IMAGE_CODINGTYPE eCompressionFormat;
    OMX_COLOR_FORMATTYPE eColorFormat;
} OMX_IMAGE_PORTDEFINITIONTYPE;
	
      

The last two fields are the current values set for the port. The possible values are obtained from the next structure OMX_IMAGE_PARAM_PORTFORMATTYPE so we discuss it in the next paragraph. The major fields we get here though are paramaters about the image size: the nFrameWidth, OMX_U32 nFrameHeight, OMX_S32 nStride and OMX_U32 nSliceHeight. These are parameters we often need if we are, say, saving a decoded image in a different format such as TGA.

The OMX_IMAGE_PARAM_PORTFORMATTYPE is defined (Section 4.4.4 in the 1.1.2 specification) as

	
typedef struct OMX_IMAGE_PARAM_PORTFORMATTYPE {
    OMX_U32 nSize;
    OMX_VERSIONTYPE nVersion;
    OMX_U32 nPortIndex;
    OMX_U32 nIndex;
    OMX_IMAGE_CODINGTYPE eCompressionFormat;
    OMX_COLOR_FORMATTYPE eColorFormat;
} OMX_IMAGE_PARAM_PORTFORMATTYPE;
	
      

The first two fields are common to all OpenMAX structures. The nPortIndex is the port we are looking at. The nIndex field is to distinguish between all of the different format types supported by this port. The eCompressionFormat and eColorFormat give information about the format.

The values for OMX_IMAGE_CODINGTYPE are given in Table 4-66 of the 1.1.2 Specification and on the RPi are given in the file /opt/vc/include/IL/OMX_Image.h as

	
typedef enum OMX_IMAGE_CODINGTYPE {
    OMX_IMAGE_CodingUnused,      /** Value when format is N/A */
    OMX_IMAGE_CodingAutoDetect,  /** Auto detection of image format */
    OMX_IMAGE_CodingJPEG,        /** JPEG/JFIF image format */
    OMX_IMAGE_CodingJPEG2K,      /** JPEG 2000 image format */
    OMX_IMAGE_CodingEXIF,        /** EXIF image format */
    OMX_IMAGE_CodingTIFF,        /** TIFF image format */
    OMX_IMAGE_CodingGIF,         /** Graphics image format */
    OMX_IMAGE_CodingPNG,         /** PNG image format */
    OMX_IMAGE_CodingLZW,         /** LZW image format */
    OMX_IMAGE_CodingBMP,         /** Windows Bitmap format */
    OMX_IMAGE_CodingKhronosExtensions = 0x6F000000, /** Reserved region for introducing Khronos Standard Extensions */ 
    OMX_IMAGE_CodingVendorStartUnused = 0x7F000000, /** Reserved region for introducing Vendor Extensions */

    OMX_IMAGE_CodingTGA,
    OMX_IMAGE_CodingPPM,

    OMX_IMAGE_CodingMax = 0x7FFFFFFF
} OMX_IMAGE_CODINGTYPE;
	
      

Unfortunately on the RPi, the only value given is OMX_IMAGE_CodingUnused so this field does not give useful information on queries. However, the values here may need to be set in the OMX_IMAGE_PORTDEFINITIONTYPE to set the data compression type.

The type OMX_COLOR_FORMATTYPE is more informative. it is defined in the file /opt/vc/include/IL/OMX_IVCommon.h as

	
typedef enum OMX_COLOR_FORMATTYPE {
    OMX_COLOR_FormatUnused,
    OMX_COLOR_FormatMonochrome,
    OMX_COLOR_Format8bitRGB332,
    OMX_COLOR_Format12bitRGB444,
    OMX_COLOR_Format16bitARGB4444,
    OMX_COLOR_Format16bitARGB1555,
    OMX_COLOR_Format16bitRGB565,
    OMX_COLOR_Format16bitBGR565,
    OMX_COLOR_Format18bitRGB666,
    OMX_COLOR_Format18bitARGB1665,
    OMX_COLOR_Format19bitARGB1666, 
    OMX_COLOR_Format24bitRGB888,
    OMX_COLOR_Format24bitBGR888,
    OMX_COLOR_Format24bitARGB1887,
    OMX_COLOR_Format25bitARGB1888,
    OMX_COLOR_Format32bitBGRA8888,
    OMX_COLOR_Format32bitARGB8888,
    OMX_COLOR_FormatYUV411Planar,
    OMX_COLOR_FormatYUV411PackedPlanar,
    OMX_COLOR_FormatYUV420Planar,
    OMX_COLOR_FormatYUV420PackedPlanar,
    OMX_COLOR_FormatYUV420SemiPlanar,
    OMX_COLOR_FormatYUV422Planar,
    OMX_COLOR_FormatYUV422PackedPlanar,
    OMX_COLOR_FormatYUV422SemiPlanar,
    OMX_COLOR_FormatYCbYCr,
    OMX_COLOR_FormatYCrYCb,
    OMX_COLOR_FormatCbYCrY,
    OMX_COLOR_FormatCrYCbY,
    OMX_COLOR_FormatYUV444Interleaved,
    OMX_COLOR_FormatRawBayer8bit,
    OMX_COLOR_FormatRawBayer10bit,
    OMX_COLOR_FormatRawBayer8bitcompressed,
    OMX_COLOR_FormatL2, 
    OMX_COLOR_FormatL4, 
    OMX_COLOR_FormatL8, 
    OMX_COLOR_FormatL16, 
    OMX_COLOR_FormatL24, 
    OMX_COLOR_FormatL32,
    OMX_COLOR_FormatYUV420PackedSemiPlanar,
    OMX_COLOR_FormatYUV422PackedSemiPlanar,
    OMX_COLOR_Format18BitBGR666,
    OMX_COLOR_Format24BitARGB6666,
    OMX_COLOR_Format24BitABGR6666,
    OMX_COLOR_FormatKhronosExtensions = 0x6F000000, /** Reserved region for introducing Khronos Standard Extensions */ 
    OMX_COLOR_FormatVendorStartUnused = 0x7F000000, /** Reserved region for introducing Vendor Extensions */
    OMX_COLOR_Format32bitABGR8888,
    OMX_COLOR_Format8bitPalette,
    OMX_COLOR_FormatYUVUV128,
    OMX_COLOR_FormatRawBayer12bit,
    OMX_COLOR_FormatBRCMEGL,
    OMX_COLOR_FormatBRCMOpaque,
    OMX_COLOR_FormatYVU420PackedPlanar,
    OMX_COLOR_FormatYVU420PackedSemiPlanar,
    OMX_COLOR_FormatMax = 0x7FFFFFFF
} OMX_COLOR_FORMATTYPE;
	
      

A rather large number of formats!

Running the program info from the Components chapter shows for the image_decode component

	
Image ports:
  Ports start on 320
  There are 2 open ports
  Port 320 has 3 buffers (minimum 2) of size 81920
  Direction is input
    Supported image formats are:
      No coding format returned
  Port 321 has 1 buffers (minimum 1) of size 541696
  Direction is output
    Supported image formats are:
      Image format compression format 0
      Image format color encoding 0x6
      Image format compression format 0
      Image format color encoding 0x14
      Image format compression format 0
      Image format color encoding 0x17
      Image format compression format 0
      Image format color encoding 0x7F000001
      Image format compression format 0
      Image format color encoding 0x10
      No more formats supported
	
      

The color codings are

This looks good: both YUV and RGB formats supported. But if you look into the Broadcom specification for the image_decode component you get this statement for the OMX_IndexParamImagePortFormat

The input port supports several compressed formats, the output port supports several colour formats. However, the component doesn't support colour format changing, so the colour format will be set to match that emitted by the encountered image format.

In other words, you only get out the format that you put in. If the image input is RGB then so is the output while if the image input is YUV then so is the output. Format changing is supported by the Broadcom component resize and will be shown in a later example.

Decoding a JPEG image

In the Buffers chapter we gave an example which decoded a JPEG image from a file. There the emphasis was on buffer management; now we will redo it with the emphasis on decoding images.

Well actually we don't do much more. We just get the output image parameters by the call

	
void get_output_port_settings(COMPONENT_T *component) {
    OMX_PARAM_PORTDEFINITIONTYPE portdef;

    printf("Port settings changed\n");
    // need to setup the input for the resizer with the output of the
    // decoder
    portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
    portdef.nVersion.nVersion = OMX_VERSION;
    portdef.nPortIndex = 321;
    OMX_GetParameter(ilclient_get_handle(component),
                     OMX_IndexParamPortDefinition, &portdef);

    unsigned int uWidth =
        (unsigned int) portdef.format.image.nFrameWidth;
    unsigned int uHeight =
        (unsigned int) portdef.format.image.nFrameHeight;
    unsigned int uStride =
        (unsigned int) portdef.format.image.nStride;
    unsigned int uSliceHeight =
        (unsigned int) portdef.format.image.nSliceHeight;
    printf("Frame width %d, frame height %d, stride %d, slice height %d\n",
	   uWidth,
	   uHeight,
	   uStride,
	   uSliceHeight);
    printf("Getting format Compression 0x%x Color Format: 0x%x\n",
	   (unsigned int) portdef.format.image.eCompressionFormat,
	   (unsigned int) portdef.format.image.eColorFormat);
}
	
      

This call should appear immediately after the port settings event has been received. The code is in il_decode_image_redone.c (not shown).

The new output from one JPEG file is

	
Port settings changed
Frame width 3072, frame height 2304, stride 3072, slice height 2304
Getting format Compression 0x0 Color Format: 0x14
	
      

so that the output color format is OMX_COLOR_FormatYUV420PackedPlanar.

Tunnelling

The previous example just used a single component. This won't be the norm: to display an image will need a decode component and a render component, for example. The IL client can look after all of the input and output buffers, copying data or pointers between them. Alternatively, the components may be able to exchange information directly between themselves, a technique called tunnelling. (There is a third mechanism "proprietary communication" that we won't spend time over.).

These communication models are shown in Figure 2-2 of the 1.1.2 Specification, reproduced here:

The Broadcom components support tunnelling. This can make it much easier to write an application using multiple components (but not that easy of course!). The basic simplification is that you don't have to set up buffers between tunnelled ports, you don't have to negotiate data formats between them and you don't have to move data between the buffers; the components look after this themselves.

You still have to feed data into the first component's input buffers and take data out of the last component's output buffers. You also have to look after state management of the components and their ports - even the tunnelled ones.

The pseudo-code for a single component was given in the previous chapter as

	
create component in loaded state with all ports disabled
move the component to idle state
enable input ports (creating input port buffers)
enable output ports (creating output port buffers)
move the component to executing state

while there is more input data
    if there is an empty input buffer
         put data into the input buffer
         call EmptyThis Buffer
    if there is a full output buffer
         process its data
         call FillThisBuffer

while there is another full output buffer
     process its data
     call FillThisBuffer
	
      

The modification for two components A and B with a tunnel between them is

	
create component A in loaded state with all ports disabled
move the component to idle state
enable input ports (creating input port buffers)

create component B in loaded state with all ports disabled
move the component to idle state
enable output ports (creating output port buffers)

setup a tunnel between the output ports of A and the
    input ports of B
// note: the components will have created their own buffers

enable output ports of A (NOT creating output port buffers)
enable input ports of B (NOT creating input port buffers)

move component A to executing state
move component B to executing state

while there is more input data
    if there is an empty input buffer on A
         put data into the input buffer
         call EmptyThis Buffer
    if there is a full output buffer on B
         process its data
         call FillThisBuffer

while there is another full output buffer on B
     process its data
     call FillThisBuffer
	
      

Essentially we just split the input and output sides, with OpenMAX doing the tunnel communication between the components.

The above pseudo-code is fine if the sizes of buffers are fixed and known from the beginning. This isn't the case with components such as image_decode: the size of the output buffers isn't known until at least one input buffer has been processed. This complicates the code as the tunnel now cannot be established until a port settings changed event has been received on the first component's output port. A tunnel can only be established if the components are in the Loaded state or if their relevant ports are disabled.

There is a convenience function ilclient_setup_tunnel in the IL Client which can handle the state issues of the components and their ports. It also has a timeout parameter which if non-zero will manage waiting for a PortSettingsEvent. We shall ignore this option for now, as it hides too much what is going on.

The IL Client tunnelling calls use a struct

	
typedef struct {
   COMPONENT_T *source;  /** The source component */
   int source_port;      /** The output port index on the source component */
   COMPONENT_T *sink;    /** The sink component */
   int sink_port;        /** The input port index on the sink component */
} TUNNEL_T;
			      
      

and a convenience macro set_tunnel to set parameters as in

	
    TUNNEL_T tunnel;
    set_tunnel(&tunnel, decodeComponent, 321, renderComponent, 90);
	
      

The function ilclient_setup_tunnel takes three parameters

	
int ilclient_setup_tunnel(TUNNEL_T *tunnel,
                          unsigned int portStream,
                          int timeout);
	
      

The third parameter we will set to zero. The timeout will be zero if we want to handle PortSettingsChanged events ourselves.

The revised pseudo-code for this case is

	
create component A in loaded state with all ports disabled
move the component to idle state
enable input ports (creating input port buffers)

create component B in loaded state with all ports disabled
move the component to idle state
enable output ports (creating output port buffers)

move component A to executing state

put data into an input buffer of component A
call EmptyThis Buffer

wait for port settings changed on the output port of component A

// ilclient_setup_tunnel will handle these steps
// move component A back to idle state
// disable output ports of component A

setup the tunnel between the ports of components A and B
// note: the components will have created their own buffers

enable output ports of component A (NOT creating output port buffers)
enable output ports of component B (NOT creating input port buffers)

while there is more input data
    if there is an empty input buffer on A
         put data into the input buffer
         call EmptyThis Buffer
    if there is a full output buffer on B
         process its data
         call FillThisBuffer

while there is another full output buffer on B
     process its data
     call FillThisBuffer
	
      

Rendering an image using tunnelling

In the first program in this chapter we used the image_decode component to decode a JPEG image into YUV format. To render this, we need a render component. The OpenMAX specification surprisingly does not specify any render components. However, the Broadcom implementation has the component video_render which will render both single images and videos.

The video_render component has one input port, 60, and no output ports (its output goes to the render device). So we can simplify the pseudo-code as we don't have any data output buffers and can just concentrate on the input side.

The program to decode and render an image is il_render_image.c:

#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>

#include <OMX_Core.h>
#include <OMX_Component.h>

#include <bcm_host.h>
#include <ilclient.h>

#define IMG  "cimg0135.jpg"
//#define IMG "hype.jpg"

void printState(OMX_HANDLETYPE handle) {
    OMX_STATETYPE state;
    OMX_ERRORTYPE err;

    err = OMX_GetState(handle, &state);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error on getting state\n");
        exit(1);
    }
    switch (state) {
    case OMX_StateLoaded:           printf("StateLoaded\n"); break;
    case OMX_StateIdle:             printf("StateIdle\n"); break;
    case OMX_StateExecuting:        printf("StateExecuting\n"); break;
    case OMX_StatePause:            printf("StatePause\n"); break;
    case OMX_StateWaitForResources: printf("StateWait\n"); break;
    case OMX_StateInvalid:          printf("StateInvalid\n"); break;
    default:                        printf("State unknown\n"); break;
    }
}

char *err2str(int err) {
    switch (err) {
    case OMX_ErrorInsufficientResources: return "OMX_ErrorInsufficientResources";
    case OMX_ErrorUndefined: return "OMX_ErrorUndefined";
    case OMX_ErrorInvalidComponentName: return "OMX_ErrorInvalidComponentName";
    case OMX_ErrorComponentNotFound: return "OMX_ErrorComponentNotFound";
    case OMX_ErrorInvalidComponent: return "OMX_ErrorInvalidComponent";
    case OMX_ErrorBadParameter: return "OMX_ErrorBadParameter";
    case OMX_ErrorNotImplemented: return "OMX_ErrorNotImplemented";
    case OMX_ErrorUnderflow: return "OMX_ErrorUnderflow";
    case OMX_ErrorOverflow: return "OMX_ErrorOverflow";
    case OMX_ErrorHardware: return "OMX_ErrorHardware";
    case OMX_ErrorInvalidState: return "OMX_ErrorInvalidState";
    case OMX_ErrorStreamCorrupt: return "OMX_ErrorStreamCorrupt";
    case OMX_ErrorPortsNotCompatible: return "OMX_ErrorPortsNotCompatible";
    case OMX_ErrorResourcesLost: return "OMX_ErrorResourcesLost";
    case OMX_ErrorNoMore: return "OMX_ErrorNoMore";
    case OMX_ErrorVersionMismatch: return "OMX_ErrorVersionMismatch";
    case OMX_ErrorNotReady: return "OMX_ErrorNotReady";
    case OMX_ErrorTimeout: return "OMX_ErrorTimeout";
    case OMX_ErrorSameState: return "OMX_ErrorSameState";
    case OMX_ErrorResourcesPreempted: return "OMX_ErrorResourcesPreempted";
    case OMX_ErrorPortUnresponsiveDuringAllocation: return "OMX_ErrorPortUnresponsiveDuringAllocation";
    case OMX_ErrorPortUnresponsiveDuringDeallocation: return "OMX_ErrorPortUnresponsiveDuringDeallocation";
    case OMX_ErrorPortUnresponsiveDuringStop: return "OMX_ErrorPortUnresponsiveDuringStop";
    case OMX_ErrorIncorrectStateTransition: return "OMX_ErrorIncorrectStateTransition";
    case OMX_ErrorIncorrectStateOperation: return "OMX_ErrorIncorrectStateOperation";
    case OMX_ErrorUnsupportedSetting: return "OMX_ErrorUnsupportedSetting";
    case OMX_ErrorUnsupportedIndex: return "OMX_ErrorUnsupportedIndex";
    case OMX_ErrorBadPortIndex: return "OMX_ErrorBadPortIndex";
    case OMX_ErrorPortUnpopulated: return "OMX_ErrorPortUnpopulated";
    case OMX_ErrorComponentSuspended: return "OMX_ErrorComponentSuspended";
    case OMX_ErrorDynamicResourcesUnavailable: return "OMX_ErrorDynamicResourcesUnavailable";
    case OMX_ErrorMbErrorsInFrame: return "OMX_ErrorMbErrorsInFrame";
    case OMX_ErrorFormatNotDetected: return "OMX_ErrorFormatNotDetected";
    case OMX_ErrorContentPipeOpenFailed: return "OMX_ErrorContentPipeOpenFailed";
    case OMX_ErrorContentPipeCreationFailed: return "OMX_ErrorContentPipeCreationFailed";
    case OMX_ErrorSeperateTablesUsed: return "OMX_ErrorSeperateTablesUsed";
    case OMX_ErrorTunnelingUnsupported: return "OMX_ErrorTunnelingUnsupported";
    default: return "unknown error";
    }
}

void eos_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    fprintf(stderr, "Got eos event\n");
}

void error_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    fprintf(stderr, "OMX error %s\n", err2str(data));
}

int get_file_size(char *fname) {
    struct stat st;

    if (stat(fname, &st) == -1) {
	    perror("Stat'ing img file");
	    return -1;
	}
    return(st.st_size);
}

unsigned int uWidth;
unsigned int uHeight;

OMX_ERRORTYPE read_into_buffer_and_empty(FILE *fp,
					 COMPONENT_T *component,
					 OMX_BUFFERHEADERTYPE *buff_header,
					 int *toread) {
    OMX_ERRORTYPE r;

    int buff_size = buff_header->nAllocLen;
    int nread = fread(buff_header->pBuffer, 1, buff_size, fp);

    printf("Read %d\n", nread);

    buff_header->nFilledLen = nread;
    *toread -= nread;
    if (*toread <= 0) {
	printf("Setting EOS on input\n");
	buff_header->nFlags |= OMX_BUFFERFLAG_EOS;
    }
    r = OMX_EmptyThisBuffer(ilclient_get_handle(component),
			buff_header);
    if (r != OMX_ErrorNone) {
	fprintf(stderr, "Empty buffer error %s\n",
		err2str(r));
    }
    return r;
}

static void set_image_decoder_input_format(COMPONENT_T *component) {
   // set input image format
    printf("Setting image decoder format\n");
    OMX_IMAGE_PARAM_PORTFORMATTYPE imagePortFormat;
    //setHeader(&imagePortFormat,  sizeof(OMX_IMAGE_PARAM_PORTFORMATTYPE));
    memset(&imagePortFormat, 0, sizeof(OMX_IMAGE_PARAM_PORTFORMATTYPE));
    imagePortFormat.nSize = sizeof(OMX_IMAGE_PARAM_PORTFORMATTYPE);
    imagePortFormat.nVersion.nVersion = OMX_VERSION;

    imagePortFormat.nPortIndex = 320;
    imagePortFormat.eCompressionFormat = OMX_IMAGE_CodingJPEG;
    OMX_SetParameter(ilclient_get_handle(component),
                     OMX_IndexParamImagePortFormat, &imagePortFormat);

}

void setup_decodeComponent(ILCLIENT_T  *handle, 
			   char *decodeComponentName, 
			   COMPONENT_T **decodeComponent) {
    int err;

    err = ilclient_create_component(handle,
                                decodeComponent,
                                decodeComponentName,
                                ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    /* |
				    ILCLIENT_ENABLE_OUTPUT_BUFFERS
				    */
                                );
    if (err == -1) {
	fprintf(stderr, "DecodeComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*decodeComponent));

    err = ilclient_change_component_state(*decodeComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*decodeComponent));

    // must be before we enable buffers
    set_image_decoder_input_format(*decodeComponent);
}

void setup_renderComponent(ILCLIENT_T  *handle, 
			      char *renderComponentName, 
			      COMPONENT_T **renderComponent) {
    int err;

   err = ilclient_create_component(handle,
                                renderComponent,
                                renderComponentName,
                                ILCLIENT_DISABLE_ALL_PORTS
				   /* |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				   */
                                );
    if (err == -1) {
	fprintf(stderr, "RenderComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*renderComponent));

    err = ilclient_change_component_state(*renderComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*renderComponent));
}

int main(int argc, char** argv) {

    int i;
    char *decodeComponentName;
    char *renderComponentName;
    int err;
    ILCLIENT_T  *handle;
    COMPONENT_T *decodeComponent;
    COMPONENT_T *renderComponent;
    FILE *fp = fopen(IMG, "r");
    int toread = get_file_size(IMG);
    OMX_BUFFERHEADERTYPE *buff_header;

    decodeComponentName = "image_decode";
    renderComponentName = "video_render";

    bcm_host_init();

    handle = ilclient_init();
    if (handle == NULL) {
	fprintf(stderr, "IL client init failed\n");
	exit(1);
    }

   if (OMX_Init() != OMX_ErrorNone) {
        ilclient_destroy(handle);
        fprintf(stderr, "OMX init failed\n");
	exit(1);
    }

   ilclient_set_error_callback(handle,
			       error_callback,
			       NULL);
   ilclient_set_eos_callback(handle,
			     eos_callback,
			     NULL);


   setup_decodeComponent(handle, decodeComponentName, &decodeComponent);
   setup_renderComponent(handle, renderComponentName, &renderComponent);
   // both components now in Idle state, no buffers, ports disabled

    // input port
    ilclient_enable_port_buffers(decodeComponent, 320, 
				 NULL, NULL, NULL);
    ilclient_enable_port(decodeComponent, 320);


    err = ilclient_change_component_state(decodeComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Executing\n");
	exit(1);
    }
    printState(ilclient_get_handle(decodeComponent));


    // Read the first block so that the decodeComponent can get
    // the dimensions of the image and call port settings
    // changed on the output port to configure it
    buff_header = 
	ilclient_get_input_buffer(decodeComponent,
				  320,
				  1 /* block */);
    if (buff_header != NULL) {
	read_into_buffer_and_empty(fp,
				   decodeComponent,
				   buff_header,
				   &toread);

	// If all the file has been read in, then
	// we have to re-read this first block.
	// Broadcom bug?
	if (toread <= 0) {
	    printf("Rewinding\n");
	    // wind back to start and repeat
	    fp = freopen(IMG, "r", fp);
	    toread = get_file_size(IMG);
	}
    }

    // wait for first input block to set params for output port
    ilclient_wait_for_event(decodeComponent, 
			    OMX_EventPortSettingsChanged, 
			    321, 0, 0, 1,
			    ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED, 
			    5);

    printf("Port settings changed\n");

    TUNNEL_T tunnel;
    set_tunnel(&tunnel, decodeComponent, 321, renderComponent, 90);
    if ((err = ilclient_setup_tunnel(&tunnel, 0, 0)) < 0) {
	fprintf(stderr, "Error setting up tunnel %X\n", err);
	exit(1);
    } else {
	printf("Tunnel set up ok\n");
    }	

   // Okay to go back to processing data
   // enable the decode output ports
   
   OMX_SendCommand(ilclient_get_handle(decodeComponent), 
		   OMX_CommandPortEnable, 321, NULL);
   
   ilclient_enable_port(decodeComponent, 321);

   // enable the render output ports
   /*
   OMX_SendCommand(ilclient_get_handle(renderComponent), 
		   OMX_CommandPortEnable, 90, NULL);
   */
   ilclient_enable_port(renderComponent, 90);

   // set both components to executing state
    err = ilclient_change_component_state(decodeComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    err = ilclient_change_component_state(renderComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }

    // now work through the file
    while (toread > 0) {
	OMX_ERRORTYPE r;

	// do we have a decode input buffer we can fill and empty?
	buff_header = 
	    ilclient_get_input_buffer(decodeComponent,
				      320,
				      1 /* block */);
	if (buff_header != NULL) {
	    read_into_buffer_and_empty(fp,
				   decodeComponent,
				   buff_header,
				   &toread);
	}
    }

    ilclient_wait_for_event(renderComponent, 
			    OMX_EventBufferFlag, 
			    90, 0, OMX_BUFFERFLAG_EOS, 0,
			    ILCLIENT_BUFFER_FLAG_EOS, 10000);
    printf("EOS on render\n");

    sleep(100);

    exit(0);
}

      

Rendering an image without tunnelling

If we choose not to use tunnelling, then we have to handle all of the intermediate steps in managing the transfer of information from the decoder to the renderer. This adds a measure of complexity, but may be needed if, say, you want to overlay text onto a decoded image. The basic mechanism is to create buffers for one component using OMX_AllocateBuffer or ilclient_enable_port_buffers and to share the resultant buffers using OMX_UseBuffer.

In the tunnelling case, we had to wait until one input buffer had been processed before we knew enough about the input image and could establish the tunnel. Similarly, we have to wait for the first buffer before we can set up our non-tunnelled equivalent.

The first major difference lies in structure formats: the decoder uses the image type OMX_IMAGE_PORTDEFINITIONTYPE while the renderer uses the video type OMX_VIDEO_PORTDEFINITIONTYPE. While these share many common fields, they are not all aligned in the same place in memory. So some fields have to be copied across from the image format to the video format.

This involves getting the OMX_IMAGE_PORTDEFINITIONTYPE structure for each component, changing some fields in the video component and setting them back into the video component. The code to do this is

	
int setup_shared_buffer_format(COMPONENT_T *decodeComponent, 
			       COMPONENT_T *renderComponent) {
    OMX_PARAM_PORTDEFINITIONTYPE portdef,  rportdef;;
    int ret;
    OMX_ERRORTYPE err;

    // need to setup the input for the render with the output of the
    // decoder
    portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
    portdef.nVersion.nVersion = OMX_VERSION;
    portdef.nPortIndex = 321;
    OMX_GetParameter(ilclient_get_handle(decodeComponent),
                     OMX_IndexParamPortDefinition, &portdef);

    // Get default values of render
    rportdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
    rportdef.nVersion.nVersion = OMX_VERSION;
    rportdef.nPortIndex = 90;
    rportdef.nBufferSize = portdef.nBufferSize;
    nBufferSize = portdef.nBufferSize;

    err = OMX_GetParameter(ilclient_get_handle(renderComponent),
                           OMX_IndexParamPortDefinition, &rportdef);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error getting render port params %s\n", err2str(err));
        return err;
    }

    // tell render input what the decoder output will be providing
    //Copy some
    rportdef.format.video.nFrameWidth = portdef.format.image.nFrameWidth;
    rportdef.format.video.nFrameHeight = portdef.format.image.nFrameHeight;
    rportdef.format.video.nStride = portdef.format.image.nStride;
    rportdef.format.video.nSliceHeight = portdef.format.image.nSliceHeight;

    err = OMX_SetParameter(ilclient_get_handle(renderComponent),
                           OMX_IndexParamPortDefinition, &rportdef);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error setting render port params %s\n", err2str(err));
        return err;
    } else {
        printf("Render port params set up ok, buf size %d\n",
	       portdef.nBufferSize);
    }

    return  OMX_ErrorNone;
}
	
      

Once the first input buffer has been processed we can call the above function to synchronise formats, and also create output buffers (only one!) for the decoder. We can use the standard IL Client call to create the buffers ilclient_enable_port_buffers and enable this port by

	
    ilclient_enable_port_buffers(decodeComponent, 321, 
				 NULL, NULL, NULL);
    ilclient_enable_port(decodeComponent, 321);
	
      

There is no need at this point to share the buffer. Anyway, we don't yet have access to the decoder's buffers: not until a non-blocking call to ilclient_get_output_buffer has succeeded in returning an output buffer from the decoder. Once that has been done, a buffer header of type OMX_BUFFERHEADERTYPE * is made available to the client. We can then re-use the buffer in this header.

The following function is a bit messy: we only have one output buffer header from the decoder, but three input buffer headers for the renderer. We disable two of them by setting their buffers to NULL. For the shared one, we can only share the buffer, the buffer header must be distinct. The call OMX_UseBuffer creates a distinct header, but shares the buffer. Unfortunately, we must also copy across information from one buffer header to the other. This type of behaviour isn't documented, just something you discover painfully after an hour or so with the debugger and guessing a lot. What has to be copied is the length of the filled buffer:


	ppRenderInputBufferHeader[0]->nFilledLen = 
                            buff_header->nFilledLen;

There is one documented behaviour in the 1.1.2 Specification section 3.2.2.14:

OMX_UseBuffer macro shall be executed under the following conditions:

The problem is that this is either wrong, misleading or the Broadcom implementation is broken: even when the state is Executing, you have to enable the port, not disable it! Ho hum, another 2 days wasted figuring that out.

The relevant code is:

	
OMX_BUFFERHEADERTYPE *pOutputBufferHeader = NULL;
OMX_BUFFERHEADERTYPE **ppRenderInputBufferHeader;

int use_buffer(COMPONENT_T *renderComponent, 
		OMX_BUFFERHEADERTYPE *buff_header) {
    int ret;
    OMX_PARAM_PORTDEFINITIONTYPE portdef;

    ppRenderInputBufferHeader =
	(OMX_BUFFERHEADERTYPE **) malloc(sizeof(void) *
					 3);

    OMX_SendCommand(ilclient_get_handle(renderComponent), 
		    OMX_CommandPortEnable, 90, NULL);
    
    ilclient_wait_for_event(renderComponent,
			    OMX_EventCmdComplete,
			    OMX_CommandPortEnable, 1,
			    90, 1, 0,
			    5000);
    
    printState(ilclient_get_handle(renderComponent));

    ret = OMX_UseBuffer(ilclient_get_handle(renderComponent),
			&ppRenderInputBufferHeader[0],
			90,
			NULL,
			nBufferSize,
			buff_header->pBuffer);
    if (ret != OMX_ErrorNone) {
	fprintf(stderr, "Eror sharing buffer %s\n", err2str(ret));
	return ret;
    } else {
	printf("Sharing buffer ok\n");
    }

    ppRenderInputBufferHeader[0]->nAllocLen =
	buff_header->nAllocLen;

    int n;
    for (n = 1; n < 3; n++) {
	printState(ilclient_get_handle(renderComponent));
	ret = OMX_UseBuffer(ilclient_get_handle(renderComponent),
			    &ppRenderInputBufferHeader[n],
			    90,
			    NULL,
			    0,
			    NULL);
	if (ret != OMX_ErrorNone) {
	    fprintf(stderr, "Eror sharing null buffer %s\n", err2str(ret));
	    return ret;
	}
    }

    ilclient_enable_port(renderComponent, 90);

    ret = ilclient_change_component_state(renderComponent,
					  OMX_StateExecuting);
    if (ret < 0) {
	fprintf(stderr, "Couldn't change render state to Executing\n");
	exit(1);
    }
    return 0;
	
      

Those are the major issues. Playing games with component state and port state until it works are standard. The program is il_render_image_notunnel.c

#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>

#include <OMX_Component.h>

#include <bcm_host.h>
#include <ilclient.h>

//#define IMG  "cimg0135.jpg"
//#define IMG  "jan.jpg";
#define IMG "hype.jpg"
char *img_file = IMG;

void printState(OMX_HANDLETYPE handle) {
    OMX_STATETYPE state;
    OMX_ERRORTYPE err;

    err = OMX_GetState(handle, &state);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error on getting state\n");
        exit(1);
    }
    switch (state) {
    case OMX_StateLoaded:           printf("StateLoaded\n"); break;
    case OMX_StateIdle:             printf("StateIdle\n"); break;
    case OMX_StateExecuting:        printf("StateExecuting\n"); break;
    case OMX_StatePause:            printf("StatePause\n"); break;
    case OMX_StateWaitForResources: printf("StateWait\n"); break;
    case OMX_StateInvalid:          printf("StateInvalid\n"); break;
    default:                        printf("State unknown\n"); break;
    }
}

char *err2str(int err) {
    switch (err) {
    case OMX_ErrorInsufficientResources: return "OMX_ErrorInsufficientResources";
    case OMX_ErrorUndefined: return "OMX_ErrorUndefined";
    case OMX_ErrorInvalidComponentName: return "OMX_ErrorInvalidComponentName";
    case OMX_ErrorComponentNotFound: return "OMX_ErrorComponentNotFound";
    case OMX_ErrorInvalidComponent: return "OMX_ErrorInvalidComponent";
    case OMX_ErrorBadParameter: return "OMX_ErrorBadParameter";
    case OMX_ErrorNotImplemented: return "OMX_ErrorNotImplemented";
    case OMX_ErrorUnderflow: return "OMX_ErrorUnderflow";
    case OMX_ErrorOverflow: return "OMX_ErrorOverflow";
    case OMX_ErrorHardware: return "OMX_ErrorHardware";
    case OMX_ErrorInvalidState: return "OMX_ErrorInvalidState";
    case OMX_ErrorStreamCorrupt: return "OMX_ErrorStreamCorrupt";
    case OMX_ErrorPortsNotCompatible: return "OMX_ErrorPortsNotCompatible";
    case OMX_ErrorResourcesLost: return "OMX_ErrorResourcesLost";
    case OMX_ErrorNoMore: return "OMX_ErrorNoMore";
    case OMX_ErrorVersionMismatch: return "OMX_ErrorVersionMismatch";
    case OMX_ErrorNotReady: return "OMX_ErrorNotReady";
    case OMX_ErrorTimeout: return "OMX_ErrorTimeout";
    case OMX_ErrorSameState: return "OMX_ErrorSameState";
    case OMX_ErrorResourcesPreempted: return "OMX_ErrorResourcesPreempted";
    case OMX_ErrorPortUnresponsiveDuringAllocation: return "OMX_ErrorPortUnresponsiveDuringAllocation";
    case OMX_ErrorPortUnresponsiveDuringDeallocation: return "OMX_ErrorPortUnresponsiveDuringDeallocation";
    case OMX_ErrorPortUnresponsiveDuringStop: return "OMX_ErrorPortUnresponsiveDuringStop";
    case OMX_ErrorIncorrectStateTransition: return "OMX_ErrorIncorrectStateTransition";
    case OMX_ErrorIncorrectStateOperation: return "OMX_ErrorIncorrectStateOperation";
    case OMX_ErrorUnsupportedSetting: return "OMX_ErrorUnsupportedSetting";
    case OMX_ErrorUnsupportedIndex: return "OMX_ErrorUnsupportedIndex";
    case OMX_ErrorBadPortIndex: return "OMX_ErrorBadPortIndex";
    case OMX_ErrorPortUnpopulated: return "OMX_ErrorPortUnpopulated";
    case OMX_ErrorComponentSuspended: return "OMX_ErrorComponentSuspended";
    case OMX_ErrorDynamicResourcesUnavailable: return "OMX_ErrorDynamicResourcesUnavailable";
    case OMX_ErrorMbErrorsInFrame: return "OMX_ErrorMbErrorsInFrame";
    case OMX_ErrorFormatNotDetected: return "OMX_ErrorFormatNotDetected";
    case OMX_ErrorContentPipeOpenFailed: return "OMX_ErrorContentPipeOpenFailed";
    case OMX_ErrorContentPipeCreationFailed: return "OMX_ErrorContentPipeCreationFailed";
    case OMX_ErrorSeperateTablesUsed: return "OMX_ErrorSeperateTablesUsed";
    case OMX_ErrorTunnelingUnsupported: return "OMX_ErrorTunnelingUnsupported";
    default: return "unknown error";
    }
}

void eos_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    fprintf(stderr, "Got eos event\n");
}

void error_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    fprintf(stderr, "OMX error %s\n", err2str(data));
}

int get_file_size(char *fname) {
    struct stat st;

    if (stat(fname, &st) == -1) {
	perror("Stat'ing img file");
	return -1;
    }
    return(st.st_size);
}

unsigned int uWidth;
unsigned int uHeight;

unsigned int nBufferSize;

OMX_ERRORTYPE read_into_buffer_and_empty(FILE *fp,
					 COMPONENT_T *component,
					 OMX_BUFFERHEADERTYPE *buff_header,
					 int *toread) {
    OMX_ERRORTYPE r;

    int buff_size = buff_header->nAllocLen;
    int nread = fread(buff_header->pBuffer, 1, buff_size, fp);

    printf("Read %d\n", nread);

    buff_header->nFilledLen = nread;
    *toread -= nread;
    if (*toread <= 0) {
	printf("Setting EOS on input\n");
	buff_header->nFlags |= OMX_BUFFERFLAG_EOS;
    }
    r = OMX_EmptyThisBuffer(ilclient_get_handle(component),
			    buff_header);
    if (r != OMX_ErrorNone) {
	fprintf(stderr, "Empty buffer error %s\n",
		err2str(r));
    }
    return r;
}

static void set_image_decoder_input_format(COMPONENT_T *component) {

    printf("Setting image decoder format\n");
    OMX_IMAGE_PARAM_PORTFORMATTYPE imagePortFormat;

    memset(&imagePortFormat, 0, sizeof(OMX_IMAGE_PARAM_PORTFORMATTYPE));
    imagePortFormat.nSize = sizeof(OMX_IMAGE_PARAM_PORTFORMATTYPE);
    imagePortFormat.nVersion.nVersion = OMX_VERSION;

    imagePortFormat.nPortIndex = 320;
    imagePortFormat.eCompressionFormat = OMX_IMAGE_CodingJPEG;
    OMX_SetParameter(ilclient_get_handle(component),
                     OMX_IndexParamImagePortFormat, &imagePortFormat);

}

void setup_decodeComponent(ILCLIENT_T  *handle, 
			   char *decodeComponentName, 
			   COMPONENT_T **decodeComponent) {
    int err;

    err = ilclient_create_component(handle,
				    decodeComponent,
				    decodeComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    |
				    ILCLIENT_ENABLE_OUTPUT_BUFFERS
				    );
    if (err == -1) {
	fprintf(stderr, "DecodeComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*decodeComponent));

    err = ilclient_change_component_state(*decodeComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*decodeComponent));

    // must be before we enable buffers
    set_image_decoder_input_format(*decodeComponent);
}

void setup_renderComponent(ILCLIENT_T  *handle, 
			   char *renderComponentName, 
			   COMPONENT_T **renderComponent) {
    int err;

    err = ilclient_create_component(handle,
				    renderComponent,
				    renderComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    );
    if (err == -1) {
	fprintf(stderr, "RenderComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*renderComponent));

    err = ilclient_change_component_state(*renderComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*renderComponent));
}

OMX_BUFFERHEADERTYPE *pOutputBufferHeader = NULL;
OMX_BUFFERHEADERTYPE **ppRenderInputBufferHeader;

int setup_shared_buffer_format(COMPONENT_T *decodeComponent, 
			       COMPONENT_T *renderComponent) {
    OMX_PARAM_PORTDEFINITIONTYPE portdef,  rportdef;;
    int ret;
    OMX_ERRORTYPE err;

    // need to setup the input for the render with the output of the
    // decoder
    portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
    portdef.nVersion.nVersion = OMX_VERSION;
    portdef.nPortIndex = 321;
    OMX_GetParameter(ilclient_get_handle(decodeComponent),
                     OMX_IndexParamPortDefinition, &portdef);

    // Get default values of render
    rportdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
    rportdef.nVersion.nVersion = OMX_VERSION;
    rportdef.nPortIndex = 90;
    rportdef.nBufferSize = portdef.nBufferSize;
    nBufferSize = portdef.nBufferSize;

    err = OMX_GetParameter(ilclient_get_handle(renderComponent),
                           OMX_IndexParamPortDefinition, &rportdef);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error getting render port params %s\n", err2str(err));
        return err;
    }

    // tell render input what the decoder output will be providing
    //Copy some
    rportdef.format.video.nFrameWidth = portdef.format.image.nFrameWidth;
    rportdef.format.video.nFrameHeight = portdef.format.image.nFrameHeight;
    rportdef.format.video.nStride = portdef.format.image.nStride;
    rportdef.format.video.nSliceHeight = portdef.format.image.nSliceHeight;

    err = OMX_SetParameter(ilclient_get_handle(renderComponent),
                           OMX_IndexParamPortDefinition, &rportdef);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error setting render port params %s\n", err2str(err));
        return err;
    } else {
        printf("Render port params set up ok, buf size %d\n",
	       portdef.nBufferSize);
    }

    return  OMX_ErrorNone;
}

int use_buffer(COMPONENT_T *renderComponent, 
		OMX_BUFFERHEADERTYPE *buff_header) {
    int ret;
    OMX_PARAM_PORTDEFINITIONTYPE portdef;

    ppRenderInputBufferHeader =
	(OMX_BUFFERHEADERTYPE **) malloc(sizeof(void) *
					 3);

    OMX_SendCommand(ilclient_get_handle(renderComponent), 
		    OMX_CommandPortEnable, 90, NULL);
    
    ilclient_wait_for_event(renderComponent,
			    OMX_EventCmdComplete,
			    OMX_CommandPortEnable, 1,
			    90, 1, 0,
			    5000);
    
    printState(ilclient_get_handle(renderComponent));

    ret = OMX_UseBuffer(ilclient_get_handle(renderComponent),
			&ppRenderInputBufferHeader[0],
			90,
			NULL,
			nBufferSize,
			buff_header->pBuffer);
    if (ret != OMX_ErrorNone) {
	fprintf(stderr, "Eror sharing buffer %s\n", err2str(ret));
	return ret;
    } else {
	printf("Sharing buffer ok\n");
    }

    ppRenderInputBufferHeader[0]->nAllocLen =
	buff_header->nAllocLen;

    int n;
    for (n = 1; n < 3; n++) {
	printState(ilclient_get_handle(renderComponent));
	ret = OMX_UseBuffer(ilclient_get_handle(renderComponent),
			    &ppRenderInputBufferHeader[n],
			    90,
			    NULL,
			    0,
			    NULL);
	if (ret != OMX_ErrorNone) {
	    fprintf(stderr, "Eror sharing null buffer %s\n", err2str(ret));
	    return ret;
	}
    }

    ilclient_enable_port(renderComponent, 90);

    ret = ilclient_change_component_state(renderComponent,
					  OMX_StateExecuting);
    if (ret < 0) {
	fprintf(stderr, "Couldn't change render state to Executing\n");
	exit(1);
    }
    return 0;
}

int main(int argc, char** argv) {

    int i;
    char *decodeComponentName;
    char *renderComponentName;
    int err;
    ILCLIENT_T  *handle;
    COMPONENT_T *decodeComponent;
    COMPONENT_T *renderComponent;

    if (argc == 2) {
	img_file = argv[1];
    }
    FILE *fp = fopen(img_file, "r");
    int toread = get_file_size(img_file);
  
    OMX_BUFFERHEADERTYPE *buff_header;

    decodeComponentName = "image_decode";
    renderComponentName = "video_render";

    bcm_host_init();

    handle = ilclient_init();
    if (handle == NULL) {
	fprintf(stderr, "IL client init failed\n");
	exit(1);
    }

    if (OMX_Init() != OMX_ErrorNone) {
	ilclient_destroy(handle);
	fprintf(stderr, "OMX init failed\n");
	exit(1);
    }

    ilclient_set_error_callback(handle,
				error_callback,
				NULL);
    ilclient_set_eos_callback(handle,
			      eos_callback,
			      NULL);


    setup_decodeComponent(handle, decodeComponentName, &decodeComponent);
    setup_renderComponent(handle, renderComponentName, &renderComponent);
    // both components now in Idle state, no buffers, ports disabled

    // input port
    ilclient_enable_port_buffers(decodeComponent, 320, 
				 NULL, NULL, NULL);
    ilclient_enable_port(decodeComponent, 320);


    err = ilclient_change_component_state(decodeComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change decode state to Executing\n");
	exit(1);
    }
    printState(ilclient_get_handle(decodeComponent));


    // Read the first block so that the decodeComponent can get
    // the dimensions of the image and call port settings
    // changed on the output port to configure it
    buff_header = 
	ilclient_get_input_buffer(decodeComponent,
				  320,
				  1 /* block */);
    if (buff_header != NULL) {
	read_into_buffer_and_empty(fp,
				   decodeComponent,
				   buff_header,
				   &toread);

	// If all the file has been read in, then
	// we have to re-read this first block.
	// Broadcom bug?
	if (toread <= 0) {
	    printf("Rewinding\n");
	    // wind back to start and repeat
	    fp = freopen(img_file, "r", fp);
	    toread = get_file_size(img_file);
	}
    }

    // wait for first input block to set params for output port
    ilclient_wait_for_event(decodeComponent, 
			    OMX_EventPortSettingsChanged, 
			    321, 0, 0, 1,
			    ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED, 
			    10000);
    printf("Port settings changed\n");

    setup_shared_buffer_format(decodeComponent, renderComponent);

   ilclient_enable_port_buffers(decodeComponent, 321, 
				 NULL, NULL, NULL);
    ilclient_enable_port(decodeComponent, 321);

    // set decoder only to executing state
    err = ilclient_change_component_state(decodeComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Executing\n");
	exit(1);
    }

    // now work through the file
    while (toread > 0) {
	OMX_ERRORTYPE r;

	// do we have a decode input buffer we can fill and empty?
	buff_header = 
	    ilclient_get_input_buffer(decodeComponent,
				      320,
				      1 /* block */);
	if (buff_header != NULL) {
	    read_into_buffer_and_empty(fp,
				       decodeComponent,
				       buff_header,
				       &toread);
	}

	// do we have an output buffer that has been filled?
	buff_header = 
	    ilclient_get_output_buffer(decodeComponent,
				      321,
				      0 /* no block */);
	if (buff_header != NULL) {
	    printf("Got an output buffer length %d\n",
		   buff_header->nFilledLen);
	    if (buff_header->nFlags & OMX_BUFFERFLAG_EOS) {
		printf("Got EOS\n");
	    }
	    if (pOutputBufferHeader == NULL) {
		use_buffer(renderComponent, buff_header);
		pOutputBufferHeader = buff_header;
	    }

	    if (buff_header->nFilledLen > 0) {
		OMX_EmptyThisBuffer(ilclient_get_handle(renderComponent),
				    buff_header);
	    }

	    OMX_FillThisBuffer(ilclient_get_handle(decodeComponent), 
			       buff_header); 
	}
    }

    int done = 0;
    while ( !done ) {
	printf("Getting last output buffers\n");
	buff_header = 
	    ilclient_get_output_buffer(decodeComponent,
				       321,
				       1 /* block */);
	printf("Got a final output buffer length %d\n",
	       buff_header->nFilledLen);
	if (buff_header->nFlags & OMX_BUFFERFLAG_EOS) {
	    printf("Got EOS\n");
	    done = 1;
	}

	if (pOutputBufferHeader == NULL) {
	    use_buffer(renderComponent, buff_header);
	    pOutputBufferHeader = buff_header;
	}

	ppRenderInputBufferHeader[0]->nFilledLen = buff_header->nFilledLen;
	OMX_EmptyThisBuffer(ilclient_get_handle(renderComponent),
			    ppRenderInputBufferHeader[0]);
   }

    sleep(100);

    exit(0);
}

      

Conclusion

This chapter has shown basic image handling using multiple components. There are more complex things that can be done, which will be addressed in later versions of this book.


      

Copyright © Jan Newmarch, jan@newmarch.name
Creative Commons License
" Programming AudioVideo on the Raspberry Pi GPU " by Jan Newmarch is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License .
Based on a work at https://jan.newmarch.name/RPi/ .

If you like this book, please contribute using PayPal

Or Flattr me:
Flattr this book