Upto: Table of Contents of full book "Programming the Raspberry Pi GPU"

OpenMAX Video Processing on the Raspberry Pi

This chapter looks at video processing using the Broadcom GPU on the Raspberry Pi.

Resources

Files

Files used are here

Video components

The Raspberry Pi has a number of OpenMAX components specifically for video processing:

Video formats

OpenMAX has a number of data structures used to get and set information about components. We have seen some of these before

Some of these were discussed in the Components chapter.

We haven't looked at the field OMX_VIDEO_PORTDEFINITIONTYPE which is part of the port definition information. It contains the following relevant fields

	
typedef struct OMX_VIDEO_PORTDEFINITIONTYPE {
    OMX_STRING cMIMEType;
    OMX_NATIVE_DEVICETYPE pNativeRender;
    OMX_U32 nFrameWidth;
    OMX_U32 nFrameHeight;
    OMX_S32 nStride;
    OMX_U32 nSliceHeight;
    OMX_U32 nBitrate;
    OMX_U32 xFramerate;
    OMX_BOOL bFlagErrorConcealment;
    OMX_VIDEO_CODINGTYPE eCompressionFormat;
    OMX_COLOR_FORMATTYPE eColorFormat;
    OMX_NATIVE_WINDOWTYPE pNativeWindow;
} OMX_VIDEO_PORTDEFINITIONTYPE;
	
      

The last two but one fields are the current values set for the port. The possible values are obtained from the next structure OMX_VIDEO_PARAM_PORTFORMATTYPE so we discuss it in the next paragraph. The major fields we get here though are paramaters about the video size: the nFrameWidth, OMX_U32 nFrameHeight, OMX_S32 nStride and OMX_U32 nSliceHeight. These are parameters we often need if we are, say, saving a decoded video in a different format such as MPEG4.

The OMX_VIDEO_PARAM_PORTFORMATTYPE is defined (Section 4.3.5 in the 1.1.2 specification) as

	
typedef struct OMX_VIDEO_PARAM_PORTFORMATTYPE {
    OMX_U32 nSize;
    OMX_VERSIONTYPE nVersion;
    OMX_U32 nPortIndex;
    OMX_U32 nIndex;
    OMX_VIDEO_CODINGTYPE eCompressionFormat;
    OMX_COLOR_FORMATTYPE eColorFormat;
    OMX_U32 xFramerate;
} OMX_VIDEO_PARAM_PORTFORMATTYPE;
	
      

The first two fields are common to all OpenMAX structures. The nPortIndex is the port we are looking at. The nIndex field is to distinguish between all of the different format types supported by this port. The eCompressionFormat and eColorFormat give information about the format, while the last field gives the frame rate of the video in frames per second.

The values for OMX_VIDEO_CODINGTYPE are given in Table 4-66 of the 1.1.2 Specification and on the RPi are given in the file /opt/vc/include/IL/OMX_Video.h as

	
typedef enum OMX_VIDEO_CODINGTYPE {
    OMX_VIDEO_CodingUnused,     /** Value when coding is N/A */
    OMX_VIDEO_CodingAutoDetect, /** Autodetection of coding type */
    OMX_VIDEO_CodingMPEG2,      /** AKA: H.262 */
    OMX_VIDEO_CodingH263,       /** H.263 */
    OMX_VIDEO_CodingMPEG4,      /** MPEG-4 */
    OMX_VIDEO_CodingWMV,        /** all versions of Windows Media Video */
    OMX_VIDEO_CodingRV,         /** all versions of Real Video */
    OMX_VIDEO_CodingAVC,        /** H.264/AVC */
    OMX_VIDEO_CodingMJPEG,      /** Motion JPEG */
    OMX_VIDEO_CodingKhronosExtensions = 0x6F000000, /** Reserved region for introducing Khronos Standard Extensions */ 
    OMX_VIDEO_CodingVendorStartUnused = 0x7F000000, /** Reserved region for introducing Vendor Extensions */

    OMX_VIDEO_CodingVP6,        /** On2 VP6 */
    OMX_VIDEO_CodingVP7,        /** On2 VP7 */
    OMX_VIDEO_CodingVP8,        /** On2 VP8 */
    OMX_VIDEO_CodingYUV,        /* raw YUV video */
    OMX_VIDEO_CodingSorenson,   /** Sorenson */
    OMX_VIDEO_CodingTheora,     /** Theora */
    OMX_VIDEO_CodingMVC,        /** H.264/MVC */       
    
    OMX_VIDEO_CodingMax = 0x7FFFFFFF
} OMX_VIDEO_CODINGTYPE;
	
      

The type OMX_COLOR_FORMATTYPE is more informative. it is defined in the file /opt/vc/include/IL/OMX_IVCommon.h as

	
typedef enum OMX_COLOR_FORMATTYPE {
    OMX_COLOR_FormatUnused,
    OMX_COLOR_FormatMonochrome,
    OMX_COLOR_Format8bitRGB332,
    OMX_COLOR_Format12bitRGB444,
    OMX_COLOR_Format16bitARGB4444,
    OMX_COLOR_Format16bitARGB1555,
    OMX_COLOR_Format16bitRGB565,
    OMX_COLOR_Format16bitBGR565,
    OMX_COLOR_Format18bitRGB666,
    OMX_COLOR_Format18bitARGB1665,
    OMX_COLOR_Format19bitARGB1666, 
    OMX_COLOR_Format24bitRGB888,
    OMX_COLOR_Format24bitBGR888,
    OMX_COLOR_Format24bitARGB1887,
    OMX_COLOR_Format25bitARGB1888,
    OMX_COLOR_Format32bitBGRA8888,
    OMX_COLOR_Format32bitARGB8888,
    OMX_COLOR_FormatYUV411Planar,
    OMX_COLOR_FormatYUV411PackedPlanar,
    OMX_COLOR_FormatYUV420Planar,
    OMX_COLOR_FormatYUV420PackedPlanar,
    OMX_COLOR_FormatYUV420SemiPlanar,
    OMX_COLOR_FormatYUV422Planar,
    OMX_COLOR_FormatYUV422PackedPlanar,
    OMX_COLOR_FormatYUV422SemiPlanar,
    OMX_COLOR_FormatYCbYCr,
    OMX_COLOR_FormatYCrYCb,
    OMX_COLOR_FormatCbYCrY,
    OMX_COLOR_FormatCrYCbY,
    OMX_COLOR_FormatYUV444Interleaved,
    OMX_COLOR_FormatRawBayer8bit,
    OMX_COLOR_FormatRawBayer10bit,
    OMX_COLOR_FormatRawBayer8bitcompressed,
    OMX_COLOR_FormatL2, 
    OMX_COLOR_FormatL4, 
    OMX_COLOR_FormatL8, 
    OMX_COLOR_FormatL16, 
    OMX_COLOR_FormatL24, 
    OMX_COLOR_FormatL32,
    OMX_COLOR_FormatYUV420PackedSemiPlanar,
    OMX_COLOR_FormatYUV422PackedSemiPlanar,
    OMX_COLOR_Format18BitBGR666,
    OMX_COLOR_Format24BitARGB6666,
    OMX_COLOR_Format24BitABGR6666,
    OMX_COLOR_FormatKhronosExtensions = 0x6F000000, /** Reserved region for introducing Khronos Standard Extensions */ 
    OMX_COLOR_FormatVendorStartUnused = 0x7F000000, /** Reserved region for introducing Vendor Extensions */
    OMX_COLOR_Format32bitABGR8888,
    OMX_COLOR_Format8bitPalette,
    OMX_COLOR_FormatYUVUV128,
    OMX_COLOR_FormatRawBayer12bit,
    OMX_COLOR_FormatBRCMEGL,
    OMX_COLOR_FormatBRCMOpaque,
    OMX_COLOR_FormatYVU420PackedPlanar,
    OMX_COLOR_FormatYVU420PackedSemiPlanar,
    OMX_COLOR_FormatMax = 0x7FFFFFFF
} OMX_COLOR_FORMATTYPE;
	
      

A rather large number of formats!

Running the program info from the Components chapter shows for the video_decode component

	
Video ports:
  Ports start on 130
  There are 2 open ports
  Port 130 has 20 buffers (minimum 1) of size 81920
  Direction is input
    Supported video formats are:
    Video format encoding 0x14
    Video compression format 0x4
    Video format encoding 0x14
    Video compression format 0x7
    Video format encoding 0x14
    Video compression format 0x7F000007
    Video format encoding 0x14
    Video compression format 0x8
    Video format encoding 0x14
    Video compression format 0x5
    Video format encoding 0x14
    Video compression format 0x3
    Video format encoding 0x14
    Video compression format 0x2
    Video format encoding 0x14
    Video compression format 0x7F000001
    Video format encoding 0x14
    Video compression format 0x7F000002
    Video format encoding 0x14
    Video compression format 0x7F000003
    Video format encoding 0x14
    Video compression format 0x6
    Video format encoding 0x14
    Video compression format 0x7F000004
    Video format encoding 0x14
    Video compression format 0x7F000005
    Video format encoding 0x14
    Video compression format 0x7F000006
    Video format encoding 0x14
    Video compression format 0x0
    No more formats supported
  Port 131 has 1 buffers (minimum 1) of size 115200
  Direction is output
    Supported video formats are:
    Video format encoding 0x14
    Video compression format 0x0
    No more formats supported
	
      

The compression formats are

At present I have not confirmed any of these except for H.264.

Decoding an H.264 file

The program to decode a video into uncompressed format is essentially the same as the one for decoding an image. There is an essential difference: reading one block is not enough for the component to get information about the video file format, and multiple block reads generally have to occur.

In this example, we just slap a loop around reading the initial blocks until a PortSettingsChanged event occurs. The call to ilclient_wait_for_event will timeout for the first set of calls, so we drop the timeout period from 10 seconds down to 2 seconds. This isn't any good for interactive viewing, of course, but here we are just decoding.

The significant change from the image decoding program is this initial loop:

	
    int port_settings_changed = 0;
    while (!port_settings_changed) {
	buff_header = 
	    ilclient_get_input_buffer(component,
				      130,
				      1 /* block */);
	if (buff_header != NULL) {
	    read_into_buffer_and_empty(fp,
				       component,
				       buff_header,
				       &toread);

	    // If all the file has been read in, then
	    // we have to re-read this first block.
	    // Broadcom bug?
	    if (toread <= 0) {
		printf("Rewinding\n");
		// wind back to start and repeat
		fp = freopen(VIDEO, "r", fp);
		toread = get_file_size(VIDEO);
	    }
	}

	// try if this block sets params for output port
	err = ilclient_wait_for_event(component, 
				      OMX_EventPortSettingsChanged, 
				      131, 0, 0, 1,
				      ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED, 
				      2000);
	if (err < 0) {
	    printf("Wait for port settings changed timed out\n");
	} else {
	    port_settings_changed = 1;
	}
    }
	
      

The full program is il_decode_video.c

#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>

#include <OMX_Core.h>
#include <OMX_Component.h>

#include <bcm_host.h>
#include <ilclient.h>

#define IMG  "/opt/vc/src/hello_pi/hello_video/test.h264"
//#define IMG "hype.jpg"

void printState(OMX_HANDLETYPE handle) {
    OMX_STATETYPE state;
    OMX_ERRORTYPE err;

    err = OMX_GetState(handle, &state);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error on getting state\n");
        exit(1);
    }
    switch (state) {
    case OMX_StateLoaded:           printf("StateLoaded\n"); break;
    case OMX_StateIdle:             printf("StateIdle\n"); break;
    case OMX_StateExecuting:        printf("StateExecuting\n"); break;
    case OMX_StatePause:            printf("StatePause\n"); break;
    case OMX_StateWaitForResources: printf("StateWait\n"); break;
    case OMX_StateInvalid:          printf("StateInvalid\n"); break;
    default:                        printf("State unknown\n"); break;
    }
}

char *err2str(int err) {
    switch (err) {
    case OMX_ErrorInsufficientResources: return "OMX_ErrorInsufficientResources";
    case OMX_ErrorUndefined: return "OMX_ErrorUndefined";
    case OMX_ErrorInvalidComponentName: return "OMX_ErrorInvalidComponentName";
    case OMX_ErrorComponentNotFound: return "OMX_ErrorComponentNotFound";
    case OMX_ErrorInvalidComponent: return "OMX_ErrorInvalidComponent";
    case OMX_ErrorBadParameter: return "OMX_ErrorBadParameter";
    case OMX_ErrorNotImplemented: return "OMX_ErrorNotImplemented";
    case OMX_ErrorUnderflow: return "OMX_ErrorUnderflow";
    case OMX_ErrorOverflow: return "OMX_ErrorOverflow";
    case OMX_ErrorHardware: return "OMX_ErrorHardware";
    case OMX_ErrorInvalidState: return "OMX_ErrorInvalidState";
    case OMX_ErrorStreamCorrupt: return "OMX_ErrorStreamCorrupt";
    case OMX_ErrorPortsNotCompatible: return "OMX_ErrorPortsNotCompatible";
    case OMX_ErrorResourcesLost: return "OMX_ErrorResourcesLost";
    case OMX_ErrorNoMore: return "OMX_ErrorNoMore";
    case OMX_ErrorVersionMismatch: return "OMX_ErrorVersionMismatch";
    case OMX_ErrorNotReady: return "OMX_ErrorNotReady";
    case OMX_ErrorTimeout: return "OMX_ErrorTimeout";
    case OMX_ErrorSameState: return "OMX_ErrorSameState";
    case OMX_ErrorResourcesPreempted: return "OMX_ErrorResourcesPreempted";
    case OMX_ErrorPortUnresponsiveDuringAllocation: return "OMX_ErrorPortUnresponsiveDuringAllocation";
    case OMX_ErrorPortUnresponsiveDuringDeallocation: return "OMX_ErrorPortUnresponsiveDuringDeallocation";
    case OMX_ErrorPortUnresponsiveDuringStop: return "OMX_ErrorPortUnresponsiveDuringStop";
    case OMX_ErrorIncorrectStateTransition: return "OMX_ErrorIncorrectStateTransition";
    case OMX_ErrorIncorrectStateOperation: return "OMX_ErrorIncorrectStateOperation";
    case OMX_ErrorUnsupportedSetting: return "OMX_ErrorUnsupportedSetting";
    case OMX_ErrorUnsupportedIndex: return "OMX_ErrorUnsupportedIndex";
    case OMX_ErrorBadPortIndex: return "OMX_ErrorBadPortIndex";
    case OMX_ErrorPortUnpopulated: return "OMX_ErrorPortUnpopulated";
    case OMX_ErrorComponentSuspended: return "OMX_ErrorComponentSuspended";
    case OMX_ErrorDynamicResourcesUnavailable: return "OMX_ErrorDynamicResourcesUnavailable";
    case OMX_ErrorMbErrorsInFrame: return "OMX_ErrorMbErrorsInFrame";
    case OMX_ErrorFormatNotDetected: return "OMX_ErrorFormatNotDetected";
    case OMX_ErrorContentPipeOpenFailed: return "OMX_ErrorContentPipeOpenFailed";
    case OMX_ErrorContentPipeCreationFailed: return "OMX_ErrorContentPipeCreationFailed";
    case OMX_ErrorSeperateTablesUsed: return "OMX_ErrorSeperateTablesUsed";
    case OMX_ErrorTunnelingUnsupported: return "OMX_ErrorTunnelingUnsupported";
    default: return "unknown error";
    }
}

void eos_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    fprintf(stderr, "Got eos event\n");
}

void error_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    fprintf(stderr, "OMX error %s\n", err2str(data));
}

int get_file_size(char *fname) {
    struct stat st;

    if (stat(fname, &st) == -1) {
	    perror("Stat'ing img file");
	    return -1;
	}
    return(st.st_size);
}

static void set_video_decoder_input_format(COMPONENT_T *component) {
   // set input video format
    printf("Setting video decoder format\n");
    OMX_VIDEO_PARAM_PORTFORMATTYPE videoPortFormat;
    //setHeader(&videoPortFormat,  sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));
    memset(&videoPortFormat, 0, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));
    videoPortFormat.nSize = sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE);
    videoPortFormat.nVersion.nVersion = OMX_VERSION;

    videoPortFormat.nPortIndex = 130;
    videoPortFormat.eCompressionFormat = OMX_VIDEO_CodingAVC;
    OMX_SetParameter(ilclient_get_handle(component),
                     OMX_IndexParamVideoPortFormat, &videoPortFormat);

}

OMX_ERRORTYPE read_into_buffer_and_empty(FILE *fp,
					 COMPONENT_T *component,
					 OMX_BUFFERHEADERTYPE *buff_header,
					 int *toread) {
    OMX_ERRORTYPE r;

    int buff_size = buff_header->nAllocLen;
    int nread = fread(buff_header->pBuffer, 1, buff_size, fp);



    buff_header->nFilledLen = nread;
    *toread -= nread;
    printf("Read %d, %d still left\n", nread, *toread);

    if (*toread <= 0) {
	printf("Setting EOS on input\n");
	buff_header->nFlags |= OMX_BUFFERFLAG_EOS;
    }
    r = OMX_EmptyThisBuffer(ilclient_get_handle(component),
			buff_header);
    if (r != OMX_ErrorNone) {
	fprintf(stderr, "Empty buffer error %s\n",
		err2str(r));
    }
    return r;
}

OMX_ERRORTYPE save_info_from_filled_buffer(COMPONENT_T *component,
					   OMX_BUFFERHEADERTYPE * buff_header) {
    OMX_ERRORTYPE r;

    printf("Got a filled buffer with %d, allocated %d\n", 
	   buff_header->nFilledLen,
	   buff_header->nAllocLen);
    if (buff_header->nFlags & OMX_BUFFERFLAG_EOS) {
	printf("Got EOS on output\n");
	exit(0);
    }

    // do something here, like save the data - do nothing this time

    // and then refill it
    r = OMX_FillThisBuffer(ilclient_get_handle(component), 
			   buff_header); 
    if (r != OMX_ErrorNone) {
	fprintf(stderr, "Fill buffer error %s\n",
		err2str(r));
    }
    return r;
}

void get_output_port_settings(COMPONENT_T *component) {
    OMX_PARAM_PORTDEFINITIONTYPE portdef;

    printf("Port settings changed\n");
    // need to setup the input for the resizer with the output of the
    // decoder
    portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
    portdef.nVersion.nVersion = OMX_VERSION;
    portdef.nPortIndex = 131;
    OMX_GetParameter(ilclient_get_handle(component),
                     OMX_IndexParamPortDefinition, &portdef);

    unsigned int uWidth =
        (unsigned int) portdef.format.video.nFrameWidth;
    unsigned int uHeight =
        (unsigned int) portdef.format.video.nFrameHeight;
    unsigned int uStride =
        (unsigned int) portdef.format.video.nStride;
    unsigned int uSliceHeight =
        (unsigned int) portdef.format.video.nSliceHeight;
    printf("Frame width %d, frame height %d, stride %d, slice height %d\n",
	   uWidth,
	   uHeight,
	   uStride,
	   uSliceHeight);
    printf("Getting format Compression 0x%x Color Format: 0x%x\n",
	   (unsigned int) portdef.format.video.eCompressionFormat,
	   (unsigned int) portdef.format.video.eColorFormat);
}

int main(int argc, char** argv) {

    int i;
    char *componentName;
    int err;
    ILCLIENT_T  *handle;
    COMPONENT_T *component;
    FILE *fp = fopen(IMG, "r");
    int toread = get_file_size(IMG);
    OMX_BUFFERHEADERTYPE *buff_header;

    componentName = "video_decode";


    bcm_host_init();

    handle = ilclient_init();
    if (handle == NULL) {
	fprintf(stderr, "IL client init failed\n");
	exit(1);
    }

   if (OMX_Init() != OMX_ErrorNone) {
        ilclient_destroy(handle);
        fprintf(stderr, "OMX init failed\n");
	exit(1);
    }

   ilclient_set_error_callback(handle,
			       error_callback,
			       NULL);
   ilclient_set_eos_callback(handle,
			     eos_callback,
			     NULL);


    err = ilclient_create_component(handle,
                                &component,
                                componentName,
                                ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    |
				    ILCLIENT_ENABLE_OUTPUT_BUFFERS
                                );
    if (err == -1) {
	fprintf(stderr, "Component create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(component));

    err = ilclient_change_component_state(component,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(component));

    // must be before we enable buffers
    set_video_decoder_input_format(component);

    // input port
    ilclient_enable_port_buffers(component, 130, 
				 NULL, NULL, NULL);
    ilclient_enable_port(component, 130);




    err = ilclient_change_component_state(component,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Executing\n");
	exit(1);
    }
    printState(ilclient_get_handle(component));


    // Read the first block so that the component can get
    // the dimensions of the video and call port settings
    // changed on the output port to configure it
    int port_settings_changed = 0;
    while (!port_settings_changed) {
    buff_header = 
	ilclient_get_input_buffer(component,
				  130,
				  1 /* block */);
    if (buff_header != NULL) {
	read_into_buffer_and_empty(fp,
				   component,
				   buff_header,
				   &toread);

	// If all the file has been read in, then
	// we have to re-read this first block.
	// Broadcom bug?
	if (toread <= 0) {
	    printf("Rewinding\n");
	    // wind back to start and repeat
	    fp = freopen(IMG, "r", fp);
	    toread = get_file_size(IMG);
	}
    }

    // wait for first input block to set params for output port
    err = ilclient_wait_for_event(component, 
				  OMX_EventPortSettingsChanged, 
				  131, 0, 0, 1,
				  ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED, 
				  2000);
    if (err < 0) {
	printf("Wait for port settings changed timed out\n");
    } else {
	port_settings_changed = 1;
    }
    err = ilclient_remove_event(component, 
				  OMX_EventPortSettingsChanged, 
				  131, 0, 0, 1);
    if (err < 0) {
	printf("Wait for remove port settings changed timed out\n");
    } else {
	port_settings_changed = 1;
    }
    }

    get_output_port_settings(component);

    // now enable output port since port params have been set
    ilclient_enable_port_buffers(component, 131, 
				 NULL, NULL, NULL);
    ilclient_enable_port(component, 131);

    // now work through the file
    while (toread > 0) {
	OMX_ERRORTYPE r;

	// do we have an input buffer we can fill and empty?
	buff_header = 
	    ilclient_get_input_buffer(component,
				      130,
				      1 /* block */);
	if (buff_header != NULL) {
	    read_into_buffer_and_empty(fp,
				   component,
				   buff_header,
				   &toread);
	}

	// do we have an output buffer that has been filled?
	buff_header = 
	    ilclient_get_output_buffer(component,
				      131,
				      0 /* no block */);
	if (buff_header != NULL) {
	    save_info_from_filled_buffer(component,
					 buff_header);
	} else {
	    printf("No filled buffer\n");
	}
    }

    while (1) {
	printf("Getting last output buffers\n");
	buff_header = 
	    ilclient_get_output_buffer(component,
				       131,
				       1 /* block */);
	if (buff_header != NULL) {
	    save_info_from_filled_buffer(component,
					 buff_header);
	}
    }
    exit(0);
}

      

Rendering an H.264 video

In a similar manner to decoding an image, rendering a video closely follows the code for rendering an image. However, there is now a major difference in how we wait for the initial PortSettingsChanged event. We have seen two possibilities so far:

Neither of these two are acceptable for immediate rendering of videos: it may take more than one read, and waiting on timeouts is too slow. Instead, we need to adopt an alternative approach based on the ilclient_remove_event function, which returns true if it can remove an event from the list of those events already seen.

The revised wait for PortSettingsChanged event is

	
while there is more data
    read a new buffer
    empty the buffer

    if a PortSettingsChanged event has occurred
        break

    if there is no more data
        block waiting for a PortSettingsChanged event
        if failed on timeout
            error
        else
            break
	
      

(This algorithm should be generally applicable. But it breaks on the tunnelled image rendering, reason unknown so far.)

The video render program using this is il_render_video.c

#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>

#include <OMX_Core.h>
#include <OMX_Component.h>

#include <bcm_host.h>
#include <ilclient.h>

//#define IMG  "cimg0135.jpg"
//#define IMG "/opt/vc/src/hello_pi/hello_video/test.h264"
//#define IMG "/home/pi/timidity/short.mpg"
#define IMG "small.ogv"

void printState(OMX_HANDLETYPE handle) {
    OMX_STATETYPE state;
    OMX_ERRORTYPE err;

    err = OMX_GetState(handle, &state);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error on getting state\n");
        exit(1);
    }
    switch (state) {
    case OMX_StateLoaded:           printf("StateLoaded\n"); break;
    case OMX_StateIdle:             printf("StateIdle\n"); break;
    case OMX_StateExecuting:        printf("StateExecuting\n"); break;
    case OMX_StatePause:            printf("StatePause\n"); break;
    case OMX_StateWaitForResources: printf("StateWait\n"); break;
    case OMX_StateInvalid:          printf("StateInvalid\n"); break;
    default:                        printf("State unknown\n"); break;
    }
}

char *err2str(int err) {
    switch (err) {
    case OMX_ErrorInsufficientResources: return "OMX_ErrorInsufficientResources";
    case OMX_ErrorUndefined: return "OMX_ErrorUndefined";
    case OMX_ErrorInvalidComponentName: return "OMX_ErrorInvalidComponentName";
    case OMX_ErrorComponentNotFound: return "OMX_ErrorComponentNotFound";
    case OMX_ErrorInvalidComponent: return "OMX_ErrorInvalidComponent";
    case OMX_ErrorBadParameter: return "OMX_ErrorBadParameter";
    case OMX_ErrorNotImplemented: return "OMX_ErrorNotImplemented";
    case OMX_ErrorUnderflow: return "OMX_ErrorUnderflow";
    case OMX_ErrorOverflow: return "OMX_ErrorOverflow";
    case OMX_ErrorHardware: return "OMX_ErrorHardware";
    case OMX_ErrorInvalidState: return "OMX_ErrorInvalidState";
    case OMX_ErrorStreamCorrupt: return "OMX_ErrorStreamCorrupt";
    case OMX_ErrorPortsNotCompatible: return "OMX_ErrorPortsNotCompatible";
    case OMX_ErrorResourcesLost: return "OMX_ErrorResourcesLost";
    case OMX_ErrorNoMore: return "OMX_ErrorNoMore";
    case OMX_ErrorVersionMismatch: return "OMX_ErrorVersionMismatch";
    case OMX_ErrorNotReady: return "OMX_ErrorNotReady";
    case OMX_ErrorTimeout: return "OMX_ErrorTimeout";
    case OMX_ErrorSameState: return "OMX_ErrorSameState";
    case OMX_ErrorResourcesPreempted: return "OMX_ErrorResourcesPreempted";
    case OMX_ErrorPortUnresponsiveDuringAllocation: return "OMX_ErrorPortUnresponsiveDuringAllocation";
    case OMX_ErrorPortUnresponsiveDuringDeallocation: return "OMX_ErrorPortUnresponsiveDuringDeallocation";
    case OMX_ErrorPortUnresponsiveDuringStop: return "OMX_ErrorPortUnresponsiveDuringStop";
    case OMX_ErrorIncorrectStateTransition: return "OMX_ErrorIncorrectStateTransition";
    case OMX_ErrorIncorrectStateOperation: return "OMX_ErrorIncorrectStateOperation";
    case OMX_ErrorUnsupportedSetting: return "OMX_ErrorUnsupportedSetting";
    case OMX_ErrorUnsupportedIndex: return "OMX_ErrorUnsupportedIndex";
    case OMX_ErrorBadPortIndex: return "OMX_ErrorBadPortIndex";
    case OMX_ErrorPortUnpopulated: return "OMX_ErrorPortUnpopulated";
    case OMX_ErrorComponentSuspended: return "OMX_ErrorComponentSuspended";
    case OMX_ErrorDynamicResourcesUnavailable: return "OMX_ErrorDynamicResourcesUnavailable";
    case OMX_ErrorMbErrorsInFrame: return "OMX_ErrorMbErrorsInFrame";
    case OMX_ErrorFormatNotDetected: return "OMX_ErrorFormatNotDetected";
    case OMX_ErrorContentPipeOpenFailed: return "OMX_ErrorContentPipeOpenFailed";
    case OMX_ErrorContentPipeCreationFailed: return "OMX_ErrorContentPipeCreationFailed";
    case OMX_ErrorSeperateTablesUsed: return "OMX_ErrorSeperateTablesUsed";
    case OMX_ErrorTunnelingUnsupported: return "OMX_ErrorTunnelingUnsupported";
    default: return "unknown error";
    }
}

void eos_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    fprintf(stderr, "Got eos event\n");
}

void error_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    fprintf(stderr, "OMX error %s\n", err2str(data));
}

int get_file_size(char *fname) {
    struct stat st;

    if (stat(fname, &st) == -1) {
	perror("Stat'ing img file");
	return -1;
    }
    return(st.st_size);
}

unsigned int uWidth;
unsigned int uHeight;

OMX_ERRORTYPE read_into_buffer_and_empty(FILE *fp,
					 COMPONENT_T *component,
					 OMX_BUFFERHEADERTYPE *buff_header,
					 int *toread) {
    OMX_ERRORTYPE r;

    int buff_size = buff_header->nAllocLen;
    int nread = fread(buff_header->pBuffer, 1, buff_size, fp);


    buff_header->nFilledLen = nread;
    *toread -= nread;
    printf("Read %d, %d still left\n", nread, *toread);

    if (*toread <= 0) {
	printf("Setting EOS on input\n");
	buff_header->nFlags |= OMX_BUFFERFLAG_EOS;
    }
    r = OMX_EmptyThisBuffer(ilclient_get_handle(component),
			    buff_header);
    if (r != OMX_ErrorNone) {
	fprintf(stderr, "Empty buffer error %s\n",
		err2str(r));
    }
    return r;
}

static void set_video_decoder_input_format(COMPONENT_T *component) {
    int err;

    // set input video format
    printf("Setting video decoder format\n");
    OMX_VIDEO_PARAM_PORTFORMATTYPE videoPortFormat;
    //setHeader(&videoPortFormat,  sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));
    memset(&videoPortFormat, 0, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));
    videoPortFormat.nSize = sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE);
    videoPortFormat.nVersion.nVersion = OMX_VERSION;

    videoPortFormat.nPortIndex = 130;
    videoPortFormat.eCompressionFormat = OMX_VIDEO_CodingAVC;

    err = OMX_SetParameter(ilclient_get_handle(component),
			   OMX_IndexParamVideoPortFormat, &videoPortFormat);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error setting video decoder format %s\n", err2str(err));
        return err;
    } else {
        printf("Video decoder format set up ok\n");
    }


}

void setup_decodeComponent(ILCLIENT_T  *handle, 
			   char *decodeComponentName, 
			   COMPONENT_T **decodeComponent) {
    int err;

    err = ilclient_create_component(handle,
				    decodeComponent,
				    decodeComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    |
				    ILCLIENT_ENABLE_OUTPUT_BUFFERS
				    );
    if (err == -1) {
	fprintf(stderr, "DecodeComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*decodeComponent));

    err = ilclient_change_component_state(*decodeComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*decodeComponent));

    // must be before we enable buffers
    set_video_decoder_input_format(*decodeComponent);
}

void setup_renderComponent(ILCLIENT_T  *handle, 
			   char *renderComponentName, 
			   COMPONENT_T **renderComponent) {
    int err;

    err = ilclient_create_component(handle,
				    renderComponent,
				    renderComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    );
    if (err == -1) {
	fprintf(stderr, "RenderComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*renderComponent));

    err = ilclient_change_component_state(*renderComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*renderComponent));
}

int main(int argc, char** argv) {

    int i;
    char *decodeComponentName;
    char *renderComponentName;
    int err;
    ILCLIENT_T  *handle;
    COMPONENT_T *decodeComponent;
    COMPONENT_T *renderComponent;
    FILE *fp = fopen(IMG, "r");
    int toread = get_file_size(IMG);
    OMX_BUFFERHEADERTYPE *buff_header;

    decodeComponentName = "video_decode";
    renderComponentName = "video_render";

    bcm_host_init();

    handle = ilclient_init();
    if (handle == NULL) {
	fprintf(stderr, "IL client init failed\n");
	exit(1);
    }

    if (OMX_Init() != OMX_ErrorNone) {
        ilclient_destroy(handle);
        fprintf(stderr, "OMX init failed\n");
	exit(1);
    }

    ilclient_set_error_callback(handle,
				error_callback,
				NULL);
    ilclient_set_eos_callback(handle,
			      eos_callback,
			      NULL);


    setup_decodeComponent(handle, decodeComponentName, &decodeComponent);
    setup_renderComponent(handle, renderComponentName, &renderComponent);
    // both components now in Idle state, no buffers, ports disabled

    // input port
    ilclient_enable_port_buffers(decodeComponent, 130, 
				 NULL, NULL, NULL);
    ilclient_enable_port(decodeComponent, 130);


    err = ilclient_change_component_state(decodeComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Executing\n");
	exit(1);
    }
    printState(ilclient_get_handle(decodeComponent));


    // Read the first block so that the decodeComponent can get
    // the dimensions of the video and call port settings
    // changed on the output port to configure it
    while (toread > 0) {
	buff_header = 
	    ilclient_get_input_buffer(decodeComponent,
				      130,
				      1 /* block */);
	if (buff_header != NULL) {
	    read_into_buffer_and_empty(fp,
				       decodeComponent,
				       buff_header,
				       &toread);

	    // If all the file has been read in, then
	    // we have to re-read this first block.
	    // Broadcom bug?
	    if (toread <= 0) {
		printf("Rewinding\n");
		// wind back to start and repeat
		fp = freopen(IMG, "r", fp);
		toread = get_file_size(IMG);
	    }
	}

	if (toread > 0 && ilclient_remove_event(decodeComponent, 
						OMX_EventPortSettingsChanged, 
						131, 0, 0, 1) == 0) {
	    printf("Removed port settings event\n");
	    break;
	} else {
	    printf("No portr settting seen yet\n");
	}
	// wait for first input block to set params for output port
	if (toread == 0) {
	    // wait for first input block to set params for output port
	    err = ilclient_wait_for_event(decodeComponent, 
					 OMX_EventPortSettingsChanged, 
					 131, 0, 0, 1,
					 ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED, 
					 2000);
	    if (err < 0) {
		fprintf(stderr, "No port settings change\n");
		//exit(1);
	    } else {
		printf("Port settings changed\n");
		break;
	    }
	}
    }

    // set the decode component to idle and disable its ports
    err = ilclient_change_component_state(decodeComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    ilclient_disable_port(decodeComponent, 131);
    ilclient_disable_port_buffers(decodeComponent, 131, 
				  NULL, NULL, NULL);
 
    // set up the tunnel between decode and render ports
    err = OMX_SetupTunnel(ilclient_get_handle(decodeComponent),
			  131,
			  ilclient_get_handle(renderComponent),
			  90);
    if (err != OMX_ErrorNone) {
	fprintf(stderr, "Error setting up tunnel %X\n", err);
	exit(1);
    } else {
	printf("Tunnel set up ok\n");
    }

    // Okay to go back to processing data
    // enable the decode output ports
   
    OMX_SendCommand(ilclient_get_handle(decodeComponent), 
		    OMX_CommandPortEnable, 131, NULL);
   
    ilclient_enable_port(decodeComponent, 131);

    // enable the render output ports
   
    OMX_SendCommand(ilclient_get_handle(renderComponent), 
		    OMX_CommandPortEnable, 90, NULL);
   
    ilclient_enable_port(renderComponent, 90);

    // set both components to executing state
    err = ilclient_change_component_state(decodeComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    err = ilclient_change_component_state(renderComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }

    // now work through the file
    while (toread > 0) {
	OMX_ERRORTYPE r;

	// do we have a decode input buffer we can fill and empty?
	buff_header = 
	    ilclient_get_input_buffer(decodeComponent,
				      130,
				      1 /* block */);
	if (buff_header != NULL) {
	    read_into_buffer_and_empty(fp,
				       decodeComponent,
				       buff_header,
				       &toread);
	}
    }

    ilclient_wait_for_event(renderComponent, 
			    OMX_EventBufferFlag, 
			    90, 0, OMX_BUFFERFLAG_EOS, 0,
			    ILCLIENT_BUFFER_FLAG_EOS, 10000);
    printf("EOS on render\n");

    sleep(100);

    exit(0);
}

      

Conclusion

This chapter has looked at decoding and rendering H264 video streams.


      

Copyright © Jan Newmarch, jan@newmarch.name
Creative Commons License
" Programming AudioVideo on the Raspberry Pi GPU " by Jan Newmarch is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License .
Based on a work at https://jan.newmarch.name/RPi/ .

If you like this book, please contribute using PayPal

Or Flattr me:
Flattr this book