Upto: Table of Contents of full book "Programming the Raspberry Pi GPU"

Playing multimedia files on the Raspberry Pi

This chapter considers playing multiplexed files on the Raspberry Pi such as MP4 files, which consist of both audio and video streams. OpenMAX does not support de-multiplexing such files, so a software demuxer such as FFMPEG or Libav must be used. The libraries available on the RPi do not support decoding of encoded audio streams, so a software decoder needs to be used. Finally, synchronisation of video and audio streams needs to be managed.

This chapter would not have been possible without seeing what was done in omxplayer. Many thanks to its authors!

Resources

Files

Files used are here

Desperate debugging

Sometimes just nothing works. Debugging using gdb or similar gets you nowhere. Broadcom have a command vcdbg which can report on what the GPU is doing. It is run by


      vcgencmd cache_flush
      vcgencmd set_logging level=0x40
      sudo vcdbg log msg

In code, you can call


vcos_log_set_level(VCOS_LOG_CATEGORY, VCOS_LOG_TRACE);

More information is at RPI vcgencmd usage and Video Core tools

Multimedia files

Multimedia files containing both audio and video are container files. Typically they will have header information and then streams of audio and video data, usually interleaved in some way, where each stream has its own appropriate format.

There are many such formats (see for example Wikipedia's Digital container format article). We will just look at the MP4 format. An MP4 file will typically contain both audio and video streams but may also contain subtitles and still images.

Although an MP4 file can contain audio and video streams of many different formats, the most common are AAC (Advanced Audio Coding) for audio and H.264 for video.

The program mediainfo can show a huge amount of information about a file. For example, for the sample file Big Buck Bunny it shows

	
General
Complete name                            : big_buck_bunny.mp4
Format                                   : MPEG-4
Format profile                           : Base Media / Version 2
Codec ID                                 : mp42
File size                                : 5.26 MiB
Duration                                 : 1mn 0s
Overall bit rate                         : 734 Kbps
Encoded date                             : UTC 2010-02-09 01:55:39
Tagged date                              : UTC 2010-02-09 01:55:40

Video
ID                                       : 2
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Baseline@L3.0
Format settings, CABAC                   : No
Format settings, ReFrames                : 2 frames
Format settings, GOP                     : M=1, N=64
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 1mn 0s
Duration_LastFrame                       : 95ms
Bit rate                                 : 613 Kbps
Width                                    : 640 pixels
Height                                   : 360 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 24.000 fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.111
Stream size                              : 4.39 MiB (83%)
Language                                 : English
Encoded date                             : UTC 2010-02-09 01:55:39
Tagged date                              : UTC 2010-02-09 01:55:40
Color primaries                          : BT.601 NTSC
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.601

Audio
ID                                       : 1
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format profile                           : LC
Codec ID                                 : 40
Duration                                 : 1mn 0s
Source duration                          : 1mn 0s
Bit rate mode                            : Constant
Bit rate                                 : 64.0 Kbps
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 22.05 KHz
Compression mode                         : Lossy
Stream size                              : 478 KiB (9%)
Source stream size                       : 478 KiB (9%)
Language                                 : English
Encoded date                             : UTC 2010-02-09 01:55:39
Tagged date                              : UTC 2010-02-09 01:55:40
	
      

The main points for now are the video format of Advanced Video Codec (H.264) and audio format AAC.

Demuxing multimedia files

In order to play a multimedia file, the file must be split into two streams, or de-multiplexed. Typically the streams will be interleaved, as a sequence of packets of each type. Due to the large variety of possible multimedia file formats, none of them are supported by the Broadcom GPU, although there have been repeated requests for at least some to be supported (e.g. Proposal for OMX.broadcom.read_media ).

To demux a file, a software package such as ffmpeg or libav must be used. FFmpeg and Libav forked some years ago, through a vitriolic and ongoing brawl. The Debian package for the RPi uses Libav while the superb multimedia omxplayer uses FFmpeg. In addition, there is continual evolution of code, formats, codecs, etc, so it can be traumatic figuring out what code to build. Added to the difficulties of OpenMAX itself, this is not an easy area to navigate.

Decoding the audio stream using Libav

In this section we just look at managing an audio stream using Libav/FFmpeg. This is a continually changing area, and this is the working version as of March, 2015 using Libav codec version 54.35. Adding in OpenMAX will be done in the following section.

Libav forked from FFmpeg acrimoniously (use Google to dig up the details). Debian and most Debian-derived systems use Libav. Omxplayer doesn't: it uses FFmpeg. The two forks align much of the time, but not always. Headers are different, sometimes the APIs differ and of course there may be semantic differences. I'm going to use Libav since that is what is installed in the Raspbian package. I take no sides, other than to comment that we know the packages must keep evolving with developments in AV codecs, containers, etc, but to add the complexity of two parallel packages is an added burden on the developer.

Whatever you do with either package involves setting up a format context. This is initialised by a call to avformat_open which takes a filename as parameter. A (container) file will contain one or more streams, and information about all of these can be found by avformat_find_stream_info. This can be printed out by av_dump_format.

The audio stream can be found through the call av_find_best_stream which returns an index into the array of streams maintained by the context.

If we only want to demultiplex the streams, this is sufficient for setup. This is the case for the video AVC streams as the Broadcom GPU can decode and render these. For audio, the situation is not so good: the Broadcom GPU will only handle WAV or raw PCM data. So we have to decode the AAC stream in software.

The decoder and its context are found from additional calls to avcodec_find_decoder and avcodec_alloc_context3.

Some codecs pull out extra information from the audio stream that the codec context didn't see. However, it turns out that the context context may need those extra pieces of information. This is a bit yucky: the extra data has to be installed from the codec into its context (shouldn't this be done by avcodec_alloc_context?). After this has been done, the codec can be opened by avcodec_open2.

This initialisation code is

	
int setup_demuxer(const char *filename) {
    // Register all formats and codecs
    av_register_all();
    if(avformat_open_input(&pFormatCtx, filename, NULL, NULL)!=0) {
	fprintf(stderr, "Can't get format\n");
        return -1; // Couldn't open file
    }
    // Retrieve stream information
    if (avformat_find_stream_info(pFormatCtx, NULL) < 0) {
	return -1; // Couldn't find stream information
    }
    printf("Format:\n");
    av_dump_format(pFormatCtx, 0, filename, 0);

    int ret;
    ret = av_find_best_stream(pFormatCtx, AVMEDIA_TYPE_AUDIO, -1, -1, NULL, 0);
    if (ret >= 0) {
	audio_stream_idx = ret;

	audio_stream = pFormatCtx->streams[audio_stream_idx];
	audio_dec_ctx = audio_stream->codec;

	AVCodec *codec = avcodec_find_decoder(audio_stream->codec->codec_id);
	codec_context = avcodec_alloc_context3(codec);

	// copy across info from codec about extradata
	codec_context->extradata =  audio_stream->codec->extradata;
	codec_context->extradata_size =  audio_stream->codec->extradata_size;


	if (codec) {
	    printf("Codec name %s\n", codec->name);
	}

	if (!avcodec_open2(codec_context, codec, NULL) < 0) {
	    fprintf(stderr, "Could not find open the needed codec");
	    exit(1);
	}
    }
    return 0;
}
	
      

The main AV processing loop reads packets using av_read_frame. If they are audio packets then they need to be decoded into frames in a slightly convulted way by avcodec_decode_audio4. The returned value is not the size of the decoded frame, but is the size of the packet consumed by the decoding. Strictly, we should loop until the amount consumed equals the size of the read packet, but we ignore that here, assuming that the total packet is consumed.

At one time it appears that Libav/FFmpeg decoded AAC frames into raw PCM data, with each channel interleaved by each sample, AV_SAMPLE_FMT_S16. Now they decode into a format AV_SAMPLE_FMT_FLTP which is a fixed point planar format (multiple samples in each channel are together). These need to be converted, using the package AVResample, which uses a different API in Libav and FFmpeg. Using Libav, we create a buffer for stereo big enough to hold enough 16-bit samples by av_samples_alloc and then convert the samples by avresample_convert.

For this exercise, we then save the 16-bit stereo samples to a file. The code is

	
    FILE *out = fopen("tmp.s16", "w");

    // for converting from AV_SAMPLE_FMT_FLTP to AV_SAMPLE_FMT_S16
    // Set up SWR context once you've got codec information
    AVAudioResampleContext *swr = avresample_alloc_context();
    av_opt_set_int(swr, "in_channel_layout",  audio_dec_ctx->channel_layout, 0);
    av_opt_set_int(swr, "out_channel_layout", audio_dec_ctx->channel_layout,  0);
    av_opt_set_int(swr, "in_sample_rate",     audio_dec_ctx->sample_rate, 0);
    av_opt_set_int(swr, "out_sample_rate",    audio_dec_ctx->sample_rate, 0);
    av_opt_set_int(swr, "in_sample_fmt",  AV_SAMPLE_FMT_FLTP, 0);
    av_opt_set_int(swr, "out_sample_fmt", AV_SAMPLE_FMT_S16,  0);
    avresample_open(swr);

    int buffer_size;
    int num_ret;

    /* read frames from the file */
    AVFrame *frame = avcodec_alloc_frame(); // av_frame_alloc
    while (av_read_frame(pFormatCtx, &pkt) >= 0) {
	// printf("Read pkt %d\n", pkt.size);

	AVPacket orig_pkt = pkt;
	if (pkt.stream_index == audio_stream_idx) {
	    // printf("  read audio pkt %d\n", pkt.size);
	    // fwrite(pkt.data, 1, pkt.size, out);

	    AVPacket avpkt;
	    int got_frame;
	    av_init_packet(&avpkt);
	    avpkt.data = pkt.data;
	    avpkt.size = pkt.size;


	    uint8_t *buffer;
	    if (((err = avcodec_decode_audio4(codec_context,
					      frame,
					      &got_frame,
					      &avpkt)) < 0)  || !got_frame) {
		fprintf(stderr, "Error decoding %d\n", err);
		continue;
	    }

	    int out_linesize;
	    /*
	    av_samples_get_buffer_size(&out_linesize, 2, frame->nb_samples,
					   AV_SAMPLE_FMT_S16, 0);
	    */
	    av_samples_alloc(&buffer, &out_linesize, 2, frame->nb_samples,
			     AV_SAMPLE_FMT_S16, 0);

	    avresample_convert(swr, &buffer, frame->linesize[0], 
			       frame->nb_samples, frame->extended_data, 
			       frame->linesize[0], frame->nb_samples);
	    /*
	    printf("Pkt size %d, decoded to %d line size %d\n",
		   pkt.size, err, out_linesize);
	    printf("Samples: pkt size %d nb_samples %d\n",
		  pkt.size, frame->nb_samples);
	    printf("Buffer is (decoded size %d)  %d %d %d %d %d\n", 
		   required_decoded_size,
		   buffer[0], buffer[1], buffer[2], buffer[3], err);
	    */
	    fwrite(buffer, 1, frame->nb_samples*4, out);
	}
	av_free_packet(&orig_pkt);
    }
	
      

The complete program is ffmpeg_demux_decode_audio.c

The resultant raw PCM file can be played by e.g.

	
aplay -c 2 -r 22050 -f  S16_LE tmp.s16
	
      

or it can be converted to WAV format by e.g.

	
sox -c 2 -r 22050 tmp.s16 tmp.wav
	
      

and then played by any player such as mplayer. Note: the audio file contains raw PCM data only and this does not contain any information such as the sampling rate of 22050 Hz. You have to supply this yourself, using information from av_dump_format or mediainfo. Playing using aplay requires this explicitly, while conversion to WAV format by sox adds in WAV header information to the PCM data.

Rendering an audio stream

Now that we have successfully demuxed and decoded the audio stream into a stream of PCM data, sending it to the Broadcom audio render device is straightforward using the techniques of the OpenMAX Audio chapter.

Essentially, we set up the OMX audio renderer using information from the Libav decoder and then feed the decoded packets into the audio renderer. It's just a looong program by this stage!

The full program is il_ffmpeg_demux_render_audio.c

#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>

#include <OMX_Core.h>
#include <OMX_Component.h>

#include <bcm_host.h>
#include <vcos_logging.h>

#define VCOS_LOG_CATEGORY (&il_ffmpeg_log_category)
static VCOS_LOG_CAT_T il_ffmpeg_log_category;

#include <ilclient.h>

#include "libavcodec/avcodec.h"
#include <libavformat/avformat.h>
#include "libavutil/mathematics.h"
#include "libavutil/samplefmt.h"
#include "libavresample/avresample.h"

#define INBUF_SIZE 4096
#define AUDIO_INBUF_SIZE 20480
#define AUDIO_REFILL_THRESH 4096

char *IMG = "taichi.mp4";

static AVCodecContext *audio_dec_ctx = NULL;
static AVStream *audio_stream = NULL;
static AVPacket pkt;
AVFormatContext *pFormatCtx = NULL;
AVAudioResampleContext *swr;

int sample_rate;
int channels;

static int audio_stream_idx = -1;

AVCodec *codec;

void setPCMMode(OMX_HANDLETYPE handle, int startPortNumber) {
    OMX_AUDIO_PARAM_PCMMODETYPE sPCMMode;
    OMX_ERRORTYPE err;
 
    memset(&sPCMMode, 0, sizeof(OMX_AUDIO_PARAM_PCMMODETYPE));
    sPCMMode.nSize = sizeof(OMX_AUDIO_PARAM_PCMMODETYPE);
    sPCMMode.nVersion.nVersion = OMX_VERSION;

    sPCMMode.nPortIndex = startPortNumber;

    err = OMX_GetParameter(handle, OMX_IndexParamAudioPcm, &sPCMMode);
    printf("Sampling rate %d, channels %d\n",
	   sPCMMode.nSamplingRate, 
	   sPCMMode.nChannels);

    //sPCMMode.nSamplingRate = 44100; // for taichi
    //sPCMMode.nSamplingRate = 22050; // for big buck bunny
    sPCMMode.nSamplingRate = sample_rate; // for anything
    sPCMMode.nChannels = channels;

    err = OMX_SetParameter(handle, OMX_IndexParamAudioPcm, &sPCMMode);
    if(err != OMX_ErrorNone){
	fprintf(stderr, "PCM mode unsupported\n");
	return;
    } else {
	fprintf(stderr, "PCM mode supported\n");
	fprintf(stderr, "PCM sampling rate %d\n", sPCMMode.nSamplingRate);
	fprintf(stderr, "PCM nChannels %d\n", sPCMMode.nChannels);
    } 
}

static void set_audio_render_input_format(COMPONENT_T *component) {
    // set input audio format
    printf("Setting audio render format\n");
    OMX_AUDIO_PARAM_PORTFORMATTYPE audioPortFormat;
    //setHeader(&audioPortFormat,  sizeof(OMX_AUDIO_PARAM_PORTFORMATTYPE));
    memset(&audioPortFormat, 0, sizeof(OMX_AUDIO_PARAM_PORTFORMATTYPE));
    audioPortFormat.nSize = sizeof(OMX_AUDIO_PARAM_PORTFORMATTYPE);
    audioPortFormat.nVersion.nVersion = OMX_VERSION;

    audioPortFormat.nPortIndex = 100;

    OMX_GetParameter(ilclient_get_handle(component),
                     OMX_IndexParamAudioPortFormat, &audioPortFormat);

    audioPortFormat.eEncoding = OMX_AUDIO_CodingPCM;
    //audioPortFormat.eEncoding = OMX_AUDIO_CodingMP3;
    OMX_SetParameter(ilclient_get_handle(component),
                     OMX_IndexParamAudioPortFormat, &audioPortFormat);

    setPCMMode(ilclient_get_handle(component), 100);

}

void printState(OMX_HANDLETYPE handle) {
    OMX_STATETYPE state;
    OMX_ERRORTYPE err;

    err = OMX_GetState(handle, &state);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error on getting state\n");
        exit(1);
    }
    switch (state) {
    case OMX_StateLoaded:           printf("StateLoaded\n"); break;
    case OMX_StateIdle:             printf("StateIdle\n"); break;
    case OMX_StateExecuting:        printf("StateExecuting\n"); break;
    case OMX_StatePause:            printf("StatePause\n"); break;
    case OMX_StateWaitForResources: printf("StateWait\n"); break;
    case OMX_StateInvalid:          printf("StateInvalid\n"); break;
    default:                        printf("State unknown\n"); break;
    }
}

char *err2str(int err) {
    switch (err) {
    case OMX_ErrorInsufficientResources: return "OMX_ErrorInsufficientResources";
    case OMX_ErrorUndefined: return "OMX_ErrorUndefined";
    case OMX_ErrorInvalidComponentName: return "OMX_ErrorInvalidComponentName";
    case OMX_ErrorComponentNotFound: return "OMX_ErrorComponentNotFound";
    case OMX_ErrorInvalidComponent: return "OMX_ErrorInvalidComponent";
    case OMX_ErrorBadParameter: return "OMX_ErrorBadParameter";
    case OMX_ErrorNotImplemented: return "OMX_ErrorNotImplemented";
    case OMX_ErrorUnderflow: return "OMX_ErrorUnderflow";
    case OMX_ErrorOverflow: return "OMX_ErrorOverflow";
    case OMX_ErrorHardware: return "OMX_ErrorHardware";
    case OMX_ErrorInvalidState: return "OMX_ErrorInvalidState";
    case OMX_ErrorStreamCorrupt: return "OMX_ErrorStreamCorrupt";
    case OMX_ErrorPortsNotCompatible: return "OMX_ErrorPortsNotCompatible";
    case OMX_ErrorResourcesLost: return "OMX_ErrorResourcesLost";
    case OMX_ErrorNoMore: return "OMX_ErrorNoMore";
    case OMX_ErrorVersionMismatch: return "OMX_ErrorVersionMismatch";
    case OMX_ErrorNotReady: return "OMX_ErrorNotReady";
    case OMX_ErrorTimeout: return "OMX_ErrorTimeout";
    case OMX_ErrorSameState: return "OMX_ErrorSameState";
    case OMX_ErrorResourcesPreempted: return "OMX_ErrorResourcesPreempted";
    case OMX_ErrorPortUnresponsiveDuringAllocation: return "OMX_ErrorPortUnresponsiveDuringAllocation";
    case OMX_ErrorPortUnresponsiveDuringDeallocation: return "OMX_ErrorPortUnresponsiveDuringDeallocation";
    case OMX_ErrorPortUnresponsiveDuringStop: return "OMX_ErrorPortUnresponsiveDuringStop";
    case OMX_ErrorIncorrectStateTransition: return "OMX_ErrorIncorrectStateTransition";
    case OMX_ErrorIncorrectStateOperation: return "OMX_ErrorIncorrectStateOperation";
    case OMX_ErrorUnsupportedSetting: return "OMX_ErrorUnsupportedSetting";
    case OMX_ErrorUnsupportedIndex: return "OMX_ErrorUnsupportedIndex";
    case OMX_ErrorBadPortIndex: return "OMX_ErrorBadPortIndex";
    case OMX_ErrorPortUnpopulated: return "OMX_ErrorPortUnpopulated";
    case OMX_ErrorComponentSuspended: return "OMX_ErrorComponentSuspended";
    case OMX_ErrorDynamicResourcesUnavailable: return "OMX_ErrorDynamicResourcesUnavailable";
    case OMX_ErrorMbErrorsInFrame: return "OMX_ErrorMbErrorsInFrame";
    case OMX_ErrorFormatNotDetected: return "OMX_ErrorFormatNotDetected";
    case OMX_ErrorContentPipeOpenFailed: return "OMX_ErrorContentPipeOpenFailed";
    case OMX_ErrorContentPipeCreationFailed: return "OMX_ErrorContentPipeCreationFailed";
    case OMX_ErrorSeperateTablesUsed: return "OMX_ErrorSeperateTablesUsed";
    case OMX_ErrorTunnelingUnsupported: return "OMX_ErrorTunnelingUnsupported";
    default: return "unknown error";
    }
}

void eos_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    printf("Got eos event\n");
}

void error_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    printf("OMX error %s\n", err2str(data));
}

void port_settings_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    printf("Got port Settings event\n");
    // exit(0);
}

void empty_buffer_done_callback(void *userdata, COMPONENT_T *comp) {
    // printf("Got empty buffer done\n");
}


int get_file_size(char *fname) {
    struct stat st;

    if (stat(fname, &st) == -1) {
	perror("Stat'ing img file");
	return -1;
    }
    return(st.st_size);
}

OMX_ERRORTYPE read_audio_into_buffer_and_empty(AVFrame *decoded_frame,
					       COMPONENT_T *component,
					       // OMX_BUFFERHEADERTYPE *buff_header,
					       int total_len) {
    OMX_ERRORTYPE r;
    OMX_BUFFERHEADERTYPE *buff_header = NULL;

#if AUDIO_DECODE
    int port_index = 120;
#else
    int port_index = 100;
#endif

    int required_decoded_size = 0;
    
    int out_linesize;
    required_decoded_size = 
	av_samples_get_buffer_size(&out_linesize, 2, 
				   decoded_frame->nb_samples,
				   AV_SAMPLE_FMT_S16, 0);
    uint8_t *buffer, *start_buffer;
    av_samples_alloc(&buffer, &out_linesize, 2, decoded_frame->nb_samples,
		     AV_SAMPLE_FMT_S16, 0);
    start_buffer = buffer;
    avresample_convert(swr, &buffer, 
		       decoded_frame->linesize[0], 
		       decoded_frame->nb_samples, 
		       // decoded_frame->extended_data, 
		       decoded_frame->data, 
		       decoded_frame->linesize[0], 
		       decoded_frame->nb_samples);
    // printf("Decoded audio size %d\n", required_decoded_size);

    /* printf("Audio timestamp %lld\n", (decoded_frame->pkt_pts * USECS_IN_SEC *
       atime_base_num /
       atime_base_den));
    */
    //print_clock_time("OMX Audio empty timestamp");

    while (required_decoded_size > 0) {
	buff_header = 
	    ilclient_get_input_buffer(component,
				      port_index,
				      1 /* block */);

	/*
	buff_header->nTimeStamp = ToOMXTime((uint64_t)
					    (decoded_frame->pkt_pts * USECS_IN_SEC *
					     atime_base_num /
					     atime_base_den));
	*/
	int len = buff_header->nAllocLen;
	    
	if (required_decoded_size > len) {
	    // fprintf(stderr, "Buffer not big enough %d, looping\n", len);
	    memcpy(buff_header->pBuffer,
		   buffer, len);
	    buff_header->nFilledLen = len;
	    buffer += len;
	} else {
	    memcpy(buff_header->pBuffer,
		   buffer, required_decoded_size);
	    buff_header->nFilledLen = required_decoded_size;
	}
	// gettimeofday(&tv, NULL);
	// printf("Time audio empty start %ld\n", 
	// tv.tv_sec * USECS_IN_SEC + tv.tv_usec - starttime);
	r = OMX_EmptyThisBuffer(ilclient_get_handle(component),
				buff_header);


	// gettimeofday(&tv, NULL);
	// printf("Time audio empty stop  %ld\n", 
	//  tv.tv_sec * USECS_IN_SEC + tv.tv_usec - starttime);

	//exit_on_omx_error(r, "Empty buffer error %s\n");

	required_decoded_size -= len;   
    }
    //av_freep(&start_buffer[0]);
    av_free(&start_buffer[0]);
    return r;
}

AVCodecContext* codec_context;

int setup_demuxer(const char *filename) {
    // Register all formats and codecs
    av_register_all();
    if(avformat_open_input(&pFormatCtx, filename, NULL, NULL)!=0) {
	fprintf(stderr, "Can't get format\n");
        return -1; // Couldn't open file
    }
    // Retrieve stream information
    if (avformat_find_stream_info(pFormatCtx, NULL) < 0) {
	return -1; // Couldn't find stream information
    }
    printf("Format:\n");
    av_dump_format(pFormatCtx, 0, filename, 0);

    int ret;
    ret = av_find_best_stream(pFormatCtx, AVMEDIA_TYPE_AUDIO, -1, -1, NULL, 0);
    if (ret >= 0) {
	audio_stream_idx = ret;
	// audio_stream_idx = 2; // JN

	audio_stream = pFormatCtx->streams[audio_stream_idx];
	audio_dec_ctx = audio_stream->codec;

	sample_rate = audio_dec_ctx->sample_rate;
	channels =  audio_dec_ctx->channels;
	printf("Sample rate is %d channels %d\n", sample_rate, channels);

	AVCodec *codec = avcodec_find_decoder(audio_stream->codec->codec_id);
	codec_context = avcodec_alloc_context3(codec);

	// copy across info from codec about extradata
	codec_context->extradata =  audio_stream->codec->extradata;
	codec_context->extradata_size =  audio_stream->codec->extradata_size;


	if (codec) {
	    printf("Codec name %s\n", codec->name);
	}

	if (!avcodec_open2(codec_context, codec, NULL) < 0) {
	    fprintf(stderr, "Could not find open the needed codec");
	    exit(1);
	}
    }
    return 0;
}

/* For the RPi name can be "hdmi" or "local" */
void setOutputDevice(OMX_HANDLETYPE handle, const char *name) {
    OMX_ERRORTYPE err;
    OMX_CONFIG_BRCMAUDIODESTINATIONTYPE arDest;

    if (name && strlen(name) < sizeof(arDest.sName)) {
	memset(&arDest, 0, sizeof(OMX_CONFIG_BRCMAUDIODESTINATIONTYPE));
	arDest.nSize = sizeof(OMX_CONFIG_BRCMAUDIODESTINATIONTYPE);
	arDest.nVersion.nVersion = OMX_VERSION;

	strcpy((char *)arDest.sName, name);
       
	err = OMX_SetParameter(handle, OMX_IndexConfigBrcmAudioDestination, &arDest);
	if (err != OMX_ErrorNone) {
	    fprintf(stderr, "Error on setting audio destination\n");
	    exit(1);
	}
    }
}

void setup_audio_renderComponent(ILCLIENT_T  *handle, 
				 char *componentName, 
				 COMPONENT_T **component) {
    int err;
    err = ilclient_create_component(handle,
				    component,
				    componentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    );
    if (err == -1) {
	fprintf(stderr, "Component create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*component));

    err = ilclient_change_component_state(*component,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*component));

    // must be before we enable buffers
    set_audio_render_input_format(*component);

    setOutputDevice(ilclient_get_handle(*component), "local");
    //setOutputDevice(ilclient_get_handle(*component), "hdmi");

    // input port
    ilclient_enable_port_buffers(*component, 100, 
				 NULL, NULL, NULL);
    ilclient_enable_port(*component, 100);




    err = ilclient_change_component_state(*component,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Executing\n");
	exit(1);
    }
    printState(ilclient_get_handle(*component));
}

int main(int argc, char** argv) {

    char *renderComponentName;

    int err;
    ILCLIENT_T  *handle;
    COMPONENT_T *renderComponent;

    if (argc > 1) {
	IMG = argv[1];
    }

    OMX_BUFFERHEADERTYPE *buff_header;

    setup_demuxer(IMG);

    renderComponentName = "audio_render";

    bcm_host_init();

    handle = ilclient_init();
    // vcos_log_set_level(VCOS_LOG_CATEGORY, VCOS_LOG_TRACE);
    if (handle == NULL) {
	fprintf(stderr, "IL client init failed\n");
	exit(1);
    }

    if (OMX_Init() != OMX_ErrorNone) {
        ilclient_destroy(handle);
        fprintf(stderr, "OMX init failed\n");
	exit(1);
    }

    ilclient_set_error_callback(handle,
				error_callback,
				NULL);
    ilclient_set_eos_callback(handle,
			      eos_callback,
			      NULL);
    ilclient_set_port_settings_callback(handle,
					port_settings_callback,
					NULL);
    ilclient_set_empty_buffer_done_callback(handle,
					    empty_buffer_done_callback,
					    NULL);

    setup_audio_renderComponent(handle, renderComponentName, &renderComponent);

    FILE *out = fopen("tmp.se16", "wb");

    // for converting from AV_SAMPLE_FMT_FLTP to AV_SAMPLE_FMT_S16
    // Set up SWR context once you've got codec information
    //AVAudioResampleContext *swr = avresample_alloc_context();
    swr = avresample_alloc_context();

    av_opt_set_int(swr, "in_channel_layout", 
		   av_get_default_channel_layout(audio_dec_ctx->channels) , 0);
    av_opt_set_int(swr, "out_channel_layout", AV_CH_LAYOUT_STEREO,  0);
    av_opt_set_int(swr, "in_sample_rate",     audio_dec_ctx->sample_rate, 0);
    av_opt_set_int(swr, "out_sample_rate",    audio_dec_ctx->sample_rate, 0);
    av_opt_set_int(swr, "in_sample_fmt", audio_dec_ctx->sample_fmt, 0);
    av_opt_set_int(swr, "out_sample_fmt", AV_SAMPLE_FMT_S16,  0);
    avresample_open(swr);

    fprintf(stderr, "Num channels for resmapling %d\n",
	   av_get_channel_layout_nb_channels(AV_CH_LAYOUT_STEREO));

    int buffer_size;
    int num_ret;

    /* read frames from the file */
    AVFrame *frame = avcodec_alloc_frame(); // av_frame_alloc
    while (av_read_frame(pFormatCtx, &pkt) >= 0) {
	// printf("Read pkt %d\n", pkt.size);

	AVPacket orig_pkt = pkt;
	if (pkt.stream_index == audio_stream_idx) {
	    // printf("  read audio pkt %d\n", pkt.size);
	    //fwrite(pkt.data, 1, pkt.size, out);

	    AVPacket avpkt;
	    int got_frame;
	    av_init_packet(&avpkt);
	    avpkt.data = pkt.data;
	    avpkt.size = pkt.size;


	    uint8_t *buffer;
	    if (((err = avcodec_decode_audio4(codec_context,
					      frame,
					      &got_frame,
					      &avpkt)) < 0)  || !got_frame) {
		fprintf(stderr, "Error decoding %d\n", err);
		continue;
	    }
	    int required_decoded_size = 0;

	    if (audio_dec_ctx->sample_fmt == AV_SAMPLE_FMT_S16) {
		buffer = frame->data;
	    } else {
		read_audio_into_buffer_and_empty(frame,
					 renderComponent,
					 required_decoded_size
					 );
	    }
	    fwrite(buffer, 1, frame->nb_samples*4, out);
	}

	av_free_packet(&orig_pkt);
    }

    exit(0);
}

      

Rendering the video stream (at full speed)

An MP4 file will usually contain an H.264 video stream and we will restrict discussion to that type. This can be decoded by the GPU using the Broadcom component OMX.broadcom.video_decode. There is no need to decode this in software. So a first cut at rendering the video stream is to first demux it using FFmpeg/Libav, pass it into the Broadcom video decoder and then into the Broadcom video renderer, OMX.broadcom.video_render.

It works: but with no controls it will play the video as fast as it can render it, just as though it was playing every frame on fast forward. Nevertheless, it is worth looking at as there are some wrinkles arising from the decoding to be addressed.

Demuxing an MP4 file takes place as earlier described. Instead of selecting on audio frames, we now select on video frames. But on feeding it to an OMX component we are now back in the situation of needing to wait for a PortSettingsChanged event to occur before setting up the tunnel between decode and render components. The code follows the structure of earlier examples, looping through the file until we get the event, setting up the tunnel and then looping through the rest of the file. Nothing new.

Setting up the demuxer however requires code specific to video rather than audio. Once we have found the "best video stream" we can extract video data from it such as image width and height. But also, we may get extradata, more information about the video stream. (This may include things like Huffman tables, but we don't need to look into the details, fortunately.).

This is similar to passing extra data from the audio codec to the audio context, but it isn't so simple: first we have to capture it and then send it into the video decoder through an OMX input buffer.

The code to setup the demuxer is

int setup_demuxer(const char *filename) {
    // Register all formats and codecs
    av_register_all();
    if(avformat_open_input(&pFormatCtx, filename, NULL, NULL)!=0) {
	fprintf(stderr, "Can't get format\n");
        return -1; // Couldn't open file
    }
    // Retrieve stream information
    if (avformat_find_stream_info(pFormatCtx, NULL) < 0) {
	return -1; // Couldn't find stream information
    }
    printf("Format:\n");
    av_dump_format(pFormatCtx, 0, filename, 0);

    int ret;
    ret = av_find_best_stream(pFormatCtx, AVMEDIA_TYPE_VIDEO, -1, -1, NULL, 0);
    if (ret >= 0) {
	video_stream_idx = ret;

	video_stream = pFormatCtx->streams[video_stream_idx];
	video_dec_ctx = video_stream->codec;

	img_width         = video_stream->codec->width;
	img_height        = video_stream->codec->height;
	extradata         = video_stream->codec->extradata;
	extradatasize     = video_stream->codec->extradata_size;

	AVCodec *codec = avcodec_find_decoder(video_stream->codec->codec_id);
	
	if (codec) {
	    printf("Codec name %s\n", codec->name);
	}
    }
    return 0;
}
      

The extra data needs to be fed into the video decoder, just like the extra data for audio needed to be fed into the audio decoder. This time, the video decoder is the OpenMAX component, and sending the data to it means getting a buffer, copying the data into it and emptying the buffer, before sending any data frames:

int SendDecoderConfig(COMPONENT_T *component)
{
  OMX_ERRORTYPE omx_err   = OMX_ErrorNone;

  /* send decoder config */
  if(extradatasize > 0 && extradata != NULL)
  {
      //fwrite(extradata, 1, extradatasize, out);

      OMX_BUFFERHEADERTYPE *omx_buffer = ilclient_get_input_buffer(component,
					  130,
					  1 /* block */);

    if(omx_buffer == NULL)
    {
	fprintf(stderr, "%s - buffer error 0x%08x", __func__, omx_err);
      return 0;
    }

    omx_buffer->nOffset = 0;
    omx_buffer->nFilledLen = extradatasize;
    if(omx_buffer->nFilledLen > omx_buffer->nAllocLen)
    {
	fprintf(stderr, "%s - omx_buffer->nFilledLen > omx_buffer-nAllocLen",  __func__);
      return 0;
    }

    memset((unsigned char *)omx_buffer->pBuffer, 0x0, omx_buffer->nAllocLen);
    memcpy((unsigned char *)omx_buffer->pBuffer, extradata, omx_buffer->nFilledLen);
    omx_buffer->nFlags = OMX_BUFFERFLAG_CODECCONFIG | OMX_BUFFERFLAG_ENDOFFRAME;
  
    omx_err =  OMX_EmptyThisBuffer(ilclient_get_handle(component),
				   omx_buffer);
    if (omx_err != OMX_ErrorNone)
    {
	fprintf(stderr, "%s - OMX_EmptyThisBuffer() failed with result(0x%x)\n", __func__, omx_err);
      return 0;
    } else {
	printf("Config sent, emptying buffer %d\n", extradatasize);
    }
  }
  return 1;
}
      

That is the essential extra point in decoding the video and sending it to the renderer. The code is at il_ffmpeg_demux_render_video_full_speed.c and as mentioned before, plays the video as fast as it can.

Rendering the video stream (with scheduling)

To play the video stream at a reasonable speed adds substantial layers of complexity. We need to add a clock and a scheduler to begin with. The scheduler is interposed between the decoder and the renderer, and the scheduler takes times from the clock.

There are now four OpenMAX components to set up:

The code for the first three is straightforward:

void setup_decodeComponent(ILCLIENT_T  *handle, 
			   char *decodeComponentName, 
			   COMPONENT_T **decodeComponent) {
    int err;

    err = ilclient_create_component(handle,
				    decodeComponent,
				    decodeComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    |
				    ILCLIENT_ENABLE_OUTPUT_BUFFERS
				    );
    if (err == -1) {
	fprintf(stderr, "DecodeComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*decodeComponent));

    err = ilclient_change_component_state(*decodeComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*decodeComponent));

    // must be before we enable buffers
    set_video_decoder_input_format(*decodeComponent);
}

void setup_renderComponent(ILCLIENT_T  *handle, 
			   char *renderComponentName, 
			   COMPONENT_T **renderComponent) {
    int err;

    err = ilclient_create_component(handle,
				    renderComponent,
				    renderComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    );
    if (err == -1) {
	fprintf(stderr, "RenderComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*renderComponent));

    err = ilclient_change_component_state(*renderComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*renderComponent));
}

void setup_schedulerComponent(ILCLIENT_T  *handle, 
			   char *schedulerComponentName, 
			   COMPONENT_T **schedulerComponent) {
    int err;

    err = ilclient_create_component(handle,
				    schedulerComponent,
				    schedulerComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    );
    if (err == -1) {
	fprintf(stderr, "SchedulerComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*schedulerComponent));

    err = ilclient_change_component_state(*schedulerComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*schedulerComponent));
}
     

The clock component is new. The clock can can use a variety of sources for its timing information. For our case we are dealing with video (and later audio). The clock can use a video clock or an audio clock as its time reference source. Here we only want to deal with video, so we use a video clock

OMX_TIME_CONFIG_ACTIVEREFCLOCKTYPE refClock;
refClock.nSize = sizeof(OMX_TIME_CONFIG_ACTIVEREFCLOCKTYPE);
refClock.nVersion.nVersion = OMX_VERSION;
refClock.eClock = OMX_TIME_RefClockVideo;

OMX_SetConfig(ilclient_get_handle(*clockComponent), 
	      OMX_IndexConfigTimeActiveRefClock, &refClock);
      

How the clock locates its reference clock doesn't seem to be specified anywhere: it seems to be a bit of 'behind the scenes' magic done implicitly by OMX implementors. There are hints in the 1.1.2 specification in Section 6.2.6 Audio-Video File Playback Example Use Case which shows that it is probably the scheduler that plays this role.

The clock needs to have a rate set. This is set in the field xScale of an OMX_TIME_CONFIG_SCALETYPE struct. The values do not seem to be clearly specified by OpenMAX. The wonderful omxplayer seems to have tried various values for this in different versions. The only clue is that 3.2.2.11.2 Sample Code Showing Calling Sequence of the 1.1.2 specification shows

	
oScale.xScale = 0x00020000; /*2x*/
	
      

from which one can guess that the value of normal speed could be 0x00010000 - but by juggling other parameters this need not be the case (which is what omxplayer currently does). Assuming this value, code is

OMX_TIME_CONFIG_SCALETYPE scaleType;
scaleType.nSize = sizeof(OMX_TIME_CONFIG_SCALETYPE);
scaleType.nVersion.nVersion = OMX_VERSION;
scaleType.xScale = 0x00010000;

OMX_SetConfig(ilclient_get_handle(*clockComponent), 
			OMX_IndexConfigTimeScale, &scaleType);
      

The complete clock setup is

void setup_clockComponent(ILCLIENT_T  *handle, 
			   char *clockComponentName, 
			   COMPONENT_T **clockComponent) {
    int err;

    err = ilclient_create_component(handle,
				    clockComponent,
				    clockComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    );
    if (err == -1) {
	fprintf(stderr, "ClockComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*clockComponent));

    err = ilclient_change_component_state(*clockComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*clockComponent));
    printClockState(*clockComponent);

    OMX_TIME_CONFIG_ACTIVEREFCLOCKTYPE refClock;
    refClock.nSize = sizeof(OMX_TIME_CONFIG_ACTIVEREFCLOCKTYPE);
    refClock.nVersion.nVersion = OMX_VERSION;
    refClock.eClock = OMX_TIME_RefClockVideo;

    err = OMX_SetConfig(ilclient_get_handle(*clockComponent), 
			OMX_IndexConfigTimeActiveRefClock, &refClock);
    if(err != OMX_ErrorNone) {
	fprintf(stderr, "COMXCoreComponent::SetConfig - %s failed with omx_err(0x%x)\n", 
              "clock", err);
    }

    OMX_TIME_CONFIG_SCALETYPE scaleType;
    scaleType.nSize = sizeof(OMX_TIME_CONFIG_SCALETYPE);
    scaleType.nVersion.nVersion = OMX_VERSION;
    scaleType.xScale = (1 << 16);

    err = OMX_SetConfig(ilclient_get_handle(*clockComponent), 
			OMX_IndexConfigTimeScale, &scaleType);
    if(err != OMX_ErrorNone) {
	fprintf(stderr, "COMXCoreComponent::SetConfig - %s failed with omx_err(0x%x)\n", 
              "clock", err);
    }

}
     

Execution reads from the file, demuxing frames using FFmpeg/Libav and sending frames to the decoder until a PortSettingsChanged event occurs. Then it sets up tunnels. There are three of these:

as in

    TUNNEL_T decodeTunnel;
    set_tunnel(&decodeTunnel, decodeComponent, 131, schedulerComponent, 10);
    if ((err = ilclient_setup_tunnel(&decodeTunnel, 0, 0)) < 0) {
	fprintf(stderr, "Error setting up decode tunnel %X\n", err);
	exit(1);
    } else {
	printf("Decode tunnel set up ok\n");
    }

    TUNNEL_T schedulerTunnel;
    set_tunnel(&schedulerTunnel, schedulerComponent, 11, renderComponent, 90);
    if ((err = ilclient_setup_tunnel(&schedulerTunnel, 0, 0)) < 0) {
	fprintf(stderr, "Error setting up scheduler tunnel %X\n", err);
	exit(1);
    } else {
	printf("Scheduler tunnel set up ok\n");
    }

    TUNNEL_T clockTunnel;
    set_tunnel(&clockTunnel, clockComponent, 80, schedulerComponent, 12);
    if ((err = ilclient_setup_tunnel(&clockTunnel, 0, 0)) < 0) {
	fprintf(stderr, "Error setting up clock tunnel %X\n", err);
	exit(1);
    } else {
	printf("Clock tunnel set up ok\n");
    }
    startClock(clockComponent);
     

The clock is started by

void startClock(COMPONENT_T *clockComponent) {
    OMX_ERRORTYPE err = OMX_ErrorNone;
    OMX_TIME_CONFIG_CLOCKSTATETYPE clockState;

    memset(&clockState, 0, sizeof( OMX_TIME_CONFIG_CLOCKSTATETYPE));
    clockState.nSize = sizeof( OMX_TIME_CONFIG_CLOCKSTATETYPE);
    clockState.nVersion.nVersion = OMX_VERSION;

    err = OMX_GetConfig(ilclient_get_handle(clockComponent), 
			OMX_IndexConfigTimeClockState, &clockState);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error getting clock state %s\n", err2str(err));
        return;
    }
    clockState.eState = OMX_TIME_ClockStateRunning;
    err = OMX_SetConfig(ilclient_get_handle(clockComponent), 
			OMX_IndexConfigTimeClockState, &clockState);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error starting clock %s\n", err2str(err));
        return;
    }

}
     

Now we turn to the tricky part of the program: matching timing of the Libav/FFmpeg packets to the OMX decoder. The relevant information from Libav/FFmpeg is contained in the structures

	
typedef struct AVPacket {
    int64_t pts;
    int64_t dts;
    ...
} AVPacket;

typedef struct AVStream {
    AVRational r_frame_rate;
    AVRational time_base;
    ...
} AVStream;
	
      

The field we have to match it to is

	
typedef struct OMX_BUFFERHEADERTYPE
{
    OMX_TICKS nTimeStamp;
    ...
} OMX_BUFFERHEADERTYPE;
	
      

The fields dts and pts refer to 'decode timestamp' and 'presentation timestamp' respectively. A good explanation is given by An ffmpeg and SDL Tutorial: Tutorial 05: Synching Video by dranger. The conclusion is that we aren't actually interested in the decode time, only the presentation time, pts.

The values of these from some typical files are useful in explaining these fields. Note that the type AVRational represents a number with numerator (num) and denominator (den).

File time_base r_frame_rate pts increment Frame rate
num den num (fps rate) den (fps scale)
Big Buck Bunny 1 600 24 1 25 23.96
Tai Chi 1 29969 29969 1000 1000 29.97
ClipCanvas 1 600 25 1 24 25
small 1 90000 30 1 3000 30
Wildlife 1 2997 2997 100 100 29.97
Sintel 1 24 24 1 1 24

The times for LibAV/FFmpeg are in seconds. So for Big Buck Bunny, the timebase is 1/600 second and the frame rate is 24 frames per second, while the pts increment is 25 timebase units. The timebase and the frame rate are given from the demuxer:

	
video_stream = pFormatCtx->streams[video_stream_idx];

fpsscale          = video_stream->r_frame_rate.den;
fpsrate           = video_stream->r_frame_rate.num;
time_base_num     = video_stream->time_base.num;
time_base_den     = video_stream->time_base.den;
	
      

The pts values are given in the decoded packets. They may not be in order, so you may have to look at several values in order to obtain the highest common factor.

The relation between these fields is

	
pts increment * timebase = 1 / frame rate
	
      

so that

Each frame occurs at the time pts * timebase seconds.

Turning to OMX, Section 6.2.1 Timestamps of the 1.1.2 specification states that the timestamp structure "... shall be interpreted as a signed 64-bit value representing microseconds." That means that any seconds value from LibAV/FFmpeg needs to be multiplied by 1,000,000 to convert it to OMX timestamp microseconds.

This relates LibAV/FFmpeg to OMX timestamps by

	
buff_header->nTimeStamp = pts * 1000000 * time_base_num / time_base_den;
	
      

On 32-bit CPUs like the RPi, the 64-bit timestamp needs to be unpacked into a struct with two 32-bit fields. This is done by

	
OMX_TICKS ToOMXTime(int64_t pts)
{
    OMX_TICKS ticks;
    ticks.nLowPart = pts;
    ticks.nHighPart = pts >> 32;
    return ticks;
}
	
      

Once the tunnels are set up and the clock is running, the program reads and demuxes frames and then sends them to the decoder. The complete program is il_ffmpeg_demux_render_video.c :

#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>

#include <OMX_Core.h>
#include <OMX_Component.h>

#include <bcm_host.h>
#include <vcos_logging.h>

#define VCOS_LOG_CATEGORY (&il_ffmpeg_log_category)
static VCOS_LOG_CAT_T il_ffmpeg_log_category;

#include <ilclient.h>

#include "libavcodec/avcodec.h"
#include <libavformat/avformat.h>
#include "libavutil/mathematics.h"
#include "libavutil/samplefmt.h"

#define INBUF_SIZE 4096
#define AUDIO_INBUF_SIZE 20480
#define AUDIO_REFILL_THRESH 4096

char *IMG = "taichi.mp4";

static AVCodecContext *video_dec_ctx = NULL;
static AVStream *video_stream = NULL;
static AVPacket pkt;
AVFormatContext *pFormatCtx = NULL;

static int video_stream_idx = -1;

uint8_t extradatasize;
void *extradata;

AVCodec *codec;

void printState(OMX_HANDLETYPE handle) {
    OMX_STATETYPE state;
    OMX_ERRORTYPE err;

    err = OMX_GetState(handle, &state);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error on getting state\n");
        exit(1);
    }
    switch (state) {
    case OMX_StateLoaded:           printf("StateLoaded\n"); break;
    case OMX_StateIdle:             printf("StateIdle\n"); break;
    case OMX_StateExecuting:        printf("StateExecuting\n"); break;
    case OMX_StatePause:            printf("StatePause\n"); break;
    case OMX_StateWaitForResources: printf("StateWait\n"); break;
    case OMX_StateInvalid:          printf("StateInvalid\n"); break;
    default:                        printf("State unknown\n"); break;
    }
}

char *err2str(int err) {
    switch (err) {
    case OMX_ErrorInsufficientResources: return "OMX_ErrorInsufficientResources";
    case OMX_ErrorUndefined: return "OMX_ErrorUndefined";
    case OMX_ErrorInvalidComponentName: return "OMX_ErrorInvalidComponentName";
    case OMX_ErrorComponentNotFound: return "OMX_ErrorComponentNotFound";
    case OMX_ErrorInvalidComponent: return "OMX_ErrorInvalidComponent";
    case OMX_ErrorBadParameter: return "OMX_ErrorBadParameter";
    case OMX_ErrorNotImplemented: return "OMX_ErrorNotImplemented";
    case OMX_ErrorUnderflow: return "OMX_ErrorUnderflow";
    case OMX_ErrorOverflow: return "OMX_ErrorOverflow";
    case OMX_ErrorHardware: return "OMX_ErrorHardware";
    case OMX_ErrorInvalidState: return "OMX_ErrorInvalidState";
    case OMX_ErrorStreamCorrupt: return "OMX_ErrorStreamCorrupt";
    case OMX_ErrorPortsNotCompatible: return "OMX_ErrorPortsNotCompatible";
    case OMX_ErrorResourcesLost: return "OMX_ErrorResourcesLost";
    case OMX_ErrorNoMore: return "OMX_ErrorNoMore";
    case OMX_ErrorVersionMismatch: return "OMX_ErrorVersionMismatch";
    case OMX_ErrorNotReady: return "OMX_ErrorNotReady";
    case OMX_ErrorTimeout: return "OMX_ErrorTimeout";
    case OMX_ErrorSameState: return "OMX_ErrorSameState";
    case OMX_ErrorResourcesPreempted: return "OMX_ErrorResourcesPreempted";
    case OMX_ErrorPortUnresponsiveDuringAllocation: return "OMX_ErrorPortUnresponsiveDuringAllocation";
    case OMX_ErrorPortUnresponsiveDuringDeallocation: return "OMX_ErrorPortUnresponsiveDuringDeallocation";
    case OMX_ErrorPortUnresponsiveDuringStop: return "OMX_ErrorPortUnresponsiveDuringStop";
    case OMX_ErrorIncorrectStateTransition: return "OMX_ErrorIncorrectStateTransition";
    case OMX_ErrorIncorrectStateOperation: return "OMX_ErrorIncorrectStateOperation";
    case OMX_ErrorUnsupportedSetting: return "OMX_ErrorUnsupportedSetting";
    case OMX_ErrorUnsupportedIndex: return "OMX_ErrorUnsupportedIndex";
    case OMX_ErrorBadPortIndex: return "OMX_ErrorBadPortIndex";
    case OMX_ErrorPortUnpopulated: return "OMX_ErrorPortUnpopulated";
    case OMX_ErrorComponentSuspended: return "OMX_ErrorComponentSuspended";
    case OMX_ErrorDynamicResourcesUnavailable: return "OMX_ErrorDynamicResourcesUnavailable";
    case OMX_ErrorMbErrorsInFrame: return "OMX_ErrorMbErrorsInFrame";
    case OMX_ErrorFormatNotDetected: return "OMX_ErrorFormatNotDetected";
    case OMX_ErrorContentPipeOpenFailed: return "OMX_ErrorContentPipeOpenFailed";
    case OMX_ErrorContentPipeCreationFailed: return "OMX_ErrorContentPipeCreationFailed";
    case OMX_ErrorSeperateTablesUsed: return "OMX_ErrorSeperateTablesUsed";
    case OMX_ErrorTunnelingUnsupported: return "OMX_ErrorTunnelingUnsupported";
    default: return "unknown error";
    }
}

void printClockState(COMPONENT_T *clockComponent) {
    OMX_ERRORTYPE err = OMX_ErrorNone;
    OMX_TIME_CONFIG_CLOCKSTATETYPE clockState;

    memset(&clockState, 0, sizeof( OMX_TIME_CONFIG_CLOCKSTATETYPE));
    clockState.nSize = sizeof( OMX_TIME_CONFIG_CLOCKSTATETYPE);
    clockState.nVersion.nVersion = OMX_VERSION;

    err = OMX_GetConfig(ilclient_get_handle(clockComponent), 
			OMX_IndexConfigTimeClockState, &clockState);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error getting clock state %s\n", err2str(err));
        return;
    }
    switch (clockState.eState) {
    case OMX_TIME_ClockStateRunning:
	printf("Clock running\n");
	break;
    case OMX_TIME_ClockStateWaitingForStartTime:
	printf("Clock waiting for start time\n");
	break;
    case OMX_TIME_ClockStateStopped:
	printf("Clock stopped\n");
	break;
    default:
	printf("Clock in other state\n");
    }
}

void startClock(COMPONENT_T *clockComponent) {
    OMX_ERRORTYPE err = OMX_ErrorNone;
    OMX_TIME_CONFIG_CLOCKSTATETYPE clockState;

    memset(&clockState, 0, sizeof( OMX_TIME_CONFIG_CLOCKSTATETYPE));
    clockState.nSize = sizeof( OMX_TIME_CONFIG_CLOCKSTATETYPE);
    clockState.nVersion.nVersion = OMX_VERSION;

    err = OMX_GetConfig(ilclient_get_handle(clockComponent), 
			OMX_IndexConfigTimeClockState, &clockState);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error getting clock state %s\n", err2str(err));
        return;
    }
    clockState.eState = OMX_TIME_ClockStateRunning;
    err = OMX_SetConfig(ilclient_get_handle(clockComponent), 
			OMX_IndexConfigTimeClockState, &clockState);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error starting clock %s\n", err2str(err));
        return;
    }

}

void eos_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    printf("Got eos event\n");
}

void error_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    printf("OMX error %s\n", err2str(data));
}

void port_settings_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {
    printf("Got port Settings event\n");
    // exit(0);
}

void empty_buffer_done_callback(void *userdata, COMPONENT_T *comp) {
    printf("Got empty buffer done\n");
}


int get_file_size(char *fname) {
    struct stat st;

    if (stat(fname, &st) == -1) {
	perror("Stat'ing img file");
	return -1;
    }
    return(st.st_size);
}

unsigned int uWidth;
unsigned int uHeight;

unsigned int fpsscale;
unsigned int fpsrate;
unsigned int time_base_num;
unsigned int time_base_den;

#ifdef OMX_SKIP64BIT
OMX_TICKS ToOMXTime(int64_t pts)
{
    OMX_TICKS ticks;
    ticks.nLowPart = pts;
    ticks.nHighPart = pts >> 32;
    return ticks;
}
#else
#define FromOMXTime(x) (x)
#endif

OMX_ERRORTYPE copy_into_buffer_and_empty(AVPacket *pkt,
					 COMPONENT_T *component,
					 OMX_BUFFERHEADERTYPE *buff_header) {
    OMX_ERRORTYPE r;

    int buff_size = buff_header->nAllocLen;
    int size = pkt->size;
    uint8_t *content = pkt->data;

    while (size > 0) {
	buff_header->nFilledLen = (size > buff_header->nAllocLen-1) ?
	    buff_header->nAllocLen-1 : size;
	memset(buff_header->pBuffer, 0x0, buff_header->nAllocLen);
	memcpy(buff_header->pBuffer, content, buff_header->nFilledLen);
	size -= buff_header->nFilledLen;
	content += buff_header->nFilledLen;
    

	/*
	if (size < buff_size) {
	    memcpy((unsigned char *)buff_header->pBuffer, 
		   pkt->data, size);
	} else {
	    printf("Buffer not big enough %d %d\n", buff_size, size);
	    return -1;
	}
	
	buff_header->nFilledLen = size;
	*/

	buff_header->nFlags = 0;
	if (size <= 0) 
	    buff_header->nFlags |= OMX_BUFFERFLAG_ENDOFFRAME;

	printf("  DTS is %s %ld\n", "str", pkt->dts);
	printf("  PTS is %s %ld\n", "str", pkt->pts);

	if (pkt->dts == 0) {
	    buff_header->nFlags |= OMX_BUFFERFLAG_STARTTIME;
	} else {
	    buff_header->nTimeStamp = ToOMXTime((uint64_t)
						(pkt->pts * 1000000/
						 time_base_den));

	    printf("Time stamp %d\n", 	buff_header->nTimeStamp);
	}

	r = OMX_EmptyThisBuffer(ilclient_get_handle(component),
				buff_header);
	if (r != OMX_ErrorNone) {
	    fprintf(stderr, "Empty buffer error %s\n",
		    err2str(r));
	} else {
	    printf("Emptying buffer %p\n", buff_header);
	}
	if (size > 0) {
	     buff_header = 
		ilclient_get_input_buffer(component,
					  130,
					  1 /* block */);
	}
    }
    return r;
}

int img_width, img_height;

int SendDecoderConfig(COMPONENT_T *component, FILE *out)
{
    OMX_ERRORTYPE omx_err   = OMX_ErrorNone;

    /* send decoder config */
    if(extradatasize > 0 && extradata != NULL)
	{
	    fwrite(extradata, 1, extradatasize, out);

	    OMX_BUFFERHEADERTYPE *omx_buffer = ilclient_get_input_buffer(component,
									 130,
									 1 /* block */);

	    if(omx_buffer == NULL)
		{
		    fprintf(stderr, "%s - buffer error 0x%08x", __func__, omx_err);
		    return 0;
		}

	    omx_buffer->nOffset = 0;
	    omx_buffer->nFilledLen = extradatasize;
	    if(omx_buffer->nFilledLen > omx_buffer->nAllocLen)
		{
		    fprintf(stderr, "%s - omx_buffer->nFilledLen > omx_buffer->nAllocLen",  __func__);
		    return 0;
		}

	    memset((unsigned char *)omx_buffer->pBuffer, 0x0, omx_buffer->nAllocLen);
	    memcpy((unsigned char *)omx_buffer->pBuffer, extradata, omx_buffer->nFilledLen);
	    omx_buffer->nFlags = OMX_BUFFERFLAG_CODECCONFIG | OMX_BUFFERFLAG_ENDOFFRAME;
  
	    omx_err =  OMX_EmptyThisBuffer(ilclient_get_handle(component),
					   omx_buffer);
	    if (omx_err != OMX_ErrorNone)
		{
		    fprintf(stderr, "%s - OMX_EmptyThisBuffer() failed with result(0x%x)\n", __func__, omx_err);
		    return 0;
		} else {
		printf("Config sent, emptying buffer %d\n", extradatasize);
	    }
	}
    return 1;
}

OMX_ERRORTYPE set_video_decoder_input_format(COMPONENT_T *component) {
    int err;

    // set input video format
    printf("Setting video decoder format\n");
    OMX_VIDEO_PARAM_PORTFORMATTYPE videoPortFormat;

    memset(&videoPortFormat, 0, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));
    videoPortFormat.nSize = sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE);
    videoPortFormat.nVersion.nVersion = OMX_VERSION;
    videoPortFormat.nPortIndex = 130;

    err = OMX_GetParameter(ilclient_get_handle(component),
			   OMX_IndexParamVideoPortFormat, &videoPortFormat);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error getting video decoder format %s\n", err2str(err));
        return err;
    }

    videoPortFormat.nPortIndex = 130;
    videoPortFormat.nIndex = 0;
    videoPortFormat.eCompressionFormat = OMX_VIDEO_CodingAVC;
    videoPortFormat.eColorFormat = OMX_COLOR_FormatUnused;
    videoPortFormat.xFramerate = 0;

#if 1 // doesn't seem to make any difference!!!
    if (fpsscale > 0 && fpsrate > 0) {
	videoPortFormat.xFramerate = 
	    (long long)(1<<16)*fpsrate / fpsscale;
    } else {
	videoPortFormat.xFramerate = 25 * (1<<16);
    }
    printf("FPS num %d den %d\n", fpsrate, fpsscale);
    printf("Set frame rate to %d\n", videoPortFormat.xFramerate);
#endif
    err = OMX_SetParameter(ilclient_get_handle(component),
			   OMX_IndexParamVideoPortFormat, &videoPortFormat);
    if (err != OMX_ErrorNone) {
        fprintf(stderr, "Error setting video decoder format %s\n", err2str(err));
        return err;
    } else {
        printf("Video decoder format set up ok\n");
    }

    OMX_PARAM_PORTDEFINITIONTYPE portParam;
    memset(&portParam, 0, sizeof( OMX_PARAM_PORTDEFINITIONTYPE));
    portParam.nSize = sizeof( OMX_PARAM_PORTDEFINITIONTYPE);
    portParam.nVersion.nVersion = OMX_VERSION;

    portParam.nPortIndex = 130;

    err =  OMX_GetParameter(ilclient_get_handle(component),
			    OMX_IndexParamPortDefinition, &portParam);
    if(err != OMX_ErrorNone)
	{
	    fprintf(stderr, "COMXVideo::Open error OMX_IndexParamPortDefinition omx_err(0x%08x)\n", err);
	    return err;
	}

    printf("Default framerate %d\n", portParam.format.video.xFramerate);

    portParam.nPortIndex = 130;

    portParam.format.video.nFrameWidth  = img_width;
    portParam.format.video.nFrameHeight = img_height;

    err =  OMX_SetParameter(ilclient_get_handle(component),
			    OMX_IndexParamPortDefinition, &portParam);
    if(err != OMX_ErrorNone)
	{
	    fprintf(stderr, "COMXVideo::Open error OMX_IndexParamPortDefinition omx_err(0x%08x)\n", err);
	    return err;
	}

    return OMX_ErrorNone;
}

int setup_demuxer(const char *filename) {
    // Register all formats and codecs
    av_register_all();
    if(avformat_open_input(&pFormatCtx, filename, NULL, NULL)!=0) {
	fprintf(stderr, "Can't get format\n");
        return -1; // Couldn't open file
    }
    // Retrieve stream information
    if (avformat_find_stream_info(pFormatCtx, NULL) < 0) {
	return -1; // Couldn't find stream information
    }
    printf("Format:\n");
    av_dump_format(pFormatCtx, 0, filename, 0);

    int ret;
    ret = av_find_best_stream(pFormatCtx, AVMEDIA_TYPE_VIDEO, -1, -1, NULL, 0);
    if (ret >= 0) {
	video_stream_idx = ret;

	video_stream = pFormatCtx->streams[video_stream_idx];
	video_dec_ctx = video_stream->codec;

	img_width         = video_stream->codec->width;
	img_height        = video_stream->codec->height;
	extradata         = video_stream->codec->extradata;
	extradatasize     = video_stream->codec->extradata_size;
	fpsscale          = video_stream->r_frame_rate.den;
	fpsrate           = video_stream->r_frame_rate.num;
	time_base_num         = video_stream->time_base.num;
	time_base_den         = video_stream->time_base.den;

	printf("Rate %d scale %d time base %d %d\n",
	       video_stream->r_frame_rate.num,
	       video_stream->r_frame_rate.den,
	       video_stream->time_base.num,
	       video_stream->time_base.den);

	AVCodec *codec = avcodec_find_decoder(video_stream->codec->codec_id);
	
	if (codec) {
	    printf("Codec name %s\n", codec->name);
	}
    }
    return 0;
}

void setup_decodeComponent(ILCLIENT_T  *handle, 
			   char *decodeComponentName, 
			   COMPONENT_T **decodeComponent) {
    int err;

    err = ilclient_create_component(handle,
				    decodeComponent,
				    decodeComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    |
				    ILCLIENT_ENABLE_OUTPUT_BUFFERS
				    );
    if (err == -1) {
	fprintf(stderr, "DecodeComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*decodeComponent));

    err = ilclient_change_component_state(*decodeComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*decodeComponent));

    // must be before we enable buffers
    set_video_decoder_input_format(*decodeComponent);
}

void setup_renderComponent(ILCLIENT_T  *handle, 
			   char *renderComponentName, 
			   COMPONENT_T **renderComponent) {
    int err;

    err = ilclient_create_component(handle,
				    renderComponent,
				    renderComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    );
    if (err == -1) {
	fprintf(stderr, "RenderComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*renderComponent));

    err = ilclient_change_component_state(*renderComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*renderComponent));
}

void setup_schedulerComponent(ILCLIENT_T  *handle, 
			      char *schedulerComponentName, 
			      COMPONENT_T **schedulerComponent) {
    int err;

    err = ilclient_create_component(handle,
				    schedulerComponent,
				    schedulerComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    |
				    ILCLIENT_ENABLE_INPUT_BUFFERS
				    );
    if (err == -1) {
	fprintf(stderr, "SchedulerComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*schedulerComponent));

    err = ilclient_change_component_state(*schedulerComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*schedulerComponent));
}

void setup_clockComponent(ILCLIENT_T  *handle, 
			  char *clockComponentName, 
			  COMPONENT_T **clockComponent) {
    int err;

    err = ilclient_create_component(handle,
				    clockComponent,
				    clockComponentName,
				    ILCLIENT_DISABLE_ALL_PORTS
				    );
    if (err == -1) {
	fprintf(stderr, "ClockComponent create failed\n");
	exit(1);
    }
    printState(ilclient_get_handle(*clockComponent));

    err = ilclient_change_component_state(*clockComponent,
					  OMX_StateIdle);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    printState(ilclient_get_handle(*clockComponent));
    printClockState(*clockComponent);

    OMX_COMPONENTTYPE*clock = ilclient_get_handle(*clockComponent);

    OMX_TIME_CONFIG_ACTIVEREFCLOCKTYPE refClock;
    refClock.nSize = sizeof(OMX_TIME_CONFIG_ACTIVEREFCLOCKTYPE);
    refClock.nVersion.nVersion = OMX_VERSION;
    refClock.eClock = OMX_TIME_RefClockVideo; // OMX_CLOCKPORT0;

    err = OMX_SetConfig(ilclient_get_handle(*clockComponent), 
			OMX_IndexConfigTimeActiveRefClock, &refClock);
    if(err != OMX_ErrorNone) {
	fprintf(stderr, "COMXCoreComponent::SetConfig - %s failed with omx_err(0x%x)\n", 
		"clock", err);
    }

    OMX_TIME_CONFIG_SCALETYPE scaleType;
    scaleType.nSize = sizeof(OMX_TIME_CONFIG_SCALETYPE);
    scaleType.nVersion.nVersion = OMX_VERSION;
    scaleType.xScale = 0x00010000;

    err = OMX_SetConfig(ilclient_get_handle(*clockComponent), 
			OMX_IndexConfigTimeScale, &scaleType);
    if(err != OMX_ErrorNone) {
	fprintf(stderr, "COMXCoreComponent::SetConfig - %s failed with omx_err(0x%x)\n", 
		"clock", err);
    }
}

int main(int argc, char** argv) {

    char *decodeComponentName;
    char *renderComponentName;
    char *schedulerComponentName;
    char *clockComponentName;
    int err;
    ILCLIENT_T  *handle;
    COMPONENT_T *decodeComponent;
    COMPONENT_T *renderComponent;
    COMPONENT_T *schedulerComponent;
    COMPONENT_T *clockComponent;

    if (argc > 1) {
	IMG = argv[1];
    }

    OMX_BUFFERHEADERTYPE *buff_header;

    setup_demuxer(IMG);

    decodeComponentName = "video_decode";
    renderComponentName = "video_render";
    schedulerComponentName = "video_scheduler";
    clockComponentName = "clock";

    bcm_host_init();

    handle = ilclient_init();
    vcos_log_set_level(VCOS_LOG_CATEGORY, VCOS_LOG_TRACE);
    if (handle == NULL) {
	fprintf(stderr, "IL client init failed\n");
	exit(1);
    }

    if (OMX_Init() != OMX_ErrorNone) {
        ilclient_destroy(handle);
        fprintf(stderr, "OMX init failed\n");
	exit(1);
    }

    ilclient_set_error_callback(handle,
				error_callback,
				NULL);
    ilclient_set_eos_callback(handle,
			      eos_callback,
			      NULL);
    ilclient_set_port_settings_callback(handle,
					port_settings_callback,
					NULL);
    ilclient_set_empty_buffer_done_callback(handle,
					    empty_buffer_done_callback,
					    NULL);


    setup_decodeComponent(handle, decodeComponentName, &decodeComponent);
    setup_renderComponent(handle, renderComponentName, &renderComponent);
    setup_schedulerComponent(handle, schedulerComponentName, &schedulerComponent);
    setup_clockComponent(handle, clockComponentName, &clockComponent);
    // both components now in Idle state, no buffers, ports disabled

    // input port
    err = ilclient_enable_port_buffers(decodeComponent, 130, 
				       NULL, NULL, NULL);
    if (err < 0) {
	fprintf(stderr, "Couldn't enable buffers\n");
	exit(1);
    }
    ilclient_enable_port(decodeComponent, 130);

    err = ilclient_change_component_state(decodeComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Executing\n");
	exit(1);
    }
    printState(ilclient_get_handle(decodeComponent));
 
    FILE *out = fopen("tmp.h264", "wb");
    SendDecoderConfig(decodeComponent, out);


    /* read frames from the file */
    while (av_read_frame(pFormatCtx, &pkt) >= 0) {
	printf("Read pkt %d\n", pkt.size);

	AVPacket orig_pkt = pkt;
	if (pkt.stream_index == video_stream_idx) {
	    printf("  read video pkt %d\n", pkt.size);
	    fwrite(pkt.data, 1, pkt.size, out);
	    buff_header = 
		ilclient_get_input_buffer(decodeComponent,
					  130,
					  1 /* block */);
	    if (buff_header != NULL) {
		copy_into_buffer_and_empty(&pkt,
					   decodeComponent,
					   buff_header);
	    } else {
		fprintf(stderr, "Couldn't get a buffer\n");
	    }


	    err = ilclient_wait_for_event(decodeComponent, 
					  OMX_EventPortSettingsChanged, 
					  131, 0, 0, 1,
					  ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED, 
					  0);
	    if (err < 0) {
		printf("No port settings change\n");
		//exit(1);
	    } else {
		printf("Port settings changed\n");
		// exit(0);
		break;
	    }


	    if (ilclient_remove_event(decodeComponent, 
				      OMX_EventPortSettingsChanged, 
				      131, 0, 0, 1) == 0) {
		printf("Removed port settings event\n");
		//exit(0);
		break;
	    } else {
		printf("No portr settting seen yet\n");
	    }
	}
	av_free_packet(&orig_pkt);
    }


    TUNNEL_T decodeTunnel;
    set_tunnel(&decodeTunnel, decodeComponent, 131, schedulerComponent, 10);
    if ((err = ilclient_setup_tunnel(&decodeTunnel, 0, 0)) < 0) {
	fprintf(stderr, "Error setting up decode tunnel %X\n", err);
	exit(1);
    } else {
	printf("Decode tunnel set up ok\n");
    }

    TUNNEL_T schedulerTunnel;
    set_tunnel(&schedulerTunnel, schedulerComponent, 11, renderComponent, 90);
    if ((err = ilclient_setup_tunnel(&schedulerTunnel, 0, 0)) < 0) {
	fprintf(stderr, "Error setting up scheduler tunnel %X\n", err);
	exit(1);
    } else {
	printf("Scheduler tunnel set up ok\n");
    }

    TUNNEL_T clockTunnel;
    set_tunnel(&clockTunnel, clockComponent, 80, schedulerComponent, 12);
    if ((err = ilclient_setup_tunnel(&clockTunnel, 0, 0)) < 0) {
	fprintf(stderr, "Error setting up clock tunnel %X\n", err);
	exit(1);
    } else {
	printf("Clock tunnel set up ok\n");
    }
    startClock(clockComponent);
    printClockState(clockComponent);

    // Okay to go back to processing data
    // enable the decode output ports
   
    OMX_SendCommand(ilclient_get_handle(decodeComponent), 
		    OMX_CommandPortEnable, 131, NULL);
   
    ilclient_enable_port(decodeComponent, 131);

    // enable the clock output ports
    OMX_SendCommand(ilclient_get_handle(clockComponent), 
		    OMX_CommandPortEnable, 80, NULL);
   
    ilclient_enable_port(clockComponent, 80);

    // enable the scheduler ports
    OMX_SendCommand(ilclient_get_handle(schedulerComponent), 
		    OMX_CommandPortEnable, 10, NULL);
   
    ilclient_enable_port(schedulerComponent, 10);

    OMX_SendCommand(ilclient_get_handle(schedulerComponent), 
		    OMX_CommandPortEnable, 11, NULL);

    ilclient_enable_port(schedulerComponent, 11);


    OMX_SendCommand(ilclient_get_handle(schedulerComponent), 
		    OMX_CommandPortEnable, 12, NULL);
   
    ilclient_enable_port(schedulerComponent, 12);

    // enable the render input ports
   
    OMX_SendCommand(ilclient_get_handle(renderComponent), 
		    OMX_CommandPortEnable, 90, NULL);
   
    ilclient_enable_port(renderComponent, 90);



    // set both components to executing state
    err = ilclient_change_component_state(decodeComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }
    err = ilclient_change_component_state(renderComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }

    err = ilclient_change_component_state(schedulerComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }

    err = ilclient_change_component_state(clockComponent,
					  OMX_StateExecuting);
    if (err < 0) {
	fprintf(stderr, "Couldn't change state to Idle\n");
	exit(1);
    }

    // now work through the file
    while (av_read_frame(pFormatCtx, &pkt) >= 0) {
	printf("Read pkt after port settings %d\n", pkt.size);
	fwrite(pkt.data, 1, pkt.size, out);
	
	if (pkt.stream_index != video_stream_idx) {
	    continue;
	}
	printf("  is video pkt\n");
	//printf("  Best timestamp is %d\n", );

	// do we have a decode input buffer we can fill and empty?
	buff_header = 
	    ilclient_get_input_buffer(decodeComponent,
				      130,
				      1 /* block */);
	if (buff_header != NULL) {
	    copy_into_buffer_and_empty(&pkt,
				       decodeComponent,
				       buff_header);
	}

	err = ilclient_wait_for_event(decodeComponent, 
				      OMX_EventPortSettingsChanged, 
				      131, 0, 0, 1,
				      ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED, 
				      0);
	if (err >= 0) {
	    printf("Another port settings change\n");
	}

    }

    ilclient_wait_for_event(renderComponent, 
			    OMX_EventBufferFlag, 
			    90, 0, OMX_BUFFERFLAG_EOS, 0,
			    ILCLIENT_BUFFER_FLAG_EOS, 10000);
    printf("EOS on render\n");

    exit(0);
}

      

Rendering both audio and video

(The code is getting loooong and meesssssyyy. I've done a cleanup to bring in some macros and other functions to simplify creating and manipulating OMX components. I've also collapsed the two while loops into one, checking - usually redundantly - the port settings changed event. This simplifies the code and doesn't cost much.).

We now have both the audio and video decode/render streams. We picture this as

The audio stream has

In the video stream

We also have an OMX clock. People are apparently much more sensitive to aberrations in audio than in video, so all recommendations are to set the clock to the audio stream rather than to the video stream.

Use the audio decoder?

The RPi can apparently decode some audio formats, AC3 and DTS. It doesn't need to use software decoding for these. So it can use the OMX audio decoder for some formats. The omxplayer includes the audio decoder in the audio pipeline, even for PCM data.

The audio decoder can't actually deal with PCM data, which is what will come out of the the Libav decoder: it can manage WAV data, which has the additional header information of sample size, etc. So if the OMX audio decoder is inserted into the pipeline, additional code must first send the appropriate WAV header information to the OMX decoder before sending the PCM data.

	
// from omxplayer linux/PlatformDefs.h
// header for PCM audio data
typedef unsigned short  WORD;
typedef unsigned int  DWORD;
typedef struct tWAVEFORMATEX {
    WORD    format_tag;
    WORD    channels;
    DWORD   samples_per_sec;
    DWORD   avg_bytes_per_sec;
    WORD    block_align;
    WORD    bits_per_sample;
    WORD    cb_size;
} __attribute__((__packed__)) WAVEFORMATEX, *PWAVEFORMATEX, *LPWAVEFORMATEX;

int send_audio_decoder_config(COMPONENT_T *component, FILE *out)
{
    WAVEFORMATEX wave_header = 
	{.format_tag =  0x0001, //WAVE_FORMAT_PCM
	 .channels = channels,
         .samples_per_sec = sample_rate, 
	 .avg_bytes_per_sec = 0,
         .block_align = block_align, 
	 .bits_per_sample = bits_per_sample, 
	 .cb_size = 0};

    OMX_ERRORTYPE omx_err   = OMX_ErrorNone;

    OMX_BUFFERHEADERTYPE *omx_buffer = ilclient_get_input_buffer(component,
								 120,
								 1 /* block */);
    
    if(omx_buffer == NULL) {
	    fprintf(stderr, "%s - buffer error 0x%08x", __func__, omx_err);
	    return 0;
    }
    
    omx_buffer->nOffset = 0;
    omx_buffer->nFilledLen = sizeof(wave_header);
    if(omx_buffer->nFilledLen > omx_buffer->nAllocLen) {
	    fprintf(stderr, "%s - omx_buffer->nFilledLen > omx_buffer->nAllocLen",  __func__);
	    return 0;
    }
    
    memset((unsigned char *)omx_buffer->pBuffer, 0x0, 
	   omx_buffer->nAllocLen);
    memcpy((unsigned char *)omx_buffer->pBuffer, &wave_header, 
	   sizeof(wave_header));
    omx_buffer->nFlags = OMX_BUFFERFLAG_CODECCONFIG | OMX_BUFFERFLAG_ENDOFFRAME;
    
    omx_err =  OMX_EmptyThisBuffer(ilclient_get_handle(component),
				   omx_buffer);
    exit_on_omx_error(omx_err, "Error setting up audio config %s\n");

    return 1;
}
	
      

In the program that follows, a switch AUDIO_DECODE is used to toggle whether or not the audio decode component is inserted into the audio pipeline. I've done that in case you want to use the code specifically in one or the other modes.


      

Conclusion

This chapter has looked at playing files containing both audio and video streams. We have only considered a limited number of formats: MP4 files containing H.264 video and AAC audio. The player omxplayer deals with far more cases and hence has increased complexity. Nevertheless, the concepts discussed and demonstrated here are applicable in these other cases too.


      

Copyright © Jan Newmarch, jan@newmarch.name
Creative Commons License
" Programming AudioVideo on the Raspberry Pi GPU " by Jan Newmarch is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License .
Based on a work at https://jan.newmarch.name/RPi/ .

If you like this book, please contribute using PayPal

Or Flattr me:
Flattr this book