Upto: Table of Contents of full book "Programming and Using Linux Sound"

Displaying video with overlays using Gtk and FFMpeg

This chapter has nothing to do with sound. However, many sound applications use a GUI environment playing a video (or similar), possibly with text overlays such as subtitles. This diversion is about programming the video side of this, using FFMpeg, Gtk, Cairo and Pango. I assume that you are familiar with the concepts of widgets, events and event handlers and so on that underlie all current GUI frameworks.

Introduction

Videos often accompany audio. Karaoke oftens overlays the video with lyrics. Building an application to include video as well as audio takes us into the realm of graphical user interfaces (GUIs). This is a complex area in its own right, and deserves (and has!) many books, including my own from many years back on the X Window System and Motif.

Motif lost its status as a major GUI for Linux/Unix systems a long time ago. There are many alternatives now, including Gtk (Gimp toolkit), tcl/Tk, Java Swing, KDE, XFCE, ... They each have their own adherents, domains of use, quirks, idiosyncrasies, ... No single GUI will satisfy everyone.

In this chapter I deal with Gtk. The reasons are threefold:

Nevertheless, as I struggled to get my head around Gtk, versions 2.0 versus 3.0, Cairo, Pango, Glib, etc. I thought it might have been easier just to fix the Java MIDI engine! This was not a pleasant experience, as the sequel will show :-(.

FFMpeg

To play Mpeg files, OGV files or similar, a decoder is required. The main contenders seem to be GStreamer and FFMpeg. For no particular reason I chose FFMpeg.

The following program reads from a video file and stores the first five frames to disk. It is taken directly from An ffmpeg and SDL Tutorial by Stephen Dranger. The program is play_video.c:

	
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>

/* Requires
   libavcodec-dev
   libavformat-dev
   libswscale
*/

void SaveFrame(AVFrame *pFrame, int width, int height, int iFrame) {
    FILE *pFile;
    char szFilename[32];
    int  y;
  
    // Open file
    sprintf(szFilename, "frame%d.ppm", iFrame);
    pFile=fopen(szFilename, "wb");
    if(pFile==NULL)
	return;
  
    // Write header
    fprintf(pFile, "P6\n%d %d\n255\n", width, height);
  
    // Write pixel data
    for(y=0; y<height; y++)
	fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, width*3, pFile);
  
    // Close file
    fclose(pFile);
}

main(int argc, char **argv) {
    AVFormatContext *pFormatCtx = NULL;
    int i, videoStream;
    AVCodecContext *pCodecCtx = NULL;
    AVCodec *pCodec = NULL;
    AVFrame *pFrame = NULL;
    AVFrame *pFrameRGB = NULL;
    AVPacket packet;
    int frameFinished;
    int numBytes;
    uint8_t *buffer = NULL;

    AVDictionary *optionsDict = NULL;
    struct SwsContext *sws_ctx = NULL;
  

    if(argc < 2) {
	printf("Please provide a movie file\n");
	return -1;
    }
    // Register all formats and codecs
    av_register_all();
  
    // Open video file
    if(avformat_open_input(&pFormatCtx, argv[1], NULL, NULL)!=0)
	return -1; // Couldn't open file
  
    // Retrieve stream information
    if(avformat_find_stream_info(pFormatCtx, NULL)<0)
	return -1; // Couldn't find stream information
  
    // Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, argv[1], 0);
  
    // Find the first video stream
    videoStream=-1;
    for(i=0; i<pFormatCtx->nb_streams; i++)
	if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
	    videoStream=i;
	    break;
	}
    if(videoStream==-1)
	return -1; // Didn't find a video stream
  
    // Get a pointer to the codec context for the video stream
    pCodecCtx=pFormatCtx->streams[videoStream]->codec;
  
    // Find the decoder for the video stream
    pCodec=avcodec_find_decoder(pCodecCtx->codec_id);
    if(pCodec==NULL) {
	fprintf(stderr, "Unsupported codec!\n");
	return -1; // Codec not found
    }
    // Open codec
    if(avcodec_open2(pCodecCtx, pCodec, &optionsDict)<0)
	return -1; // Could not open codec
  
    // Allocate video frame
    pFrame=avcodec_alloc_frame();
  
    // Allocate an AVFrame structure
    pFrameRGB=avcodec_alloc_frame();
    if(pFrameRGB==NULL)
	return -1;
  
    // Determine required buffer size and allocate buffer
    numBytes=avpicture_get_size(PIX_FMT_RGB24, pCodecCtx->width,
				pCodecCtx->height);
    buffer=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));

    
    sws_ctx =
	sws_getContext
	(
	 pCodecCtx->width,
	 pCodecCtx->height,
	 pCodecCtx->pix_fmt,
	 pCodecCtx->width,
	 pCodecCtx->height,
	 PIX_FMT_RGB24,
	 SWS_BILINEAR,
	 NULL,
	 NULL,
	 NULL
	 );
    

    // Assign appropriate parts of buffer to image planes in pFrameRGB
    // Note that pFrameRGB is an AVFrame, but AVFrame is a superset
    // of AVPicture
    avpicture_fill((AVPicture *)pFrameRGB, buffer, PIX_FMT_RGB24,
		   pCodecCtx->width, pCodecCtx->height);

    // Read frames and save first five frames to disk
    i=0;
    while(av_read_frame(pFormatCtx, &packet)>=0) {
	// Is this a packet from the video stream?
	if(packet.stream_index==videoStream) {
	    // Decode video frame
	    avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished,
				  &packet);
      
	    // Did we get a video frame?
	    if(frameFinished) {
		// Convert the image from its native format to RGB
		sws_scale
		    (
		     sws_ctx,
		     (uint8_t const * const *)pFrame->data,
		     pFrame->linesize,
		     0,
		     pCodecCtx->height,
		     pFrameRGB->data,
		     pFrameRGB->linesize
		     );

		printf("Read frame\n");
		// Save the frame to disk
		if(++i<=5)
		    SaveFrame(pFrameRGB, pCodecCtx->width, pCodecCtx->height,
			      i);
		else
		    break;
	    }
	}
    
	// Free the packet that was allocated by av_read_frame
	av_free_packet(&packet);
    }
  
    // Free the RGB image
    av_free(buffer);
    av_free(pFrameRGB);
  
    // Free the YUV frame
    av_free(pFrame);
  
    // Close the codec
    avcodec_close(pCodecCtx);
  
    // Close the video file
    avformat_close_input(&pFormatCtx);
  
    return 0;

}


      

Basic Gtk

Gtk is a fairly standard GUI toolkit. Simple programs are described in many tutorials such as First programs in GTK+ Refer to such tutorials for the basics in Gtk programming.

We just include the following example without explanation which uses three child widgets, two buttons and one label. The label will hold an integer number. The buttons will increase or decrease this number.

	#include <gtk/gtk.h>

	gint count = 0;
	char buf[5];

	void increase(GtkWidget *widget, gpointer label)
	{
  	    count++;

	    sprintf(buf, "%d", count);
	    gtk_label_set_text(GTK_LABEL(label), buf);
	}

	void decrease(GtkWidget *widget, gpointer label)
	{
	    count--;

	    sprintf(buf, "%d", count);
	    gtk_label_set_text(GTK_LABEL(label), buf);
	}

	int main(int argc, char** argv) {

	    GtkWidget *label;
	    GtkWidget *window;
	    GtkWidget *frame;
	    GtkWidget *plus;
	    GtkWidget *minus;

	    gtk_init(&argc, &argv);

	    window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
	    gtk_window_set_position(GTK_WINDOW(window), GTK_WIN_POS_CENTER);
	    gtk_window_set_default_size(GTK_WINDOW(window), 250, 180);
	    gtk_window_set_title(GTK_WINDOW(window), "+-");

	    frame = gtk_fixed_new();
	    gtk_container_add(GTK_CONTAINER(window), frame);

	    plus = gtk_button_new_with_label("+");
	    gtk_widget_set_size_request(plus, 80, 35);
	    gtk_fixed_put(GTK_FIXED(frame), plus, 50, 20);

	    minus = gtk_button_new_with_label("-");
	    gtk_widget_set_size_request(minus, 80, 35);
	    gtk_fixed_put(GTK_FIXED(frame), minus, 50, 80);

	    label = gtk_label_new("0");
	    gtk_fixed_put(GTK_FIXED(frame), label, 190, 58); 

	    gtk_widget_show_all(window);

	    g_signal_connect(window, "destroy",
	    G_CALLBACK (gtk_main_quit), NULL);

	    g_signal_connect(plus, "clicked", 
	    G_CALLBACK(increase), label);

	    g_signal_connect(minus, "clicked", 
	    G_CALLBACK(decrease), label);

	    gtk_main();

	    return 0;
	}
      

Gtk, like every other GUI toolkit, has a large number of widgets. These are listed in the GTK+ 3 Reference Manual . This includes the widget GtkImage . As would be expected from the name, it can take a set of pixels from somewhere and build them into an image which can be displayed.

The following example is from CS 23 Software Design and Implementation Lecture notes GTK+ Programming and shows an image loaded from a file:

#include <gtk/gtk.h>

int main( int argc, char *argv[])
{
	GtkWidget *window, *image;

	gtk_init(&argc, &argv);

	window = gtk_window_new(GTK_WINDOW_TOPLEVEL);

	gtk_window_set_position(GTK_WINDOW(window), GTK_WIN_POS_CENTER);
	gtk_window_set_default_size(GTK_WINDOW(window), 230, 150);
	gtk_window_set_title(GTK_WINDOW(window), "Image");
	gtk_window_set_resizable(GTK_WINDOW(window), FALSE);

	gtk_container_set_border_width(GTK_CONTAINER(window), 2);

	image = gtk_image_new_from_file("pic/60cm.gif");
	gtk_container_add(GTK_CONTAINER(window), image);

	g_signal_connect(G_OBJECT(window), "destroy", G_CALLBACK(gtk_main_quit), NULL);
	gtk_widget_show_all(window);
	gtk_main();

	return 0;
}
      

Versions of Gtk

Gtk currently (at July 2013) has major versions 2 and 3. The macro GTK_MAJOR_VERSION can be used to detect version 2 or 3. However, Gtk also depends on a number of other libraries and it can get very confusing working out which documentation pages you should be looking at. Here is a list of the principal libraries and their primary API pages:

Gtk 3
Gtk 2

Displaying the video using Gtk

We want to take the images produced by FFMpeg as AVFrame's and display them in a GtkImage. We don't want to use code that reads from a file, because reading and writing files at 30 frames per second would be ludicrous. Instead we want some in-memory representation of the frames to load into the GtkImage.

Here is where we hit our first snag: the suitable in-memory representation changed in an incompatable way between Gtk 2.0 and Gtk 3.0. I'm only going to talk in the language of the X Window System since I don't know about other underlying systems such as Microsoft Windows.

See Migrating from GTK+ 2.x to GTK+ 3 for a description of some of the changes between these versions.

Pixmaps

The X Window System architecture model is a client-server model which has clients (applications) talking to servers (devices with graphic displays and input devices). At the lowest level (Xlib) a client will send basic requests such as "draw a line from here to there" to the server. The server will draw the line using information on the server side such as current line thickness, colour, etc.

If you want to keep an array of pixels to represent an image, then this array is usually kept on the X Window server in a pixmap. Pixmaps can be created and modified by applications by sending messages across the wire from the client to the server. Even a simple modification such as changing the value of a single pixel involves a network roundtrip and this can obviously become very expensive if done often.

Pixbufs

Pixbufs are client side equivalents of pixmaps. They can be manipulated by the client without round trips to the X server. This reduces time and network overheads in manipulating them. However, it means that information that would have been kept on the server now has to built and maintained on the client application side.

X, Wayland and Mir

The X Window System is nearly 30 years old. During that time it has evolved to meet changes in hardware and in software requirements, while still maintaining backward compatability.

Significant changes have occurred in hardware during this 30 years: multi-core systems are now prevalent and GPUs have brought changes in processing video. And generally the amount of memory (cache and RAM) means that memory is no longer such an issue.

At the same time, the software side has evolved. It is now common to make use of a "compositing window manager" such as Compiz so that you can have effects such as Wobbly Windows. This is not good for the X Window model: requests from the application go to the X server but then a requested image has to be passed to the compositing window manager which will perform its effects and then send images back to the X server. This is big increase in network traffic, in which the X server is now just playing the role of display rather than compositor.

Application libraries have now evolved so that much of the work that was formerly done by the X server can now be done on the application side by libraries such as cairo, pixman, freetype, fontconfig and pango.

All of these changes have led to proposals for new backend servers which live cooperatively in this evolved world. This has been sparked by the development of Wayland, but is a bit messed up by Ubuntu forking this to develop Mir. We won't buy into those arguments - just google for "mir and wayland"...

In a very simplistic sense, what it means here is that in future pixmaps are out while pixbufs are in.

Gtk 3.0

With Gtk 3.0, pixmaps no longer exist. You only have pixbufs, in the data structure GdkPixbuf. To display the FFMpeg decoded video we pick up after the image has been transcoded to the picture_RGB, translate it into a GdkPixbuf and create the GtkImage

	pixbuf = gdk_pixbuf_new_from_data(picture_RGB->data[0], GDK_COLORSPACE_RGB,
	                                  0, 8, width, height, 
	                                  picture_RGB->linesize[0], 
                                          pixmap_destroy_notify,
	                                  NULL);
	gtk_image_set_from_pixbuf((GtkImage*) image, pixbuf);
      

Gtk 2.0

Gtk 2.0 still had pixmaps, in the structure GdkPixmap. In theory it should be possible to have code similar to the Gtk 3.0 code using the function GdkPixmap *gdk_pixmap_create_from_data(GdkDrawable *drawable, const gchar *data, gint width, gint height, gint depth, const GdkColor *fg, const GdkColor *bg) which is documented in the GDK 2 Reference Manual at Bitmaps and Pixmaps and then call void gtk_image_set_from_pixmap(GtkImage *image, GdkPixmap *pixmap, GdkBitmap *mask) documented in the Gtk 2.6 reference manual at GtkImage .

The only problem is that I couldn't get the function gdk_pixmap_create_from_data to work. No matter what argument I tried for the drawable, the call always barfed on either its type or its value. For example, a documented value is NULL but this always caused an assertion error ("should not be NULL").

So what does work? Well, all I could find was a bit of a mess of both pixmaps and pixbufs: create a pixbuf filled with video data, create a pixmap, write the pixbuf data into the pixmap, and then fill the image with the pixmap data:

	pixbuf = gdk_pixbuf_new_from_data(picture_RGB->data[0], GDK_COLORSPACE_RGB,
             	                          0, 8, width, height, 
	                                  picture_RGB->linesize[0], 
	                                  pixmap_destroy_notify,
	                                  NULL);
	pixmap = gdk_pixmap_new(window->window, width, height, -1);
	gdk_draw_pixbuf((GdkDrawable *) pixmap, NULL,
	                pixbuf, 
	                0, 0, 0, 0, wifth, height,
	                GDK_RGB_DITHER_NORMAL, 0, 0);

	gtk_image_set_from_pixmap((GtkImage*) image, pixmap, NULL);
      

Threads and Gtk

The video will need to play in its own thread. Gtk will set up a GUI processing loop in its thread. Since this is Linux, we will use Posix pthreads. The video thread will need to be started explicitly by

	pthread_t tid;
	pthread_create(&tid, NULL, play_background, NULL);
      

where the function play_background calls the FFMpeg code to decode the video file. Note that the thread should not be started until the application has been realised, or it will attempt to draw into non-existent windows.

The Gtk thread will be started by the call to

	gtk_widget_show (window);
      

That's straightforward enough. But now we have to handle the video thread making calls into the GUI thread in order to draw the image. The best document I have found on this is Is GTK+ thread safe? How do I write multi-threaded GTK+ applications? Basically it states that code which can affect the Gtk thread should be enclosed by a gdk_threads_enter() ... gdk_threads_leave() pair.

That's okay for Gtk 2.0. What about Gtk 3.0? Ooops! Those calls are now deprecated. So what are you supposed to do? So far (as at 3 July, 2013) all that seems to exist are developer dialogs such as Re: deprecating gdk threads which states "We have never done a great job of explaining when gdk_threads_enter/leave are required, it seems - as a consequence, a good number of the critical sections I've seen marked throughout my jhbuild checkout are unnecessary. If your application doesn't call gdk_threads_init or gdk_threads_set_lock_functions, there's no need to use enter/leave. Libraries are a different story, of course". But who should call gdk_threads_init? And what's this about libraries? I'll continue to use them until I know better.

The code

Finally, the code to play a video in a Gtk application which works with both Gtk 2.0 and Gtk 3.0! It is gtk_play_video.c:

	
#include <gtk/gtk.h>
#include <gdk/gdkx.h>

#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>

GtkWidget *image;
GtkWidget *window;

#if GTK_MAJOR_VERSION == 2
GdkPixmap *pixmap;
#endif

/* FFMpeg vbls */
AVFormatContext *pFormatCtx = NULL;
AVCodecContext *pCodecCtx = NULL;
int videoStream;
struct SwsContext *sws_ctx = NULL;
AVCodec *pCodec = NULL;

#define WIDTH 192
#define HEIGHT 144

static void pixmap_destroy_notify(guchar *pixels,
				  gpointer data) {
    printf("Destroy pixmap - not sure how\n");
}

static gpointer play_background(gpointer args) {
    /* based on code from
       http://www.cs.dartmouth.edu/~xy/cs23/gtk.html
       http://cdry.wordpress.com/2009/09/09/using-custom-io-callbacks-with-ffmpeg/
    */

    int i;
    AVPacket packet;
    int frameFinished;
    AVFrame *pFrame = NULL;

    /* initialize packet, set data to NULL, let the demuxer fill it */
    /* http://ffmpeg.org/doxygen/trunk/doc_2examples_2demuxing_8c-example.html#a80 */
    av_init_packet(&packet);
    packet.data = NULL;
    packet.size = 0;

    int bytesDecoded;
    GdkPixbuf *pixbuf;
    AVFrame *picture_RGB;
    char *buffer;

    pFrame=avcodec_alloc_frame();

    i=0;
    picture_RGB = avcodec_alloc_frame();
    buffer = malloc (avpicture_get_size(PIX_FMT_RGB24, WIDTH, HEIGHT));
    avpicture_fill((AVPicture *)picture_RGB, buffer, PIX_FMT_RGB24, WIDTH, HEIGHT);

    while(av_read_frame(pFormatCtx, &packet)>=0) {
	if(packet.stream_index==videoStream) {
	    usleep(33670);  // 29.7 frames per second
	    // Decode video frame
	    avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished,
				  &packet);

	    int width = pCodecCtx->width;
	    int height = pCodecCtx->height;
	    
	    sws_ctx = sws_getContext(width, height, pCodecCtx->pix_fmt, width, height, 
				     PIX_FMT_RGB24, SWS_BICUBIC, NULL, NULL, NULL);

	    if (frameFinished) {
		printf("Frame %d width %d height %d\n", i++, width, height);
		
		sws_scale(sws_ctx,  (uint8_t const * const *) pFrame->data, pFrame->linesize, 0, height, picture_RGB->data, picture_RGB->linesize);
		
		printf("old width %d new width %d\n",  pCodecCtx->width, picture_RGB->width);
		pixbuf = gdk_pixbuf_new_from_data(picture_RGB->data[0], GDK_COLORSPACE_RGB,
						  0, 8, width, height, 
						  picture_RGB->linesize[0], pixmap_destroy_notify,
						  NULL);
				


		/* start GTK thread lock for drawing */
		gdk_threads_enter();    

#if  GTK_MAJOR_VERSION == 2
		/* this leaks memory somehow */
		pixmap = gdk_pixmap_new(window->window, WIDTH, HEIGHT, -1);
	
		gdk_draw_pixbuf((GdkDrawable *) pixmap, NULL,
				pixbuf, 
				0, 0, 0, 0, WIDTH, HEIGHT,
				GDK_RGB_DITHER_NORMAL, 0, 0);

		gtk_image_set_from_pixmap((GtkImage*) image, pixmap, NULL);

		//gtk_widget_queue_draw(image);
#elif GTK_MAJOR_VERSION == 3
		gtk_image_set_from_pixbuf((GtkImage*) image, pixbuf);
		
		//gtk_widget_queue_draw(image);
		
#endif	      
		//g_object_unref(pixbuf);
		//sws_freeContext(sws_ctx);
		/* release GTK thread lock */
		gdk_threads_leave();
	    }
	}
	av_free_packet(&packet);
	g_thread_yield();
    }

    printf("Video over!\n");
    exit(0);
}

/* Called when the windows are realized
 */
static void realize_cb (GtkWidget *widget, gpointer data) {
    /* start the video playing in its own thread */
    GThread *tid;
    tid = g_thread_new("video",
		       play_background, 
		       NULL);
}

static gboolean delete_event( GtkWidget *widget,
                              GdkEvent  *event,
                              gpointer   data )
{
    /* If you return FALSE in the "delete-event" signal handler,
     * GTK will emit the "destroy" signal. Returning TRUE means
     * you don't want the window to be destroyed.
     * This is useful for popping up 'are you sure you want to quit?'
     * type dialogs. */

    g_print ("delete event occurred\n");

    /* Change TRUE to FALSE and the main window will be destroyed with
     * a "delete-event". */
    return TRUE;
}

/* Another callback */
static void destroy( GtkWidget *widget,
                     gpointer   data )
{
    gtk_main_quit ();
}

int main(int argc, char** argv)
{
    // Is this necessary?
    XInitThreads();

    int i;


    /* FFMpeg stuff */

    AVFrame *pFrame = NULL;
    AVPacket packet;

    AVDictionary *optionsDict = NULL;

    av_register_all();

    if(avformat_open_input(&pFormatCtx, "/home/httpd/html/ComputersComputing/simpson.mpg", NULL, NULL)!=0)
	return -1; // Couldn't open file
  
    // Retrieve stream information
    if(avformat_find_stream_info(pFormatCtx, NULL)<0)
	return -1; // Couldn't find stream information
  
    // Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, argv[1], 0);
  
    // Find the first video stream
    videoStream=-1;
    for(i=0; i<pFormatCtx->nb_streams; i++)
	if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
	    videoStream=i;
	    break;
	}
    if(videoStream==-1)
	return -1; // Didn't find a video stream

    for(i=0; i<pFormatCtx->nb_streams; i++)
	if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO) {
	    printf("Found an audio stream too\n");
	    break;
	}

  
    // Get a pointer to the codec context for the video stream
    pCodecCtx=pFormatCtx->streams[videoStream]->codec;
  
    // Find the decoder for the video stream
    pCodec=avcodec_find_decoder(pCodecCtx->codec_id);
    if(pCodec==NULL) {
	fprintf(stderr, "Unsupported codec!\n");
	return -1; // Codec not found
    }
  
    // Open codec
    if(avcodec_open2(pCodecCtx, pCodec, &optionsDict)<0)
	return -1; // Could not open codec

    sws_ctx =
	sws_getContext
	(
	 pCodecCtx->width,
	 pCodecCtx->height,
	 pCodecCtx->pix_fmt,
	 pCodecCtx->width,
	 pCodecCtx->height,
	 PIX_FMT_YUV420P,
	 SWS_BILINEAR,
	 NULL,
	 NULL,
	 NULL
	 );

    /* GTK stuff now */

     /* This is called in all GTK applications. Arguments are parsed
     * from the command line and are returned to the application. */
    gtk_init (&argc, &argv);
    
    /* create a new window */
    window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    
    /* When the window is given the "delete-event" signal (this is given
     * by the window manager, usually by the "close" option, or on the
     * titlebar), we ask it to call the delete_event () function
     * as defined above. The data passed to the callback
     * function is NULL and is ignored in the callback function. */
    g_signal_connect (window, "delete-event",
		      G_CALLBACK (delete_event), NULL);
    
    /* Here we connect the "destroy" event to a signal handler.  
     * This event occurs when we call gtk_widget_destroy() on the window,
     * or if we return FALSE in the "delete-event" callback. */
    g_signal_connect (window, "destroy",
		      G_CALLBACK (destroy), NULL);

    g_signal_connect (window, "realize", G_CALLBACK (realize_cb), NULL);
    
    /* Sets the border width of the window. */
    gtk_container_set_border_width (GTK_CONTAINER (window), 10);

    image = gtk_image_new();
    gtk_widget_show (image);

    /* This packs the button into the window (a gtk container). */
    gtk_container_add (GTK_CONTAINER (window), image);
        
    /* and the window */
    gtk_widget_show (window);
    
    /* All GTK applications must have a gtk_main(). Control ends here
     * and waits for an event to occur (like a key press or
     * mouse event). */
    gtk_main ();
    
    return 0;
}


      

Overlaying an image on top of an image

It is common in a movie on TV to see a fixed image layered on top of the video. Subtitles can be an example of dynamic images but may be text overlaid instead. This section just considers one image on top of another.

In Gtk 2.0 it is surprisingly easy: draw one pixbuf into a pixmap and then draw the overlay pixbuf into the same pixmap.

pixmap = gdk_pixmap_new(window->window, 720, 480, -1);
	
gdk_draw_pixbuf((GdkDrawable *) pixmap, NULL,
		pixbuf, 
		0, 0, 0, 0, 720, 480,
		GDK_RGB_DITHER_NORMAL, 0, 0);

// overlay another pixbuf
gdk_draw_pixbuf((GdkDrawable *) pixmap, NULL,
                overlay_pixbuf, 
                0, 0, 0, 0, overlay_width, overlay_height,
                GDK_RGB_DITHER_NORMAL, 0, 0);

gtk_image_set_from_pixmap((GtkImage*) image, pixmap, NULL);

gtk_widget_queue_draw(image);
      

Gtk 3.0 does not seem so straightforward as pixmaps have disappeared. Various pages suggest using Cairo surfaces instead and later sections will look at that. But the page on the The GdkPixbuf Structure suggests that - as long as you get the data types aligned - you can just write the pixels of the second image into the pixbuf data of the first. The page (although old) Gdk-pixbuf is a useful tutorial on Gdk pixbufs. One of the details you have to get right is the rowstride of each image: the two-dimensional image is stored as a linear array of bytes and the rowstride tells how many bytes make up a row. Typically there are 3 or 4 bytes per pixel (for RGB or RGB+alpha) and these also need to be matched between the images.

The program gtk_play_video_overlay.c is

	
#include <gtk/gtk.h>
#include <gdk/gdkx.h>

#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>

//#define OVERLAY_IMAGE "/home/newmarch/jannewmarch.signature.png"
#define OVERLAY_IMAGE "jan-small.png"
#define VIDEO "short.mpg"
//#define VIDEO "video1.mp4"

GtkWidget *image;
GtkWidget *window;

#if GTK_MAJOR_VERSION == 2
GdkPixmap *pixmap;
#endif

/* FFMpeg vbls */
AVFormatContext *pFormatCtx = NULL;
AVCodecContext *pCodecCtx = NULL;
int videoStream;
struct SwsContext *sws_ctx = NULL;
AVCodec *pCodec = NULL;



static void pixmap_destroy_notify(guchar *pixels,
				  gpointer data) {
    printf("Destroy pixmap - not sure how\n");
}

#if  GTK_MAJOR_VERSION == 3
static void overlay(GdkPixbuf *pixbuf, GdkPixbuf *overlay_pixbuf, 
			 int height_offset, int width_offset) {
    int overlay_width, overlay_height, overlay_rowstride, overlay_n_channels;
    guchar *overlay_pixels, *overlay_p;
    guchar red, green, blue, alpha;
    int m, n;
    int rowstride, n_channels, width, height;
    guchar *pixels, *p;

    if (overlay_pixbuf == NULL) {
        return;
    }

    /* get stuff out of overlay pixbuf */
    overlay_n_channels = gdk_pixbuf_get_n_channels (overlay_pixbuf);
    n_channels =  gdk_pixbuf_get_n_channels(pixbuf);
    printf("Overlay has %d channels, destination has %d channels\n",
           overlay_n_channels, n_channels);
    overlay_width = gdk_pixbuf_get_width (overlay_pixbuf);
    overlay_height = gdk_pixbuf_get_height (overlay_pixbuf);

    overlay_rowstride = gdk_pixbuf_get_rowstride (overlay_pixbuf);
    overlay_pixels = gdk_pixbuf_get_pixels (overlay_pixbuf);

    rowstride = gdk_pixbuf_get_rowstride (pixbuf);
    width = gdk_pixbuf_get_width (pixbuf);
    pixels = gdk_pixbuf_get_pixels (pixbuf);

    printf("Overlay: width %d str8ide %d\n", overlay_width, overlay_rowstride);
    printf("Dest: width  str8ide %d\n", rowstride);

    for (m = 0; m < overlay_width; m++) {
        for (n = 0; n < overlay_height; n++) {
            overlay_p = overlay_pixels + n * overlay_rowstride + m * overlay_n_channels;
            red = overlay_p[0];
            green = overlay_p[1];
            blue = overlay_p[2];
            if (overlay_n_channels == 4)
		alpha = overlay_p[3];
	    else
		alpha = 0;

            p = pixels + (n+height_offset) * rowstride + (m+width_offset) * n_channels;
            p[0] = red;
            p[1] = green;
            p[2] = blue;
            if (n_channels == 4)
                p[3] = alpha;
        }
    }
}
#endif /* GTK_MAJOR_VERSION == 3 */

static void *play_background(void *args) {
    /* based on code from
       http://www.cs.dartmouth.edu/~xy/cs23/gtk.html
       http://cdry.wordpress.com/2009/09/09/using-custom-io-callbacks-with-ffmpeg/
    */

    int i;
    AVPacket packet;
    int frameFinished;
    AVFrame *pFrame = NULL;

    int bytesDecoded;
    GdkPixbuf *pixbuf;
    GdkPixbuf *overlay_pixbuf;
    AVFrame *picture_RGB;
    char *buffer;

    GError *error = NULL;
    overlay_pixbuf = gdk_pixbuf_new_from_file(OVERLAY_IMAGE, &error);
    if (!overlay_pixbuf) {
	fprintf(stderr, "%s\n", error->message);
	g_error_free(error);
	exit(1);
    }
    int overlay_width = gdk_pixbuf_get_width(overlay_pixbuf);
    int overlay_height =  gdk_pixbuf_get_height(overlay_pixbuf);

    pFrame=avcodec_alloc_frame();

    i=0;
    picture_RGB = avcodec_alloc_frame();
    buffer = malloc (avpicture_get_size(PIX_FMT_RGB24, 720, 576));
    avpicture_fill((AVPicture *)picture_RGB, buffer, PIX_FMT_RGB24, 720, 576);

    while(av_read_frame(pFormatCtx, &packet)>=0) {
	if(packet.stream_index==videoStream) {
	    usleep(33670);  // 29.7 frames per second
	    // Decode video frame
	    avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished,
				  &packet);
	    int width = pCodecCtx->width;
	    int height = pCodecCtx->height;
	    
	    sws_ctx = sws_getContext(width, height, pCodecCtx->pix_fmt, width, height, 
				     PIX_FMT_RGB24, SWS_BICUBIC, NULL, NULL, NULL);

	    if (frameFinished) {
		printf("Frame %d\n", i++);
		
		sws_scale(sws_ctx,  (uint8_t const * const *) pFrame->data, pFrame->linesize, 0, height, picture_RGB->data, picture_RGB->linesize);
		
		printf("old width %d new width %d\n",  pCodecCtx->width, picture_RGB->width);
		pixbuf = gdk_pixbuf_new_from_data(picture_RGB->data[0], GDK_COLORSPACE_RGB,
						  0, 8, width, height, 
						  picture_RGB->linesize[0], pixmap_destroy_notify,
						  NULL);



		/* start GTK thread lock for drawing */
		gdk_threads_enter();    

#if  GTK_MAJOR_VERSION == 2
		pixmap = gdk_pixmap_new(window->window, 720, 480, -1);
	
		gdk_draw_pixbuf((GdkDrawable *) pixmap, NULL,
				pixbuf, 
				0, 0, 0, 0, 720, 480,
				GDK_RGB_DITHER_NORMAL, 0, 0);

		// overlay another pixbuf
		gdk_draw_pixbuf((GdkDrawable *) pixmap, NULL,
                                overlay_pixbuf, 
                                0, 0, 0, 0, overlay_width, overlay_height,
                                GDK_RGB_DITHER_NORMAL, 0, 0);

		gtk_image_set_from_pixmap((GtkImage*) image, pixmap, NULL);

		gtk_widget_queue_draw(image);
#elif GTK_MAJOR_VERSION == 3
		overlay(pixbuf, overlay_pixbuf, 300, 200);

		gtk_image_set_from_pixbuf((GtkImage*) image, pixbuf); 

		gtk_widget_queue_draw(image);
#endif	      
		//g_object_unref(pixbuf);
		//sws_freeContext(sws_ctx);
		/* release GTK thread lock */
		gdk_threads_leave();
	    }
	}
	av_free_packet(&packet);
    }

    printf("Video over!\n");
    exit(0);
}

/* Called when the windows are realized
 */
static void realize_cb (GtkWidget *widget, gpointer data) {
    /* start the video playing in its own thread */
    pthread_t tid;
    pthread_create(&tid, NULL, play_background, NULL);
}

static gboolean delete_event( GtkWidget *widget,
                              GdkEvent  *event,
                              gpointer   data )
{
    /* If you return FALSE in the "delete-event" signal handler,
     * GTK will emit the "destroy" signal. Returning TRUE means
     * you don't want the window to be destroyed.
     * This is useful for popping up 'are you sure you want to quit?'
     * type dialogs. */

    g_print ("delete event occurred\n");

    /* Change TRUE to FALSE and the main window will be destroyed with
     * a "delete-event". */
    return TRUE;
}

/* Another callback */
static void destroy( GtkWidget *widget,
                     gpointer   data )
{
    gtk_main_quit ();
}

int main(int argc, char** argv)
{
    // Is this necessary?
    XInitThreads();

    int i;


    /* FFMpeg stuff */

    AVFrame *pFrame = NULL;
    AVPacket packet;

    AVDictionary *optionsDict = NULL;

    av_register_all();

    if(avformat_open_input(&pFormatCtx, VIDEO, NULL, NULL)!=0)
	return -1; // Couldn't open file
  
    // Retrieve stream information
    if(avformat_find_stream_info(pFormatCtx, NULL)<0)
	return -1; // Couldn't find stream information
  
    // Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, argv[1], 0);
  
    // Find the first video stream
    videoStream=-1;
    for(i=0; i<pFormatCtx->nb_streams; i++)
	if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
	    videoStream=i;
	    break;
	}
    if(videoStream==-1)
	return -1; // Didn't find a video stream

    for(i=0; i<pFormatCtx->nb_streams; i++)
	if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO) {
	    printf("Found an audio stream too\n");
	    break;
	}

  
    // Get a pointer to the codec context for the video stream
    pCodecCtx=pFormatCtx->streams[videoStream]->codec;
  
    // Find the decoder for the video stream
    pCodec=avcodec_find_decoder(pCodecCtx->codec_id);
    if(pCodec==NULL) {
	fprintf(stderr, "Unsupported codec!\n");
	return -1; // Codec not found
    }
  
    // Open codec
    if(avcodec_open2(pCodecCtx, pCodec, &optionsDict)<0)
	return -1; // Could not open codec

    sws_ctx =
	sws_getContext
	(
	 pCodecCtx->width,
	 pCodecCtx->height,
	 pCodecCtx->pix_fmt,
	 pCodecCtx->width,
	 pCodecCtx->height,
	 PIX_FMT_YUV420P,
	 SWS_BILINEAR,
	 NULL,
	 NULL,
	 NULL
	 );

    /* GTK stuff now */

    /* This is called in all GTK applications. Arguments are parsed
     * from the command line and are returned to the application. */
    gtk_init (&argc, &argv);
    
    /* create a new window */
    window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    
    /* When the window is given the "delete-event" signal (this is given
     * by the window manager, usually by the "close" option, or on the
     * titlebar), we ask it to call the delete_event () function
     * as defined above. The data passed to the callback
     * function is NULL and is ignored in the callback function. */
    g_signal_connect (window, "delete-event",
		      G_CALLBACK (delete_event), NULL);
    
    /* Here we connect the "destroy" event to a signal handler.  
     * This event occurs when we call gtk_widget_destroy() on the window,
     * or if we return FALSE in the "delete-event" callback. */
    g_signal_connect (window, "destroy",
		      G_CALLBACK (destroy), NULL);

    g_signal_connect (window, "realize", G_CALLBACK (realize_cb), NULL);
    
    /* Sets the border width of the window. */
    gtk_container_set_border_width (GTK_CONTAINER (window), 10);

    image = gtk_image_new();
    gtk_widget_show (image);

    /* This packs the button into the window (a gtk container). */
    gtk_container_add (GTK_CONTAINER (window), image);
        
    /* and the window */
    gtk_widget_show (window);
    
    /* All GTK applications must have a gtk_main(). Control ends here
     * and waits for an event to occur (like a key press or
     * mouse event). */
    gtk_main ();
    
    return 0;
}


      

Alpha channel

An overlay image may have some "transparent" parts in it. You don't want such parts to be overlaid onto the underlying image. But such parts will need to have a value in the array of pixels. Even zero is a value - black! Some images will have another byte per pixel allocated as the "alpha channel". This has a value to show how "transparent" the pixel is. A value of 255 means not transparent at all, a value of zero means totally transparent.

The simplest way of combining a "transparent" pixel with the underlying pixel is simply to not do so: leave the underlying pixel untouched. More complex algorithms are pointed to by the Wikipedia Alpha compositing page.

Converting an image which doesn't have an alpha channel to one which does can be done using the function gdk_pixbuf_add_alpha. This can also be used to set the value of the alpha channel by matching against a colour. For example, the following should set the alpha value to 0 for any white pixels and to 255 for all others:

pixbuf = gdk_pixbuf_add_alpha(pixbuf, TRUE, 255, 255, 255);
      

Unfortunately it seems to want to leave an "edge" of pixels which should be marked as transparent.

With alpha marking in place, a simple test can be used in the overlay function as to whether or not to perform the overlay:

	
if (alpha < 128) {
    continue;
 }
	
      

It's not worth giving a complete program just for a couple of changed lines. It is gtk_play_video_overlay_alpha.c.

Using Cairo to draw on an image

With the disappearance of pixmaps from Gtk 3.0, Cairo is now the only real way of assembling multiple components into an image. General Cairo information is at Cairo documentation , a tutorial is at Cairo graphics tutorial and information about overlaying onto images is at Images in Cairo .

Cairo takes sources and a destination. The sources can be changed, and frequently are: from an image source, to a colour source, etc. The destination is where the drawn stuff ends up.

Destinations can be in memory or at a variety of backends. We want in an in-memory destination so that we can extract a pixbuf from it, with all operations done on the client side. We create a destination as a surface of type cairo_surface_t and set it into a Cairo context of typecairo_t by

cairo_surface_t *surface = cairo_image_surface_create (CAIRO_FORMAT_ARGB32, 
						       width, height);
cairo_t *cr = cairo_create(surface);
      

The cairo context cr is then used to set sources, perform drawing, etc. At the end of this we will extract a pixmap from the surface.

The first step is to set the source to the pixbuf for each frame of the video and to paint this to the destination by

gdk_cairo_set_source_pixbuf(cr, pixbuf, 0, 0);
cairo_paint (cr);
      

We can overlay another image on top of this by changing the source to the overlay image and painting that:

gdk_cairo_set_source_pixbuf(cr, overlay_pixbuf, 300, 200);
cairo_paint (cr);
      

Note that Cairo will do any alpha blending that is required if the overlay has "transparent" pixels.

To draw the text, we need to reset the source to an RGB surface, set all the parameters for the text, and draw the text into the destination. This is done by

// white text
cairo_set_source_rgb(cr, 1.0, 1.0, 1.0); 
// this is a standard font for Cairo 
cairo_select_font_face (cr, "cairo:serif",
			CAIRO_FONT_SLANT_NORMAL, 
			CAIRO_FONT_WEIGHT_BOLD);
cairo_set_font_size (cr, 20);
cairo_move_to(cr, 10.0, 50.0);
cairo_show_text (cr, "hello");
      

Finally we want to extract the fnal image from the destination and set it into the GdkImage for display. Here there is another difference between Gtk 2.0 and Gtk 3.0: Gtk 3.0 has a function gdk_pixbuf_get_from_surface which will return a GdKPixbuf; Gtk 2.0 has no such function. We only look at the Gtk 3.0 version for now

pixbuf = gdk_pixbuf_get_from_surface(surface,
				     0,
				     0,
				     width,
				     height);

gtk_image_set_from_pixbuf((GtkImage*) image, pixbuf); 
      

The complete program is gtk_play_video_cairo.c

	
#include <gtk/gtk.h>

#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>

#define OVERLAY_IMAGE "jan-small.png"

GtkWidget *image;
GtkWidget *window;

#if GTK_MAJOR_VERSION == 2
GdkPixmap *pixmap;
#endif

/* FFMpeg vbls */
AVFormatContext *pFormatCtx = NULL;
AVCodecContext *pCodecCtx = NULL;
int videoStream;
struct SwsContext *sws_ctx = NULL;
AVCodec *pCodec = NULL;



static void pixmap_destroy_notify(guchar *pixels,
				  gpointer data) {
    printf("Destroy pixmap - not sure how\n");
}

static void *play_background(void *args) {
    /* based on code from
       http://www.cs.dartmouth.edu/~xy/cs23/gtk.html
       http://cdry.wordpress.com/2009/09/09/using-custom-io-callbacks-with-ffmpeg/
    */

    int i;
    AVPacket packet;
    int frameFinished;
    AVFrame *pFrame = NULL;

    int bytesDecoded;
    GdkPixbuf *pixbuf;
    GdkPixbuf *overlay_pixbuf;
    AVFrame *picture_RGB;
    char *buffer;

    GError *error = NULL;
    overlay_pixbuf = gdk_pixbuf_new_from_file(OVERLAY_IMAGE, &error);
    if (!overlay_pixbuf) {
	fprintf(stderr, "%s\n", error->message);
	g_error_free(error);
	exit(1);
    }

    // add an alpha layer for a white background
    overlay_pixbuf = gdk_pixbuf_add_alpha(overlay_pixbuf, TRUE, 255, 255, 255);

    int overlay_width = gdk_pixbuf_get_width(overlay_pixbuf);
    int overlay_height =  gdk_pixbuf_get_height(overlay_pixbuf);

    pFrame=avcodec_alloc_frame();

    i=0;
    picture_RGB = avcodec_alloc_frame();
    buffer = malloc (avpicture_get_size(PIX_FMT_RGB24, 720, 576));
    avpicture_fill((AVPicture *)picture_RGB, buffer, PIX_FMT_RGB24, 720, 576);

    while(av_read_frame(pFormatCtx, &packet)>=0) {
	if(packet.stream_index==videoStream) {
	    usleep(33670);  // 29.7 frames per second
	    // Decode video frame
	    avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished,
				  &packet);

	    int width = pCodecCtx->width;
	    int height = pCodecCtx->height;
	    
	    sws_ctx = sws_getContext(width, height, pCodecCtx->pix_fmt, width, height, 
				     PIX_FMT_RGB24, SWS_BICUBIC, NULL, NULL, NULL);

	    if (frameFinished) {
		printf("Frame %d\n", i++);
		
		sws_scale(sws_ctx,  (uint8_t const * const *) pFrame->data, pFrame->linesize, 0, height, picture_RGB->data, picture_RGB->linesize);
		
		printf("old width %d new width %d\n",  pCodecCtx->width, picture_RGB->width);
		pixbuf = gdk_pixbuf_new_from_data(picture_RGB->data[0], GDK_COLORSPACE_RGB,
						  0, 8, width, height, 
						  picture_RGB->linesize[0], pixmap_destroy_notify,
						  NULL);

		// Create the destination surface
		cairo_surface_t *surface = cairo_image_surface_create (CAIRO_FORMAT_ARGB32, 
								       width, height);
		cairo_t *cr = cairo_create(surface);

		// draw the background image
		gdk_cairo_set_source_pixbuf(cr, pixbuf, 0, 0);
		cairo_paint (cr);

		// overlay an image on top
		// alpha blending will be done by Cairo
		gdk_cairo_set_source_pixbuf(cr, overlay_pixbuf, 300, 200);
		cairo_paint (cr);

		// draw some white text on top
		cairo_set_source_rgb(cr, 1.0, 1.0, 1.0);
		// this is a standard font for Cairo
		cairo_select_font_face (cr, "cairo:serif",
					CAIRO_FONT_SLANT_NORMAL, 
					CAIRO_FONT_WEIGHT_BOLD);
		cairo_set_font_size (cr, 20);
		cairo_move_to(cr, 10.0, 50.0);
		cairo_show_text (cr, "hello");

		/* start GTK thread lock for drawing */
		gdk_threads_enter();    

#if  GTK_MAJOR_VERSION == 2
		int pwidth, pheight, stride;
		unsigned char *data;

		data = cairo_image_surface_get_data (surface);
		pwidth = cairo_image_surface_get_width (surface);
		pheight = cairo_image_surface_get_height (surface);
		stride = cairo_image_surface_get_stride (surface);
		
		// this function doesn't work properly
		// code doesn't work
		pixmap = gdk_pixmap_create_from_data(NULL, data,
					      pwidth, pheight,
					      8, NULL, NULL);

		gtk_image_set_from_pixmap((GtkImage*) image, pixmap, NULL);

		gtk_widget_queue_draw(image);
#elif GTK_MAJOR_VERSION == 3
		pixbuf = gdk_pixbuf_get_from_surface(surface,
						     0,
						     0,
						     width,
						     height);

		gtk_image_set_from_pixbuf((GtkImage*) image, pixbuf); 

		//gtk_widget_queue_draw(image);
#endif	      
		/* reclaim memory */
		g_object_unref(pixbuf);
		//sws_freeContext(sws_ctx);
		cairo_surface_destroy(surface);
		cairo_destroy(cr);

		/* release GTK thread lock */
		gdk_threads_leave();
	    }
	}
	av_free_packet(&packet);
    }

    printf("Video over!\n");
    exit(0);
}

/* Called when the windows are realized
 */
static void realize_cb (GtkWidget *widget, gpointer data) {
    /* start the video playing in its own thread */
    pthread_t tid;
    pthread_create(&tid, NULL, play_background, NULL);
}

static gboolean delete_event( GtkWidget *widget,
                              GdkEvent  *event,
                              gpointer   data )
{
    /* If you return FALSE in the "delete-event" signal handler,
     * GTK will emit the "destroy" signal. Returning TRUE means
     * you don't want the window to be destroyed.
     * This is useful for popping up 'are you sure you want to quit?'
     * type dialogs. */

    g_print ("delete event occurred\n");

    /* Change TRUE to FALSE and the main window will be destroyed with
     * a "delete-event". */
    return TRUE;
}

/* Another callback */
static void destroy( GtkWidget *widget,
                     gpointer   data )
{
    gtk_main_quit ();
}

int main(int argc, char** argv)
{
    // Is this necessary?
    XInitThreads();

    int i;


    /* FFMpeg stuff */

    AVFrame *pFrame = NULL;
    AVPacket packet;

    AVDictionary *optionsDict = NULL;

    av_register_all();

    if(avformat_open_input(&pFormatCtx, "short.mpg", NULL, NULL)!=0)
	return -1; // Couldn't open file
  
    // Retrieve stream information
    if(avformat_find_stream_info(pFormatCtx, NULL)<0)
	return -1; // Couldn't find stream information
  
    // Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, argv[1], 0);
  
    // Find the first video stream
    videoStream=-1;
    for(i=0; i<pFormatCtx->nb_streams; i++)
	if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
	    videoStream=i;
	    break;
	}
    if(videoStream==-1)
	return -1; // Didn't find a video stream

    for(i=0; i<pFormatCtx->nb_streams; i++)
	if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO) {
	    printf("Found an audio stream too\n");
	    break;
	}

  
    // Get a pointer to the codec context for the video stream
    pCodecCtx=pFormatCtx->streams[videoStream]->codec;
  
    // Find the decoder for the video stream
    pCodec=avcodec_find_decoder(pCodecCtx->codec_id);
    if(pCodec==NULL) {
	fprintf(stderr, "Unsupported codec!\n");
	return -1; // Codec not found
    }
  
    // Open codec
    if(avcodec_open2(pCodecCtx, pCodec, &optionsDict)<0)
	return -1; // Could not open codec

    sws_ctx =
	sws_getContext
	(
	 pCodecCtx->width,
	 pCodecCtx->height,
	 pCodecCtx->pix_fmt,
	 pCodecCtx->width,
	 pCodecCtx->height,
	 PIX_FMT_YUV420P,
	 SWS_BILINEAR,
	 NULL,
	 NULL,
	 NULL
	 );

    /* GTK stuff now */

    /* This is called in all GTK applications. Arguments are parsed
     * from the command line and are returned to the application. */
    gtk_init (&argc, &argv);
    
    /* create a new window */
    window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    
    /* When the window is given the "delete-event" signal (this is given
     * by the window manager, usually by the "close" option, or on the
     * titlebar), we ask it to call the delete_event () function
     * as defined above. The data passed to the callback
     * function is NULL and is ignored in the callback function. */
    g_signal_connect (window, "delete-event",
		      G_CALLBACK (delete_event), NULL);
    
    /* Here we connect the "destroy" event to a signal handler.  
     * This event occurs when we call gtk_widget_destroy() on the window,
     * or if we return FALSE in the "delete-event" callback. */
    g_signal_connect (window, "destroy",
		      G_CALLBACK (destroy), NULL);

    g_signal_connect (window, "realize", G_CALLBACK (realize_cb), NULL);
    
    /* Sets the border width of the window. */
    gtk_container_set_border_width (GTK_CONTAINER (window), 10);

    image = gtk_image_new();
    gtk_widget_show (image);

    /* This packs the button into the window (a gtk container). */
    gtk_container_add (GTK_CONTAINER (window), image);
        
    /* and the window */
    gtk_widget_show (window);
    
    /* All GTK applications must have a gtk_main(). Control ends here
     * and waits for an event to occur (like a key press or
     * mouse event). */
    gtk_main ();
    
    return 0;
}


      

Drawing text using Pango

While Cairo can draw any form of text, the functions such as cairo_show_text do not have much flexibility. To draw in, say, multiple colours will involve much work. Pango is a library for handling all aspects of text. There is a Pango Reference Manual . A good tutorial is at The Pango connection: Part 2 .

The simplest way of colouring text (and some other effects) is to create the text marked up with HTML such as

gchar *markup_text = "<span foreground=\"red\">hello </span><span foreground=\"black\">world</span>";
      

which has "hello" in red and "world" in black. This is then parsed into the text itself "ed black" and a set of attribute markups.

gchar *markup_text = "<span foreground=\"red\">hello </span><span foreground=\"black\">world</span>";
PangoAttrList *attrs;
gchar *text;

pango_parse_markup (markup_text, -1,0, &attrs, &text, NULL, NULL);
      

This can be rendered into a Cairo context by creating a PangoLayout from the Cairo context, laying out the text with its attributes in the Pango layout and then showing this layout in the Cairo context:

PangoLayout *layout;
PangoFontDescription *desc;

cairo_move_to(cr, 300.0, 50.0);
layout = pango_cairo_create_layout (cr);
pango_layout_set_text (layout, text, -1);
pango_layout_set_attributes(layout, attrs);
pango_cairo_update_layout (cr, layout);
pango_cairo_show_layout (cr, layout);
      

(Yes, there is a lot of jumping around between libraries in all of this!).

Just as before, once all content has been drawn into the Cairo context, it can be extracted as a pixbuf from the Cairo surface destination, set into the GtkImage and added to the Gtk event queue.

The complete program is gtk_play_video_pango.c:

	
#include <gtk/gtk.h>

#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>

#define OVERLAY_IMAGE "jan-small.png"

GtkWidget *image;
GtkWidget *window;

#if GTK_MAJOR_VERSION == 2
GdkPixmap *pixmap;
#endif

/* FFMpeg vbls */
AVFormatContext *pFormatCtx = NULL;
AVCodecContext *pCodecCtx = NULL;
int videoStream;
struct SwsContext *sws_ctx = NULL;
AVCodec *pCodec = NULL;



static void pixmap_destroy_notify(guchar *pixels,
				  gpointer data) {
    printf("Destroy pixmap - not sure how\n");
}

static void *play_background(void *args) {
    /* based on code from
       http://www.cs.dartmouth.edu/~xy/cs23/gtk.html
       http://cdry.wordpress.com/2009/09/09/using-custom-io-callbacks-with-ffmpeg/
    */

    int i;
    AVPacket packet;
    int frameFinished;
    AVFrame *pFrame = NULL;

    /* initialize packet, set data to NULL, let the demuxer fill it */
    /* http://ffmpeg.org/doxygen/trunk/doc_2examples_2demuxing_8c-example.html#a80 */
    av_init_packet(&packet);
    packet.data = NULL;
    packet.size = 0;

    int bytesDecoded;
    GdkPixbuf *pixbuf;
    GdkPixbuf *overlay_pixbuf;
    AVFrame *picture_RGB;
    char *buffer;
    
    // Pango marked up text, half red, half black
    gchar *markup_text = "<span foreground=\"red\">hello</span><span foreground=\"black\">world</span>";
    PangoAttrList *attrs;
    gchar *text;

    pango_parse_markup (markup_text, -1,0, &attrs, &text, NULL, NULL);


    GError *error = NULL;
    overlay_pixbuf = gdk_pixbuf_new_from_file(OVERLAY_IMAGE, &error);
    if (!overlay_pixbuf) {
	fprintf(stderr, "%s\n", error->message);
	g_error_free(error);
	exit(1);
    }

    // add an alpha layer for a white background
    overlay_pixbuf = gdk_pixbuf_add_alpha(overlay_pixbuf, TRUE, 255, 255, 255);

    int overlay_width = gdk_pixbuf_get_width(overlay_pixbuf);
    int overlay_height =  gdk_pixbuf_get_height(overlay_pixbuf);

    pFrame=avcodec_alloc_frame();

    i=0;
    picture_RGB = avcodec_alloc_frame();
    buffer = malloc (avpicture_get_size(PIX_FMT_RGB24, 720, 576));
    avpicture_fill((AVPicture *)picture_RGB, buffer, PIX_FMT_RGB24, 720, 576);

    while(av_read_frame(pFormatCtx, &packet)>=0) {
	if(packet.stream_index==videoStream) {
	    usleep(33670);  // 29.7 frames per second
	    // Decode video frame
	    avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished,
				  &packet);

	    int width = pCodecCtx->width;
	    int height = pCodecCtx->height;
	    
	    sws_ctx = sws_getContext(width, height, pCodecCtx->pix_fmt, width, height, 
				     PIX_FMT_RGB24, SWS_BICUBIC, NULL, NULL, NULL);

	    if (frameFinished) {
		printf("Frame %d\n", i++);
		
		sws_scale(sws_ctx,  (uint8_t const * const *) pFrame->data, pFrame->linesize, 0, height, picture_RGB->data, picture_RGB->linesize);
		
		printf("old width %d new width %d\n",  pCodecCtx->width, picture_RGB->width);
		pixbuf = gdk_pixbuf_new_from_data(picture_RGB->data[0], GDK_COLORSPACE_RGB,
						  0, 8, width, height, 
						  picture_RGB->linesize[0], pixmap_destroy_notify,
						  NULL);

		// Create the destination surface
		cairo_surface_t *surface = cairo_image_surface_create (CAIRO_FORMAT_ARGB32, 
								       width, height);
		cairo_t *cr = cairo_create(surface);

		// draw the background image
		gdk_cairo_set_source_pixbuf(cr, pixbuf, 0, 0);
		cairo_paint (cr);

		// overlay an image on top
		// alpha blending will be done by Cairo
		gdk_cairo_set_source_pixbuf(cr, overlay_pixbuf, 300, 200);
		cairo_paint (cr);

		// draw some white text on top
		cairo_set_source_rgb(cr, 1.0, 1.0, 1.0);
		// this is a standard font for Cairo
		cairo_select_font_face (cr, "cairo:serif",
					CAIRO_FONT_SLANT_NORMAL, 
					CAIRO_FONT_WEIGHT_BOLD);
		cairo_set_font_size (cr, 20);
		cairo_move_to(cr, 10.0, 50.0);
		cairo_show_text (cr, "hello");

		// draw Pango text
		PangoLayout *layout;
		PangoFontDescription *desc;

		cairo_move_to(cr, 300.0, 50.0);
		layout = pango_cairo_create_layout (cr);
		pango_layout_set_text (layout, text, -1);
		pango_layout_set_attributes(layout, attrs);
		pango_cairo_update_layout (cr, layout);
		pango_cairo_show_layout (cr, layout);


		/* start GTK thread lock for drawing */
		gdk_threads_enter();    

#if  GTK_MAJOR_VERSION == 2
		int pwidth, pheight, stride;
		unsigned char *data;

		data = cairo_image_surface_get_data (surface);
		pwidth = cairo_image_surface_get_width (surface);
		pheight = cairo_image_surface_get_height (surface);
		stride = cairo_image_surface_get_stride (surface);
		
		// this function doesn't work properly
		// code doesn't work
		pixmap = gdk_pixmap_create_from_data(NULL, data,
					      pwidth, pheight,
					      8, NULL, NULL);

		gtk_image_set_from_pixmap((GtkImage*) image, pixmap, NULL);

		gtk_widget_queue_draw(image);
#elif GTK_MAJOR_VERSION == 3
		pixbuf = gdk_pixbuf_get_from_surface(surface,
						     0,
						     0,
						     width,
						     height);

		gtk_image_set_from_pixbuf((GtkImage*) image, pixbuf); 

		gtk_widget_queue_draw(image);
#endif	      
		/* reclaim memory */
		g_object_unref(pixbuf);
		//sws_freeContext(sws_ctx);
		g_object_unref(layout);
		cairo_surface_destroy(surface);
		cairo_destroy(cr);

		/* release GTK thread lock */
		gdk_threads_leave();
	    }
	}
	av_free_packet(&packet);
    }

    printf("Video over!\n");
    exit(0);
}

/* Called when the windows are realized
 */
static void realize_cb (GtkWidget *widget, gpointer data) {
    /* start the video playing in its own thread */
    pthread_t tid;
    pthread_create(&tid, NULL, play_background, NULL);
}

static gboolean delete_event( GtkWidget *widget,
                              GdkEvent  *event,
                              gpointer   data )
{
    /* If you return FALSE in the "delete-event" signal handler,
     * GTK will emit the "destroy" signal. Returning TRUE means
     * you don't want the window to be destroyed.
     * This is useful for popping up 'are you sure you want to quit?'
     * type dialogs. */

    g_print ("delete event occurred\n");

    /* Change TRUE to FALSE and the main window will be destroyed with
     * a "delete-event". */
    return TRUE;
}

/* Another callback */
static void destroy( GtkWidget *widget,
                     gpointer   data )
{
    gtk_main_quit ();
}

int main(int argc, char** argv)
{
    // Is this necessary?
    XInitThreads();

    int i;


    /* FFMpeg stuff */

    AVFrame *pFrame = NULL;
    AVPacket packet;

    AVDictionary *optionsDict = NULL;

    av_register_all();

    if(avformat_open_input(&pFormatCtx, "short.mpg", NULL, NULL)!=0)
	return -1; // Couldn't open file
  
    // Retrieve stream information
    if(avformat_find_stream_info(pFormatCtx, NULL)<0)
	return -1; // Couldn't find stream information
  
    // Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, argv[1], 0);
  
    // Find the first video stream
    videoStream=-1;
    for(i=0; i<pFormatCtx->nb_streams; i++)
	if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
	    videoStream=i;
	    break;
	}
    if(videoStream==-1)
	return -1; // Didn't find a video stream

    for(i=0; i<pFormatCtx->nb_streams; i++)
	if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO) {
	    printf("Found an audio stream too\n");
	    break;
	}

  
    // Get a pointer to the codec context for the video stream
    pCodecCtx=pFormatCtx->streams[videoStream]->codec;
  
    // Find the decoder for the video stream
    pCodec=avcodec_find_decoder(pCodecCtx->codec_id);
    if(pCodec==NULL) {
	fprintf(stderr, "Unsupported codec!\n");
	return -1; // Codec not found
    }
  
    // Open codec
    if(avcodec_open2(pCodecCtx, pCodec, &optionsDict)<0)
	return -1; // Could not open codec

    sws_ctx =
	sws_getContext
	(
	 pCodecCtx->width,
	 pCodecCtx->height,
	 pCodecCtx->pix_fmt,
	 pCodecCtx->width,
	 pCodecCtx->height,
	 PIX_FMT_YUV420P,
	 SWS_BILINEAR,
	 NULL,
	 NULL,
	 NULL
	 );

    /* GTK stuff now */

    /* This is called in all GTK applications. Arguments are parsed
     * from the command line and are returned to the application. */
    gtk_init (&argc, &argv);
    
    /* create a new window */
    window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    
    /* When the window is given the "delete-event" signal (this is given
     * by the window manager, usually by the "close" option, or on the
     * titlebar), we ask it to call the delete_event () function
     * as defined above. The data passed to the callback
     * function is NULL and is ignored in the callback function. */
    g_signal_connect (window, "delete-event",
		      G_CALLBACK (delete_event), NULL);
    
    /* Here we connect the "destroy" event to a signal handler.  
     * This event occurs when we call gtk_widget_destroy() on the window,
     * or if we return FALSE in the "delete-event" callback. */
    g_signal_connect (window, "destroy",
		      G_CALLBACK (destroy), NULL);

    g_signal_connect (window, "realize", G_CALLBACK (realize_cb), NULL);
    
    /* Sets the border width of the window. */
    gtk_container_set_border_width (GTK_CONTAINER (window), 10);

    image = gtk_image_new();
    gtk_widget_show (image);

    /* This packs the button into the window (a gtk container). */
    gtk_container_add (GTK_CONTAINER (window), image);
        
    /* and the window */
    gtk_widget_show (window);
    
    /* All GTK applications must have a gtk_main(). Control ends here
     * and waits for an event to occur (like a key press or
     * mouse event). */
    gtk_main ();
    
    return 0;
}


      

Conclusion

Getting to grips with some aspects of the Gtk toolkit is not trivial. We will use some of this material in later chapters, which is why it has been pulled out of the sound sections of this book and placed in a "diversions" section. Those not interested in Linux sound may nevertheless find it useful.

	
      

Copyright © Jan Newmarch, jan@newmarch.name
Creative Commons License
"Programming and Using Linux Sound - in depth" by Jan Newmarch is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License .
Based on a work at https://jan.newmarch.name/LinuxSound/ .

If you like this book, please contribute using PayPal

Or Flattr me:
Flattr this book