Upto: Table of Contents of full book "Programming and Using Linux Sound"

TiMidity

TiMidity is designed as a standalone application with a particular kind of extensibility. Out of the box it can sort-of play Karaoke but not well. This chapter looks at how to work with TiMidity to build a Karaoke system.

Files

Source files in this chapter are here .

Introduction

TiMidity is a MIDI player, not a Karaoke player. By default it just plays the MIDI music, with the lyrics printed out:

	
$timidity ../54154.mid
Requested buffer size 32768, fragment size 8192
ALSA pcm 'default' set buffer size 32768, period size 8192 bytes
Playing ../54154.mid
MIDI file: ../54154.mid
Format: 1  Tracks: 1  Divisions: 30
No instrument mapped to tone bank 0, program 92 - this instrument will not be heard
#0001
@@00@12
@Here Comes The Sun
@
@@Beatles
Here comes the sun
doo doo doo doo
Here comes the sun
I said it's alright
Little
darling
	
      

But it has a number of alternative interfaces which give different displays. If you run timidity with the -h (help) option, it will show a screen including something like

	
Available interfaces (-i, --interface option):
  -in          ncurses interface
  -ie          Emacs interface (invoked from `M-x timidity')
  -ia          XAW interface
  -id          dumb interface
  -ir          remote interface
  -iA          ALSA sequencer interface
	
      
.

The default interface is "dumb", but if you run with, say, the XAW interface you get a display like

There is, however, one unfortunate effect: the lyrics are displayed before they are due to be played! To get the lyrics played just as they should be sung, you need to turn on the --trace option. From the man page, "Toggles trace mode. In trace mode, TiMidity++ attempts to display its current state in real time." You may find the link between documentation and behaviour a lttle less than obvious...

	
timidity --trace ../54154.mid
	
      

This now works fine for MIDI files, the lyrics are displayed when they should be sung. But it doesn't display the lyrics for .KAR files. For that you need the --trace-text-meta option:

	
timidity --trace --trace-text-meta ../54154.kar
	
      

So by this stage, TiMidity will display the lyrics on the screen in realtime for Karaoke files (and MIDI files with LYRIC events). To have our own control over this display, we need to build our own TiMidity interface.

TiMidity and Jack

In the MIDI chapter on User space tools we discussed playing MIDI files using Jack. Jack is designed to link audio sources and sinks in arbitrary configurations. By running qjackctl you can link, for example, microphone outputs to speaker inputs. This is done by dragging "capture_1" to "playback_1" etc and appears like

If TiMidity is then run with Jack output, you get instant Karaoke. You can also see the lyrics played in real time using the --trace option:

	
timidity -Oj --trace 54154.mid
	
      

The connections are shown in qjackctl as

The lyric display is klunky, and will be improved later.

TiMidity interface

You will need to have the TiMidity source downloaded from SourceForge TiMidity++ .

In the earlier chapter on MIDI and TiMidity we discussed two alternative ways of building applications using TiMidity:

Both options are possible here, with one wrinkle: if we want to capture MIDI events then we have to do so as a back-end to TiMidity, which requires that we build a TiMidity interface.

To recap on this, the different interface files for TiMidity are stored in the directory interface and include files such as dumb_c.c for the dumb interface. They all revolve around a data structure ControlMode defined in timidity/controls.h:

	
typedef struct {
  char *id_name, id_character;
  char *id_short_name;
  int verbosity, trace_playing, opened;

  int32 flags;

  int  (*open)(int using_stdin, int using_stdout);
  void (*close)(void);
  int (*pass_playing_list)(int number_of_files, char *list_of_files[]);
  int  (*read)(int32 *valp);
  int  (*write)(char *buf, int32 size);
  int  (*cmsg)(int type, int verbosity_level, char *fmt, ...);
  void (*event)(CtlEvent *ev);  /* Control events */
} ControlMode;
	
      

.

For the simplest values of the functions in this structure, see the code for the dumb interface in interface/dumb_c.c.

For dealing with lyrics, the main field to set is the function event(). This will be passed a pointer to a CtlEvent which is defined in timidity/controls.h:

	
typedef struct _CtlEvent {
    int type;           /* See above */
    ptr_size_t v1, v2, v3, v4;/* Event value */
} CtlEvent;
	
      

The type field distinguishes a large number of event types such as CTLE_NOW_LOADING and CTLE_PITCH_BEND. The type of interest to us is CTLE_LYRIC.

Typical code to handle this is in interface/dumb_c.c which prints event information to output:
	
static void ctl_event(CtlEvent *e)
{
    switch(e->type) {
      case CTLE_LYRIC:
        ctl_lyric((int)e->v1);
        break;
   }
}

static void ctl_lyric(int lyricid)
{
    char *lyric;

    lyric = event2string(lyricid);
    if(lyric != NULL)
    {
        if(lyric[0] == ME_KARAOKE_LYRIC)
        {
            if(lyric[1] == '/' || lyric[1] == '\\')
            {
                fprintf(outfp, "\n%s", lyric + 2);
                fflush(outfp);
            }
            else if(lyric[1] == '@')
            {
                if(lyric[2] == 'L')
                    fprintf(outfp, "\nLanguage: %s\n", lyric + 3);
                else if(lyric[2] == 'T')
                   fprintf(outfp, "Title: %s\n", lyric + 3);
                else
                    fprintf(outfp, "%s\n", lyric + 1);
            }
            else
            {
                fputs(lyric + 1, outfp);
                fflush(outfp);
            }
        }
        else
        {
            if(lyric[0] == ME_CHORUS_TEXT || lyric[0] == ME_INSERT_TEXT)
                fprintf(outfp, "\r");
            fputs(lyric + 1, outfp);
            fflush(outfp);
        }
    }
}
	
      

Getting the list of lyrics

The failing of the current interfaces in TiMidity with regard to Karaoke is that while they can show the lyrics as they are played, they don't show the lyric lines and progressively highlight them as they are played. For that, you need the set of lyrics.

TiMidity in fact builds a list of lyrics, and makes them accessible. It has a function event2string() which takes an integer parameter from one upwards. For each value it returns the string of a lyric or text event, finally returning NULL on the end of the list. The first character returned is a type parameter, the rest is the string. Using GLib functions, we can build up an array of lines for a KAR file by

	
struct _lyric_t {
    gchar *lyric;
    long tick; // not used here
};
typedef struct _lyric_t lyric_t;

struct _lyric_lines_t {
    char *language;
    char *title;
    char *performer;
    GArray *lines; // array of GString *
};
typedef struct _lyric_lines_t lyric_lines_t;

GArray *lyrics;
lyric_lines_t lyric_lines;

static void build_lyric_lines() {
    int n;
    lyric_t *plyric;
    GString *line = g_string_new("");
    GArray *lines =  g_array_sized_new(FALSE, FALSE, sizeof(GString *), 64);

    lyric_lines.title = NULL;

    n = 1;
    char *evt_str;
    while ((evt_str = event2string(n++)) != NULL) {
        gchar *lyric = evt_str+1;

        if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'L')) {
            lyric_lines.language =  lyric + 2;
            continue;
        }

        if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'T')) {
            if (lyric_lines.title == NULL) {
                lyric_lines.title = lyric + 2;
            } else {
                lyric_lines.performer = lyric + 2;
            }
            continue;
        }

        if (lyric[0] == '@') {
            // some other stuff like @KMIDI KARAOKE FILE
            continue;
        }

        if ((lyric[0] == '/') || (lyric[0] == '\\')) {
            // start of a new line
            // add to lines
            g_array_append_val(lines, line);
            line = g_string_new(lyric + 1);
        }  else {
            line = g_string_append(line, lyric);
        }
    }
    lyric_lines.lines = lines;
    
    printf("Title is %s, performer is %s, language is %s\n", 
           lyric_lines.title, lyric_lines.performer, lyric_lines.language);
    for (n = 0; n < lines->len; n++) {
        printf("Line is %s\n", g_array_index(lines, GString *, n)->str);
    }
}
	
      

The function build_lyric_lines() should be called from the CTLE_LOADING_DONE branch of ctl_event().

TiMidity options

If we choose to use TiMidity as front-end then we need to run it

with suitable options. These include turning tracing on and also dynamically loading our new interface. This can be done for example by
	
timidity -d. -iv --trace  --trace-text-meta ...
	
      

for a "v" interface in the current directory.

The alternative is building a main program that calls TiMidity as a library. The command line parameters to TiMidity then have to be included as hard-coded parameters in the application. One is easy: the CtlMode has a field trace_playing and setting that to one turns tracing on. Including Text events as Lyric events requires digging a bit deeper into TiMidity, but just requires (shortly after initialising the library)

	
extern int opt_trace_text_meta_event;
opt_trace_text_meta_event = 1;
	
      

Playing lyrics using Pnago + Cairo + Xlib

I want to be able to play my Karaoke files on the Raspberry Pi and similar Systems on a Chip (SOCs). Unfortuanately the rapsberry Pi has a grossly underpowered CPU, so I have ended up using a CubieBoard 2.

Even then I don't know how to program the GPU, so anything involving heavy graphics is not possible on this CPU. Any of the MIDI players hit close to (or over) 100% CPU usage just be themselves. So the system discussed in the next section, showing background video, isn't feasible.

In this section we use TiMidity as MIDI player with a minimal backend to display the lyrics as they are played. The lowest level of GUI support is used, namely Xlib. This can be used to draw text using low-level Xlib calls such as XDrawImageString. This wrks fine with ASCII languages, and with appropriate font choices, with other languages in the ISO-8859 family.

Asian languages are harder to deal with in standard C. They involve one or two byte characters when using an encoding such as UTF-8. To manage these. it is easiest to switch to a library designed to handle them such, such as Cairo.

Cairo is good for drawing simple text. For e.g. Chinese characters you have to find a font that will allow you to draw them. Alternatively, you can jump up one further level to Pango. Pango looks after all the font issues and produces glyphs which are sent to the X server.

that approach is adopted in the following interface, x_code.c


#include <X11/Xlib.h>
#include <X11/Xutil.h>

#include <gtk/gtk.h>

#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>

#include "mytimidity.h"

#define WIDTH 720
#define HEIGHT 480

#define NUM_LINES 4

Display *display;
Window window;
GC gc, color_gc;

struct _lyric_t {
    gchar *lyric;
    long tick;

};
typedef struct _lyric_t lyric_t;

struct _lyric_lines_t {
    char *language;
    char *title;
    char *performer;
    GArray *lines; // array of GString *
};
typedef struct _lyric_lines_t lyric_lines_t;

GArray *lyrics;
GString *lyrics_array[NUM_LINES];

lyric_lines_t lyric_lines;

typedef struct _coloured_line_t {
    gchar *line;
    gchar *front_of_line;
    gchar *marked_up_line;
    PangoAttrList *attrs;
} coloured_line_t;

int height_lyric_pixbufs[] = {100, 200, 300, 400}; // vertical offset of lyric in video
int coloured_text_offset;

// fluid_player_t* player;

// int current_panel = 1;  // panel showing current lyric line
int current_line = 0;  // which line is the current lyric
gchar *current_lyric;   // currently playing lyric line
GString *front_of_lyric;  // part of lyric to be coloured red
//GString *end_of_lyric;    // part of lyrci to not be coloured


gchar *markup[] = {"<span font=\"28\" foreground=\"RED\">",
		   "</span><span font=\"28\" foreground=\"white\">",
		   "</span>"};
gchar *markup_newline[] = {"<span foreground=\"black\">",
			   "</span>"};
GString *marked_up_label;

PangoFontDescription *font_description;

cairo_surface_t *surface;
cairo_t *cr;

extern ControlMode  *ctl;

ControlMode video_ctl=
    {
	"x interface", 'N',
	"x",
	1,          /* verbosity */
	1,          /* trace playing */
	0,          /* opened */
	0,          /* flags */
	ctl_open,
	ctl_close,
	pass_playing_list,
	ctl_read,
	NULL,       /* write */
	cmsg,
	ctl_event
    };

static FILE *outfp;
int video_error_count;
static char *current_file;
struct midi_file_info *current_file_info;

static int pass_playing_list(int number_of_files, char *list_of_files[]) {
    int n;

    for (n = 0; n < number_of_files; n++) {
	printf("Playing list %s\n", list_of_files[n]);
	
	current_file = list_of_files[n];
	play_midi_file( list_of_files[n]);
    }
    XCloseDisplay(display);
    exit(0);
    return 0;
}

static void paint_background() {
    cr = cairo_create(surface);
    cairo_set_source_rgb(cr, 0.0, 0.8, 0.0);
    cairo_paint(cr);
    cairo_destroy(cr);
}

static void set_font() {
    font_description = pango_font_description_new ();
    pango_font_description_set_family (font_description, "serif");
    pango_font_description_set_weight (font_description, PANGO_WEIGHT_BOLD);
    pango_font_description_set_absolute_size (font_description, 32 * PANGO_SCALE);
}

static int draw_text(char *text, float red, float green, float blue, int height, int offset) {
  // See http://cairographics.org/FAQ/
  PangoLayout *layout;
  int width, ht;
  cairo_text_extents_t extents;

  layout = pango_cairo_create_layout (cr);
  pango_layout_set_font_description (layout, font_description);
  pango_layout_set_text (layout, text, -1);

  if (offset == 0) {
      pango_layout_get_size(layout, &width, &ht);
      offset = (WIDTH - (width/PANGO_SCALE)) / 2;
  }

  cairo_set_source_rgb (cr, red, green, blue);
  cairo_move_to (cr, offset, height);
  pango_cairo_show_layout (cr, layout);

  g_object_unref (layout);
  return offset;
}

static void init_X() {
    int screenNumber;
    unsigned long foreground, background;
    int screen_width, screen_height;
    Screen *screen;
    XSizeHints hints;
    char **argv = NULL;
    XGCValues gcValues;
    Colormap colormap;
    XColor rgb_color, hw_color;
    Font font;
    //char *FNAME = "hanzigb24st";
    char *FNAME = "-misc-fixed-medium-r-normal--0-0-100-100-c-0-iso10646-1";

    display = XOpenDisplay(NULL);
    if (display == NULL) {
	fprintf(stderr, "Can't open dsplay\n");
	exit(1);
    }
    screenNumber = DefaultScreen(display);
    foreground = BlackPixel(display, screenNumber);
    background = WhitePixel(display, screenNumber);

    screen = DefaultScreenOfDisplay(display);
    screen_width = WidthOfScreen(screen);
    screen_height = HeightOfScreen(screen);

    hints.x = (screen_width - WIDTH) / 2;
    hints.y = (screen_height - HEIGHT) / 2;
    hints.width = WIDTH;
    hints.height = HEIGHT;
    hints.flags = PPosition | PSize;

    window = XCreateSimpleWindow(display,
				 DefaultRootWindow(display),
				 hints.x, hints.y, WIDTH, HEIGHT, 10,
				 foreground, background);

    XSetStandardProperties(display, window, 
			   "TiMidity", "TiMidity", 
			   None,
			   argv, 0,
			   &hints);

    XMapWindow(display, window);


    set_font();
    surface = cairo_xlib_surface_create(display, window,
					DefaultVisual(display, 0), WIDTH, HEIGHT);
    cairo_xlib_surface_set_size(surface, WIDTH, HEIGHT);

    paint_background();

    /*
    cr = cairo_create(surface);
    draw_text(g_array_index(lyric_lines.lines, GString *, 0)->str,
	      0.0, 0.0, 1.0, height_lyric_pixbufs[0]);
    draw_text(g_array_index(lyric_lines.lines, GString*, 1)->str,
	      0.0, 0.0, 1.0, height_lyric_pixbufs[0]);
    cairo_destroy(cr);
    */
    XFlush(display);
}


static int inited_video = 0;
/*ARGSUSED*/
static int ctl_open(int using_stdin, int using_stdout)
{
  init_X();

    // dont know what this function does
    /*
      if (current_file != NULL) {
      current_file_info = get_midi_file_info(current_file, 1);
      printf("Opening info for %s\n", current_file);
      } else {
      printf("Current is NULL\n");
      }
    */
    ctl->opened = 1;
    return 0;
}

static void ctl_close(void)
{
    fflush(outfp);
    video_ctl.opened=0;
    exit(0);
}

/*ARGSUSED*/
static int ctl_read(int32 *valp)
{
    return RC_NONE;
}

static int cmsg(int type, int verbosity_level, char *fmt, ...)
{
    /*
      va_list ap;

      if ((type==CMSG_TEXT || type==CMSG_INFO || type==CMSG_WARNING) &&
      video_ctl.verbosity<verbosity_level)
      return 0;
      va_start(ap, fmt);
      if(type == CMSG_WARNING || type == CMSG_ERROR || type == CMSG_FATAL)
      video_error_count++;
      if (!video_ctl.opened)
      {
      vfprintf(stderr, fmt, ap);
      fputs(NLS, stderr);
      }
      else
      {
      vfprintf(outfp, fmt, ap);
      fputs(NLS, outfp);
      fflush(outfp);
      }
      va_end(ap);
    */
    return 0;
}

static void ctl_total_time(long tt)
{
    /*
      int mins, secs;
      if (video_ctl.trace_playing)
      {
      secs=(int)(tt/play_mode->rate);
      mins=secs/60;
      secs-=mins*60;
      cmsg(CMSG_INFO, VERB_NORMAL,
      "Total playing time: %3d min %02d s", mins, secs);
      }
    */
}

static void ctl_file_name(char *name)
{
    current_file = name;

    if (video_ctl.verbosity>=0 || video_ctl.trace_playing)
	cmsg(CMSG_INFO, VERB_NORMAL, "Playing %s", name);
}

static void ctl_current_time(int secs)
{
    int mins;
    static int prev_secs = -1;

#ifdef __W32__
    if(wrdt->id == 'w')
	return;
#endif /* __W32__ */
    if (ctl->trace_playing && secs != prev_secs)
	{
	    prev_secs = secs;
	    mins=secs/60;
	    secs-=mins*60;
	    fprintf(stdout, "\r%3d:%02d", mins, secs);
	}
}

void build_lyric_lines() {
    int n;
    lyric_t *plyric;
    GString *line = g_string_new("");
    GArray *lines =  g_array_sized_new(FALSE, FALSE, sizeof(GString *), 64);

    lyric_lines.title = NULL;

    n = 1;
    char *evt_str;
    while ((evt_str = event2string(n++)) != NULL) {

        gchar *lyric = evt_str+1;
	printf("Building line %s\n", lyric);

	if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'L')) {
	    lyric_lines.language =  lyric + 2;
	    continue;
	}

	if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'T')) {
	    if (lyric_lines.title == NULL) {
		lyric_lines.title = lyric + 2;
	    } else {
		lyric_lines.performer = lyric + 2;
	    }
	    continue;
	}

	if (lyric[0] == '@') {
	    // some other stuff like @KMIDI KARAOKE FILE
	    continue;
	}

	if ((lyric[0] == '/') || (lyric[0] == '\\')) {
	    // start of a new line
	    // add to lines
	    printf("New line\n");
	    g_array_append_val(lines, line);
	    line = g_string_new(lyric + 1);
	}  else {
	    line = g_string_append(line, lyric);
	}
    }
    lyric_lines.lines = lines;
    
    printf("Title is %s, performer is %s, language is %s\n", 
	   lyric_lines.title, lyric_lines.performer, lyric_lines.language);
    for (n = 0; n < lines->len; n++) {
	printf("Line is %s\n", g_array_index(lines, GString *, n)->str);
    }
    
}

static void ctl_lyric(int lyricid)
{
    char *lyric;

    current_file_info = get_midi_file_info(current_file, 1);

    lyric = event2string(lyricid);
    if(lyric != NULL)
	lyric++;
    printf("Got a lyric %s\n", lyric);


    if ((*lyric == '\\') || (*lyric == '/')) {
	// int next_panel = (current_panel+2) % NUM_LINES; // really (current_panel+2)%2
	int next_line = current_line + NUM_LINES;
	gchar *next_lyric;

	if (current_line + NUM_LINES < lyric_lines.lines->len) {
	    current_line += 1;
	    
	    //lyrics_array[(next_line-1) % NUM_LINES] = 
	    //	g_array_index(lyric_lines.lines, GString *, next_line);
	    
	    // update label for next line after this one
	    next_lyric = g_array_index(lyric_lines.lines, GString *, next_line)->str;

	} else {
	    current_line += 1;
	    lyrics_array[(next_line-1) % NUM_LINES] = NULL;
	    next_lyric = "";
	}

	// set up new line as current line
	if (current_line < lyric_lines.lines->len) {
	    GString *gstr = g_array_index(lyric_lines.lines, GString *, current_line);
	    current_lyric = gstr->str;
	    front_of_lyric = g_string_new(lyric+1); // lose	slosh
	}	  
	printf("New line. Setting front to %s end to \"%s\"\n", lyric+1, current_lyric); 


	// Now draw stuff
	paint_background();

	cr = cairo_create(surface);

	int n;
	for (n = 0; n < NUM_LINES; n++) {
	    //lyrics_array[n] = g_array_index(lyric_lines.lines, GString *, n+1);
	    if (lyrics_array[n] != NULL) {
		draw_text(lyrics_array[n]->str,
			  0.0, 0.0, 0.5, height_lyric_pixbufs[n], 0);
	    }
	}
	// redraw current and next lines
	if (current_line < lyric_lines.lines->len) {
	    if (current_line >= 2) {
		// redraw last line still in red
		GString *gstr = lyrics_array[(current_line-2) % NUM_LINES];
		if (gstr != NULL) {
		    draw_text(gstr->str,
			      1.0, 0.0, 00, 
			      height_lyric_pixbufs[(current_line-2) % NUM_LINES],
			      0);
		}
	    }
	    // draw next line in brighter blue
	    coloured_text_offset = draw_text(lyrics_array[(current_line-1) % NUM_LINES]->str,
		      0.0, 0.0, 1.0, height_lyric_pixbufs[(current_line-1) % NUM_LINES], 0);
	    printf("coloured text offset %d\n", coloured_text_offset);
	}

	//try
	if (next_line < lyric_lines.lines->len) {
	    lyrics_array[(next_line-1) % NUM_LINES] = 
		g_array_index(lyric_lines.lines, GString *, next_line);
	}
	

	    /*
	draw_text(current_lyric, 0.0, 0.0, 1.0, 
		  height_lyric_pixbufs[(current_line-1) % NUM_LINES]);


	draw_text(next_lyric, 0.0, 0.0, 1.0, 
		  height_lyric_pixbufs[(next_line-1) % NUM_LINES]);
	    */
	cairo_destroy(cr);
	XFlush(display);


    } else {
	// change text colour as chars are played
	if ((front_of_lyric != NULL) && (lyric != NULL)) {
	    g_string_append(front_of_lyric, lyric);
	    char *s = front_of_lyric->str;
	    //coloured_lines[current_panel].front_of_line = s;

	    cairo_t *cr = cairo_create(surface);

	    // See http://cairographics.org/FAQ/
	    draw_text(s, 1.0, 0.0, 0.0, 
		      height_lyric_pixbufs[(current_line-1) % NUM_LINES], 
		      coloured_text_offset);

	    cairo_destroy(cr);
	    XFlush(display);

	}
    }
}


static void ctl_event(CtlEvent *e)
{
    //printf("Got ctl event %d\n", e->type);
    switch(e->type)
	{
	case CTLE_NOW_LOADING:
	    ctl_file_name((char *)e->v1);
	    break;
	case CTLE_LOADING_DONE:
	    // MIDI file is loaded, about to play
	    current_file_info = get_midi_file_info(current_file, 1);
	    if (current_file_info != NULL) {
		printf("file info not NULL\n");
	    } else {
		printf("File info is NULL\n");
	    }

	    int n = 1;
	    char *evt_str;
	    while ((evt_str = event2string(n++)) != NULL) {
		printf("Event in tabel: %s\n", evt_str);
	    }

	    build_lyric_lines();
	    cr = cairo_create(surface);

	    // draw line to be sung slightly brighter
	    // than the rest
	    for (n = 0; n < NUM_LINES; n++) {
		lyrics_array[n] = g_array_index(lyric_lines.lines, GString *, n+1);
		draw_text(lyrics_array[n]->str,
			  0.0, 0.0, 0.5, height_lyric_pixbufs[n], 0);
	    }
	    draw_text(lyrics_array[0]->str,
		      0.0, 0.0, 1.0, height_lyric_pixbufs[0], 0);

	    /*
	    draw_text(g_array_index(lyric_lines.lines, GString *, 1)->str,
		      0.0, 0.0, 1.0, height_lyric_pixbufs[0]);
	    draw_text(g_array_index(lyric_lines.lines, GString*, 2)->str,
		      0.0, 0.0, 1.0, height_lyric_pixbufs[1]);
	    draw_text(g_array_index(lyric_lines.lines, GString*, 3)->str,
		      0.0, 0.0, 1.0, height_lyric_pixbufs[2]);
	    */
	    cairo_destroy(cr);
	    XFlush(display);
	    
	    break;
	case CTLE_PLAY_START:

	    ctl_total_time(e->v1);
	    break;
        case CTLE_PLAY_END:
	    printf("Exiting, play ended\n");
	    exit(0);
	    break;
	case CTLE_CURRENT_TIME:
	    ctl_current_time((int)e->v1);
	    break;
	case CTLE_LYRIC:
	    ctl_lyric((int)e->v1);
	    break;
	    /*
	      case CTLE_REFRESH:
	      printf("Refresh\n");
	      break;
	    */
	default:
	    0;
	    //printf("Other event\n");
	}
}

/*
 * interface_<id>_loader();
 */
ControlMode *interface_x_loader(void)
{
    return &video_ctl;
}

      

it is compiled by

        gcc  -fPIC $(CFLAGS) -c -o x_code.o x_code.c $(LIBS)
        gcc -shared -o if_x.so x_code.o $(LIBS)
      

and run by

timidity -d. -ix --trace --trace-text-meta ...
      

Playing a background video with Gtk

In the chapter on FluidSynth case: we discussed a program to show lyrics overlaid onto a movie. Apart from the above considerations, the rest of the application follows similarly to the FluidSynth case: build a set of lyric lines, display them using Pango over Gtk Pixbufs, and when a new lyric event occurs update the corresponding colours in the lyric line.

All of the dynamic action needs to occur out of the back-end of TiMidity. particularly in the function ctl_event. Other parts such as initialising FFMpeg and Gtk must also occur in the back-end when using standard TiMidity. If TiMidity is used as a library, this initialisation could occur in the front or the back. For simplicity, we just place it all in the back in the file video_code.c:


#include <gtk/gtk.h>

#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>

#include "mytimidity.h"

#define USE_PIXBUF

// saving as pixbufs leaks memory
//#define USE_PIXBUF

/* run by
   gtkkaraoke_player_video_pango /usr/share/sounds/sf2/FluidR3_GM.sf2 /home/newmarch/Music/karaoke/sonken/songs/54154.kar
*/

/*
 * APIs:
 * GString: https://developer.gnome.org/glib/2.28/glib-Strings.html
 * Pango text attributes: https://developer.gnome.org/pango/stable/pango-Text-Attributes.html#pango-parse-markup
 * Pango layout: http://www.gtk.org/api/2.6/pango/pango-Layout-Objects.html
 * Cairo rendering: https://developer.gnome.org/pango/stable/pango-Cairo-Rendering.html#pango-cairo-create-layout
 * Cairo surface_t: http://cairographics.org/manual/cairo-cairo-surface-t.html
 * GTK+ 3 Reference Manual: https://developer.gnome.org/gtk3/3.0/
 * Gdk Pixbufs: https://developer.gnome.org/gdk/stable/gdk-Pixbufs.html
 */

struct _lyric_t {
    gchar *lyric;
    long tick;

};
typedef struct _lyric_t lyric_t;

struct _lyric_lines_t {
    char *language;
    char *title;
    char *performer;
    GArray *lines; // array of GString *
};
typedef struct _lyric_lines_t lyric_lines_t;

GArray *lyrics;

lyric_lines_t lyric_lines;

typedef struct _coloured_line_t {
    gchar *line;
    gchar *front_of_line;
    gchar *marked_up_line;
    PangoAttrList *attrs;
#ifdef USE_PIXBUF
    GdkPixbuf *pixbuf;
#endif
} coloured_line_t;

coloured_line_t coloured_lines[2];

// fluid_synth_t* synth;

GtkWidget *image;
#if GTK_MAJOR_VERSION == 2
GdkPixmap *dbuf_pixmap;
#endif

int height_lyric_pixbufs[] = {300, 400}; // vertical offset of lyric in video

// fluid_player_t* player;

int current_panel = 1;  // panel showing current lyric line
int current_line = 0;  // which line is the current lyric
gchar *current_lyric;   // currently playing lyric line
GString *front_of_lyric;  // part of lyric to be coloured red
//GString *end_of_lyric;    // part of lyrci to not be coloured



// Colours seem to get mixed up when putting a pixbuf onto a pixbuf
#ifdef USE_PIXBUF
#define RED blue
#else
#define RED red
#endif

gchar *markup[] = {"<span font=\"28\" foreground=\"RED\">",
		   "</span><span font=\"28\" foreground=\"white\">",
		   "</span>"};
gchar *markup_newline[] = {"<span foreground=\"black\">",
			   "</span>"};
GString *marked_up_label;

/* FFMpeg vbls */
AVFormatContext *pFormatCtx = NULL;
AVCodecContext *pCodecCtx = NULL;
int videoStream;
struct SwsContext *sws_ctx = NULL;
AVCodec *pCodec = NULL;


void markup_line(coloured_line_t *line) {
    GString *str =  g_string_new(markup[0]);
    g_string_append(str, line->front_of_line);
    g_string_append(str, markup[1]);
    g_string_append(str, line->line + strlen(line->front_of_line));
    g_string_append(str, markup[2]);
    printf("Marked up label \"%s\"\n", str->str);

    line->marked_up_line = str->str;
    // we have to free line->marked_up_line

    pango_parse_markup(str->str, -1,0, &(line->attrs), NULL, NULL, NULL);
    g_string_free(str, FALSE);
}

#ifdef USE_PIXBUF
void update_line_pixbuf(coloured_line_t *line) {
    //return;
    cairo_surface_t *surface;
    cairo_t *cr;
            
    int lyric_width = 480;
    int lyric_height = 60;
    surface = cairo_image_surface_create (CAIRO_FORMAT_ARGB32, 
                                          lyric_width, lyric_height);
    cr = cairo_create (surface);

    PangoLayout *layout;
    PangoFontDescription *desc;
    
    // draw the attributed text
    layout = pango_cairo_create_layout (cr);
    pango_layout_set_text (layout, line->line, -1);
    pango_layout_set_attributes(layout, line->attrs);

    // centre the image in the surface
    int width, height;   
    pango_layout_get_pixel_size(layout,
				&width,
				&height);
    cairo_move_to(cr, (lyric_width-width)/2, 0);    

    pango_cairo_update_layout (cr, layout);
    pango_cairo_show_layout (cr, layout);

    // pull the pixbuf out of the surface
    unsigned char *data = cairo_image_surface_get_data(surface);
    width = cairo_image_surface_get_width(surface);
    height = cairo_image_surface_get_height(surface);
    int stride = cairo_image_surface_get_stride(surface);
    printf("Text surface width %d height %d stride %d\n", width, height, stride);

    GdkPixbuf *old_pixbuf = line->pixbuf;
    line->pixbuf = gdk_pixbuf_new_from_data(data, GDK_COLORSPACE_RGB, 1, 8, width, height, stride, NULL, NULL);
    cairo_surface_destroy(surface);
    g_object_unref(old_pixbuf);
}
#endif

extern ControlMode  *ctl;

ControlMode video_ctl=
    {
	"video interface", 'v',
	"video",
	1,          /* verbosity */
	1,          /* trace playing */
	0,          /* opened */
	0,          /* flags */
	ctl_open,
	ctl_close,
	pass_playing_list,
	ctl_read,
	NULL,       /* write */
	cmsg,
	ctl_event
    };

static void init_timidity() {
    int err;

    timidity_start_initialize();

    if ((err = timidity_pre_load_configuration()) != 0) {
	printf("couldn't pre-load configuration file\n");
	exit(1);
    }

    err += timidity_post_load_configuration();

    if (err) {
	printf("couldn't post-load configuration file\n");
	exit(1);
    }

    timidity_init_player();

    
    extern int opt_trace_text_meta_event;
    opt_trace_text_meta_event = 1;

    //ctl = &video_ctl;
    //ctl->trace_playing = 1;
    //opt_trace_text_meta_event = 1;
    
}

static FILE *outfp;
int video_error_count;
static char *current_file;
struct midi_file_info *current_file_info;

static int pass_playing_list(int number_of_files, char *list_of_files[]) {
    int n;

    for (n = 0; n < number_of_files; n++) {
	printf("Playing list %s\n", list_of_files[n]);
	
	current_file = list_of_files[n];
	/*
	  current_file_info = get_midi_file_info(current_file, 1);
	  if (current_file_info != NULL) {
	  printf("file info not NULL\n");
	  } else {
	  printf("File info is NULL\n");
	  }
	*/
	play_midi_file( list_of_files[n]);
    }
    return 0;
}

static int inited_video = 0;
/*ARGSUSED*/
static int ctl_open(int using_stdin, int using_stdout)
{
    if (! inited_video) {
	init_video();
	inited_video = 1;
    }

    // dont know what this function does
    /*
      if (current_file != NULL) {
      current_file_info = get_midi_file_info(current_file, 1);
      printf("Opening info for %s\n", current_file);
      } else {
      printf("Current is NULL\n");
      }
    */
    ctl->opened = 1;
    return 0;
}

static void ctl_close(void)
{
    fflush(outfp);
    video_ctl.opened=0;
}

/*ARGSUSED*/
static int ctl_read(int32 *valp)
{
    return RC_NONE;
}

static int cmsg(int type, int verbosity_level, char *fmt, ...)
{
    /*
      va_list ap;

      if ((type==CMSG_TEXT || type==CMSG_INFO || type==CMSG_WARNING) &&
      video_ctl.verbosity<verbosity_level)
      return 0;
      va_start(ap, fmt);
      if(type == CMSG_WARNING || type == CMSG_ERROR || type == CMSG_FATAL)
      video_error_count++;
      if (!video_ctl.opened)
      {
      vfprintf(stderr, fmt, ap);
      fputs(NLS, stderr);
      }
      else
      {
      vfprintf(outfp, fmt, ap);
      fputs(NLS, outfp);
      fflush(outfp);
      }
      va_end(ap);
    */
    return 0;
}

static void ctl_total_time(long tt)
{
    /*
      int mins, secs;
      if (video_ctl.trace_playing)
      {
      secs=(int)(tt/play_mode->rate);
      mins=secs/60;
      secs-=mins*60;
      cmsg(CMSG_INFO, VERB_NORMAL,
      "Total playing time: %3d min %02d s", mins, secs);
      }
    */
}

static void ctl_file_name(char *name)
{
    current_file = name;

    if (video_ctl.verbosity>=0 || video_ctl.trace_playing)
	cmsg(CMSG_INFO, VERB_NORMAL, "Playing %s", name);
}

static void ctl_current_time(int secs)
{
    int mins;
    static int prev_secs = -1;

#ifdef __W32__
    if(wrdt->id == 'w')
	return;
#endif /* __W32__ */
    if (ctl->trace_playing && secs != prev_secs)
	{
	    prev_secs = secs;
	    mins=secs/60;
	    secs-=mins*60;
	    fprintf(stdout, "\r%3d:%02d", mins, secs);
	}
}

void build_lyric_lines() {
    int n;
    lyric_t *plyric;
    GString *line = g_string_new("");
    GArray *lines =  g_array_sized_new(FALSE, FALSE, sizeof(GString *), 64);

    lyric_lines.title = NULL;

    n = 1;
    char *evt_str;
    while ((evt_str = event2string(n++)) != NULL) {

        gchar *lyric = evt_str+1;
	printf("Building line %s\n", lyric);

	if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'L')) {
	    lyric_lines.language =  lyric + 2;
	    continue;
	}

	if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'T')) {
	    if (lyric_lines.title == NULL) {
		lyric_lines.title = lyric + 2;
	    } else {
		lyric_lines.performer = lyric + 2;
	    }
	    continue;
	}

	if (lyric[0] == '@') {
	    // some other stuff like @KMIDI KARAOKE FILE
	    continue;
	}

	if ((lyric[0] == '/') || (lyric[0] == '\\')) {
	    // start of a new line
	    // add to lines
	    g_array_append_val(lines, line);
	    line = g_string_new(lyric + 1);
	}  else {
	    line = g_string_append(line, lyric);
	}
    }
    lyric_lines.lines = lines;
    
    printf("Title is %s, performer is %s, language is %s\n", 
	   lyric_lines.title, lyric_lines.performer, lyric_lines.language);
    for (n = 0; n < lines->len; n++) {
	printf("Line is %s\n", g_array_index(lines, GString *, n)->str);
    }
    
}

static void overlay_lyric(cairo_t *cr,
			  coloured_line_t *line,
			  int ht) {
    PangoLayout *layout;
    int height, width;

    if (line->line == NULL) {
	return;
    }

    layout = pango_cairo_create_layout (cr);
    pango_layout_set_text (layout, line->line, -1);
    pango_layout_set_attributes(layout, line->attrs);
    pango_layout_get_pixel_size(layout,
				&width,
				&height);
    cairo_move_to(cr, (720-width)/2, ht);

    pango_cairo_update_layout (cr, layout);
    pango_cairo_show_layout (cr, layout);

    g_object_unref(layout);
}

static void pixmap_destroy_notify(guchar *pixels,
				  gpointer data) {
    printf("Ddestroy pixmap\n");
}

static void *play_background(void *args) {
    /* based on code from
       http://www.cs.dartmouth.edu/~xy/cs23/gtk.html
       http://cdry.wordpress.com/2009/09/09/using-custom-io-callbacks-with-ffmpeg/
    */

    int i;
    AVPacket packet;
    int frameFinished;
    AVFrame *pFrame = NULL;

    int oldSize;
    char *oldData;
    int bytesDecoded;
    GdkPixbuf *pixbuf;
    AVFrame *picture_RGB;
    char *buffer;

#if GTK_MAJOR_VERSION == 2
    GdkPixmap *pixmap;
    GdkBitmap *mask;
#endif

    pFrame=avcodec_alloc_frame();

    i=0;
    picture_RGB = avcodec_alloc_frame();
    buffer = malloc (avpicture_get_size(PIX_FMT_RGB24, 720, 576));
    avpicture_fill((AVPicture *)picture_RGB, buffer, PIX_FMT_RGB24, 720, 576);

    int width = pCodecCtx->width;
    int height = pCodecCtx->height;
    
    sws_ctx = sws_getContext(pCodecCtx->width, pCodecCtx->height, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height, PIX_FMT_RGB24, SWS_BICUBIC, NULL, NULL, NULL);
    
    while(av_read_frame(pFormatCtx, &packet)>=0) {
	if(packet.stream_index==videoStream) {
	    //printf("Frame %d\n", i++);
	    usleep(33670);  // 29.7 frames per second
	    // Decode video frame
	    avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished,
				  &packet);


	    if (frameFinished) {
		//printf("Frame %d\n", i++);
		
		sws_scale(sws_ctx,  (uint8_t const * const *) pFrame->data, pFrame->linesize, 0, pCodecCtx->height, picture_RGB->data, picture_RGB->linesize);
		

		pixbuf = gdk_pixbuf_new_from_data(picture_RGB->data[0], GDK_COLORSPACE_RGB, 0, 8, 720, 480, picture_RGB->linesize[0], pixmap_destroy_notify, NULL);


		/* start GTK thread lock for drawing */
		gdk_threads_enter();    

#define SHOW_LYRIC
#ifdef SHOW_LYRIC
                // Create the destination surface
                cairo_surface_t *surface = cairo_image_surface_create (CAIRO_FORMAT_ARGB32, 
                                                                       width, height);
                cairo_t *cr = cairo_create(surface);

                // draw the background image
                gdk_cairo_set_source_pixbuf(cr, pixbuf, 0, 0);
                cairo_paint (cr);

#ifdef USE_PIXBUF
		// draw the lyric
		GdkPixbuf *lyric_pixbuf = coloured_lines[current_panel].pixbuf;
		if (lyric_pixbuf != NULL) {
		    int width = gdk_pixbuf_get_width(lyric_pixbuf);
		    gdk_cairo_set_source_pixbuf(cr, 
						lyric_pixbuf, 
						(720-width)/2, 
						height_lyric_pixbufs[current_panel]);
		    cairo_paint (cr);
		}

		int next_panel = (current_panel+1) % 2;
		lyric_pixbuf = coloured_lines[next_panel].pixbuf;
		if (lyric_pixbuf != NULL) {
		    int width = gdk_pixbuf_get_width(lyric_pixbuf);
		    gdk_cairo_set_source_pixbuf(cr, 
						lyric_pixbuf, 
						(720-width)/2, 
						height_lyric_pixbufs[next_panel]);
		    cairo_paint (cr);
		}
#else

		overlay_lyric(cr, 
			      coloured_lines+current_panel, 
			      height_lyric_pixbufs[current_panel]);

		int next_panel = (current_panel+1) % 2;
		overlay_lyric(cr, 
			      coloured_lines+next_panel, 
			      height_lyric_pixbufs[next_panel]);
#endif
		pixbuf = gdk_pixbuf_get_from_surface(surface,
						     0,
						     0,
						     width,
						     height);

		gtk_image_set_from_pixbuf((GtkImage*) image, pixbuf);

		g_object_unref(pixbuf);		/* reclaim memory */
		//sws_freeContext(sws_ctx);
		//g_object_unref(layout);
		cairo_surface_destroy(surface);
		cairo_destroy(cr);
#else
		gtk_image_set_from_pixbuf((GtkImage*) image, pixbuf);
#endif /* SHOW_LYRIC */

		/* release GTK thread lock */
		gdk_threads_leave();
	    }
	}
	av_free_packet(&packet);
    }
    sws_freeContext(sws_ctx);

    printf("Video over!\n");
    exit(0);
}

static void ctl_lyric(int lyricid)
{
    char *lyric;

    current_file_info = get_midi_file_info(current_file, 1);

    lyric = event2string(lyricid);
    if(lyric != NULL)
	lyric++;
    printf("Got a lyric %s\n", lyric);
    if (*lyric == '\\') {
	int next_panel = current_panel; // really (current_panel+2)%2
	int next_line = current_line + 2;
	gchar *next_lyric;

	if (current_line + 2 >= lyric_lines.lines->len) {
	    return;
	}
	current_line += 1;
	current_panel = (current_panel + 1) % 2;

	// set up new line as current line
	current_lyric = g_array_index(lyric_lines.lines, GString *, current_line)->str;
	front_of_lyric = g_string_new(lyric+1); // lose \
	printf("New line. Setting front to %s end to \"%s\"\n", lyric+1, current_lyric); 

	coloured_lines[current_panel].line = current_lyric;
	coloured_lines[current_panel].front_of_line = lyric+1;
	markup_line(coloured_lines+current_panel);
#ifdef USE_PIXBUF
	update_line_pixbuf(coloured_lines+current_panel);
#endif
	// update label for next line after this one
	next_lyric = g_array_index(lyric_lines.lines, GString *, next_line)->str;
	    
	marked_up_label = g_string_new(markup_newline[0]);

	g_string_append(marked_up_label, next_lyric);
	g_string_append(marked_up_label, markup_newline[1]);
	PangoAttrList *attrs;
	gchar *text;
	pango_parse_markup (marked_up_label->str, -1,0, &attrs, &text, NULL, NULL);
	    
	coloured_lines[next_panel].line = next_lyric;
	coloured_lines[next_panel].front_of_line = "";
	markup_line(coloured_lines+next_panel);
#ifdef USE_PIXBUF
	update_line_pixbuf(coloured_lines+next_panel);
#endif
    } else {
	// change text colour as chars are played
	if ((front_of_lyric != NULL) && (lyric != NULL)) {
	    g_string_append(front_of_lyric, lyric);
	    char *s = front_of_lyric->str;
	    coloured_lines[current_panel].front_of_line = s;
	    markup_line(coloured_lines+current_panel);
#ifdef USE_PIXBUF
	    update_line_pixbuf(coloured_lines+current_panel);
#endif
	}
    }
}


static void ctl_event(CtlEvent *e)
{
    //printf("Got ctl event %d\n", e->type);
    switch(e->type)
	{
	case CTLE_NOW_LOADING:
	    ctl_file_name((char *)e->v1);
	    break;
	case CTLE_LOADING_DONE:
	    // MIDI file is loaded, about to play
	    current_file_info = get_midi_file_info(current_file, 1);
	    if (current_file_info != NULL) {
		printf("file info not NULL\n");
	    } else {
		printf("File info is NULL\n");
	    }

	    int n = 1;
	    char *evt_str;
	    while ((evt_str = event2string(n++)) != NULL) {
		printf("Event in tabel: %s\n", evt_str);
	    }

	    build_lyric_lines();

	    break;
	case CTLE_PLAY_START:

	    ctl_total_time(e->v1);
	    break;
	case CTLE_CURRENT_TIME:
	    ctl_current_time((int)e->v1);
	    break;
	case CTLE_LYRIC:
	    ctl_lyric((int)e->v1);
	    break;
	    /*
	      case CTLE_REFRESH:
	      printf("Refresh\n");
	      break;
	    */
	default:
	    0;
	    //printf("Other event\n");
	}
}



static void *play_timidity(void *args) {
    char *argv[] = {"54154.kar"};
    
    timidity_play_main(1, argv);
    printf("Play timidity finished\n");
}

/* Called when the windows are realized
 */
static void realize_cb (GtkWidget *widget, gpointer data) {
    /* start the video playing in its own thread */
    pthread_t back_id;
    pthread_create(&back_id, NULL, play_background, NULL);

#if 0
    pthread_t timidity_id;
    pthread_create(&timidity_id, NULL, play_timidity, NULL);
#endif
}

static gboolean delete_event( GtkWidget *widget,
                              GdkEvent  *event,
                              gpointer   data )
{
    /* If you return FALSE in the "delete-event" signal handler,
     * GTK will emit the "destroy" signal. Returning TRUE means
     * you don't want the window to be destroyed.
     * This is useful for popping up 'are you sure you want to quit?'
     * type dialogs. */

    g_print ("delete event occurred\n");

    /* Change TRUE to FALSE and the main window will be destroyed with
     * a "delete-event". */

    return TRUE;
}

/* Another callback */
static void destroy( GtkWidget *widget,
                     gpointer   data )
{
    gtk_main_quit ();
}

void *play_gtk(void *args) {
    printf("About to start gtk_main\n");
    gtk_main();
}

int init_video() {
    #define FNAME "short.mpg"

    lyrics = g_array_sized_new(FALSE, FALSE, sizeof(lyric_t *), 1024);

    /* FFMpeg stuff */

    AVFrame *pFrame = NULL;
    AVPacket packet;

    AVDictionary *optionsDict = NULL;

    av_register_all();

    if(avformat_open_input(&pFormatCtx, FNAME, NULL, NULL)!=0) {
	printf("Couldn't open video file called short.mpg\n");
	return -1; // Couldn't open file
    } else {
	printf("Opened short.mpg\n");
    }
  
    // Retrieve stream information
    if(avformat_find_stream_info(pFormatCtx, NULL)<0) {
	printf("Couldn't find stream information\n");
	return -1; // Couldn't find stream information
    }

    // Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, FNAME, 0);
  
    // Find the first video stream
    videoStream=-1;

    int i;
    for(i=0; i<pFormatCtx->nb_streams; i++)
	if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
	    videoStream=i;
	    break;
	}
    if(videoStream==-1)
	return -1; // Didn't find a video stream

    for(i=0; i<pFormatCtx->nb_streams; i++)
	if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO) {
	    printf("Found an audio stream too\n");
	    break;
	}

 
    // Get a pointer to the codec context for the video stream
    pCodecCtx=pFormatCtx->streams[videoStream]->codec;
  
    // Find the decoder for the video stream
    pCodec=avcodec_find_decoder(pCodecCtx->codec_id);
    if(pCodec==NULL) {
	fprintf(stderr, "Unsupported codec!\n");
	return -1; // Codec not found
    }
  
    // Open codec
    if(avcodec_open2(pCodecCtx, pCodec, &optionsDict)<0) {
	printf("Could not open codec\n");
	return -1; // Could not open codec
    }

    /* GTK stuff now */

    /* GtkWidget is the storage type for widgets */
    GtkWidget *window;
    GtkWidget *button;
    GtkWidget *lyrics_box;

    
    /* This is called in all GTK applications. Arguments are parsed
     * from the command line and are returned to the application. */
    gtk_init (0, NULL);
    
    /* create a new window */
    window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    
    /* When the window is given the "delete-event" signal (this is given
     * by the window manager, usually by the "close" option, or on the
     * titlebar), we ask it to call the delete_event () function
     * as defined above. The data passed to the callback
     * function is NULL and is ignored in the callback function. */
    g_signal_connect (window, "delete-event",
		      G_CALLBACK (delete_event), NULL);
    
    /* Here we connect the "destroy" event to a signal handler.  
     * This event occurs when we call gtk_widget_destroy() on the window,
     * or if we return FALSE in the "delete-event" callback. */
    g_signal_connect (window, "destroy",
		      G_CALLBACK (destroy), NULL);

    g_signal_connect (window, "realize", G_CALLBACK (realize_cb), NULL);
    
    /* Sets the border width of the window. */
    gtk_container_set_border_width (GTK_CONTAINER (window), 10);

    lyrics_box = gtk_vbox_new(TRUE, 1);
    gtk_widget_show(lyrics_box);

    /*
      char *str = "     ";
      lyric_labels[0] = gtk_label_new(str);
      str =  "World";
      lyric_labels[1] = gtk_label_new(str);
    */

    image = gtk_image_new();

    //image_drawable = gtk_drawing_area_new();
    //gtk_widget_set_size_request (canvas, 720, 480);
    //gtk_drawing_area_size((GtkDrawingArea *) image_drawable, 720, 480);

    //gtk_widget_show (lyric_labels[0]);
    //gtk_widget_show (lyric_labels[1]);

    gtk_widget_show (image);
    
    //gtk_box_pack_start (GTK_BOX (lyrics_box), lyric_labels[0], TRUE, TRUE, 0);
    //gtk_box_pack_start (GTK_BOX (lyrics_box), lyric_labels[1], TRUE, TRUE, 0);
    gtk_box_pack_start (GTK_BOX (lyrics_box), image, TRUE, TRUE, 0);
    //gtk_box_pack_start (GTK_BOX (lyrics_box), canvas, TRUE, TRUE, 0);
    //gtk_box_pack_start (GTK_BOX (lyrics_box), image_drawable, TRUE, TRUE, 0);
    
    /* This packs the button into the window (a gtk container). */
    gtk_container_add (GTK_CONTAINER (window), lyrics_box);
        
    /* and the window */
    gtk_widget_show (window);
    
    /* All GTK applications must have a gtk_main(). Control ends here
     * and waits for an event to occur (like a key press or
     * mouse event). */
    pthread_t tid_gtk;
    pthread_create(&tid_gtk, NULL, play_gtk, NULL);
    //gtk_main ();
}

/*
 * interface_<id>_loader();
 */
ControlMode *interface_v_loader(void)
{
    return &video_ctl;
}

      

Background video with TiMidity as library

The code for this follows the same structure as the code in the MIDI TiMidity chapter. It is is in the file gtkkaraoke_player_video_pango.:


#include <stdio.h>
#include <stdlib.h>

#include "sysdep.h"
#include "controls.h"


extern ControlMode  *video_ctl;
extern ControlMode  *ctl;

static void init_timidity() {
    int err;

    timidity_start_initialize();

    if ((err = timidity_pre_load_configuration()) != 0) {
	printf("couldn't pre-load configuration file\n");
	exit(1);
    }

    err += timidity_post_load_configuration();

    if (err) {
	printf("couldn't post-load configuration file\n");
	exit(1);
    }

    timidity_init_player();

    
    extern int opt_trace_text_meta_event;
    opt_trace_text_meta_event = 1;

    ctl = &video_ctl;
    //ctl->trace_playing = 1;
    //opt_trace_text_meta_event = 1;
    
}


#define MIDI_FILE "54154.kar"

static void *play_midi(void *args) {
    char *argv[1];
    argv[0] = MIDI_FILE;
    int argc = 1;

    timidity_play_main(argc, argv);

    printf("Audio finished\n");
    exit(0);
}


int main(int argc, char** argv)
{
    XInitThreads();

    int i;

    /* TiMidity */
    init_timidity();
    play_midi(NULL);


    return 0;
}


      

Background video with TiMidity as front-end

The interface needs to be built as a shared library by

	
if_video.so: video_code.c
        gcc  -fPIC $(CFLAGS) -c -o video_code.o video_code.c $(LIBS)
        gcc -shared -o if_video.so video_code.o $(LIBS)
	
      

TiMidity is then run with options

	
timidity -d. -iv --trace  --trace-text-meta 
	
      

As before, it crashes TiMidity from the Ubuntu distro but works fine with TiMidity built from source in the current Linux environment.

Adding microphone input

At this stage we have a single application which can play a MIDI file, play a background movie and display highlighted lyrics on top of the video. There is no microphone input to sing along.

Singing along can be handled either within this application or by an external process. if we want to include it in the current application then we will have to build a mixer for two audio streams. Java does this in the JavaSound package, but in C we would need to do that ourselves. Now I think that can be done in ALSA, but at present their mixer code is gobbledygook to me.

Jack makes it easy to mix audio - from different processes. The earlier section showed how to do that.

My long term goal is to include scoring, etc. I need to split out the GUI code into a process that can also deal with microphone input. The Gtk interface in TiMidity is a "separate process" model, so I'll build on that - when I have Jack under control.

This section is a place-holder for now.

Conclusion

This chapter has shown how to use TiMidity as MIDI player for a Karaoke system. On my laptop it uses less than 60% of CPU with Gtk 3.0, better than the 100% of FluidSynth.


Copyright © Jan Newmarch, jan@newmarch.name
Creative Commons License
"Programming and Using Linux Sound - in depth" by Jan Newmarch is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License .
Based on a work at https://jan.newmarch.name/LinuxSound/ .

If you like this book, please contribute using PayPal

Or Flattr me:
Flattr this book