Upto: Table of Contents of full book "Programming and Using Linux Sound"

Decoding the DKD files on the Sonken Karaoke DVD

This chapter is about getting the information off my Sonken Karaoke DVD so that I can start writing programs to play the songs. It is not directly involved in playing sound under Linux and can be skipped.

Introduction

I have two Karaoke players, a Sonken MD-388 and a Malata MDVD-6619. Between the two of them they have all the features I think I need from Karaoke players. These are

The Malata is really good in that it shows the notes of the melody and also shows the notes that you are singing. But it has a pathetic range of English songs and doesn't show the Pinyin for the Chinese songs. The Songen has a good selection of both and shows the Pinyin, but doesn't show the notes and has a simplistic scoring system.

So I want to take the songs off my Sonken DVD and either play them on the Malata or on my PC. Playing them on my PC is preferred because then I am only limited by the programs that I can write and am not so dependent on the vendor's machines. So my immediate goal is to get the songs off the Sonken DVD and start playing them in the ways that I want.

The files on the Sonken DVD are in DKD format. This is an undocumented format probably standing for Digital Karaoke Disk. Many people have worked on this format, and there has been much discussion in forums such as the Karaoke Engineering. These include Understanding the HOTDOG files on DVD of California electronics , Decoding JBK 6628 DVD Karaoke Disc and Karaoke Huyndai 99 .

When I started looking at my disk, I went about it in a different direction to many of the posters in these forums. Also, the results in the forums were presented in an adhoc and often confusing manner - as could be expected. So I ended up re-inventing a lot of what had already been discovered, as well as coming up with some new stuff.

In hindsight, I could have saved myself weeks of work if I had paid proper attention to what was said in the forums. So this chapter is my attempt to lay out the results in a simple and logical enough way so that people trying to do similar things with their own disks can easily work out what is applicable to their situation and what is different.

What this chapter will cover is

This section is not complete, as there is still more to be discovered.

Format shifting

Isn't it illegal to copy your DVDs? Not in Australia, under the right conditions ( Copyright Amendment Act 2006 - FAQs):

Will I be able to copy my music collection onto my iPod?
Yes. You can format-shift music that you own to devices such as an MP3 player, X-Box 360 or your computer.

I am just copying the music I legally bought with the Sonken DVD to my computer for personal use. That is within the revised Australian copyright act. You should check if your country allows the same rights.

Don't ask for any copies of the files off my DVD. That would be illegal, and I'm not going to do it.

Files on the DVD

My Sonken DVD disk contains these files:

	
	  BACK01.MPG
	  DTSMUS00.DKD
	  DTSMUS01.DKD
	  DTSMUS02.DKD
	  DTSMUS03.DKD
	  DTSMUS04.DKD
	  DTSMUS05.DKD
	  DTSMUS06.DKD
	  DTSMUS07.DKD
	  DTSMUS10.DKD
	  DTSMUS20.DKD
	
      

BACK01.MPG

This is the MP3 file that plays in the background

DTSMUS00.DKD - DTSMUS07.DKD

These are the song files. The number of these depends on how many songs are on the DVD.

DTSMUS10.DKD

No-one has worked out what this file is for yet.

DTSMUS20.DKD

This file contains the list of song number/song title/artist as given in the song book. The song number in this file is one less than the song number in the book.

Decoding DTSMUS20.DKD

I'm on a Linux system and I use Linux/Unix utilities and applications. Equivalents exist under other O/S's such as Windows and Apple.

Song information

The Unix command strings lists all the ASCII 8-bit encoded strings in a file that are at least 4 characters long. Running this command on all the DVD files shows that DTSMUS20.DKD is the only one with lots of english-language strings, and these strings are the song titles on the DVD.

A brief selection is

	
	  Come To Me
	  Come To Me Boy
	  Condition Of My Heart
	  Fly To The Sky
	  Cool Love
	  Count Down
	  Cowboy
	  Crazy
	
      

The actual strings that would show on your disk depends of course on the songs on it. You would need some english language titles on it for this to work, of course!

To make further progress you need a binary editor. I use bvi. emacs has a binary editor mode as well. Search in there for a song title you know is on the disk. For example, searching for the Beatles "Here Comes The Sun" shows the block

	
	  000AA920  12 D3 88 48 65 72 65 20 43 6F 6D 65 73 20 54 68 ...Here Comes Th
	  000AA930  65 20 52 61 69 6E 20 41 67 61 69 6E 00 45 75 72 e Rain Again.Eur
	  000AA940  79 74 68 6D 69 63 73 00 1F 12 D3 89 48 65 72 65 ythmics.....Here
	  000AA950  20 43 6F 6D 65 73 20 54 68 65 20 53 75 6E 00 42  Comes The Sun.B
	  000AA960  65 61 74 6C 65 73 00 1B 12 D3 8A 48 65 72 65 20 eatles.....Here
	  000AA970  46 6F 72 20 59 6F 75 00 46 69 72 65 68 6F 75 73 For You.Firehous
	
      

The string "Here Comes The Sun" starts at 0xAA94C followed by a null byte. This is followed at 0xAA95F by the null-terminated "Beatles". Immediately before this is 4 bytes. The length of these two strings (including the null bytes) and the 4 bytes is 0x1F and this is the first of the 4 preceding bytes. So the block consists of a 4-byte header followed by a null-terminated song title followed by a null-terminated artist. Byte 1 is the length of the song information block including the 4 byte header.

Byte 2 of the header block is 0x12. jim75 at Decoding JBK 6628 DVD Karaoke Disc discovered the document JBK_Manual%5B1%5D.doc . In there is a list of country codes:

	
	  00 : KOREAN
	  01 : CHINESE( reserved )
	  02 : CHINESE
	  03 : TAIWANESE
	  04 : JAPANESE
	  05 : RUSSIAN
	  06 : THAI
	  07 : TAIWANESE( reserved )
	  08 : CHINESE( reserved )
	  09 : CANTONESE
	  12 : ENGLISH
	  13 : VIETNAMESE
	  14 : PHILIPPINE
	  15 : TURKEY
	  16 : SPANISH
	  17 : INDONESIAN
	  18 : MALAYSIAN
	  19 : PORTUGUESE
	  20 : FRENCH
	  21 : INDIAN
	  22 : BRASIL
	
      

The Beatle's song has 0x12 in byte 2 of the header and this matches the country codes in the table. This is confirmed by looking at other language files (later).

I've discovered later that the WMA files have their own codes. So far I have seen

	
	  83 : CHINESE WMA
	  92 : ENGLISH WMA
	  94 : PHILIPPINE WMA
	
      

I guess you can see the pattern with the earlier ones!

Bytes 3 and 4 of the header are 0xD389 which is 54153 in decimal. This is one less than the song number in the book (54154). So bytes 3 and 4 are a 16-bit short integer, one less than the song index in the book.

This pattern is repeated throughout the file, so that each record is of this format.

Beginning/end of data

There is a long sequence of bytes near the beginning of the file "01 01 01 01 01 ...". This finishes on my file at 0x9F23. By comparing the index number with those in my song book, I confirm this is the start of the Korean songs, and probably the start of all songs. I haven't found any table giving me this start value.

Checking a number of songs gives me this table:

I can't find the Vietnamese songs, though. There don't seem to any on my disk. My song book is lying! I guess there is some table somewhere giving these start points, but I haven't found it - these were all found by looking at my song book and then in the file.

The end of the block is signalled by a sequence of "FF FF FF FF ..." at 0x136C92.

But there is lots of stuff both before and after the song information block. I don't know what it means.

Chinese songs

The first English song in my book is "Gump by Al Wierd", song number 24452. In the table of contents file DTSMUS20.DK this is at 0x9562D (611885). The entry before this is "20 03 3A 04 CE D2 B4 F2 C1 CB D2 BB CD A8 B2 BB CB B5 BB B0 B5 C4 B5 E7 BB B0 B8 F8 C4 E3 00 00". The song code is "3A 04" i.e. 14852 which is song number 14853 (one offset, remember!). When I play that song on my karaoke machine I'm in luck: the first character of the song is "我", which I recognise as the word "I" (in Pinyin: wo3). It's encoding in the file is "CE D2". I've got Chinese input installed on my computer so I can search for this Chinese character.

A Google search for "unicode value of 我" shows me

	
	  [RESOLVED] Converting Unicode Character Literal to Uint16 variable ...
	  www.codeguru.com › ... › C++ (Non Visual C++ Issues)
	  5 posts - 2 authors - 1 Jul 2011

          I've determined that the unicode character '我' has a hex value of 
          0x6211 by looking it up on the "GNOME Character Map 2.32.1" 
          and if I do this....
	
      

and then looking up 0x6211 on Unicode Search gives gold:

	
	  Unicode	6211 (25105)
	  GB Code	CED2 (4650)
	  Big 5 Code	A7DA
	  CNS Code	1-4A3C
	
      

There's the CED2 in the second line as GB Code. So there you go: the character set is GB (probably GB2312 with EUC-CN encoding) with code for 我 as CED2.

Just to make sure: using the table by Mary Ansell at GB Code Table the bytes "CE D2 B4 F2 C1 CB D2 BB CD A8 B2 BB CB B5 BB B0 B5 C4 B5 E7 BB B0 B8 F8 C4 E3" translate into "我 打 了 一 通 ..." which is indeed the song.

Other languages

I'm not familiar with other language encodings so haven't investigated the Thai, Vietnamese, etc. The Korean seems to be EUC-KR.

Programs

The earlier investigations by others have created programs in C or C++. These are generally standalone programs. I would like to build a collection of reusable modules, so I have chosen Java as implementation language.

Java goodies

Java is a good O/O language which supports good design. It includes a Midi player and Midi classes. It supports multiple language encodings so it is easy to switch from, say GB-2312 to Unicode. It has good cross-platform GUI support.

Java baddies

Java doesn't support unsigned integer types. This sucks really badly here since so many data types are unsigned for these programs. Even bytes in Java are signed :-(. Here are some of the tricks :-(.

Classes

The song class contains information about a single song and is given here: SongInformation.java

	
      


public class SongInformation {


    // Public fields of each song record
    /**
     *  Song number in the file, one less than in songbook
     */
    public long number;

    /**
     * song title in Unicode
     */
    public String title;

    /**
     * artist in Unicode
     */
    public String artist;

    /**
     * integer value of language code
     */
    public int language;

    public static final int  KOREAN = 0;
    public static final int  CHINESE1 = 1;
    public static final int  CHINESE2 = 2;
    public static final int  TAIWANESE3 = 3 ;
    public static final int  JAPANESE = 4;
    public static final int  RUSSIAN = 5;
    public static final int  THAI = 6;
    public static final int  TAIWANESE7 = 7;
    public static final int  CHINESE8 = 8;
    public static final int  CANTONESE = 9;
    public static final int  ENGLISH = 0x12;
    public static final int  VIETNAMESE = 0x13;
    public static final int  PHILIPPINE = 0x14;
    public static final int  TURKEY = 0x15;
    public static final int  SPANISH = 0x16;
    public static final int  INDONESIAN = 0x17;
    public static final int  MALAYSIAN = 0x18;
    public static final int  PORTUGUESE = 0x19;
    public static final int  FRENCH = 0x20;
    public static final int  INDIAN = 0x21;
    public static final int  BRASIL = 0x22;
    public static final int  CHINESE131 = 131;
    public static final int  ENGLISH146 = 146;
    public static final int  PHILIPPINE148 = 148;

    public SongInformation(long number,
			   String title,
			   String artist,
			   int language) {
	this.number = number;
	this.title = title;
	this.artist = artist;
	this.language = language;
    }

    public String toString() {
	return "" + (number+1) + " (" + language + ") \"" + title + "\" " + artist;
    }

    public boolean titleMatch(String pattern) {
	// System.out.println("Pattern: " + pattern);
	return title.matches("(?i).*" + pattern + ".*");
    }

    public boolean artistMatch(String pattern) {
	return artist.matches("(?i).*" + pattern + ".*");
    }

    public boolean numberMatch(String pattern) {
	Long n;
	try {
	    n = Long.parseLong(pattern) - 1;
	    //System.out.println("Long is " + n);
	} catch(Exception e) {
	    //System.out.println(e.toString());
	    return false;
	}
	return number == n;
    }


    public boolean languageMatch(int lang) {
	return language == lang;
    }
}

	
      

The song table class holds a list of song information objects and is given by SongTable.java

	
      
import java.util.Vector;
import java.io.FileInputStream;
import java.io.*;
import java.nio.charset.Charset;

// public class SongTable implements java.util.Iterator {
// public class SongTable extends  Vector<SongInformation> {
public class SongTable {

    private static final String SONG_INFO_FILE = "/home/newmarch/Music/karaoke/sonken/DTSMUS20.DKD";
    private static final long INFO_START = 0x9F23;

    public static final int ENGLISH = 0x12;
    
    private static Vector<SongInformation> allSongs;

    private Vector<SongInformation> songs = 
	new Vector<SongInformation>  ();

    public static long[] langCount = new long[0x23];

    public SongTable(Vector<SongInformation> songs) {
	this.songs = songs;
    }

    public SongTable() throws java.io.IOException, 
			      java.io.FileNotFoundException {
	FileInputStream fstream = new FileInputStream(SONG_INFO_FILE);
	fstream.skip(INFO_START);
	while (true) {
	    int len;
	    int lang;
	    long number;

	    len = fstream.read();
	    lang = fstream.read();
	    number = readShort(fstream);
	    if (len == 0xFF && lang == 0xFF && number == 0xFFFFL) {
		break;
	    }
	    byte[] bytes = new byte[len - 4];
	    fstream.read(bytes);
	    int endTitle;
	    // find null at end of title
	    for (endTitle = 0; bytes[endTitle] != 0; endTitle++)
		;
	    byte[] titleBytes = new byte[endTitle];
	    byte[] artistBytes = new byte[len - endTitle - 6];

	    System.arraycopy(bytes, 0, titleBytes, 0, titleBytes.length);
	    System.arraycopy(bytes, endTitle + 1,
			     artistBytes, 0, artistBytes.length);
	    String title = toUnicode(lang, titleBytes);
	    String artist = toUnicode(lang, artistBytes);
	    // System.out.printf("artist: %s, title: %s, lang: %d, number %d\n", artist, title, lang, number);
	    SongInformation info = new SongInformation(number,
						       title,
						       artist,
						       lang);
	    songs.add(info);

	    if (lang > 0x22) {
		//System.out.println("Illegal lang value " + lang + " at song " + number);
	    } else {
		langCount[lang]++;
	    }
	}
	allSongs = songs;
    }

    public void dumpTable() {
	for (SongInformation song: songs) {
	    System.out.println("" + (song.number+1) + " - " +
			       song.artist + " - " +
			       song.title);
	}
    }

    public java.util.Iterator<SongInformation> iterator() {
	return songs.iterator();
    }

    private int readShort(FileInputStream f)  throws java.io.IOException {
	int n1 = f.read();
	int n2 = f.read();
	return (n1 << 8) + n2;
    }

    private String toUnicode(int lang, byte[] bytes) {
	switch (lang) {
	case SongInformation.ENGLISH:
	case SongInformation.ENGLISH146:
	case SongInformation.PHILIPPINE:
	case SongInformation.PHILIPPINE148:
	    // case SongInformation.HINDI:
	case SongInformation.INDONESIAN:
	case SongInformation.SPANISH:
	    return new String(bytes);

	case SongInformation.CHINESE1:
	case SongInformation.CHINESE2:
	case SongInformation.CHINESE8:
	case SongInformation.CHINESE131:
	case SongInformation.TAIWANESE3:
	case SongInformation.TAIWANESE7:
	case SongInformation.CANTONESE:
            Charset charset = Charset.forName("gb2312");
            return new String(bytes, charset);

	case SongInformation.KOREAN:
	    charset = Charset.forName("euckr");
            return new String(bytes, charset);

	default:
	    return "";
	}
    }

    public SongInformation getNumber(long number) {
	for (SongInformation info: songs) {
	    if (info.number == number) {
		return info;
	    }
	}
	return null;
    }

    public SongTable titleMatches( String pattern) {
	Vector<SongInformation> matchSongs = 
	    new Vector<SongInformation>  ();

	for (SongInformation song: songs) {
	    if (song.titleMatch(pattern)) {
		matchSongs.add(song);
	    }
	}
	return new SongTable(matchSongs);
    }

     public SongTable artistMatches( String pattern) {
	Vector<SongInformation> matchSongs = 
	    new Vector<SongInformation>  ();

	for (SongInformation song: songs) {
	    if (song.artistMatch(pattern)) {
		matchSongs.add(song);
	    }
	}
	return new SongTable(matchSongs);
    }

      public SongTable numberMatches( String pattern) {
	Vector<SongInformation> matchSongs = 
	    new Vector<SongInformation>  ();

	for (SongInformation song: songs) {
	    if (song.numberMatch(pattern)) {
		matchSongs.add(song);
	    }
	}
	return new SongTable(matchSongs);
    }

    public String toString() {
	StringBuffer buf = new StringBuffer();
	for (SongInformation song: songs) {
	    buf.append(song.toString() + "\n");
	}
	return buf.toString();
    }
	
    public static void main(String[] args) {
	// for testing
	SongTable songs = null;
	try {
	    songs = new SongTable();
	} catch(Exception e) {
	    System.err.println(e.toString());
	    System.exit(1);
	}
	songs.dumpTable();
	System.exit(0);

	// Should print "54151 Help Yourself Tom Jones"
	System.out.println(songs.getNumber(54150).toString());

	// Should print "18062 伦巴(恋歌) 伦巴"
	System.out.println(songs.getNumber(18061).toString());

	System.out.println(songs.artistMatches("Tom Jones").toString());
	/* Prints
54151 Help Yourself Tom Jones
50213 Daughter Of Darkness Tom Jones
23914 DELILAH Tom Jones
52834 Funny Familiar Forgotten Feelings Tom Jones
54114 Green green grass of home Tom Jones
54151 Help Yourself Tom Jones
55365 I (WHO HAVE NOTHING) TOM JONES
52768 I Believe Tom Jones
55509 I WHO HAVE NOTHING TOM JONES
55594 I'll Never Fall Inlove Again Tom Jones
55609 I'm Coming Home Tom Jones
51435 It's Not Unusual Tom Jones
55817 KISS Tom Jones
52842 Little Green Apples Tom Jones
51439 Love Me Tonight Tom Jones
56212 My Elusive Dream TOM JONES
56386 ONE DAY SOON Tom Jones
22862 THAT WONDERFUL SOUND Tom Jones
57170 THE GREEN GREEN GRASS OF HOME TOM JONES
57294 The Wonderful Sound Tom Jones
23819 TILL Tom Jones
51759 What's New Pussycat Tom Jones
52862 With These Hands Tom Jones
57715 Without Love Tom Jones
57836 You're My World Tom Jones
	*/

	for (int n = 1; n < langCount.length; n++) {
	    if (langCount[n] != 0) {
		System.out.println("Count: " + langCount[n] + " of lang " + n);
	    }
	}

	// Check Russian, etc
	System.out.println("Russian " + '\u0411');
	System.out.println("Korean " + '\u0411');
	System.exit(0);
    }
}
	
      

You may need to adjust the constant values in the file-based constructor for this to work properly for you.

A Java program using Swing to allow display and searching of the song titles is SongTableSwing.java

	
      
import java.awt.*;
import java.awt.event.*;
import javax.swing.event.*;
import javax.swing.*;
import javax.swing.SwingUtilities;
import java.util.regex.*;
import java.io.*;

public class SongTableSwing extends JPanel {
   private DefaultListModel model = new DefaultListModel();
    private JList list;
    private static SongTable allSongs;

    private JTextField numberField;
    private JTextField langField;
    private JTextField titleField;
    private JTextField artistField;

    // This font displays Asian and European characters.
    // It should be in your distro.
    // Fonts displaying all Unicode are zysong.ttf and Cyberbit.ttf
    // See http://unicode.org/resources/fonts.html
    private Font font = new Font("WenQuanYi Zen Hei", Font.PLAIN, 16);
    // font = new Font("Bitstream Cyberbit", Font.PLAIN, 16);
    
    private int findIndex = -1;

    /**
     * Describe <code>main</code> method here.
     *
     * @param args a <code>String</code> value
     */
    public static final void main(final String[] args) {
	allSongs = null;
	try {
	    allSongs = new SongTable();
	} catch(Exception e) {
	    System.err.println(e.toString());
	    System.exit(1);
	}

	JFrame frame = new JFrame();
	frame.setTitle("Song Table");
	frame.setSize(1000, 800);
	frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
	
	SongTableSwing panel = new SongTableSwing(allSongs);
	frame.getContentPane().add(panel);

	frame.setVisible(true);
    }

    public SongTableSwing(SongTable songs) {

	if (font == null) {
	    System.err.println("Can't fnd font");
	}
		
	int n = 0;
	java.util.Iterator<SongInformation> iter = songs.iterator();
	while(iter.hasNext()) {
	    model.add(n++, iter.next());
	    // model.add(n++, iter.next().toString());
	}

	BorderLayout mgr = new BorderLayout();
 
	list = new JList(model);
	// list = new JList(songs);
	list.setFont(font);
	JScrollPane scrollPane = new JScrollPane(list);

	setLayout(mgr);
	add(scrollPane, BorderLayout.CENTER);

	JPanel bottomPanel = new JPanel();
	bottomPanel.setLayout(new GridLayout(2, 1));
	add(bottomPanel, BorderLayout.SOUTH);

	JPanel searchPanel = new JPanel();
	bottomPanel.add(searchPanel);
	searchPanel.setLayout(new FlowLayout());

	JLabel numberLabel = new JLabel("Number");
	numberField = new JTextField(5);

	JLabel langLabel = new JLabel("Language");
	langField = new JTextField(8);

	JLabel titleLabel = new JLabel("Title");
	titleField = new JTextField(20);
	titleField.setFont(font);

	JLabel artistLabel = new JLabel("Artist");
	artistField = new JTextField(10);
	artistField.setFont(font);

	searchPanel.add(numberLabel);
	searchPanel.add(numberField);
	// searchPanel.add(langLabel);
	// searchPanel.add(langField);
	searchPanel.add(titleLabel);
	searchPanel.add(titleField);
	searchPanel.add(artistLabel);
	searchPanel.add(artistField);

	titleField.getDocument().addDocumentListener(new DocumentListener() {
		public void changedUpdate(DocumentEvent e) {
		    // rest find to -1 to restart any find searches
		    findIndex = -1;
		    // System.out.println("reset find index");
		}
		public void insertUpdate(DocumentEvent e) {
		    findIndex = -1;
		    // System.out.println("reset insert find index");
		}
		public void removeUpdate(DocumentEvent e) {
		    findIndex = -1;
		    // System.out.println("reset remove find index");
		}
	    }
	    );
	artistField.getDocument().addDocumentListener(new DocumentListener() {
		public void changedUpdate(DocumentEvent e) {
		    // rest find to -1 to restart any find searches
		    findIndex = -1;
		    // System.out.println("reset insert find index");
		}
		public void insertUpdate(DocumentEvent e) {
		    findIndex = -1;
		    // System.out.println("reset insert find index");
		}
		public void removeUpdate(DocumentEvent e) {
		    findIndex = -1;
		    // System.out.println("reset insert find index");
		}
	    }
	    );

	titleField.addActionListener(new ActionListener(){
                public void actionPerformed(ActionEvent e){
		    filterSongs();
                }});
	artistField.addActionListener(new ActionListener(){
                public void actionPerformed(ActionEvent e){
		    filterSongs();
                }});
	numberField.addActionListener(new ActionListener(){
                public void actionPerformed(ActionEvent e){
		    filterSongs();
                }});

	JPanel buttonPanel = new JPanel();
	bottomPanel.add(buttonPanel);
	buttonPanel.setLayout(new FlowLayout());

	JButton find = new JButton("Find");
	JButton filter = new JButton("Filter");
	JButton reset = new JButton("Reset");
	JButton play = new JButton("Play");
	buttonPanel.add(find);
	buttonPanel.add(filter);
	buttonPanel.add(reset);
	buttonPanel.add(play);

	find.addActionListener(new ActionListener() {
		public void actionPerformed(ActionEvent e) {
		    findSong();
		}
	    });

	filter.addActionListener(new ActionListener() {
		public void actionPerformed(ActionEvent e) {
		    filterSongs();
		}
	    });

	reset.addActionListener(new ActionListener() {
		public void actionPerformed(ActionEvent e) {
		    resetSongs();
		}
	    });

	play.addActionListener(new ActionListener() {
		public void actionPerformed(ActionEvent e) {
		    playSong();
		}
	    });
 
     }

    public void findSong() {
	String number = numberField.getText();
	String language = langField.getText();
	String title = titleField.getText();
	String artist = artistField.getText();

	if (number.length() != 0) {
	    try {

		long num = Integer.parseInt(number) - 1;
		for (int n = 0; n < model.getSize(); n++) {
		    SongInformation info = (SongInformation) model.getElementAt(n);
		    if (info.number == num) {
			list.setSelectedIndex(n);
			list.ensureIndexIsVisible(n);
			return;
		    }
		}
	    } catch(Exception e) {
		System.err.println("Not a number");
		numberField.setText("");
	    }

	    return;
	}

	/*
	System.out.println("Title " + title + title.length() + 
			   "artist " + artist + artist.length() +
			   " find start " + findIndex +
			   " model size " + model.getSize());
	if (title.length() == 0 && artist.length() == 0) {
	    System.err.println("no search terms");
	    return;
	}
	*/

	//System.out.println("Search " + searchStr + " from index " + findIndex);
	for (int n = findIndex + 1; n < model.getSize(); n++) {
	    SongInformation info = (SongInformation) model.getElementAt(n);
	    //System.out.println(info.toString());

	    if ((title.length() != 0) && (artist.length() != 0)) {
		if (info.titleMatch(title) && info.artistMatch(artist)) {
		    // System.out.println("Found " + info.toString());
			findIndex = n;
			list.setSelectedIndex(n);
			list.ensureIndexIsVisible(n);
			break;
		}
	    } else {
		if ((title.length() != 0) && info.titleMatch(title)) {
		    // System.out.println("Found " + info.toString());
		    findIndex = n;
		    list.setSelectedIndex(n);
		    list.ensureIndexIsVisible(n);
		    break;
		} else if ((artist.length() != 0) && info.artistMatch(artist)) {
		    // System.out.println("Found " + info.toString());
		    findIndex = n;
		    list.setSelectedIndex(n);
		    list.ensureIndexIsVisible(n);
		    break;

		}
	    }

	}
    }

    public void filterSongs() {
	String title = titleField.getText();
	String artist = artistField.getText();
	String number = numberField.getText();
	SongTable filteredSongs = allSongs;

	if (allSongs == null) {
	    // System.err.println("Songs is null");
	    return;
	}

	if (title.length() != 0) {
	    filteredSongs = filteredSongs.titleMatches(title);
	}
	if (artist.length() != 0) {
	    filteredSongs = filteredSongs.artistMatches(artist);
	}
	if (number.length() != 0) {
	    filteredSongs = filteredSongs.numberMatches(number);
	}

	model.clear();
	int n = 0;
	java.util.Iterator<SongInformation> iter = filteredSongs.iterator();
	while(iter.hasNext()) {
	    model.add(n++, iter.next());
	}
    }

    public void resetSongs() {
	artistField.setText("");
	titleField.setText("");
	numberField.setText("");
	model.clear();
	int n = 0;
	java.util.Iterator<SongInformation> iter = allSongs.iterator();
	while(iter.hasNext()) {
	    model.add(n++, iter.next());
	}
    }
    /**
     * "play" a song by printing its id to standard out.
     * Can be used in a pipeline this way
     */
    public void playSong() {
	SongInformation song = (SongInformation) list.getSelectedValue();
	if (song == null) {
	    return;
	}
	long number = song.number + 1;
	System.out.println("" + number);
    }


    class SongInformationRenderer extends JLabel implements ListCellRenderer {

	public Component getListCellRendererComponent(
						      JList list,
						      Object value,
						      int index,
						      boolean isSelected,
						      boolean cellHasFocus) {
	    setText(value.toString());
	    return this;
	}
    }
}
	
      

When "play" is selected it will print the song id to standard output for use in a pipeline.

The data files

General

The files DTSMUS00.DKD - DTSMUS07.DKD contain the music files. There are two formats for the music: Microsoft WMA files and MIDI files. In my song books some songs are marked as having a singer. These turn out to be the WMA files. Those without a singer are MIDI files.

The WMA files are just that. The MIDI files are slightly compressed and have to be decoded before they can be played.

Each song block has at the beginning a section containing the lyrics. These are compressed and have to be decoded.

The data for one song forms a record of contiguous bytes. These records are collected into blocks, also contiguous. The blocks are separate. There is a "super block" of pointers to these blocks. Part of the song number is an index into the super block, selecting the block. The rest of the song number is an index of the record in the block.

My route into this

I came backwards into this and only arrived at understanding what others had accomplished after some time. So in case it helps any others, here is my route.

I used the Unix command strings to discover the songs information in DTSMUS10.DKD. On the other files it didn't seem to produce much. But there were ASCII strings in these files and some were repeated. So I wrote a shell pipeline to sort these strings and count them. The pipeline for one file was

	
	  strings DTSMUS05.DKD | sort |uniq -c | sort -n -r |less	
	
      

This produced results

	
	  1229 :^y|
	  1018 j?wK
	  843 ]/<
	  756  Seh
	  747  Ser
	  747 _\D+P
	  674 :^yt
	  234 IRI$	
	
      

The results weren't inspiring. But when I looked inside the files to see where "Ser" was occurring, I also saw:

	
	  q03C3E230  F6 01 00 00 00 02 00 16 00 57 00 69 00 6E 00 64 .........W.i.n.d
	  03C3E240  00 6F 00 77 00 73 00 20 00 4D 00 65 00 64 00 69 .o.w.s. .M.e.d.i
	  03C3E250  00 61 00 20 00 41 00 75 00 64 00 69 00 6F 00 20 .a. .A.u.d.i.o.
	  03C3E260  00 39 00 00 00 24 00 20 00 34 00 38 00 20 00 6B .9...$. .4.8. .k
	  03C3E270  00 62 00 70 00 73 00 2C 00 20 00 34 00 34 00 20 .b.p.s.,. .4.4.
	  03C3E280  00 6B 00 48 00 7A 00 2C 00 20 00 73 00 74 00 65 .k.H.z.,. .s.t.e
	  03C3E290  00 72 00 65 00 6F 00 20 00 31 00 2D 00 70 00 61 .r.e.o. .1.-.p.a
	  03C3E2A0  00 73 00 73 00 20 00 43 00 42 00 52 00 00 00 02 .s.s. .C.B.R....
	  03C3E2B0  00 61 01 91 07 DC B7 B7 A9 CF 11 8E E6 00 C0 0C .a..............
	  03C3E2C0  20 53 65 72 00 00 00 00 00 00 00 40 9E 69 F8 4D  Ser.......@.i.M	
	
      

Wow! two byte characters!

The strings command has options to look at e.g. 2-byte big-endian character strings. The command

	
	  strings -e b DTSMUS05.DKD
	
      

turned up

	
	  IsVBR
	  DeviceConformanceTemplate
	  WM/WMADRCPeakReference
	  WM/WMADRCAverageReference
	  WMFSDKVersion
	  9.00.00.2980
	  WMFSDKNeeded
	  0.0.0.0000
	
      

These are all part of the WMA format.

According to Gary Kessler's FILE SIGNATURES TABLE , the signature of a WMA file is given by the header

	
	  30 26 B2 75 8E 66 CF 11
	  A6 D9 00 AA 00 62 CE 6C	
	
      

and that pattern does occur, with the above strings appearing some time later.

The spec for the ASF/WMA file format is at Advanced Systems Format (ASF) Specification although you are advised not to read it in case you want to do anything open source with such files.

So on that basis I could indentify the start of WMA files. The 4 bytes preceding each WMA file are the length of the file. From that I could find the end of the file, which turned out to be the start of a record for the next record containing some stuff and then the next WMA file.

In these records I could see patterns I couldn't understand, but also from byte 36 on I could see strings like

	
	  AIN'T IT FUNNY HOW TIME SLIPS AWAY, Str length: 34


	  00000000  10 50 41 10 50 49 10 50 4E 10 50 27 10 50 54 10 .PA.PI.PN.P'.PT.
	  00000010  50 20 11 F1 25 12 71 05 04 61 05 05 51 21 13 01 P ..%.q..a..Q!..
	  00000020  02 05 91 2B 10 20 48 10 50 4F 10 50 57 13 40 00 ...+. H.PO.PW.@.
	  00000030  12 61 02 12 01 02 04 D1 05 04 51 3B 05 31 05 04 .a........Q;.1..
	  00000040  C1 29 10 20 50 10 51 45 10 21 28 10 21 1E 10 21 .). P.QE.!(.!..!
	  00000050  3A 14 F1 05 13 31 02 10 C1 0E 11 A1 58 15 A0 00 :....1......X...
	  00000060  15 70 00 13 A0 A9                               .p....
	
      

Can you see "A.I.N.'.T" ( as ".PA.PI.PN.P'.PT")?

But I couldn't figure out what the encoding was or how to find the table of song starts. That's when I was ready to look at the earlier stuff and understand how it applied to me. ( Understanding the HOTDOG files on DVD of California electronics , Decoding JBK 6628 DVD Karaoke Disc and Karaoke Huyndai 99 ).

The super block

The file DTSMUS00.DKD starts with a bunch of nulls. At 0x200 it starts to kick in with data. This was identified as the start of a "table of tables" i.e. a superblock. Each entry in this superblock is a 4-byte integer, which turns out to be an index to tables in the data files. The superblock is terminated by a sequence of nulls (for me at 0x5F4) and there are less than 256 indexes in the table.

The value of these superblock entries seems to have changed in different versions. In the JBK disk and also on mine, the values have to be multiplied by 0x800 to give a "virtual offset" in the data files.

To give meaning to this: on my disk at 0x200 is

	
	  00000200  00 00 00 01 00 00 08 6C 00 00 0F C1 00 00 17 7A 
	  00000210  00 00 1E 81 00 00 25 21 00 00 2B 8D 00 00 32 B7 
	
      

So the table values are 0x1, 0x86C, 0xFC1, 0x177A, ... The "virtual addresses" are 0x800, 0x436000 (0x86C * 0x800) and so on. If you go to these addresses, then before the address is a bunch of nulls, and at that address is data.

Why I call them virtual addresses is because there are 8 data files on my DVD and most addresses are larger than any of the files. The files in my case are all 1065353216L (except the last) bytes. The "obvious" solution works: the file number is address / file size, and the offset into the file is address % file size. You can check this by looking for the nulls before the address of each block.

Song start tables

Each of the tables indexed from the super block is a table of song indexes. Each table contains 4-byte indexes. Each table has at most 0x100 entries, or is terminated by a zero index. Each index is the offset from the table start of the beginning of a song entry.

Locating song entry from song number

Given a song number such as 54154 "Here Comes The Sun" we can now find the song entry. Reduce the song number by one to 54153. It is a 16-bit number. The top 8 bits are the index of the song index table in the superblock. The bottom 8 bits are the index of the song entry in the song index table.

Pseudocode:

	
	  songNumber = get number for song from DTSMUS20.DKD
	  superBlockIdx = songNumber >>
	  indexTableIdx = songNumber & 0xFF

	  seek(DTSMUS00.DKD, superBlockIdx) 
	  superBlockValue = read 4-byte int from DTSMUS00.DKD

	  locationIndexTable = superBlockValue * 0x800
	  fileNumber = locationIndexTable / fileSize
	  indexTableStart = locationIndexTable % fileSize
	  entryLocation = indexTableStart + indexTableIdx 

	  seek(fileNumber, entryLocation)
	  read song entry
	
      

Song entries

Each song entry has a header and is followed by two blocks that I call the information block and the song data block. Each header block has a 2-byte type code and a 2-byte integer length. The type code is either 0x0800 or 0x0000. The code signals the encoding of the song data: 0x0800 is a WMA file while 0x0000 is a MIDI file.

If the type code is 0x0 such as the Beatles "Help!" (song number 51765) then the information block has the length in the header block and starts 12 bytes further in. The song data block immediately follows this.

If the type code is 0x8000 then the information block starts 4 bytes in for the length given in the header. The song block starts on the next 16-byte boundary from the end of the information block.

The song block starts with a 4-byte header which is the length of the song data for all types.

Song data

If the song type is 0x8000 then the song data is a WMA file. All songs looked at have a singer included in this file.

If the song type is 0x0 then (from the book) there is no singer in the songs looked at. The file is encoded, and decodes to a MIDI file.

Decoding MIDI files

All files have a lyric block followed by a music block. The lyric block is compressed and it has been discovered that this is LZW compression. This decompresses to a set of 4-byte chunks. The first two bytes are characters of the lyric. For 1-byte encodings such as English or Vietnamese, the first byte is one character and the second is either zero or another character (two byts such as "\r\n"). For two byte encodings such as GB-2312, the two bytes form one character.

The next two bytes are the length of time the character string plays for.

Lyric block

Each lyric block starts with strings such as "#0001 @@00@12 @Help Yourself @ @@Tom Jones " The language code is in there as NN in "@00@NN". The song title, writer, singer are clear. (Note: these characters are all 4 bytes apart!). For English it is "12" and so on.

Bytes 0 and 1 of each block are a character in the lyric. Bytes 2 and 3 are the duration of each character. To turn them into MIDI data, the durations have to be turned into start/stop of each character.

My Java program to do this is SongExtracter.java

	
      
import java.io.*;
import javax.sound.midi.*;
import java.nio.charset.Charset;

public class SongExtracter {
    private static final boolean DEBUG = false;

    private String[] dataFiles = new String[] {
	"DTSMUS00.DKD", "DTSMUS01.DKD", "DTSMUS02.DKD",
	"DTSMUS03.DKD", "DTSMUS04.DKD", "DTSMUS05.DKD",
	"DTSMUS06.DKD", "DTSMUS07.DKD"};
    private String superBlockFileName = dataFiles[0];
    private static final String DATADIR = "/home/newmarch/Music/karaoke/sonken/";
    private static final String SONGDIR ="/home/newmarch/Music/karaoke/sonken/songs/";
    //private static final String SONGDIR ="/server/KARAOKE/KARAOKE/Sonken/";
    private static final long SUPERBLOCK_OFFSET = 0x200;
    private static final long BLOCK_MULTIPLIER = 0x800;
    private static final long FILE_SIZE = 0x3F800000L;

    private static final int SIZE_UINT = 4;
    private static final int SIZE_USHORT = 2;

    private static final int ENGLISH = 12;

    public RawSong getRawSong(int songNumber) 
	throws java.io.IOException, 
	       java.io.FileNotFoundException {
	if (songNumber < 1) {
	    throw new FileNotFoundException();
	}

	// song number in files is one less than song number in books, so
	songNumber--;

	long locationIndexTable = getTableIndexFromSuperblock(songNumber);
	debug("Index table at %X\n", locationIndexTable);

	long locationSongDataBlock = getSongIndex(songNumber, locationIndexTable);

	// Now we are at the start of the data block
	return readRawSongData(locationSongDataBlock);

	//debug("Data block at %X\n", songStart);
    }

    private long getTableIndexFromSuperblock(int songNumber)
	throws java.io.IOException, 
	       java.io.FileNotFoundException {
	// index into superblock of table of song offsets
	int superBlockIdx = songNumber >> 8;

	debug("Superblock index %X\n", superBlockIdx);
	    
	File superBlockFile = new File(DATADIR + superBlockFileName);

        FileInputStream fstream = new FileInputStream(superBlockFile);

	fstream.skip(SUPERBLOCK_OFFSET + superBlockIdx * SIZE_UINT);
	debug("Skipping to %X\n", SUPERBLOCK_OFFSET + superBlockIdx*4);
	long superBlockValue = readUInt(fstream);

	// virtual address of the index table for this song
	long locationIndexTable = superBlockValue * BLOCK_MULTIPLIER;

	return locationIndexTable;
    }

    /*
     * Virtual address of song data block
     */
    private long getSongIndex(int songNumber, long locationIndexTable) 
	throws java.io.IOException, 
	       java.io.FileNotFoundException {
	// index of song into table of song ofsets
	int indexTableIdx = songNumber & 0xFF;
	debug("Index into index table %X\n", indexTableIdx);

	// translate virtual address to physical address
	int whichFile = (int) (locationIndexTable / FILE_SIZE);
	long indexTableStart =  locationIndexTable % FILE_SIZE;
	debug("Which file %d index into file %X\n", whichFile, indexTableStart);

	File songDataFile = new File(DATADIR + dataFiles[whichFile]);
        FileInputStream dataStream = new FileInputStream(songDataFile);
	dataStream.skip(indexTableStart + indexTableIdx * SIZE_UINT);
	debug("Song data index is at %X\n", indexTableStart + indexTableIdx*SIZE_UINT);

	long songStart = readUInt(dataStream) + indexTableStart;

	return songStart + whichFile * FILE_SIZE;
    }

    private RawSong readRawSongData(long locationSongDataBlock) 
	throws java.io.IOException {
	int whichFile = (int) (locationSongDataBlock / FILE_SIZE);
	long dataStart =  locationSongDataBlock % FILE_SIZE;
	debug("Which song file %d  into file %X\n", whichFile, dataStart);

	File songDataFile = new File(DATADIR + dataFiles[whichFile]);
        FileInputStream dataStream = new FileInputStream(songDataFile);
	dataStream.skip(dataStart);

	RawSong rs = new RawSong();
	rs.type = readUShort(dataStream);
	rs.compressedLyricLength = readUShort(dataStream);
	// discard next short
	readUShort(dataStream);
	rs.uncompressedLyricLength = readUShort(dataStream);
	debug("Type %X, cLength %X uLength %X\n", rs.type, rs.compressedLyricLength, rs.uncompressedLyricLength);

	// don't know what the next word is for, skip it
	//dataStream.skip(4);
	readUInt(dataStream);

	// get the compressed lyric
	rs.lyric = new byte[rs.compressedLyricLength];
	dataStream.read(rs.lyric);

	long toBoundary = 0;
	long songLength = 0;
	long uncompressedSongLength = 0;

	// get the song data
	if (rs.type == 0) {
	    // Midi file starts in 4 bytes time
	    songLength = readUInt(dataStream);
	    uncompressedSongLength = readUInt(dataStream);
	    System.out.printf("Song data length %d, uncompressed %d\n", 
			      songLength, uncompressedSongLength);
	    rs.uncompressedSongLength = uncompressedSongLength;

	    // next word is language again?
	    //toBoundary = 4;
	    //dataStream.skip(toBoundary);
	    readUInt(dataStream);
	} else {
	    // WMA starts on next 16-byte boundary
	    if( (dataStart + rs.compressedLyricLength + 12) % 16 != 0) {
		// dataStart already on 16-byte boundary, so just need extra since then
		toBoundary = 16 - ((rs.compressedLyricLength + 12) % 16);
		debug("Read lyric data to %X\n", dataStart + rs.compressedLyricLength + 12);
		debug("Length %X to boundary %X\n", rs.compressedLyricLength, toBoundary);
		dataStream.skip(toBoundary);
	    }
	    songLength = readUInt(dataStream);
	}

	rs.music = new byte[(int) songLength];
	dataStream.read(rs.music);

	return rs;
    }

    private long readUInt(InputStream is) throws IOException {
	long val = 0;
	for (int n = 0; n < SIZE_UINT; n++) {
	    int c = is.read();
	    val = (val << 8) + c;
	}
	debug("ReadUInt %X\n", val);
	return val;
    }

    private int readUShort(InputStream is) throws IOException {
	int val = 0;
	for (int n = 0; n < SIZE_USHORT; n++) {
	    int c = is.read();
	    val = (val << 8) + c;
	}
	debug("ReadUShort %X\n", val);
	return val;
    }

    void debug(String f, Object ...args) {
	if (DEBUG) {
	    System.out.printf("Debug: " + f, args);
	}
    }

    public Song getSong(RawSong rs) {
	Song song;
	if (rs.type == 0x8000) {
	    song = new WMASong(rs);
	} else {
	    song = new MidiSong(rs);
	}
	return song;
    }

    public static void main(String[] args) {
	if (args.length != 1) {
	    System.err.println("Usage: java SongExtractor <song numnber>");
	    System.exit(1);
	}

	SongExtracter se = new SongExtracter();
	try {
	    RawSong rs = se.getRawSong(Integer.parseInt(args[0]));
	    rs.dumpToFile(args[0]);

	    Song song = se.getSong(rs);
	    song.dumpToFile(args[0]);
	    song.dumpLyric();
	} catch(Exception e) {
	    e.printStackTrace();
	}
    }

    private class RawSong {
	/**
	 * type == 0x0 is Midi
	 * type == 0x8000 is WMA
	 */
	public int type;
	public int compressedLyricLength;
	public int uncompressedLyricLength;
	public long uncompressedSongLength; // only needed for compressed Midi
	public byte[] lyric;
	public byte[] music;

	public void dumpToFile(String fileName) throws IOException {
	    FileOutputStream fout = new FileOutputStream(SONGDIR + fileName + ".lyric");
	    fout.write(lyric);
	    fout.close();

	    fout = new FileOutputStream(SONGDIR + fileName + ".music");
	    fout.write(music);
	    fout.close();
	    System.out.println("Dumped to " + filename):
	}
    }

    private class Song {
	public int type;
	public byte[] lyric;
	public byte[] music;
	protected Sequence sequence;
	protected int language = -1;

	public Song(RawSong rs) {
	    type = rs.type;
	    lyric = decodeLyric(rs.lyric,
				rs.uncompressedLyricLength);
	}

	/**
	 * Raw lyric is LZW compressed. Decompress it
	 */
	public byte[] decodeLyric(byte[] compressedLyric, long uncompressedLength) {
	    // uclen is short by at least 2 - other code adds 10 so we do too
	    // TODO: change LZW to use a Vector to build result so we don't have to guess at length
	    byte[] result = new byte[(int) uncompressedLength + 10];
	    LZW lzw = new LZW();
	    int len = lzw.expand(compressedLyric, compressedLyric.length, result);
	    System.out.printf("uncompressedLength %d, actual %d\n", uncompressedLength, len);
	    lyric = new byte[len];
	    System.arraycopy(result, 0, lyric, 0, (int) uncompressedLength);
	    return lyric;
	}

	public void dumpToFile(String fileName) throws IOException {
	    FileOutputStream fout = new FileOutputStream(SONGDIR + fileName + ".decodedlyric");
	    fout.write(lyric);
	    fout.close();
	    
	    fout = new FileOutputStream(SONGDIR + fileName + ".decodedmusic");
	    fout.write(music);
	    fout.close();
	    
	    fout = new FileOutputStream(SONGDIR + fileName + ".mid");
	    if (sequence == null)  {
		System.out.println("Seq is null");
	    } else {
		// type is MIDI type 0
		MidiSystem.write(sequence, 0, fout);
	    }
	}

	public void dumpLyric() {
	    for (int n = 0; n < lyric.length; n += 4) {
		if (lyric[n] == '\r') {
		    System.out.println();
		} else {
		    System.out.printf("%c", lyric[n] & 0xFF);
		}
	    }
	    System.out.println();
	    System.out.printf("Language is %X\n", getLanguageCode()); 
	}

	/**
	 * Lyric contains the language code as string @00@NN in header section
	 */
	public int getLanguageCode() {
	    int lang = 0;

	    // Look for @00@NN and return NN
	    for (int n = 0; n < lyric.length-20; n += 4) {
		if (lyric[n] == (byte) '@' &&
		    lyric[n+4] == (byte) '0' &&
		    lyric[n+8] == (byte) '0' &&
		    lyric[n+12] == (byte) '@') {
		    lang = ((lyric[n+16]-'0') << 4) + lyric[n+20]-'0';
		    break;
		}
	    }
	    return lang;
	}

	/**
	 * Lyric is in a language specific encoding. Translate to Unicode UTF-8.
	 * Not all languages are handled because I don't have a full set of examples
	 */
	public byte[] lyricToUnicode(byte[] bytes) {
	    if (language == -1) {
		language = getLanguageCode();
	    }
	    switch (language) {
	    case SongInformation.ENGLISH:
		return bytes;

	    case SongInformation.KOREAN: {
 		Charset charset = Charset.forName("gb2312");
		String str = new String(bytes, charset);
		bytes = str.getBytes();
		System.out.println(str);
		return bytes;
	    }

	    case SongInformation.CHINESE1:
	    case SongInformation.CHINESE2:
	    case SongInformation.CHINESE8:
	    case SongInformation.CHINESE131:
	    case SongInformation.TAIWANESE3:
	    case SongInformation.TAIWANESE7:
	    case SongInformation.CANTONESE:
		Charset charset = Charset.forName("gb2312");
		String str = new String(bytes, charset);
		bytes = str.getBytes();
		System.out.println(str);
		return bytes;
	    }
	    // language not handled
	    return bytes;
	}

	public void durationToOnOff() {

	}

	public Track createSequence() {
	    Track track;

	    try {
		sequence = new Sequence(Sequence.PPQ, 30);
	    } catch(InvalidMidiDataException e) {
		// help!!!
	    }
	    track = sequence.createTrack();
	    addLyricToTrack(track);
	    return track;
	}

	public void addMsgToTrack(MidiMessage msg, Track track, long tick) {
	    MidiEvent midiEvent = new MidiEvent(msg, tick);

	    
	    // No need to sort or delay insertion. From the Java API
	    // "The list of events is kept in time order, meaning that this
	    // event inserted at the appropriate place in the list"
	    track.add(midiEvent);
	}

	/**
	 * return byte as int, converting to unsigned if needed
	 */
	protected int ub2i(byte b) {
	    return  b >= 0 ? b : 256 + b;
	}

	public void addLyricToTrack(Track track) {
	    long lastDelay = 0;
	    int offset = 0;
	    int data0;
	    int data1;
	    final int LYRIC = 0x05;
	    MetaMessage msg;

	    while (offset < lyric.length-4) {
		int data3 = ub2i(lyric[offset+3]);
		int data2 = ub2i(lyric[offset+2]);
		data0 = ub2i(lyric[offset]);
		data1 = ub2i(lyric[offset+1]);

		long delay = (data3 << 8) + data2;

		offset += 4;
		byte[] data;
		int len;
		long tick;

		// 	System.out.printf("Lyric offset %X char %X after %d with delay %d made of %d %d\n", offset, data0, lastDelay, delay, lyric[offset-1], lyric[offset-2]);

		if (data1 == 0) {
		    data = new byte[] {(byte) data0}; //, (byte) MetaMessage.META};
		} else {
		    data = new byte[] {(byte) data0, (byte) data1}; // , (byte) MetaMessage.META};
		}
		data = lyricToUnicode(data);
		    
		msg = new MetaMessage();

		if (delay > 0) {
		    tick = delay;
		    lastDelay = delay;
		} else {
		    tick = lastDelay;
		}
		
		try {
		    msg.setMessage(LYRIC, data, data.length);
		} catch(InvalidMidiDataException e) {
		    e.printStackTrace();
		    continue;
		}
		addMsgToTrack(msg, track, tick);
	    }
	}

    }

    private class WMASong extends Song {

	public WMASong(RawSong rs) {
	    // We want to decode the lyric, but just copy the music data
	    super(rs);
	    music = rs.music;
	    createSequence();
	}

	public void dumpToFile(String fileName) throws IOException {
	    System.out.println("Dumping WMA to " + fileName + ".wma");
	    super.dumpToFile(fileName);
	    FileOutputStream fout = new FileOutputStream(fileName + ".wma");
	    fout.write(music);
	    fout.close();
	}

    }

    private class MidiSong extends Song {

        private String[] keyNames = {"C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"};

	public MidiSong(RawSong rs) {
	    // We want the decoded lyric plus also need to decode the music
	    // and then turn it into a Midi sequence
	    super(rs);
	    decodeMusic(rs);
	    createSequence();
	}

	public void dumpToFile(String fileName) throws IOException {
	    System.out.println("Dumping Midi to " + fileName);
	    super.dumpToFile(fileName);
	}

        public String getKeyName(int nKeyNumber)
        {
	    if (nKeyNumber > 127)
                {
		    return "illegal value";
                }
	    else
                {
		    int     nNote = nKeyNumber % 12;
		    int     nOctave = nKeyNumber / 12;
		    return keyNames[nNote] + (nOctave - 1);
                }
        }

	public byte[] decodeMusic(RawSong rs) {
	    byte[]  compressedMusic = rs.music;
	    long uncompressedSongLength = rs.uncompressedSongLength;

	    // TODO: change LZW to use a Vector to build result so we don't have to guess at length
	    byte[] expanded = new byte[(int) uncompressedSongLength + 20];
	    LZW lzw = new LZW();
	    int len = lzw.expand(compressedMusic, compressedMusic.length, expanded);
	    System.out.printf("Uncompressed %d, Actual %d\n", compressedMusic.length, len);
	    music = new byte[len];
	    System.arraycopy(expanded, 0, music, 0, (int) len);


	    return music;
	}

	public Track createSequence() {
	    Track track = super.createSequence();
	    addMusicToTrack(track);
	    return track;
	}



	public void addMusicToTrack(Track track) {
	    int timeLine = 0;
	    int offset = 0;
	    int midiChannelNumber = 1;

	    /* From http://board.midibuddy.net/showpost.php?p=533722&postcount=31
	       Block of 5 bytes :
	       xx xx xx xx xx
	       1st byte = Delay Time
	       2nd byte = Delay Time when the velocity will be 0, 
	       this one will generate another midi event 
	       with velocity 0 (see above).
	       3nd byte = Event, for example 9x : Note On for channel x+1,
	       cx for PrCh, bx for Par, ex for Pitch Bend....
	       4th byte = Note
	       5th byte = Velocity
	    */
	    System.out.println("Adding music to track");
	    while (offset < music.length - 5) {

		int startDelayTime = ub2i(music[offset++]);
		int endDelayTime = ub2i(music[offset++]);
		int event = ub2i(music[offset++]);
		int data1 = ub2i(music[offset++]);
		int data2 = ub2i(music[offset++]);


		int tick = timeLine + startDelayTime;
		System.out.printf("Offset %X event %X timeline %d\n", offset, event & 0xFF, tick);

		ShortMessage msg = new ShortMessage();
		ShortMessage msg2 = null;

		try {
		    // For Midi event types see http://www.midi.org/techspecs/midimessages.php
		    switch (event & 0xF0) {
		    case ShortMessage.CONTROL_CHANGE:  // Control Change 0xB0
		    case ShortMessage.PITCH_BEND:  // Pitch Wheel Change 0xE0
			msg.setMessage(event, data1, data2);
			/*
			  writeChannel(midiChannelNumber, chunk[2], false);
			  writeChannel(midiChannelNumber, chunk[3], false);
			  writeChannel(midiChannelNumber, chunk[4], false);
			*/
			break;

		    case ShortMessage.PROGRAM_CHANGE: // Program Change 0xC0
		    case ShortMessage.CHANNEL_PRESSURE: // Channel Pressure (After-touch) 0xD0
			msg.setMessage(event, data1, 0);
			break;

		    case 0x00:
			// case 0x90:
			// Note on
			int note = data1;
			int velocity = data2;

			/* We have to generate a pair of note on/note off.
			   The C code manages getting the order of events
			   done correctly by keeping a list of note off events
			   and sticking them into the Midi sequence when appropriate.
			   The Java add() looks after timing for us, so we'll
			   generate a note off first and add it, and then do the note on
			*/
			System.out.printf("Note on %s at %d, off at %d at offset %X channel %d\n", 
					  getKeyName(note),
					  tick, tick + endDelayTime, offset, (event &0xF)+1);
			// ON
			msg.setMessage(ShortMessage.NOTE_ON | (event & 0xF),
				       note, velocity);

			// OFF
			msg2 = new ShortMessage();
			msg2.setMessage(ShortMessage.NOTE_OFF  | (event & 0xF), 
					note, velocity);

			break;

		    case 0xF0: // System Exclusive
			// We'll write the data as is to the buffer
			offset -= 3;
			// msg = SysexMessage();
			while (music[offset] != (byte) 0xF7) // bytes only go upto 127 GRRRR!!!
			    {
				//				writeChannel(midiChannelNumber, midiData[midiOffset], false);
				System.out.printf("sysex: %x\n", music[offset]);
				offset++;
				if (offset >= music.length) {
				    System.err.println("Run off end of array while processing Sysex");
				    break;
				}
			    }
			//			writeChannel(midiChannelNumber, midiData[midiOffset], false);
			offset++;
			System.out.printf("Ignoring sysex %02X\n", event);

			// ignore the message for now
			continue;
			// break;

		    default:
			System.out.printf("Unrecognized code %02X\n", event);
			continue;
		    }
		} catch(InvalidMidiDataException e) {
		    e.printStackTrace();
		}

		addMsgToTrack(msg, track, tick);
		if (msg2 != null ) {
		    if (endDelayTime <= 0) System.out.println("Start and end at same time");
		    addMsgToTrack(msg2, track, tick + endDelayTime);
		    msg2 = null;
		}

		timeLine = tick;
	    }
	}
    }
}

	
      

with support classes LZW.java

	
      
/**
 * Based on code by Mark Nelson
 * http://marknelson.us/1989/10/01/lzw-data-compression/
 */

public class LZW {


    private final int BITS = 12;                   /* Setting the number of bits to 12, 13*/
    private final int HASHING_SHIFT = (BITS-8);    /* or 14 affects several constants.    */
    private final int MAX_VALUE = (1 << BITS) - 1; /* Note that MS-DOS machines need to   */
    private final int MAX_CODE = MAX_VALUE - 1;    /* compile their code in large model if*/
    /* 14 bits are selected.               */

    private final int TABLE_SIZE = 5021;           /* The string table size needs to be a */
    /* prime number that is somewhat larger*/
    /* than 2**BITS.                       */
    private final int NEXT_CODE = 257;

    private long[] prefix_code = new long[TABLE_SIZE];;        /* This array holds the prefix codes   */
    private int[] append_character = new int[TABLE_SIZE];  /* This array holds the appended chars */
    private int[] decode_stack; /* This array holds the decoded string */

    private int input_bit_count=0;
    private long input_bit_buffer=0; // must be 32 bits
    private int offset = 0;

    /*
    ** This routine simply decodes a string from the string table, storing
    ** it in a buffer.  The buffer can then be output in reverse order by
    ** the expansion program.
    */
    /* JN: returns size of buffer used 
     */
    private int decode_string(int idx, long code)
    {
	int i;

	i=0;
	while (code > (NEXT_CODE - 1))
	    {
		decode_stack[idx++] = append_character[(int) code];
		code=prefix_code[(int) code];
		if (i++>=MAX_CODE)
		    {
			System.err.printf("Fatal error during code expansion.\n");
			return 0;
		    }
	    }

	decode_stack[idx]= (int) code;

	return idx;
    }

    /*
    ** The following two routines are used to output variable length
    ** codes.  They are written strictly for clarity, and are not
    ** particularyl efficient.
    */

    long input_code(byte[] inputBuffer, int inputLength, int dummy_offset, boolean firstTime)
    {
	long return_value;

	//int pOffsetIdx = 0;
	if (firstTime)
	    {
		input_bit_count = 0;
		input_bit_buffer = 0;
	    }

	while (input_bit_count <= 24 && offset < inputLength)
	    {
		/*
		input_bit_buffer |= (long) inputBuffer[offset++] << (24 - input_bit_count);
		input_bit_buffer &= 0xFFFFFFFFL;
		System.out.printf("input buffer %d\n", (long) inputBuffer[offset]);
		*/
		// Java doesn't have unsigned types. Have to play stupid games when mixing
		// shifts and type coercions
		long val = inputBuffer[offset++];
		if (val < 0) {
		    val = 256 + val;
		}
		// System.out.printf("input buffer: %d\n", val);
		//		if ( ((long) inpu) < 0) System.out.println("Byte is -ve???");
		input_bit_buffer |= (((long) val) << (24 - input_bit_count)) & 0xFFFFFFFFL;
		//input_bit_buffer &= 0xFFFFFFFFL;
		// System.out.printf("input bit buffer %d\n", input_bit_buffer);

		/*
		if (input_bit_buffer < 0) {
		    System.err.println("Negative!!!");
		}
		*/

		input_bit_count  += 8;
	    }

	if (offset >= inputLength && input_bit_count < 12)
	    return MAX_VALUE;

	return_value       = input_bit_buffer >>> (32 - BITS);
	input_bit_buffer <<= BITS;
	input_bit_buffer &= 0xFFFFFFFFL;
	input_bit_count   -= BITS;

	return return_value;
    }

    void dumpLyric(int data)
    {
	System.out.printf("LZW: %d\n", data);
	if (data == 0xd)
	    System.out.printf("\n");	      
    }

    /*
    **  This is the expansion routine.  It takes an LZW format file, and expands
    **  it to an output file.  The code here should be a fairly close match to
    **  the algorithm in the accompanying article.
    */

    public int expand(byte[] intputBuffer, int inputBufferSize, byte[] outBuffer)
    {
	long next_code = NEXT_CODE;/* This is the next available code to define */
	long new_code;
	long old_code;
	int character;
	int string_idx;
	
	int offsetOut = 0;


	prefix_code      = new long[TABLE_SIZE];
	append_character = new int[TABLE_SIZE];
	decode_stack     = new int[4000];

	old_code= input_code(intputBuffer, inputBufferSize, offset, true);  /* Read in the first code, initialize the */
	character = (int) old_code;          /* character variable, and send the first */
	outBuffer[offsetOut++] = (byte) old_code;       /* code to the output file                */
	//outTest(output, old_code);
	// dumpLyric((int) old_code);

	/*
	**  This is the main expansion loop.  It reads in characters from the LZW file
	**  until it sees the special code used to inidicate the end of the data.
	*/
	while ((new_code=input_code(intputBuffer, inputBufferSize, offset, false)) != (MAX_VALUE))
	    {
		// dumpLyric((int)new_code);
		/*
		** This code checks for the special STRING+CHARACTER+STRING+CHARACTER+STRING
		** case which generates an undefined code.  It handles it by decoding
		** the last code, and adding a single character to the end of the decode string.
		*/

		if (new_code>=next_code)
		    {
			if (new_code > next_code)
			    {
				System.err.printf("Invalid code: offset:%X new:%X next:%X\n", offset, new_code, next_code);
				break;
			    }

			decode_stack[0]= (int) character;
			string_idx=decode_string(1, old_code);
		    }
		else
		    {
			/*
			** Otherwise we do a straight decode of the new code.
			*/
			string_idx=decode_string(0,new_code);
		    }

		/*
		** Now we output the decoded string in reverse order.
		*/
		character=decode_stack[string_idx];
		while (string_idx >= 0)
		    {
			int data = decode_stack[string_idx--]; 
			outBuffer[offsetOut] = (byte) data;
			//outTest(output, *string--);

			if (offsetOut % 4 == 0) {
			    //dumpLyric(data);
			}

			offsetOut++;
		    }

		/*
		** Finally, if possible, add a new code to the string table.
		*/
		if (next_code > 0xfff)
		    {
			next_code = NEXT_CODE;
			System.err.printf("*");
		    }

		// test code
		if (next_code > 0xff0 || next_code < 0x10f)
		    {
			Debug.printf("%02X ", new_code);
		    }

		prefix_code[(int) next_code]=old_code;
		append_character[(int) next_code] = (int) character;
		next_code++;

		old_code=new_code;
	    }
	Debug.printf("offset out is %d\n", offsetOut);
	return offsetOut;
    }
}
	
      

SongInformation.java

	
      


public class SongInformation {


    // Public fields of each song record
    /**
     *  Song number in the file, one less than in songbook
     */
    public long number;

    /**
     * song title in Unicode
     */
    public String title;

    /**
     * artist in Unicode
     */
    public String artist;

    /**
     * integer value of language code
     */
    public int language;

    public static final int  KOREAN = 0;
    public static final int  CHINESE1 = 1;
    public static final int  CHINESE2 = 2;
    public static final int  TAIWANESE3 = 3 ;
    public static final int  JAPANESE = 4;
    public static final int  RUSSIAN = 5;
    public static final int  THAI = 6;
    public static final int  TAIWANESE7 = 7;
    public static final int  CHINESE8 = 8;
    public static final int  CANTONESE = 9;
    public static final int  ENGLISH = 0x12;
    public static final int  VIETNAMESE = 0x13;
    public static final int  PHILIPPINE = 0x14;
    public static final int  TURKEY = 0x15;
    public static final int  SPANISH = 0x16;
    public static final int  INDONESIAN = 0x17;
    public static final int  MALAYSIAN = 0x18;
    public static final int  PORTUGUESE = 0x19;
    public static final int  FRENCH = 0x20;
    public static final int  INDIAN = 0x21;
    public static final int  BRASIL = 0x22;
    public static final int  CHINESE131 = 131;
    public static final int  ENGLISH146 = 146;
    public static final int  PHILIPPINE148 = 148;

    public SongInformation(long number,
			   String title,
			   String artist,
			   int language) {
	this.number = number;
	this.title = title;
	this.artist = artist;
	this.language = language;
    }

    public String toString() {
	return "" + (number+1) + " (" + language + ") \"" + title + "\" " + artist;
    }

    public boolean titleMatch(String pattern) {
	// System.out.println("Pattern: " + pattern);
	return title.matches("(?i).*" + pattern + ".*");
    }

    public boolean artistMatch(String pattern) {
	return artist.matches("(?i).*" + pattern + ".*");
    }

    public boolean numberMatch(String pattern) {
	Long n;
	try {
	    n = Long.parseLong(pattern) - 1;
	    //System.out.println("Long is " + n);
	} catch(Exception e) {
	    //System.out.println(e.toString());
	    return false;
	}
	return number == n;
    }


    public boolean languageMatch(int lang) {
	return language == lang;
    }
}

	
      

and Debug.java

	
      

public class Debug {

    public static final boolean DEBUG = false;

    public static void println(String str) {
	if (DEBUG) {
	    System.out.println(str);
	}
    }

    public static void printf(String format, Object... args) {
	if (DEBUG) {
	    System.out.printf(format, args);
	}
    }
}
	    
	    
	
      

To compile these, run

	
    javac SongExtracter.java LZW.java Debug.java SongInformation.java
	
      

and run by

	
java SongExtracter <song number >
	
      

The program to convert these MIDI files to Karaoke KAR files is KARConverter.java

	
      /*
 * KARConverter.java
 *
 * The output from decodnig the Sonken data is not in
 * the format required by the KAR "standard".
 * e.g. we need @T for the title,
 * and LYRIC events need to be changed to TEXT events
 * Tempo has to be changed too
 *
 */

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;

import javax.sound.midi.MidiSystem;
import javax.sound.midi.InvalidMidiDataException;
import javax.sound.midi.Sequence;
import javax.sound.midi.Track;
import javax.sound.midi.MidiEvent;
import javax.sound.midi.MidiMessage;
import javax.sound.midi.ShortMessage;
import javax.sound.midi.MetaMessage;
import javax.sound.midi.SysexMessage;
import javax.sound.midi.Receiver;




public class KARConverter {
    private static int LYRIC = 5;
    private static int TEXT = 1;

    private static boolean firstLyricEvent = true;

    public static void main(String[] args) {
	if (args.length != 1) {
	    out("KARConverter: usage:");
	    out("\tjava KARConverter <file>");
	    System.exit(1);
	}
	/*
	 *	args[0] is the common prefix of the two files
	 */
	File	inFile = new File(args[0] + ".mid");
	File	outFile = new File(args[0] + ".kar");

	/*
	 *	We try to get a Sequence object, which the content
	 *	of the MIDI file.
	 */
	Sequence	inSequence = null;
	Sequence	outSequence = null;
	try {
	    inSequence = MidiSystem.getSequence(inFile);
	} catch (InvalidMidiDataException e) {
	    e.printStackTrace();
	    System.exit(1);
	} catch (IOException e) {
	    e.printStackTrace();
	    System.exit(1);
	}

	if (inSequence == null) {
	    out("Cannot retrieve Sequence.");
	} else {
	    try {
		outSequence = new Sequence(inSequence.getDivisionType(),
					   inSequence.getResolution());
	    } catch(InvalidMidiDataException e) {
		e.printStackTrace();
		System.exit(1);
	    }
		    
	    createFirstTrack(outSequence);
	    Track[]	tracks = inSequence.getTracks();
	    fixTrack(tracks[0], outSequence);
	}
	FileOutputStream outStream = null;
	try {
	    outStream = new FileOutputStream(outFile);
	    MidiSystem.write(outSequence, 1, outStream);
	} catch(Exception e) {
	    e.printStackTrace();
	    System.exit(1);
	}
    }


    public static void fixTrack(Track oldTrack, Sequence seq) {
	Track lyricTrack = seq.createTrack();
	Track dataTrack = seq.createTrack();

	int nEvent = fixHeader(oldTrack, lyricTrack);
	System.out.println("nEvent " + nEvent);
	for ( ; nEvent < oldTrack.size(); nEvent++) {
	    MidiEvent event = oldTrack.get(nEvent);
	    if (isLyricEvent(event)) {
		event = convertLyricToText(event);
		lyricTrack.add(event);
	    } else {
		dataTrack.add(event);
	    }
	}
    }

    public static int fixHeader(Track oldTrack, Track lyricTrack) {
	int nEvent;

	// events at 0-10 are meaningless
	// events at 11, 12 should be the language code,
	// but maybe at 12, 13
	nEvent = 11;
	MetaMessage lang1 = (MetaMessage) (oldTrack.get(nEvent).getMessage());
	String val = new String(lang1.getData());
	if (val.equals("@")) {
	    // try 12
	    lang1 = (MetaMessage) (oldTrack.get(++nEvent).getMessage());
	}		
	MetaMessage lang2 = (MetaMessage) (oldTrack.get(++nEvent).getMessage());
	String lang = new String(lang1.getData()) +
	    new String(lang2.getData());
	System.out.println("Lang " + lang);
	byte[] karLang = getKARLang(lang);

	MetaMessage msg = new MetaMessage();
	try {
	    msg.setMessage(TEXT, karLang, karLang.length);
	    MidiEvent evt = new MidiEvent(msg, 0L);
	    lyricTrack.add(evt);
	} catch(InvalidMidiDataException e) {
	}

	// song title is next
	StringBuffer titleBuff = new StringBuffer();
	for (nEvent = 15; nEvent < oldTrack.size(); nEvent++) {
	    MidiEvent event = oldTrack.get(nEvent);
	    msg = (MetaMessage) (event.getMessage());
	    String contents = new String(msg.getData());
	    if (contents.equals("@")) {
		break;
	    }
	    if (contents.equals("\r\n")) {
		continue;
	    }
	    titleBuff.append(contents);
	}
	String title = "@T" + titleBuff.toString();
	System.out.println("Title '" + title +"'");
	byte[] titleBytes = title.getBytes();

	msg = new MetaMessage();
	try {
	    msg.setMessage(TEXT, titleBytes, titleBytes.length);
	    MidiEvent evt = new MidiEvent(msg, 0L);
	    lyricTrack.add(evt);
	} catch(InvalidMidiDataException e) {
	}

	
	// skip the next 2 @'s
	for (int skip = 0; skip < 2; skip++) {
	    for (++nEvent; nEvent < oldTrack.size(); nEvent++) {
		MidiEvent event = oldTrack.get(nEvent);
		msg = (MetaMessage) (event.getMessage());
		String contents = new String(msg.getData());
		if (contents.equals("@")) {
		    break;
		}
	    }
	}

	// then the singer
	StringBuffer singerBuff = new StringBuffer();
	for (++nEvent; nEvent < oldTrack.size(); nEvent++) {
	    MidiEvent event = oldTrack.get(nEvent);
	    if (event.getTick() != 0) {
		break;
	    }
	    if (! isLyricEvent(event)) {
		break;
	    }

	    msg = (MetaMessage) (event.getMessage());
	    String contents = new String(msg.getData());
	    if (contents.equals("\r\n")) {
		continue;
	    }
	    singerBuff.append(contents);
	}
	String singer = "@T" + singerBuff.toString();
	System.out.println("Singer '" + singer +"'");

	byte[] singerBytes = singer.getBytes();

	msg = new MetaMessage();
	try {
	    msg.setMessage(1, singerBytes, singerBytes.length);
	    MidiEvent evt = new MidiEvent(msg, 0L);
	    lyricTrack.add(evt);
	} catch(InvalidMidiDataException e) {
	}

	return nEvent;
    }

    public static boolean isLyricEvent(MidiEvent event) {
	if (event.getMessage() instanceof MetaMessage) {
	    MetaMessage msg = (MetaMessage) (event.getMessage());
	    if (msg.getType() == LYRIC) {
		return true;
	    }
	}
	return false;
    }

    public static MidiEvent convertLyricToText(MidiEvent event) {
	if (event.getMessage() instanceof MetaMessage) {
	    MetaMessage msg = (MetaMessage) (event.getMessage());
	    if (msg.getType() == LYRIC) {
		byte[] newMsgData = null;
		if (firstLyricEvent) {
		    // need to stick a \ at the front
		    newMsgData = new byte[msg.getData().length + 1];
		    System.arraycopy(msg.getData(), 0, newMsgData, 1, msg.getData().length);
		    newMsgData[0] = '\\';
		    firstLyricEvent = false;
		} else {
		    newMsgData = msg.getData();
		    if ((new String(newMsgData)).equals("\r\n")) {
			newMsgData = "\\".getBytes();
		    }
		}
		try {
		    /*
		    msg.setMessage(TEXT, 
				   msg.getData(), 
				   msg.getData().length);
		    */
		    msg.setMessage(TEXT, 
				   newMsgData, 
				   newMsgData.length);
		} catch(InvalidMidiDataException e) {
		    e.printStackTrace();
		}
	    }
	}
	return event;
    }

    public static byte[] getKARLang(String lang) {
	System.out.println("lang is " + lang);
	if (lang.equals("12")) {
	    return "@LENG".getBytes();
	}
	
	// don't know any other language specs, so guess
	if (lang.equals("01")) {
	    return "@LCHI".getBytes();
	}
	if (lang.equals("02")) {
	    return "@LCHI".getBytes();
	}
	if (lang.equals("08")) {
	    return "@LCHI".getBytes();
	}
	if (lang.equals("09")) {
	    return "@LCHI".getBytes();
	}
	if (lang.equals("07")) {
	    return "@LCHI".getBytes();
	}
	if (lang.equals("")) {
	    return "@L".getBytes();
	}
	if (lang.equals("")) {
	    return "@LENG".getBytes();
	}
	if (lang.equals("")) {
	    return "@LENG".getBytes();
	}
	if (lang.equals("")) {
	    return "@LENG".getBytes();
	}
	if (lang.equals("")) {
	    return "@LENG".getBytes();
	}
	if (lang.equals("")) {
	    return "@LENG".getBytes();
	}


	return ("@L" + lang).getBytes();
    }


    public static void copyNotesTrack(Track oldTrack, Sequence seq) {
	Track newTrack = seq.createTrack();

	for (int nEvent = 0; nEvent < oldTrack.size(); nEvent++)
	    {
		MidiEvent event = oldTrack.get(nEvent);

		newTrack.add(event);
	    }
    }

    public static void createFirstTrack(Sequence sequence) {
	Track track = sequence.createTrack();
	MetaMessage msg1 = new MetaMessage();
	MetaMessage msg2 = new MetaMessage();

	byte data[] = "Soft Karaoke".getBytes();
	try {
	    msg1.setMessage(3, data, data.length);
	} catch(InvalidMidiDataException e) {
	    e.printStackTrace();
	    return;
	}
	MidiEvent event = new MidiEvent(msg1, 0L);
	track.add(event);

	data = "@KMIDI KARAOKE FILE".getBytes();
	try {
	    msg2.setMessage(1, data, data.length);
	} catch(InvalidMidiDataException e) {
	    e.printStackTrace();
	    return;
	}
	MidiEvent event2 = new MidiEvent(msg2, 0L);
	track.add(event2);
    }

    public static void output(MidiEvent event)
    {
	MidiMessage	message = event.getMessage();
	long		lTicks = event.getTick();
    }



    private static void out(String strMessage)
    {
	System.out.println(strMessage);
    }
}



/*** KARConverter.java ***/


	
      

Playing MIDI files

The MIDI files extracted from the disk can be played using standard MIDI players such as Timidity. The lyrics are included and the melody line is in MIDI channel one. I've written a batch of Java programs using Swing and also the Java Sound framework which can play and do things to MIDI files. At the same time as playing MIDI files I can also do cool karaoke things like show the lyrics, show the notes that should be played and show progress through the lyrics. I'm still working on those, they will get posted later.

Playing WMA files

WMA files are "evil." They are based on two Microsoft proprietary formats. The first is the Advanced Systems Format (ASF) file format which describes the "container" for the music data. The second is the codec, Windows Media Audio 9.

The ASF is the primary problem. Microsoft have a published specification. This specification is strongly antagonistic to anything open source. The license states that if you build an implementation based on that specification then you:

And what's more, you are not allowed to begin any new implementation after January 1, 2012 - and it is already May, 2012!

Just to make it a little worse, Microsoft have Patent 6041345 "Active stream format for holding multiple media streams" filed in Mar 7, 1997. The patent appears to cover the same ground as many other such formats which were in existence at the time, so the standing of this patent (were it to be challenged) is not clear. However, it has been used to block the GPL-licensed project VirtualDub from supporting ASF. The status of patenting a file format is a little suspect anyway, but may become a little clearer after Oracle wins or loses its claim to patent the Java API.

The FFmpeg project has nevertheless done a clean-room implementation of ASF, reverse-engineering the file format and not using the ASF specification at all. It has also reverse-engineered the WMA codec. This allows players such as mplayer and VLC to play ASF/WMA files. FFmpeg itself can also convert from ASF/WMA to better formats such as Ogg Vorbis.

There is no Java handler for WMA files, and given the license there is unlikely to be one unless it is based on FFmpeg.

The WMA files that I have extracted from the DVD have the following characteristics:

The Sonken player plays the right channel if no-one is singing into the microphones, but switches to the left channel (effectively muting the lead singer) as soon as someone sings into a microphone. Simple and effective.

The lyrics are still there in the track data as MIDI and can be extracted as before. They can be played by a MIDI player. I have no idea (yet) how to synchronise playing the MIDI and the WMA files.

KAR format

The resultant MIDI files are not in KAR format. This means that Karaoke players such as pykaraoke may have problems playing them. It is not too hard to convert the files to this format: loop through the sequence, writing or modifying MIDI events as appropriate. The program is not very exciting but is downloadable as KARConverter .

Playing songs with pykar

One of the simplest ways to play Karaoke Midi files is by using pykar . Regrettably, the songs ripped fromt he Sonken disk do not play properly - this is a mixture of bugs in pykar and features required which are not supplied. The problems and their solutions follow.

Tempo

Many Midi files will set the tempo explicitly using the Meta Event Set Tempo, 0x51. These files often do not. Pykar expects a Midi file to include this event, and otherwise defaults to a tempo of zero beats per minute. As might be expected, this trhows out all timing calculations performed by PyKar.

As The Sonic Spot explains, "If no set tempo event is present, 120 beats per minute is assumed" and gives a formula for calculating the appropriate tempo value, which is 60000000/120.

This requires one change to one PyKaraoke file: change line 190 of pykar.py from

sele.Tempo = [(0, 0)]
      

to

self.Tempo = [(0, 500000)]
      

Language encoding

The file pykdb.py clains that cp1252 is the default character encoding for Karaoke files, ans uses a font DejaVuSans.tt which is appropriate for displaying such characters. This encoding adds in various European symbols such as 'á' in the top 128 bits of a byte, in addition to standard ASCII.

I'm not sure where PyKaraoke got that information from, but it certainly doesn't apply to Chinese Karaoke. I don't know what encodings Chinese, Japanese, Korean, etc use, but my code dumps them out as Unicode UTF-8. A suitable font for Unicode is Cyberbit.ttf. (See the Fonts chapter in my lecture notes on Global Software).

The file pykdb.py needs the lines

        self.KarEncoding = 'cp1252'  # Default text encoding in karaoke files
        self.KarFont = FontData("DejaVuSans.ttf")
      

changed to

        self.KarEncoding = 'utf-8'  # Default text encoding in karaoke files
        self.KarFont = FontData("Cyberbit.ttf")
      

and a copy of Cyberbit.tt copied to the directory /usr/share/pykaraoke/fonts/.

Songs with no notes

Some songs on the disk have no MIDI notes, as this is all in a WMA file. The MIDI file only has the lyrics. PyKaraoke only plays upto the last note, which is at zero!. So no lyrics are played.

Conclusion

This chapter has discussed basically a forensics issue: how to get information off a DVD when the format of the files is not known. It doesn't have anything directly to do with playing sound, although it does give me a big source of files that I have already paid for.