Converting CSV and Vector Data to Native HTK Format Using C#

http://jamesrossiter.wordpress.com/2008/11/16/converting-csv-and-vector-data-to-native-htk-format-using-c/


The output of my Principal Component Analysis in Matlab to reduce the dimensionality of gesture data is in the comma separated variable format. 57 dimension data goes in, X dimension data comes out in standard csv format. It is better to remove unnecessary information from the gesture data as it only makes the recognition of gesture (and intent, in my case) more difficult.
The problem is that the HTK, which I am going to use to perform recognition, doesn’t natively accept csv data so you have to convert to the HTK binary format parameter files. I chose to do this in c# as I’m familiar with it, but I stumbled across a few problems relating to the conversion between big-endian and little-endian binary data. HTK reads data in the opposite way to my PC (although I’m sure I read on their website somewhere that there is automatic detection for this).
The following code is pretty rough around the edges as it includes a lot of stuff to help me debug it. The program reads a directory and converts all *.csv files into HTK format binary files by reading in the data as floats, converting to bytes, writing a header and then writing the data to a binary file *.csv.bin.

static void Main(string[] args)
{
string dir = @”G:\PHD Nov 09\# programs\matlab work\test”;
DirectoryInfo di = new DirectoryInfo(dir);
FileInfo[] rgFiles = di.GetFiles(“*.csv”);
foreach (FileInfo fi in rgFiles)
{
using (TextReader tr = new StreamReader(fi.FullName))
{
string data = tr.ReadToEnd();
System.Text.ASCIIEncoding encoding=new System.Text.ASCIIEncoding();
byte[] byteArray = encoding.GetBytes(data);
string newdata = data.Replace(‘\n’,’ ‘);
string[] plit = newdata.Trim().Split(‘ ‘);
int samples = plit.Length;
int itemspersample = plit[0].Split(‘,’).Length;
// now create binary data, each sample (part of a line in the file)
// has to be converted from a float to a 4 byte array and then joined to make one
// large binary file
byte[] bytedata = new byte[samples * itemspersample * 4];
for (int i = 0; i < samples; i++)
{
for (int j = 0; j < itemspersample; j++)
{
string dd = plit[i].Split(‘,’)[j];
float f = (float)Convert.ToDouble(plit[i].Split(‘,’)[j]);
byte[] temp = new byte[4];
temp = BitConverter.GetBytes(f);
bytedata[(i * itemspersample * 4) + (j * 4)] = temp[3];
bytedata[(i * itemspersample * 4) + (j * 4) + 1] = temp[2];
bytedata[(i * itemspersample * 4) + (j * 4) + 2] = temp[1];
bytedata[(i * itemspersample * 4) + (j * 4) + 3] = temp[0];
}
}
// now create HTK header 12 bytes long
byte[] nSamples = BitConverter.GetBytes(samples);
byte[] sampPeriod = BitConverter.GetBytes(100000);
byte[] sampSize = BitConverter.GetBytes(Convert.ToInt16(itemspersample * 4));
byte[] parmKind = BitConverter.GetBytes(Convert.ToInt16(9));
using (BinaryWriter bw = new BinaryWriter(File.Open(fi.FullName + “.bin”, FileMode.Create)))
{
Array.Reverse(nSamples);
Array.Reverse(sampPeriod);
Array.Reverse(sampSize);
Array.Reverse(parmKind);
bw.Write(nSamples);
bw.Write(sampPeriod);
bw.Write(sampSize);
bw.Write(parmKind);
bw.Write(bytedata);
}
}
}

To check it works you run HList, with no config file required as the header explains to HTK everything it needs to know about the data:

G:\PHD Nov 09\# programs\matlab work\test>hlist -h EO412_10PCs.csv.bin
————————- Source: EO412_10PCs.csv.bin ————————
Sample Bytes:  40       Sample Kind:   USER
Num Comps:     10       Sample Period: 10000.0 us
Num Samples:   5        File Format:   HTK
—————————— Samples: 0->-1 ——————————
0:    1838.200 308.910-262.970 401.920 -66.737-499.370 305.260-260.250 -91.974  28.171
1:    1837.700 308.630-263.340 400.810 -67.144-499.920 305.280-260.060 -92.174  27.584
2:    1837.000 308.360-263.750 399.940 -67.870-500.510 305.540-259.960 -91.964  26.922
3:    1836.500 308.160-264.000 398.500 -68.003-501.230 305.790-259.980 -92.138  26.342
4:    1837.000 308.360-263.750 399.940 -67.870-500.510 305.540-259.960 -91.964  26.922
———————————– END ———————————–