






			 ENTROPIC RESEARCH LABORATORY, INC.


       TECHNICAL MEMORANDUM

       SUBJECT:	Data Files in the Entropic Signal Processing System (ESPS)

       ID: ETM-S-86-13:rap/jtb

       VERSION:	3.4

       DATE: 9/11/91

       AUTHOR: Alan Parker and Joe Buck

       LAST REVISED BY:	John Shore

       REFERENCES:

	[1] Entropic Signal Processing System, Programmers Reference Manual

	[2] ETM-S-86-12, Parameter and Common Files in	the  Entropic  Signal
	    Processing System (ESPS)

	[3] ETM-S-86-14, Entropic Signal Processing System Programming Guide-
	    lines

	[4] ETM-S-86-21, Introduction to the Entropic Signal Processing	 Sys-
	    tem	(ESPS)

	[5] ETM-S-86-25, Guidelines for	Implementing ESPS Feature  File	 Sub-
	    types

	[6] ESPS APPLICATIONS NOTE: File Headers and Record Keeping in ESPS

	[7] ESPS APPLICATIONS NOTE: Converting Data  to	 and  from  ESPS  FEA
	    Files

	[8] ETM-S-88--3, Machine Independent File I/O (MIIO) Module for	ESPS

       1 .  Introduction

       This document describes the Entropic Signal Processing  System  (ESPS)
       data  files  and	presents information required to understand and	write
       programs	using the ESPS data file header	structures.

       The reader must also read and understand	 the  ESPS  manual  pages  in
       reference  [1].	  These	manual pages are in the	standard UNIX format;
       ____________________
	  c
	    Copyright 1987-1990	Entropic Speech, Inc. Copyright	1991 Entropic
       Research	Laboratory, Inc.,  All rights reserved.











       ETM-S-86-13:rap/jtb					       page 2


       Section 1 describes user	commands, Section 3  describes	library	 rou-
       tines,  and  Section 5 describes	the format of ESPS files.  This	docu-
       ment together with reference [1]	is the complete	 description  of  the
       ESPS file structures.

       For additional information on ESPS programming guidelines and a sample
       program	see  reference	[3].   For a general introduction to ESPS see
       [4].

       This document also describes support for	reading	files with  NIST  and
       other foreign headers, and how non-ESPS programs	can read ESPS files.

       2 .  Design Goals

       The design goals	for ESPS files are:

	 +o  Store all relevant information about the file, including analysis
	    conditions,	 source	 files,	history, and the type of data that is
	    in the file.

	 +o  Flexibility.  We will have sampled data  files,  files  of	fixed
	    length  records,  and record files in which	a tag marking a	posi-
	    tion is associated with each record.  We may want to store 8, 16,
	    or 32-bit integers,	or 32 or 64-bit	floating point values.

	 +o  The	design should allow programs that only understand  a  few  of
	    the	 header	 items to run correctly.  For example, one could ima-
	    gine a general plotting program that uses certain header items to
	    determine where the	data starts, whether the data type is integer
	    or floating, and to	put appropriate	labels on the plot.   A	 pro-
	    gram  should  not need special knowledge about how a certain type
	    of data is stored; it should all be	in the header.

	 +o  The	header should provide enough information to repeat an experi-
	    ment or to check files for compatibility before processing.

	 +o  The	design should allow new	file types to be designed  and	added
	    without affecting existing data files of another type.

	 +o  The	design should  allow  users  to	 add  additional  ("generic")
	    header items to existing ESPS file types without affecting exist-
	    ing	data files of that type	or existing programs that  deal	 with
	    files of that type.

	 +o  The	header in a specific data file type should only	contain	items
	    that are used by that specific file	type.

	 +o  It should be possible to create user-defined ESPS file types.

	 +o  All	fields in a data record	should not have	to be the same	type;
	    _e._g. some fields are integer while others are floating.




       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					       page 3


	 +o  Provide support for	non-ESPS (foreign) headers.

	 +o  Provide reasonable support for non-ESPS programs to	 access	 ESPS
	    files.

	 +o  Provide the	ability	for ESPS files to be moved  across  different
	    computer architectures (by dealing with issues such	as byte	swap-
	    ping, etc.).

       For a general introduction to ESPS files	and file headers, see [6].

       3 .  ESPS File Types

       All ESPS	files conform to a common structure, consisting	of a standard
       header  followed	 by data records.  The header is the primary means by
       which record-keeping is performed within	ESPS.  An ESPS header include
       the values of all important parameters to the program that created the
       ESPS file, and the header is a recursive	structure  that	 permits  the
       inclusion  of  the headers of all source	files that were	inputs to the
       program.	 It follows that most ESPS files contain a  complete  history
       of  the origin of the data in the file and all intermediate processing
       steps.  ESPS file headers and data records can be viewed	by  means  of
       the program _p_s_p_s	(1-ESPS).

       Current ESPS file types in the general release include the following:

	    FILT (obsolete) digital filters

	    SCBK	    scalar quantization	codebooks

	    SD (obsolete)   sampled

	    SPEC (obsolete) spectra

	    FEA		    feature file

       For  a  complete	 description  of  the  ESPS  data  files   refer   to
       _E_S_P_S(5-ESPS),  _S_D(5-ESPS), _F_I_L_T(5-ESPS),	_S_C_B_K(5-ESPS), and _F_E_A(5-ESPS)
       in reference [1].

       The FEA type deserves special mention.  It is a general	purpose	 file
       type  that  can be used to store	arbitrary information in fixed-length
       records;	optionally, the	records	can point  to  positions  in  various
       source  files.	Unlike	the  case for the other	ESPS file types	- the
       names, sizes, and data types of the fields  in  FEA  records  are  not
       pre-defined.   This  means  that	FEA files can be used to create	user-
       defined ESPS files that have all	of the	advantages  of	the  built-in
       file  types.   For  details,  see  [5].	 We  are replacing all of the
       built-in	file types with	FEA subtypes.  Currently, the  following  are
       available:

	 FEA_SPEC
	      used for storing spectral	information, such as  power  spectra.


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					       page 4


	      Each  record  contains  a	 vector	of spectral values (powers or
	      power densities in the case of power  spectra),  together	 with
	      certain  optional	 fields	- total	power, an imaginary-part vec-
	      tor, number of frequencies, and a	vector of frequencies -	 that
	      are present or absent according to the values of certain header
	      items;

	 FEA_ANA
	      suitable for speech processing;  contains	 spectral  parameters
	      (various	representations),  pulse lengths, powers, and voicing
	      information;

	 FEA_VQ
	      contains vector quantization codebooks;

	 FEA_STAT
	      contains statistical  information	 (means,  covariances,	etc.)
	      suitable for pattern-classification;

	 FEA_SD
	      contains sampled data; supports multi-channel and	complex	data

	 FEA_FILT
	      contains digital filter coefficients

       4 .  ESPS File Headers

       ESPS data file headers consist of two primary parts; the	 common	 sec-
       tion  and  the type specific section.  All ESPS data file headers have
       the same	common section.	 This section contains	information  relating
       to  the type of the data	in the file, creation date and version infor-
       mation, housekeeping information	required by the	 header	 access	 rou-
       tines, and comments.  The type-specific section of the header contains
       data relevant to	the specific file type.	 Data fields  in  the  header
       are called _h_e_a_d_e_r _i_t_e_m_s.

       Internally, a file header is stored as a	C  structure.	In  the	 file
       itself,	there  are  two	types of header	items.	Most header items are
       assigned	a fixed	location  and  are  always  present.   The  remaining
       header  items  may  or may not be present and may be variable in	size.
       These are stored	in the file after the fixed location items, but	 pro-
       grammers	 need  not  be	concerned about	the actual data	format in the
       file.

       Analysis	conditions stored in file headers reflect the  history	of  a
       file, not commands to programs that operate on the file.	 In ESPS, the
       parameter file, the common file,	and in some cases  the	command	 line
       specify	analysis  conditions and options that affect the operation of
       many ESPS programs.  ESPS programs consult  the	parameter  file,  the
       common  file,  and  the	command	line for analysis conditions, and for
       most analysis conditions	store these in the header of the output	 data
       file.  For information about the	parameter and common files, see	[2].



       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					       page 5


       There will be cases where there are not header items to save  particu-
       lar  analysis conditions	or options.  This is particularly likely dur-
       ing experimentation or testing of  changes  to  programs.   If  it  is
       determined  that	 a new analysis	condition or option is significant it
       will likely be added to the file	header in a future release.   In  the
       meantime, experimental versions are free	to use the spare locations in
       file headers (documented	in [2, Section 5] for each file	 type),	 with
       the understanding that official ESPS releases may invalidate such use.
       To further support record keeping, all ESPS programs will  record  the
       command	line  that invoked them	in the comment field of	any generated
       output file headers, and	in the case of conditions or options  without
       corresponding  header  entries  recording the command line will be the
       record of their use.  This convention facilitates record	keeping	 when
       files  are  produced  by	 unofficial  version of	programs with special
       analysis	 options.   The	 ESPS  library	routines   _g_e_t__c_m_d__l_i_n_e	  and
       _a_d_d__c_o_m_m_e_n_t (3-ESPS) support this convention.

       To allow	a single file to contain  its  entire  history,	 headers  are
       recursive  structures.  This  means  that an entire file	header may be
       included	as an optional header item. The	C structure representing  the
       header  in memory has fields that can contain pointers to other header
       structures.

       5 .  Programming	with the ESPS Header Structures

       This section presents information needed	to write programs  using  the
       ESPS header structures.

       5 .1 .  Header Structure

       Refer to	the listing of _h_e_a_d_e_r._h	in reference [1] to  help  understand
       this  section.	Also,  see _E_S_P_S(5-ESPS).  In the following, we assume
       that the	variable "head"	is a pointer to	an ESPS	 file  header  struc-
       ture.   As mentioned above, the header structure	contains two portions
       - the common portion and	the type-specific portion.  The	 common	 por-
       tion of the header itself contains two structures, _c_o_m_m_o_n and _v_a_r_i_a_b_l_e

       The structure _c_o_m_m_o_n contains items present in every ESPS  data	file,
       regardless  of  type.   This  section  contains	the  type code of the
       header, a check value, identification of	the program and	 its  version
       that  created the header, the name of the person	who ran	that program,
       the number of data records in the file, whether	the  data  is  tagged
       (meaning	 each  record  has  a  tag that	refers to a point in a source
       file), and the type of the records in the data file.  An	example	(file
       creation	date) is

	       head->common.date.


       The structure _v_a_r_i_a_b_l_e contains pointers	to source file names,  embed-
       ded  source  file  headers,  two	text fields (one is the	comment	field
       mentioned earlier), the name of a reference file,  and  two  variables
       that  tell  the	number of source file names and	embedded headers.  An


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					       page 6


       example (reference file name) is

	       head->variable.refer.


       The variable portion also contains so-called "generic"  header  items,
       which are used by programs to store arbitrary information not included
       in the standard header.	This facility means that the basic design  of
       an  ESPS	file type does not have	to anticipate every application.  For
       example,	the SD file type does not make provision for storing  in  the
       file  header  the  average  rate	of zero	crossings, but a program that
       needs to	can do so by means of a	generic	header item.  If such an item
       was  created and	called "zero_crossing",	a pointer to the stored	value
       is given	by

	       get_genhd("zero_crossing", ih)


       The type-specific portion of the	header is provided by a	union _h_d that
       contains	 pointers to each possible type-specific structure.  They are
       declared	in a union so that additional pointers can be  added  in  the
       future  without	affecting  existing data files (since the size of the
       pointers	themselves are the same).   Programmers	 can  tell  which  of
       these  pointers	to  use	 by checking the item head->common.type.  The
       pointer names are given in the following	table:


		   _____________________________________________
		   _|V_a_l_u_e__o_f____c__o__m__m__o__n__.__t__y__p__e___P_o_i_n_t_e_r__i_n____h__d_t_o__u_s_e__|
		   |FT_SD		   _h_d_._s_d		|
		   |FT_SPEC		   _h_d_._s_p_e_c		|
		   |FT_FILT		   _h_d_._f_i_l_t		|
		   |FT_SCBK		   _h_d_._s_c_b_k		|
		   |FT_FEA		   _h_d_._f_e_a		|
		   _|____________________________________________|



       For example, the	sampling frequency from	the header of an  (old-style)
       SD file is head->hd.sd->sf.

       5 .1 .1 .  NIST (Sphere)	Headers

       ESPS programs can read sample data files	containing  the	 NIST  Shpere
       header  (such  as found on the TIMIT CD-ROM database).  This is imple-
       mented by having	the ESPS file header access routines, detect an	input
       file containing a NIST header and build an ESPS FEA_SD header from it.
       All header information in the NIST header is saved as  generic  header
       items in	the ESPS header.






       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					       page 7


       5 .1 .2 .  Foreign Headers

       It is often desirable to	maintain an existing file header  even	if  a
       file  is	 converted to ESPS.   The non-ESPS header is called a foreign
       header and it can be stored in the file between the  ESPS  header  and
       the first data record.

       Foreign headers are kept	in the ESPS header in the  following  manner:
       The generic header item _f_o_r_e_i_g_n__h_d__l_e_n_g_t_h contains the size (in bytes)
       of the foreign header.	If  this  item	is  present  (and  non-zero),
       _r_e_a_d__h_e_a_d_e_r()  will  read  this	many additional	bytes of data, put it
       into  the  header,  and	set  an	 additional   generic	header	 item
       _f_o_r_e_i_g_n__h_d__p_t_r to point to it.  From that point on, the foreign header
       is just part of the ESPS	header.

       With this mechanism, it is possible to write  programs  that  use  the
       ESPS  header and	record I/O functions while still having	access to the
       foreign header.	The procedure is simple: use  _r_e_a_d__h_e_a_d_e_r()  to	 read
       the  ESPS  header,  and	_g_e_t__g_e_n_h_d() to get the pointer to the foreign
       header.

       The conversion programs _b_t_o_s_p_s and _a_d_d_f_e_a_h_d would usually be  used  to
       create such files.   See	reference [7].

       5 .2 .  Header Access Routines

       There are several ESPS library routines that  support  access  to  the
       ESPS  data file headers.	These are described in detail in Section 3 of
       [1].   Declarations   for   all	 these	 functions   are   found   in
       <_e_s_p_s/_h_e_a_d_e_r._h>;	 the programmer	need not declare them if this file is
       included.

       The _r_e_a_d__h_e_a_d_e_r routine attempts	to read	a file header from  a  stream
       (the  fd	 parameter  is typically one from a "fopen" call).  If an I/O
       error occurs _r_e_a_d__h_e_a_d_e_r	returns	NULL. If the file does not begin with
       a  valid	 ESPS  header,	then  _r_e_a_d__h_e_a_d_e_r  will	determine if the file
       begins with a NIST header.  If a	NIST header is present,	then a FEA_SD
       header is generated and filled in with the appropriate values from the
       NIST header (sample rate, minimum  and  maximum	sample	values,	 data
       type, etc).   _R_e_a_d__h_e_a_d_e_r then returns a	pointer	to this	FEA_SD header
       in the usual way.  Programs which called	_r_e_a_d__h_e_a_d_e_r on	a  NIST	 file
       will  operate  normally	and not	know that the file wasn't initially a
       normal ESPS file.  If neither a valid ESPS header, or a NIST header in
       present,	then _r_e_a_d__h_e_a_d_e_r will check for	the _u_n_i_x environment variable
       DEF_HEADER.  If it is defined and points	to a file  containg  a	valid
       ESPS  header,  then  that header	is used	instead.  If all of the	above
       attempts	to find	a header fail, then NULL is returned.  Memory for the
       file  header  is	 allocated  from  dynamic  memory.  After _r_e_a_d__h_e_a_d_e_r
       returns,	the program should consult _c_o_m_m_o_n._t_y_p_e to determine the	 type
       of the header that has been returned.

       The _w_r_i_t_e__h_e_a_d_e_r	routine	attempts  to  write  a	file  header  onto  a
       stream.	 It  computes  and  fills in values for	hsize, fixsiz, check,


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					       page 8


       date, and hdvers	and ensures that must-be-zero fields contain zero, by
       clearing	 them.	 If an I/O error occurs, _w_r_i_t_e__h_e_a_d_e_r writes an	error
       message and the program bombs.

       The _n_e_w__h_e_a_d_e_r routine allocates	a new header from dynamic memory  and
       returns	a  pointer  to	it.   Which type-specific header is allocated
       depends on the value of the argument _t_y_p_e.    If	_t_y_p_e is	zero, then no
       type-specific  header  is  allocated,  only the common part is.	 This
       case is intended	for use	by the header access programs.

       The _c_o_p_y__h_e_a_d_e_r routine accepts a pointer to a file header and returns
       a  copy	of  the	 same  type in which all items except common.comment,
       common.prog,   common.vers,    common.date,    common.progdate,	  and
       common.hdvers  are  copied  from	the source header.  This routine par-
       tially generates	a header for an	output file, given a  header  for  an
       input  file  (any changed parameters must be filled in, along with the
       program name and	version).  It is important to realize that this	func-
       tion  can only be used when the output file is of the same type as the
       input file.

       The _a_d_d__s_o_u_r_c_e__f_i_l_e routine inserts a source file name and header into
       the next	available positions in a header. The nnames and	nheads fields
       are incremented.	Strictly speaking, only	pointers are copied,  so  the
       source parameters must not be altered before _w_r_i_t_e__h_e_a_d_e_r is called.

       The _a_d_d__c_o_m_m_e_n_t routine appends a character string  onto	 the  comment
       field  of  a  header.   For  example,  with  the	 help of _g_e_t__c_m_d__l_i_n_e
       (3-ESPS), the command line is added as a	comment. All ESPS programs by
       convention add the command line as a comment in the output header.

       For creating _z_f_u_n_c structures, the function _n_e_w__z_f_u_n_c is	supplied.  It
       allocates  dynamic memory, builds a new zfunc structure,	and returns a
       pointer to it. The zeros	and poles are also copied to dynamic  memory.
       To  assist  in  storing	_z_f_u_n_cs	as generic header items	the functions
       _a_d_d__g_e_n_z_f_u_n_c and	_g_e_t__g_e_n_z_f_u_n_c are provided.

       Generic	header	items  are  added  to  an  existing  header  by	  the
       _a_d_d__g_e_n_h_d_X (3-ESPS) routines ("X" stands for one of six	possible data
       types).	Other generic-related header access routines are  _g_e_n_h_d__l_i_s_t,
       which  returns a	list of	the defined generic header items, _g_e_n_h_d__t_y_p_e,
       which returns  the  type	 of  a	specific  generic  header  item,  and
       _g_e_t__g_e_n_h_d,  which returns a pointer to a	specific generic header	item.
       These routines make it possible for programs to	process	 the  generic
       header  items  in  ESPS	files without having to	know what they are in
       advance.

       5 .2 .1 .  Using	the Header Access Routines

       The header access routines are easy to use.   The important  thing  to
       remember	 is  that,  before _w_r_i_t_e__h_e_a_d_e_r	is called, _a_l_l _v_a_l_u_e_s _m_u_s_t _b_e
       _s_e_t _i_n _t_h_e _h_e_a_d_e_r.  The easiest error  is  to  confuse  pointers	 that
       might  be in use	pointing to several different headers, most often the
       input and the output file.  The basic model of  use  is	to  open  the


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					       page 9


       input  file  and	call _r_e_a_d__h_e_a_d_e_r on this file.	This allocates memory
       for the input file header, checks the header on the  input  file,  and
       reads to	its end.  The next read	on this	file will return data, rather
       than header, values.

       If _r_e_a_d__h_e_a_d_e_r returns without error, then it has found a  valid	 ESPS
       header.	   The	program	 should	now check _c_o_m_m_o_n._t_y_p_e to be sure that
       the file	is of the type expected	by the program.	 If it is  not,	 then
       an  error message should	be printed and the program should exit with a
       non-zero	exit code (standard convention is to  use  exit(1)).	 Some
       programs	 may  accept  several  (or any)	of the valid ESPS file types.
       These programs should still consult  _c_o_m_m_o_n._t_y_p_e	 to  determine	which
       pointer	in  the	union _h_d to use.    The	use of the wrong pointer will
       cause unpredictable results (most often	a  harmless  bus  error,  but
       worse could happen).

       Since _r_e_a_d__h_e_a_d_e_r returns a pointer to the main	header	structure,  a
       pointer must be declared	before _r_e_a_d__h_e_a_d_e_r is called:

	   struct header *h;
	   .
	   .
	   .
	   h = read_header(file);

       After the call to _r_e_a_d__h_e_a_d_e_r, items in the header  may	be  accessed.
       Here are	some examples:

	   char	*ref_file;
	   float frequency, avg_zero;
	   .
	   .
	   .
	   if (h->common.type == FT_SD)	{      /*verify	file type*/
	       fprintf(stderr, "Input file is not an SD	file.0);
	       exit(1)
	   }
	   ref_file = h->variable.refer;       /*get name of reference file for	tags*/
	   frequency = h->hd.sd.sf;	       /*sampling frequency of data*/
	   avg_zero = *(float *) get_genhd("zero_crossing", h);


       In some	ESPS  programs	(especially  newer  programs),	the  call  to
       _r_e_a_d__h_e_a_d_e_r  and	the file type checking is done within a	library	func-
       tion _e_o_p_e_n, which also opens the	file.  An example of  this  is	given
       later.

       If the output file of the program is the	same type as the  input	 file
       then  _c_o_p_y__h_e_a_d_e_r  is useful for	creating the header of the new output
       file and	filling	in most	values from the	input file.   Some  of	these
       values  might have to be	changed	before _w_r_i_t_e__h_e_a_d_e_r is called, but in
       many cases, the header of an output file	is  much  like	that  of  its
       input  file.  If	the output file	is of a	different type than the	input


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					      page 10


       file, or	the programmer decides that none or very few values  will  be
       in common, then _n_e_w__h_e_a_d_e_r should be used to create the header for the
       new output file.	 A type	code must be  specified	 when  _n_e_w__h_e_a_d_e_r  is
       called.	In either case,	when creating an output	ESPS file from source
       ESPS files, the source headers should  be  saved	 in  the  new  output
       header.	 This is done by calling _a_d_d__s_o_u_r_c_e__f_i_l_e.  Here	is an example
       of reading values from the header of an input file, and	creating  and
       writing an output file header.

	   char	*speaker_name;
	   float samp_freq, zero_crossing, *coh_thresh;
	   long	num_samples;
	   .
	   .
	   .
	   ih =	read_header(in_file);	       /* get input header */
	   speaker_name	= (char	*) get_genhd("speaker",	ih);  /*generic	example*/
	   .
	   .
	   .
	   oh =	new_header(FT_FEA)	       /* create new output header */
	   (void) add_source_file(oh,in_file,ih);	/* save	input header */
	   oh->variable.refer =	ih->variable.refer;   /*same refer file*/
	   /*add the command line as a comment*/
	   add_comment(oh, get_cmd_line(argc, argv));
	   (void) strcpy (oh->common.prog, progname);
	   (void) strcpy (oh->common.vers, Version);
	   /*add generic header	items*/
	   (void)add_genhd_f("avg. zero	crossing", &zero_crossing, 1, ih)
	   coh_thresh =	add_genhd_d("voicing threshold", (double *)NULL, 1, ih);
	   .
	   .
	   .
	   zero_crossing = . . .;
	   *coh_thresh = . . .;
	   (void) write_header(oh, ostrm);     /*ostrm is output file stream*/


       The call	to _a_d_d__s_o_u_r_c_e__f_i_l_e results in the input	header	being  stored
       on  the	new  output file.   This is done (within _a_d_d__s_o_u_r_c_e__f_i_l_e ) by
       setting a pointer (_v_a_r_i_a_b_l_e._s_r_c_h_e_a_d)  to	 point	to  the	 old  header.
       There is	a limit	to the number of headers that can be embedded in this
       way.  It	is defined as MAX_SOURCES and its current value	can be	found
       in  <_e_s_p_s/_h_e_a_d_e_r._h>.	The  header  item  _v_a_r_i_a_b_l_e._n_h_e_a_d_s  should be
       checked before calling _a_d_d__s_o_u_r_c_e__f_i_l_e  if  the	programmer  wants  to
       avoid  the chance of a run-time error because of	exceeding this limit.
       Of course, if the header	is being stored	in a new header	and only once
       in  the	program,  then it is safe to store a header without checking.
       This is the case	in the above example.

       The example above includes both	"styles"  of  adding  generic  header
       items  -	 the  _a_d_d__g_e_n_h_d_X  routines can	either allocate	space for the
       item and	return a pointer to it,	 or  accept  a	pointer	 to  existing


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					      page 11


       space.	Note that the actual values written out	to the header are the
       values that exist when _w_r_i_t_e__h_e_a_d_e_r is called,  _n_o_t  the	 values	 that
       exist  when  the	 _a_d_d__g_e_n_h_d_X routines are called.  Thus, in the	above
       example,	the two	lines prior to the _w_r_i_t_e__h_e_a_d_e_r	 call  determine  the
       values  written	to the corresponding generic header items.  There is,
       however,	a "Hi-C" way to	guarantee  that	 the  value  written  to  the
       header  is  fixed  at  the time of the call to _a_d_d__g_e_n_h_d_X routine, as
       follows:

	   *add_genhd_f("avg. zero crossing", NULL, 1, ih) = zero_crossing;

       When writing programs that exploit the generic  header  items  in  the
       input  ESPS files, it may be the	case that the presence of those	items
       is useful but not essential.  In	such  cases,  care  should  be	taken
       allow  the  program to run on ESPS files	that do	not have the required
       gereric header items.  This can be accomplished by  using  _g_e_n_h_d__l_i_s_t,
       or by checking for a NULL return	value from _g_e_t__g_e_n_h_d.

       6 .  Data Structures

       The data	structures for data records in the different  types  of	 ESPS
       files  are  specified  in the Section 5 manual pages that describe the
       file formats.  The general approach is to define	a  C  structure	 that
       represents  one	data record in the file.  Access functions are avail-
       able to support reading and writing such	records, so that  programmers
       can  deal with the C structure and ignore the issue of how that struc-
       ture is stored in the file.

       6 .1 .  Data Access Routines

       In general a function named _g_e_t_xx__r_e_c and one  named  _p_u_t_xx__r_e_c  are
       provided	 to  get  and  put  records  from/onto	an ESPS	file type xx.
       There is	also  n_a_l_l_o_xx__r_e_c function that  allocates  memory  for  one
       record.	In  most  cases,  the size of certain elements of the record,
       depend on values	in the file header.

       For example, here is an example of reading a SPEC record:

	   FILE	*ifile;
	   struct ana_data *p;
	   struct header *ih;
	   .
	   .
	   .
	   ifile = fopen(in_file,"r");	       /* open the file	*/
	   ih =	read_header(ifile);	       /* read the header */
	   if (ih->common.type != FT_SPEC) bomb();/* must be an	ANA file */
	   p = allo_ana_rec(ih);	       /* allocate the record */
	   if (!get_ana_rec(p,ih,ifile)	eof(); /* read record */
	   nfreq = p->n_frqs;		       /* get number of	frequencies*/





       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					      page 12


       The pointer p could be used in subsequent calls	to  _g_e_t__a_n_a__r_e_c	 pro-
       vided  that the program deals with only one record at a time.  If more
       than one	record is needed at the	same time within the  program,	addi-
       tional  pointers	 must  be declared and additional space	must be	allo-
       cated by	calls to _a_l_l_o__a_n_a__r_e_c.

       In the case of the built	in file	types, the type	 of  the  data	(_e._g.
       float,  integer,	etc.) is fixed and defined by the file format itself.
       The header access routine _w_r_i_t_e__h_e_a_d_e_r  automatically  sets  the	 type
       fields in the common part of the	header to their	correct	values (which
       depend on various other fields in the header).  The type	 codes	might
       be set by code like this	(this is done within _w_r_i_t_e__h_e_a_d_e_r - most ESPS
       programmers will	never have to do anything like this):

	   struct header *h;
	   {
	   . . .
	   /* the tag doesn't count here */
	   h->common.ndouble = 0;
	   h->common.nfloat = 3	* val1 + 2;
	   h->common.nlong = 0;
	   h->common.short = 1;
	   h->common.nchar = 0;
	   }


       The actual code in the library has additional logic to insure that  it
       is being	used properly.

       There is	one case where the type	code fields are	not set	 properly  by
       _w_r_i_t_e__h_e_a_d_e_r  -	old  style  SD	files.	 Since SD files	can have data
       records of different types (unlike other	current	file types, in	which
       the  types in records are pre-defined), the user	has specify what type
       of SD record is wanted.	 This  is  done	 by  means  of	the  function
       _s_e_t__s_d__t_y_p_e,  which must	be called before writing any data to the file
       and before calling _w_r_i_t_e__h_e_a_d_e_r.	 SD  files  are	 exceptional  in  one
       other  respect,	namely	that  the  "get"  and  "put" routines come in
       several flavors.	 Recognizing that programmers will sometimes want  to
       read  or	 write SD records to or	from short, float, and double arrays,
       different routines are provided to support these	cases.	For  example,
       the  routine _p_u_t__s_d__r_e_c_f	will write SD records from a program array of
       type float.  Note that the type of SD record (i.e., the type that will
       be  stored  in the SD file) does	not matter here.  Provided that	there
       has been	a previous call	to _s_e_t__s_d__t_y_p_e,	the right conversion will  be
       performed  on  output.  The new FEA_SD file type, which is intended to
       replace SD files, also allows separate specification of the data	 type
       stored  in the file and the data	type in	memory (see _i_n_i_t__f_e_a_s_d__h_d and
       _a_l_l_o__f_e_a_s_d__r_e_c_s).

       As mentioned above, FEA files provide a mechanism to define their  own
       record  structure.   That  is,  users can determine the number, types,
       sizes, and names	of the	items  stored  in  FEA	records.   FEA	files
       include	special	 support  for  labelling the records with pointers to


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					      page 13


       other source files.  The	(3-ESPS) functions that	deal with  FEA	files
       are:  _a_l_l_o__f_e_a__r_e_c,  _a_d_d__f_e_a__f_l_d, _f_e_a__d_e_c_o_d_e, _f_e_a__e_n_c_o_d_e, _g_e_t__f_e_a__p_t_r,
       _g_e_t__f_e_a__r_e_c, _p_r_i_n_t__f_e_a__r_e_c, and _p_u_t__f_e_a__r_e_c.  For a detailed introduc-
       tion to FEA files, see [5].

       6 .2 .  Using _e_o_p_e_n

       Many ESPS programs go through a sequence	 of  getting  a	 command-line
       file  name, checking to see if it is a "-" (for standard	input or out-
       put), opening the file, and checking to see that	is an ESPS file	 type
       of  the	proper	type  (and  proper subtype for FEA files).  Here's an
       example with a FEA_ANA file:

	   /*
	    * open the input file
	    */
	   if (optind <	argc) {
	       in_fea =	argv[optind++];
	       if (strcmp (in_fea, "-")	== 0)
		   in_fea = "<stdin>";
	       else
		   TRYOPEN (argv[0], in_fea, "r", infea_strm);
	       }
	   else	{
	       Fprintf(stderr, "no input file specified.0);
	       SYNTAX;
	       }
	   /*
	    * check for	FEA_ANA	ESPS file
	    */
	   if ((fea_ih = read_header(infea_strm)) == NULL)
	       ERROR_EXIT("couldn't read input FEA file	header");
	   if(fea_ih->common.type != FT_FEA)
	       ERROR_EXIT("Input file is not a FEA file");
	   if(fea_ih->hd.fea->fea_type != FEA_ANA)
	       ERROR_EXIT("Input file is not FEA_ANA type");
	   /*
	    * allocate a record; input a record, and process
	    */
	   anafea_rec =	allo_anafea_rec(fea_ih);
	   (void) get_anafea_rec(anafea_rec, fea_ih, infea_strm);
	   if (*(short *)get_genhd("spec_rep", fea_ih) != RC)
	       ERROR_EXIT("FEA_ANA file	does not have reflection coefficients");
	   rc0 = anafea_rec->spec_param[0];   /*get first reflection coefficient*/


       Often, code like	the above can be simplified by using _e_o_p_e_n,  as	 fol-
       lows:

	   if (optind <	argc)
	       in_fea =	eopen(ProgName,	argv[optind++],	"r", FT_FEA, FEA_ANA, &fea_ih, &infea_strm);
	   else	{
		   Fprintf(stderr, "no input file specified.0);


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					      page 14


		   SYNTAX;
	   }
	   /* note that	standard input is allowed, so we don't need to check in_fea
	    *
	    * allocate a record; input a record, and process
	    */
	   anafea_rec =	allo_anafea_rec(fea_ih);
	   (void) get_anafea_rec(anafea_rec, fea_ih, infea_strm);
	   if (*(short *)get_genhd("spec_rep", fea_ih) != RC)
	       ERROR_EXIT("FEA_ANA file	does not have reflection coefficients");
	   rc0 = anafea_rec->spec_param[0];   /*get first reflection coefficient*/

       6 .3 .  Reading Data Files with Generic Programs

       Before FEA files	were introduced, a different mechanism was  available
       to  allow so-called "generic" (also called "dumb") programs to operate
       on ESPS data files without knowing the details of the file format.  We
       shall describe that mechanism here, but users should avoid using	it as
       it will be phased out in	later releases.

       An example of such a "dumb" generic program is _s_t_a_t_s  (1-ESPS).	 This
       program computes	certain	statistics on records in a data	file, without
       knowledge of the	particular data	structure.  Clearly, in	order to sup-
       port  this  class of programs sufficient	information must be available
       in the file header.  To provide that information, the common  part  of
       the  header contains the	number of doubles, floats, longs, shorts, and
       characters that comprise	a record within	the file.  Moreover, the data
       in  each	record is stored in the	order listed - i.e., the given number
       of doubles followed by the given	number of floats, etc.	 These	items
       are  set	 automatically	by _w_r_i_t_e__h_e_a_d_e_r	when the ESPS file is created
       (except in the case of SD files,	where a	call to	_s_e_t__s_d__t_y_p_e  is	 also
       needed),	 and  they can be used by generic programs to properly read a
       record.	To simplify this further, the library  function	 _g_e_t__g_e_n__r_e_c_d
       is  provided  to	 read an arbitrary ESPS	record into an vector of dou-
       bles.  A	short example of reading records from a	data file in  a	 non-
       type specific manner is:

	   FILE	*ifile;
	   struct header *ih;
	   double tag;
	   double dbuf[100];
	   .
	   .
	   .
	   ifile = fopen(in_file,"r");
	   ih =	read_header(ifile);
	   if (get_gen_recd(dubf, &tag,	ih, ifile) == EOF) done();
	   a = dbuf[2];		       /* access an element */

       Programmers and	users  sometimes  need	to  know  the  correspondence
       between this generic view of an ESPS record and the type-specific view
       provided	by the C structures such as this one for PIT files  (see  _P_I_T
       (5-ESPS)):


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					      page 15


	   struct pitch	{
	       long tag;	       /*position in the reference file*/
	       float pulse_dist;       /*pulse to pulse	distance*/
	       float raw_pulse_dist;   /*raw pulse distance*/
	   }


       For example, a user of a	generic	statistics program might want to  get
       statistics  on the pulse_dist component of pitch	records, and to	do so
       must be able to state which element of the record  this	is  from  the
       viewpoint  of  generic  programs.  This information is provided in the
       section "RECORD ELEMENT FILE STRUCTURE" within  each  Section  5	 ESPS
       manual  page.   Thus,  for  example,  by	reading	_S_C_B_K (5-ESPS) you can
       determine that the final_dist component of SCBK records is the element
       number  1  from	the  generic viewpoint.	In the case of FEA files, the
       correspondence between the FEA-specific view and	the generic  view  is
       more  complicated  since	a single such table will not suffice, and for
       this reason an ESPS program is provided to translate between  the  two
       views; see _e_l_e_m_f_e_a (1-ESPS).

       7 .  The	Preamble - Reading ESPS	Files with non-ESPS Programs

       Each ESPS header	contains a fixed size structure	 which	we  call  the
       preamble.   The preamble	contains information necessary for the opera-
       tion of the _r_e_a_d__h_e_a_d_e_r function	and will not be	described  in  detail
       here.	However,  the preamble does contain information	that allows a
       non-ESPS	program	to skip	over  an  ESPS	header	to  access  the	 data
       directly.   Note	that this can also be achieved by the user level pro-
       gram _b_h_d.   The assumption here,	is that	the user needs to incorporate
       into an existing	non-ESPS program to ability to read an ESPS file.

       From the	view point of a	non-ESPS program, the preamble is the first 8
       long words (32 bytes) of	the file.   The	preamble is always written in
       the Entropic EDR	format (see reference [8]), so on a  machine  with  a
       different  default  byte	order the user program will have to byte swap
       the data.  EDR format has the same byte order as	Sun workstations,  so
       if your machine's order if different, you will have to convert it.

       This is the preamble:
       struct preamble {
	  long	       machine_code;   /* machine which	wrote file, see	header.h */
	  long	       check_code;     /* version check	code */
	  long	       data_offset;    /* data offset (in bytes, from 0) in file */
	  long	       record_size;    /* record size in bytes */
	  long	       check;	       /* ESPS magic number, same as main header */
	  long	       edr;	       /* YES if EDR_ESPS, NO if native	*/
	  long	       align_pad_size; /* alignment pad	need for some SD files */
	  long	       foreign_hd;     /* pointer to foreign header, -1	if */
				       /* there	is none	*/
       };

       The third long word contains the	offset from the	beginning of the file
       to  the	first  data  record  (in  bytes).    So	to skip	over the ESPS


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					      page 16


       header, simply seek to this location in	the  file.    The  next	 long
       word,  contains	the record size	in bytes.   For	a better detailing of
       the   record   format   use   the   program   _f_e_a__e_l_e_m_e_n_t(1-ESPS)   or
       _g_e_n__e_l_e_m_e_n_t(1-ESPS).

       All a non-ESPS need do to read an ESPS file, is to declare this struc-
       ture,  read  in	the first part of the file into	it and then seek into
       the file	to the point indicated by _d_a_t_a__o_f_f_s_e_t.	 If  the  machine  on
       which  you  are reading this file has a different byte order from EDR,
       the you will have to byte swap these fields after reading them.	  The
       field  _e_d_r  in  the preamble tells you whether you have to worry	about
       the data	format of the data records, with respect to  byte  order  and
       floating	 point format.	 If _e_d_r	is 1, then the data is in EDR format.
       This is the most	common data format in  the  workstation	 market.   If
       your machine's data formats are different than that of a	Sun, then you
       will have to convert the	data.	If the _e_d_r flag	is 0, then  the	 file
       is  in the native format	of the machine which wrote it.	You will have
       to determine that machine type (look at _m_a_c_h_i_n_e__c_o_d_e) and convert from
       that  format to that of the destination machine.	  In most cases	there
       is nothing to do, but some cases	can be	difficult.   It	 is  possible
       that  you can ensure that the ESPS file is written in the easiest for-
       mat for you to deal with	by your	non-ESPS program.  See reference  [8]
       for details on this.

       If _f_o_r_e_i_g_n__h_d in	the preamble is	not -1,	 then  the  file  contains  a
       foreign header and this is a pointer to it.   Please note that this is
       not the normal way for ESPS programs to	access	the  foreign  header.
       That    would   be   via	  the	generic	  item	 _f_o_r_e_i_g_n__h_d__p_t_r	  and
       _f_o_r_e_i_g_n__h_d__l_e_n_g_t_h.   The	foreign	header pointer	in  the	 preamble  is
       provided	just for the case of non-ESPS program access.



				       (end)





















       Version 3.4			ERL			      9/11/91



