From brideout at haystack.mit.edu  Tue Jan 18 13:04:50 2005
From: brideout at haystack.mit.edu (William Rideout)
Date: Tue Jan 18 13:04:53 2005
Subject: [gps-developers] Importing GPS data in NovAtel binary format
Message-ID: <41ED4FC2.6020504@haystack.mit.edu>

Anthea has asked me to import data into our GPS processing from a site that only 
has data in a NovAtel binary format.  On their site also is a windows 
command-line application that will convert that format into ascii.  Since the 
GPS processing has only dealt with rinex format files so far, I want to try to 
solve this problem in as generic a way as possible.

Since I do not have a program to convert this format to rinex, and can only 
convert it ascii on windows, I plan to use the public pc hermes to do this 
conversion for me.  I plan to make this solution extensible - that is, if we run 
across any new sites that only expose NovAtel binary format files, I simply need 
to add the name of that site and some other details to the python cron script 
running on hermes.

Technical details:


The job of this cron script running on the pc hermes will be to download any new 
NovAtel binary format files found in its list of urls.  In particular, this 
binary format is called an ISMRB log, and the program to extract it is called 
ParseIsmr2.exe, written by GPS Silicon Valley.  The command line is:

ParseIsmr2.exe all <input file> <output file>

This outputs a comma-separated ascii file with the following columns:

%Week, GPS TOW ,PRN, RxStatus,   Az  ,  Elv ,L1 CNo, S4 ,
S4 Cor,1SecSigma,3SecSigma,10SecSigma,30SecSigma,60SecSigma, Code-Carrier, 
C-CStdev, TEC45, TECRate45, TEC30, TECRate30, TEC15, TECRate15, TEC0, 
TECRate0,L1 LockTime,ChanStatus,L2 LockTime, L2 CNo

Of these, only the following columns are needed:

%Week - GPS week - week 0 starts at midnight, 6 Jan 1980 UTC
GPS TOW -  seconds since midnight Sunday UTC of present week
PRN -  satellite id
Az, Elv
TEC45 - tec averaged over 45 seconds

The python script will be responsible for calling ParseIsmr2.exe, and then 
converting the file into the "iono" format normally output by our algo program. 
  For reference, this ascii file format is space separated with 11 columns:

Filename: <lowercase 4 letter site name>-DDMMYY.iono

Columns:
1. time (seconds since midnight Sunday UTC of present week)
2. code
3. phase (in meters = TEC/6.128)
4. code/z (z is mapping function at particular elevation)
5. phase/z (in meters = TEC/6.128)
6. az
7. el
8. z (z is mapping function at particular elevation - always > 1)
9. pierce point lat
10. pierce point lon
11. prn (satellite id)

In order to calculate the pierce point, the python script will also need to know 
the station location.  This information will be included in the hard-coded 
dictionary that also contains the url to the data files themselves, and the 
four-letter site identifier.  This dictionary is what will need to be updated 
when a new site is added.

When all the iono files are created, they will be copied to 
/usr/oasis/experiments/gps/novatel, where the gps_retrieve.py script will be 
modified to retrieve them.  These *.iono will then be handled like any other 
*.iono created by the algo program, with the same algorithms to correct for 
receiver bias and remove bad data.

Bill

-- 
Bill Rideout
MIT Haystack Observatory
Email: brideout@haystack.mit.edu
Phone: 781 981-5624
From brideout at haystack.mit.edu  Thu Jan 20 14:51:15 2005
From: brideout at haystack.mit.edu (William Rideout)
Date: Thu Jan 20 14:51:18 2005
Subject: [gps-developers] Re: Importing GPS data in NovAtel binary format
In-Reply-To: <41ED4FC2.6020504@haystack.mit.edu>
References: <41ED4FC2.6020504@haystack.mit.edu>
Message-ID: <41F00BB3.7050105@haystack.mit.edu>

Thanks to the more stable version of the windows executable ParseIsmr.exe, the 
downloading is now running as a Scheduled Task (the Windows term for a cron job) 
on hermes.  I have it set to update once a week under the user brideout.  I 
couldn't use asgstaff since that user didn't have permission to schedule a task.

To add a new site that only has NovAtel data in the same form:

1. Edit the part of the file on hermes under brideout called build_iono.py that 
lists the sites to download.  For now this list has only one item:

### Update this data structure to add new NovAtel binary format sites ###
#  Items: 1. unique 4 letter site
#         2. site lat
#         3. site long
#         4. fnmatch type string to select files to download
#         5. site url
novAtelList = (('doua',
                 4.013771,
                 9.715404,
                 'dou*.zip',
                 'ftp://ftp.estec.esa.nl/pub/xe/bertram/douala/'),
                )

2. Add a file in /usr/oasis/experiments/gps/novatel named <site>.loc, that gives 
the site location in x,y,z form.  Example: doua.loc is:

6271339.2804067042 1073715.0505919124 446454.18061132729 APPROX POSITION XYZ



The gps toolkit has been updated to process this new data source, and is 
available at ~brideout/gps_install.tar.

Bill


William Rideout wrote:
> Anthea has asked me to import data into our GPS processing from a site 
> that only has data in a NovAtel binary format.  On their site also is a 
> windows command-line application that will convert that format into 
> ascii.  Since the GPS processing has only dealt with rinex format files 
> so far, I want to try to solve this problem in as generic a way as 
> possible.
> 
> Since I do not have a program to convert this format to rinex, and can 
> only convert it ascii on windows, I plan to use the public pc hermes to 
> do this conversion for me.  I plan to make this solution extensible - 
> that is, if we run across any new sites that only expose NovAtel binary 
> format files, I simply need to add the name of that site and some other 
> details to the python cron script running on hermes.
> 
> Technical details:
> 
> 
> The job of this cron script running on the pc hermes will be to download 
> any new NovAtel binary format files found in its list of urls.  In 
> particular, this binary format is called an ISMRB log, and the program 
> to extract it is called ParseIsmr2.exe, written by GPS Silicon Valley.  
> The command line is:
> 
> ParseIsmr2.exe all <input file> <output file>
> 
> This outputs a comma-separated ascii file with the following columns:
> 
> %Week, GPS TOW ,PRN, RxStatus,   Az  ,  Elv ,L1 CNo, S4 ,
> S4 Cor,1SecSigma,3SecSigma,10SecSigma,30SecSigma,60SecSigma, 
> Code-Carrier, C-CStdev, TEC45, TECRate45, TEC30, TECRate30, TEC15, 
> TECRate15, TEC0, TECRate0,L1 LockTime,ChanStatus,L2 LockTime, L2 CNo
> 
> Of these, only the following columns are needed:
> 
> %Week - GPS week - week 0 starts at midnight, 6 Jan 1980 UTC
> GPS TOW -  seconds since midnight Sunday UTC of present week
> PRN -  satellite id
> Az, Elv
> TEC45 - tec averaged over 45 seconds
> 
> The python script will be responsible for calling ParseIsmr2.exe, and 
> then converting the file into the "iono" format normally output by our 
> algo program.  For reference, this ascii file format is space separated 
> with 11 columns:
> 
> Filename: <lowercase 4 letter site name>-DDMMYY.iono
> 
> Columns:
> 1. time (seconds since midnight Sunday UTC of present week)
> 2. code
> 3. phase (in meters = TEC/6.128)
> 4. code/z (z is mapping function at particular elevation)
> 5. phase/z (in meters = TEC/6.128)
> 6. az
> 7. el
> 8. z (z is mapping function at particular elevation - always > 1)
> 9. pierce point lat
> 10. pierce point lon
> 11. prn (satellite id)
> 
> In order to calculate the pierce point, the python script will also need 
> to know the station location.  This information will be included in the 
> hard-coded dictionary that also contains the url to the data files 
> themselves, and the four-letter site identifier.  This dictionary is 
> what will need to be updated when a new site is added.
> 
> When all the iono files are created, they will be copied to 
> /usr/oasis/experiments/gps/novatel, where the gps_retrieve.py script 
> will be modified to retrieve them.  These *.iono will then be handled 
> like any other *.iono created by the algo program, with the same 
> algorithms to correct for receiver bias and remove bad data.
> 
> Bill
> 


-- 
Bill Rideout
MIT Haystack Observatory
Email: brideout@haystack.mit.edu
Phone: 781 981-5624