From brideout at haystack.mit.edu Mon Jan 5 13:20:09 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:08 2004 Subject: [gps-developers] TEC file inventory Message-ID: <3FF9AAD9.7000505@haystack.mit.edu> As requested by John Foster, I've created tec files for March 20 - April 20, 2001, and placed them in /usr/oasis/experiments/gps/algo/. There are now 54 days worth of tec files there, as summarized below: march2001Tec: March 20-31, 2001 april2001Tec: April 1-20 2001 april2002Tec2: March 31, April 1,2,14-21,26-28 sept2002Tec: Sept 7, 2002 oct2002Tec: Oct 1,2 2002 tec_302_2003: Oct 29-31,2003 tec_324_2003: Nov 20,21 2003 Let me know if any maps of the 2001 data set is required. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Thu Jan 8 16:25:41 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:10 2004 Subject: [gps-developers] The next improvement in gps processing References: <3FD795F1.3060201@haystack.mit.edu> Message-ID: <3FFDCAD5.4030108@haystack.mit.edu> An update on progress in gps processing: As discussed below, I've been pursuing a method to improve both receiver bias calculations and mapping functions by fitting elevation versus TEC data. This method is an extension of the one suggested by Pat Doherty at the GPS seminar. My fitter takes a sample of gps data, and tries to make the histogram of TEC versus elevation as flat as possible by adjusting both receiver bias and a mapping method parameter. Our present method of finding receiver bias makes the assumption that TEC goes to zero at some time in the sample. This introduces a systematic error in our data, since we always get a receiver bias that is higher than the real receiver bias. As a result, our TEC versus time plots show a systematic trend towards higher TEC at low elevations (scalloping up). Using the techniques described below, I can recalculate receiver bias and a mapping parameter. I've done this for 7 random sites for day 200, 2001. The removal of scalloping from the data is dramatic (I have a stack of printed plots). Listed below is the old receiver bias, the new receiver bias, and mapping fitting parameter (discussed more below): Site Old bias New bias Map parm ----------------------------------- cht1 -0.68 -1.69 0.947 riga 1.96 -1.11 0.847 agmt 6.31 6.31 0.920 ahid -19.96 -20.28 0.958 ais1 5.24 3.47 0.872 amc2 -0.87 -3.43 0.877 aml5 -3.69 -4.81 0.924 The new biases are all either lower than or in one case equal to the old biases. The greatest difference is 3.07, and the average difference is 1.4, corresponding at a night-time average baseline TEC of about 9 TEC units. Note that in one case (site agmt), the old and new receiver biases are the same. This is because the fitter actually found a slightly higher bias (6.91 versus 6.31). However, this yields a few negative TEC values, and so I constrained the fitter to lower biases only. In the other 6 cases, the fitter found the same value with or without that constaint. Some details on the fitting: I had trouble using the slab model of elevation versus z, because changing neither the slab height nor the slab thickness produced much change in the function. I instead used the function from Kevin Duh's paper, which changes much more dramatically: 1 z = ---------------------------- sqrt(1.0 - (fit * cos(el))^2) where "fit" is the fitting parameter (which was hard-coded to 0.95 in Kevin's report). The resulting values of "fit" ranged from 0.847 to 0.947. I tried using sample data from around both solar midnight and solar noon. The solar noon data actually gave the better results. Data from solar midnight did not always coverge, the solar noon data always coverged in less than ten steps (about 2-3 seconds of processing). In both cases I used plus and minus 3 hours for a total range of 6 hours of data. If nobody sees any major problems with this approach, I plan to integrate it into the automated processing code. To do this I'll add the following error checks, that mark the data as suspect: 1. If the fitter does not coverge rapidly (20 steps?) 2. If the result is a limit (other that the receiver bias upper limit). Since this fitting procedure replaces the entire second half of the processing (rerunning algo with a new reciever bias), I expect this new version to run in about the same time as the old automated process. Please let me know if you have any comments, or want to see the "before and after" plots. Bill William Rideout wrote: > Now that the satellite bias issue is settled, its time to consider the > next step in improving the processing of gps data. Our long term goal > remains processing the data using any ionospheric model, and solving for > both the ionosphere and receiver bias at once since they are coupled > problems. All the raw data needed for such a full solution is now > available, which is really just time, az, el, and effective > line-of-sight tec (along with receiver location). > > However, what steps can we take to make our present processing better > until that full solution is ready? We agree that our present method of > finding receiver bias is not perfect, since we assume TEC goes to zero > at some time. However, I'm not convinced that our use of a fixed > mapping function to get vertical TEC is not equally significant an > error. Pat Doherty suggested a method for getting receiver bias that > assumes a perfect mapping function, as I mentioned in a previous email. > I propose we improve on that suggestion to find an optimal receiver > bias and slab height. This is a small step toward the complete solution > described above, but I think that its better than considering receiver > bias alone. > > Here's the approach that Phil and I worked on this morning: The > fundamental assumption Pat Doherty makes is that TEC shouldn't depend on > elevation angle, especially at night. (I'll argue that it shouldn't > depend on elevation angle ever as long as the sample is large enough, > but that's a different email). Given that, we simply need to find the > receiver bias and mapping function that minimize elevation angle > dependence. To simplify things, I keep the slab model, and simply vary > the slab height, so that we only have two input parameters. > > There are standard methods (Levenberg-Marquardt technique) that Phil > pointed out to find the best solution for this kind of well-behaved > problem. There's no reason to expect that the solution space has more > than one minimum. These methods are available in python at > http://cars9.uchicago.edu/software/python/. So all I'd need to do is > write a python method that accepts receiver bias and slab height and > returns a measure of the flatness of tec versus elevation angle. > > Here's how I think this method might work. If I bin TEC data by > elevation, get a mean for each bin, and then plot mean versus elevation, > it should approximate a parabola (where elevation 0 is straight > overhead). Given the fitted line, > > mean = a + b*el^2 > > b is the flatness measure I'd return. To make the method fast, I'd load > each site's elevation versus line-of-sight tec into global memory from > the *.dat file. Then with each method call, I'd loop through that > array, recalculating vertical TEC given the new receiver bias and > mapping method, and binning as I went. The last step would be a fit to > a small number of data points. > > I don't think this should take more that a few days to implement, and > see how it works. > > The other method is what we have is coincidence measurements between > receivers that works for receivers that are close to other receivers. > This method has the following drawbacks: > > 1. As presently implemented in Matlab, its very slow. > 2. It assumes a perfect mapping method also (especially important when > receivers are not that close). > 3. It fails utterly for isolated receivers. > > Let me know what you think. > > Bill > > > > ------------------------------------------------------------------------ > -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Thu Jan 15 09:42:28 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:10 2004 Subject: [gps-developers] Testing of fitted bias GPS processing Message-ID: <4006A6D4.9020704@haystack.mit.edu> Here's some preliminary results and my plans for testing the new GPS processing technique of fitting elevation verses vertical tec for receiver bias. I've finished processing one full day of data (Day 275 in 2002) and here are some results: Of 999 sites, 834 processed successfully, and 165 were marked as suspect. This is a success rate of 84%, as compared to about 88% for the old approach. A site is marked as suspect if it contains too little data, or it contains discontinuities in the histogram of tec values, or if the fitter fails for both the data around solar noon, and the data around solar midnight. A fitter failure occurs if the fitter takes too many steps (>50), or if it returns a fit at a limit. The one exception is that the receiver bias upper limit is set to the upper estimate obtained by assuming 0 TEC at some point at night. The fitter is allowed to return that upper limit. Anthea worried that the assumption that TEC goes to zero is worse near the equator, so I calculated average difference between the fitted receiver bias and the zero TEC receiver bias for all sites binned by latitude: Latitude Average Sample range Bias Diff Size 0-15 2.13 22 15-30 1.52 56 30-45 0.67 597 45-60 0.54 122 60-75 0.55 32 75-90 0.0 4 This trend shows up clearly from the fitted biases. To convert these bias differences to TEC difference, multiply by 6.154, so they range from 13 TEC near the equator, to 3 at midlatitudes, to very small in the polar regions. The next test I plan to make is to see how repeatable these biases are from one day to the next, so I'll be running adjacent days. It may well be that an even better estimate of receiver bias can be had by taking a running average of the past few days. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Wed Jan 28 16:38:34 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:10 2004 Subject: [gps-developers] Inventory of GPS data Message-ID: <40182BDA.2000102@haystack.mit.edu> In order not to fill up /usr/oasis/experiments with GPS data (which I could at the drop of the hat), I have moved most of the tec files off it on to DVD's. There is a file called INVENTORY now that lists all days for which tec files are available, and it tells whether these files are there or on DVD's. This file is: /usr/oasis/experiments/gps/algo/INVENTORY This file also tells which processing method was used to process the data. The two processing methods are: 1. 0 TEC receiver bias - this is our standard method, which calculates receiver bias by assuming tec went to 0. 2. Fitted elevation vs vertical tec receiver bias - this is the new method I'm working on. I haven't released any data processed this way yet, but I will soon. I'll be writing a detailed description analyzing this approach in the next few days, but in summary its significantly better than the 0 TEC receiver bias approach, but still leaves room for improvement with John Holt's unified approach. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624